diff --git a/Gemma/[Gemma_4]Mulitmodal_inference_with_vLLM.ipynb b/Gemma/[Gemma_4]Mulitmodal_inference_with_vLLM.ipynb new file mode 100644 index 00000000..72cf6a21 --- /dev/null +++ b/Gemma/[Gemma_4]Mulitmodal_inference_with_vLLM.ipynb @@ -0,0 +1,1228 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "dxP3r4mvQRnX" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "uU6msr-hQZGj" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HI6jQZyLQouk" + }, + "source": [ + "# Gemma - 4 Multimodal Inference with vLLM.\n", + "\n", + "Google released Gemma-4 its new variant that comes with Multimodality, Multilingual support upto 140 global languages, 128K context window and also Agentic capabilites like structured responses and function calling. In this notebook, we will explore how to use Multimodal inference with vLLM via Gemma 4 E2B-it model\n", + "\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8zdBd9FqRNYi" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a L4 GPU:\n", + "\n", + "1. In the upper-right of the Colab window, select **▾ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **L4 GPU**.\n", + "\n", + "### Gemma-4 setup\n", + "\n", + "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n", + "\n", + "* Get access to Gemma on kaggle.com.\n", + "* Select a Colab runtime with sufficient resources to run\n", + " the Gemma 4 E2B-it model.\n", + "* Generate and configure a Kaggle username and an API key as Colab secrets.\n", + "\n", + "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UPG9Tt-xRXnN" + }, + "source": [ + "### Install dependencies\n", + "\n", + "Run the cell below to install all the required dependencies.\n", + "\n", + "- vLLM server\n", + "- Accelerate\n", + "- DuckDuck Go Search - for function calling" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting vllm[audio]\n", + " Downloading vllm-0.19.0-cp38-abi3-manylinux_2_31_aarch64.whl.metadata (10.0 kB)\n", + "Collecting regex (from vllm[audio])\n", + " Downloading regex-2026.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (40 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.4/40.4 kB\u001b[0m \u001b[31m300.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m297.0 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hCollecting cachetools (from vllm[audio])\n", + " Downloading cachetools-7.0.5-py3-none-any.whl.metadata (5.6 kB)\n", + "Requirement already satisfied: psutil in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (7.2.2)\n", + "Collecting sentencepiece (from vllm[audio])\n", + " Using cached sentencepiece-0.2.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (10 kB)\n", + "Collecting numpy (from vllm[audio])\n", + " Downloading numpy-2.4.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (6.6 kB)\n", + "Requirement already satisfied: requests>=2.26.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (2.33.1)\n", + "Collecting tqdm (from vllm[audio])\n", + " Using cached tqdm-4.67.3-py3-none-any.whl.metadata (57 kB)\n", + "Collecting blake3 (from vllm[audio])\n", + " Downloading blake3-1.0.8-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.6 kB)\n", + "Collecting py-cpuinfo (from vllm[audio])\n", + " Using cached py_cpuinfo-9.0.0-py3-none-any.whl.metadata (794 bytes)\n", + "Collecting transformers<5,>=4.56.0 (from vllm[audio])\n", + " Downloading transformers-4.57.6-py3-none-any.whl.metadata (43 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.0/44.0 kB\u001b[0m \u001b[31m1.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0m eta \u001b[36m-:--:--\u001b[0m\n", + "\u001b[?25hCollecting tokenizers>=0.21.1 (from vllm[audio])\n", + " Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (7.3 kB)\n", + "Collecting protobuf!=6.30.*,!=6.31.*,!=6.32.*,!=6.33.0.*,!=6.33.1.*,!=6.33.2.*,!=6.33.3.*,!=6.33.4.*,>=5.29.6 (from vllm[audio])\n", + " Downloading protobuf-7.34.1-cp310-abi3-manylinux2014_aarch64.whl.metadata (595 bytes)\n", + "Collecting fastapi>=0.115.0 (from fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading fastapi-0.135.3-py3-none-any.whl.metadata (28 kB)\n", + "Collecting aiohttp>=3.13.3 (from vllm[audio])\n", + " Downloading aiohttp-3.13.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (8.1 kB)\n", + "Collecting openai>=2.0.0 (from vllm[audio])\n", + " Downloading openai-2.30.0-py3-none-any.whl.metadata (29 kB)\n", + "Collecting pydantic>=2.12.0 (from vllm[audio])\n", + " Using cached pydantic-2.12.5-py3-none-any.whl.metadata (90 kB)\n", + "Requirement already satisfied: prometheus_client>=0.18.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (0.24.1)\n", + "Collecting pillow (from vllm[audio])\n", + " Downloading pillow-12.2.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (8.8 kB)\n", + "Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm[audio])\n", + " Using cached prometheus_fastapi_instrumentator-7.1.0-py3-none-any.whl.metadata (13 kB)\n", + "Collecting tiktoken>=0.6.0 (from vllm[audio])\n", + " Using cached tiktoken-0.12.0-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (6.7 kB)\n", + "Collecting lm-format-enforcer==0.11.3 (from vllm[audio])\n", + " Downloading lm_format_enforcer-0.11.3-py3-none-any.whl.metadata (17 kB)\n", + "Collecting llguidance<1.4.0,>=1.3.0 (from vllm[audio])\n", + " Downloading llguidance-1.3.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (10 kB)\n", + "Collecting outlines_core==0.2.11 (from vllm[audio])\n", + " Downloading outlines_core-0.2.11-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (5.8 kB)\n", + "Collecting diskcache==5.6.3 (from vllm[audio])\n", + " Using cached diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n", + "Collecting lark==1.2.2 (from vllm[audio])\n", + " Using cached lark-1.2.2-py3-none-any.whl.metadata (1.8 kB)\n", + "Collecting xgrammar<1.0.0,>=0.1.32 (from vllm[audio])\n", + " Downloading xgrammar-0.1.33-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.metadata (7.6 kB)\n", + "Requirement already satisfied: typing_extensions>=4.10 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (4.15.0)\n", + "Collecting filelock>=3.16.1 (from vllm[audio])\n", + " Downloading filelock-3.25.2-py3-none-any.whl.metadata (2.0 kB)\n", + "Collecting partial-json-parser (from vllm[audio])\n", + " Downloading partial_json_parser-0.2.1.1.post7-py3-none-any.whl.metadata (6.1 kB)\n", + "Requirement already satisfied: pyzmq>=25.0.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (27.1.0)\n", + "Collecting msgspec (from vllm[audio])\n", + " Downloading msgspec-0.20.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (5.5 kB)\n", + "Collecting gguf>=0.17.0 (from vllm[audio])\n", + " Downloading gguf-0.18.0-py3-none-any.whl.metadata (4.5 kB)\n", + "Collecting mistral_common>=1.10.0 (from mistral_common[image]>=1.10.0->vllm[audio])\n", + " Downloading mistral_common-1.11.0-py3-none-any.whl.metadata (5.9 kB)\n", + "Collecting opencv-python-headless>=4.13.0 (from vllm[audio])\n", + " Downloading opencv_python_headless-4.13.0.92-cp37-abi3-manylinux_2_28_aarch64.whl.metadata (19 kB)\n", + "Requirement already satisfied: pyyaml in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (6.0.3)\n", + "Requirement already satisfied: six>=1.16.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (1.17.0)\n", + "Collecting setuptools<81.0.0,>=77.0.3 (from vllm[audio])\n", + " Downloading setuptools-80.10.2-py3-none-any.whl.metadata (6.6 kB)\n", + "Collecting einops (from vllm[audio])\n", + " Downloading einops-0.8.2-py3-none-any.whl.metadata (13 kB)\n", + "Collecting compressed-tensors==0.14.0.1 (from vllm[audio])\n", + " Downloading compressed_tensors-0.14.0.1-py3-none-any.whl.metadata (7.1 kB)\n", + "Collecting depyf==0.20.0 (from vllm[audio])\n", + " Downloading depyf-0.20.0-py3-none-any.whl.metadata (7.3 kB)\n", + "Collecting cloudpickle (from vllm[audio])\n", + " Using cached cloudpickle-3.1.2-py3-none-any.whl.metadata (7.1 kB)\n", + "Collecting watchfiles (from vllm[audio])\n", + " Downloading watchfiles-1.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.9 kB)\n", + "Requirement already satisfied: python-json-logger in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from vllm[audio]) (4.1.0)\n", + "Collecting ninja (from vllm[audio])\n", + " Using cached ninja-1.13.0-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (5.1 kB)\n", + "Collecting pybase64 (from vllm[audio])\n", + " Downloading pybase64-1.4.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (8.7 kB)\n", + "Collecting cbor2 (from vllm[audio])\n", + " Downloading cbor2-5.9.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (5.5 kB)\n", + "Collecting ijson (from vllm[audio])\n", + " Downloading ijson-3.5.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (23 kB)\n", + "Collecting setproctitle (from vllm[audio])\n", + " Downloading setproctitle-1.3.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (10 kB)\n", + "Collecting openai-harmony>=0.0.3 (from vllm[audio])\n", + " Downloading openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (8.0 kB)\n", + "Collecting anthropic>=0.71.0 (from vllm[audio])\n", + " Downloading anthropic-0.89.0-py3-none-any.whl.metadata (3.1 kB)\n", + "Collecting model-hosting-container-standards<1.0.0,>=0.1.13 (from vllm[audio])\n", + " Downloading model_hosting_container_standards-0.1.14-py3-none-any.whl.metadata (24 kB)\n", + "Collecting mcp (from vllm[audio])\n", + " Downloading mcp-1.27.0-py3-none-any.whl.metadata (8.2 kB)\n", + "Collecting opentelemetry-sdk>=1.27.0 (from vllm[audio])\n", + " Downloading opentelemetry_sdk-1.40.0-py3-none-any.whl.metadata (1.6 kB)\n", + "Collecting opentelemetry-api>=1.27.0 (from vllm[audio])\n", + " Downloading opentelemetry_api-1.40.0-py3-none-any.whl.metadata (1.5 kB)\n", + "Collecting opentelemetry-exporter-otlp>=1.27.0 (from vllm[audio])\n", + " Downloading opentelemetry_exporter_otlp-1.40.0-py3-none-any.whl.metadata (2.4 kB)\n", + "Collecting opentelemetry-semantic-conventions-ai>=0.4.1 (from vllm[audio])\n", + " Downloading opentelemetry_semantic_conventions_ai-0.5.1-py3-none-any.whl.metadata (997 bytes)\n", + "Collecting numba==0.61.2 (from vllm[audio])\n", + " Using cached numba-0.61.2-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (2.8 kB)\n", + "Collecting torch==2.10.0 (from vllm[audio])\n", + " Using cached torch-2.10.0-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (31 kB)\n", + "Collecting torchaudio==2.10.0 (from vllm[audio])\n", + " Downloading torchaudio-2.10.0-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (6.9 kB)\n", + "Collecting torchvision==0.25.0 (from vllm[audio])\n", + " Downloading torchvision-0.25.0-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (5.4 kB)\n", + "Collecting flashinfer-python==0.6.6 (from vllm[audio])\n", + " Downloading flashinfer_python-0.6.6-py3-none-any.whl.metadata (11 kB)\n", + "Collecting flashinfer-cubin==0.6.6 (from vllm[audio])\n", + " Downloading flashinfer_cubin-0.6.6-py3-none-any.whl.metadata (1.3 kB)\n", + "Collecting nvidia-cudnn-frontend<1.19.0,>=1.13.0 (from vllm[audio])\n", + " Downloading nvidia_cudnn_frontend-1.18.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (4.4 kB)\n", + "Collecting nvidia-cutlass-dsl>=4.4.0.dev1 (from vllm[audio])\n", + " Downloading nvidia_cutlass_dsl-4.4.2-py3-none-any.whl.metadata (2.7 kB)\n", + "Collecting quack-kernels>=0.2.7 (from vllm[audio])\n", + " Downloading quack_kernels-0.3.8-py3-none-any.whl.metadata (618 bytes)\n", + "Collecting av (from vllm[audio])\n", + " Downloading av-17.0.0-cp311-abi3-manylinux_2_28_aarch64.whl.metadata (4.6 kB)\n", + "Collecting resampy (from vllm[audio])\n", + " Downloading resampy-0.4.3-py3-none-any.whl.metadata (3.0 kB)\n", + "Collecting scipy (from vllm[audio])\n", + " Using cached scipy-1.17.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (62 kB)\n", + "Collecting soundfile (from vllm[audio])\n", + " Using cached soundfile-0.13.1-py2.py3-none-manylinux_2_28_aarch64.whl.metadata (16 kB)\n", + "Collecting loguru (from compressed-tensors==0.14.0.1->vllm[audio])\n", + " Downloading loguru-0.7.3-py3-none-any.whl.metadata (22 kB)\n", + "Collecting astor (from depyf==0.20.0->vllm[audio])\n", + " Using cached astor-0.8.1-py2.py3-none-any.whl.metadata (4.2 kB)\n", + "Collecting dill (from depyf==0.20.0->vllm[audio])\n", + " Downloading dill-0.4.1-py3-none-any.whl.metadata (10 kB)\n", + "Collecting apache-tvm-ffi!=0.1.8,!=0.1.8.post0,<0.2,>=0.1.6 (from flashinfer-python==0.6.6->vllm[audio])\n", + " Downloading apache_tvm_ffi-0.1.9-cp312-abi3-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl.metadata (6.1 kB)\n", + "Collecting click (from flashinfer-python==0.6.6->vllm[audio])\n", + " Downloading click-8.3.2-py3-none-any.whl.metadata (2.6 kB)\n", + "Collecting nvidia-ml-py (from flashinfer-python==0.6.6->vllm[audio])\n", + " Downloading nvidia_ml_py-13.595.45-py3-none-any.whl.metadata (9.7 kB)\n", + "Requirement already satisfied: packaging>=24.2 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from flashinfer-python==0.6.6->vllm[audio]) (26.0)\n", + "Collecting tabulate (from flashinfer-python==0.6.6->vllm[audio])\n", + " Downloading tabulate-0.10.0-py3-none-any.whl.metadata (40 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.8/40.8 kB\u001b[0m \u001b[31m426.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hCollecting interegular>=0.3.2 (from lm-format-enforcer==0.11.3->vllm[audio])\n", + " Using cached interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB)\n", + "Collecting llvmlite<0.45,>=0.44.0dev0 (from numba==0.61.2->vllm[audio])\n", + " Using cached llvmlite-0.44.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl.metadata (5.0 kB)\n", + "Collecting numpy (from vllm[audio])\n", + " Using cached numpy-2.2.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (63 kB)\n", + "Collecting sympy>=1.13.3 (from torch==2.10.0->vllm[audio])\n", + " Using cached sympy-1.14.0-py3-none-any.whl.metadata (12 kB)\n", + "Collecting networkx>=2.5.1 (from torch==2.10.0->vllm[audio])\n", + " Using cached networkx-3.6.1-py3-none-any.whl.metadata (6.8 kB)\n", + "Requirement already satisfied: jinja2 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from torch==2.10.0->vllm[audio]) (3.1.6)\n", + "Collecting fsspec>=0.8.5 (from torch==2.10.0->vllm[audio])\n", + " Downloading fsspec-2026.3.0-py3-none-any.whl.metadata (10 kB)\n", + "Collecting aiohappyeyeballs>=2.5.0 (from aiohttp>=3.13.3->vllm[audio])\n", + " Using cached aiohappyeyeballs-2.6.1-py3-none-any.whl.metadata (5.9 kB)\n", + "Collecting aiosignal>=1.4.0 (from aiohttp>=3.13.3->vllm[audio])\n", + " Using cached aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB)\n", + "Requirement already satisfied: attrs>=17.3.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from aiohttp>=3.13.3->vllm[audio]) (26.1.0)\n", + "Collecting frozenlist>=1.1.1 (from aiohttp>=3.13.3->vllm[audio])\n", + " Using cached frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (20 kB)\n", + "Collecting multidict<7.0,>=4.5 (from aiohttp>=3.13.3->vllm[audio])\n", + " Using cached multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (5.3 kB)\n", + "Collecting propcache>=0.2.0 (from aiohttp>=3.13.3->vllm[audio])\n", + " Using cached propcache-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (13 kB)\n", + "Collecting yarl<2.0,>=1.17.0 (from aiohttp>=3.13.3->vllm[audio])\n", + " Using cached yarl-1.23.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (79 kB)\n", + "Requirement already satisfied: anyio<5,>=3.5.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from anthropic>=0.71.0->vllm[audio]) (4.13.0)\n", + "Collecting distro<2,>=1.7.0 (from anthropic>=0.71.0->vllm[audio])\n", + " Using cached distro-1.9.0-py3-none-any.whl.metadata (6.8 kB)\n", + "Collecting docstring-parser<1,>=0.15 (from anthropic>=0.71.0->vllm[audio])\n", + " Using cached docstring_parser-0.17.0-py3-none-any.whl.metadata (3.5 kB)\n", + "Requirement already satisfied: httpx<1,>=0.25.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from anthropic>=0.71.0->vllm[audio]) (0.28.1)\n", + "Collecting jiter<1,>=0.4.0 (from anthropic>=0.71.0->vllm[audio])\n", + " Using cached jiter-0.13.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (5.2 kB)\n", + "Collecting sniffio (from anthropic>=0.71.0->vllm[audio])\n", + " Using cached sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)\n", + "Collecting starlette>=0.46.0 (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading starlette-1.0.0-py3-none-any.whl.metadata (6.3 kB)\n", + "Collecting typing-inspection>=0.4.2 (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)\n", + "Collecting annotated-doc>=0.0.2 (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)\n", + "Collecting fastapi-cli>=0.0.8 (from fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading fastapi_cli-0.0.24-py3-none-any.whl.metadata (6.4 kB)\n", + "Collecting python-multipart>=0.0.18 (from fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached python_multipart-0.0.22-py3-none-any.whl.metadata (1.8 kB)\n", + "Collecting email-validator>=2.0.0 (from fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading email_validator-2.3.0-py3-none-any.whl.metadata (26 kB)\n", + "Collecting uvicorn>=0.12.0 (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading uvicorn-0.43.0-py3-none-any.whl.metadata (6.7 kB)\n", + "Collecting pydantic-settings>=2.0.0 (from fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached pydantic_settings-2.13.1-py3-none-any.whl.metadata (3.4 kB)\n", + "Collecting pydantic-extra-types>=2.0.0 (from fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading pydantic_extra_types-2.11.1-py3-none-any.whl.metadata (4.2 kB)\n", + "Requirement already satisfied: jsonschema>=4.21.1 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from mistral_common>=1.10.0->mistral_common[image]>=1.10.0->vllm[audio]) (4.26.0)\n", + "Collecting jmespath (from model-hosting-container-standards<1.0.0,>=0.1.13->vllm[audio])\n", + " Downloading jmespath-1.1.0-py3-none-any.whl.metadata (7.6 kB)\n", + "Collecting supervisor>=4.2.0 (from model-hosting-container-standards<1.0.0,>=0.1.13->vllm[audio])\n", + " Downloading supervisor-4.3.0-py2.py3-none-any.whl.metadata (87 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.9/87.9 kB\u001b[0m \u001b[31m79.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m31m64.8 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hCollecting nvidia-cutlass-dsl-libs-base==4.4.2 (from nvidia-cutlass-dsl>=4.4.0.dev1->vllm[audio])\n", + " Downloading nvidia_cutlass_dsl_libs_base-4.4.2-cp312-cp312-manylinux_2_28_aarch64.whl.metadata (2.6 kB)\n", + "Collecting cuda-python>=12.8 (from nvidia-cutlass-dsl-libs-base==4.4.2->nvidia-cutlass-dsl>=4.4.0.dev1->vllm[audio])\n", + " Downloading cuda_python-13.2.0-py3-none-any.whl.metadata (6.5 kB)\n", + "Collecting importlib-metadata<8.8.0,>=6.0 (from opentelemetry-api>=1.27.0->vllm[audio])\n", + " Using cached importlib_metadata-8.7.1-py3-none-any.whl.metadata (4.7 kB)\n", + "Collecting opentelemetry-exporter-otlp-proto-grpc==1.40.0 (from opentelemetry-exporter-otlp>=1.27.0->vllm[audio])\n", + " Downloading opentelemetry_exporter_otlp_proto_grpc-1.40.0-py3-none-any.whl.metadata (2.6 kB)\n", + "Collecting opentelemetry-exporter-otlp-proto-http==1.40.0 (from opentelemetry-exporter-otlp>=1.27.0->vllm[audio])\n", + " Downloading opentelemetry_exporter_otlp_proto_http-1.40.0-py3-none-any.whl.metadata (2.5 kB)\n", + "Collecting googleapis-common-protos~=1.57 (from opentelemetry-exporter-otlp-proto-grpc==1.40.0->opentelemetry-exporter-otlp>=1.27.0->vllm[audio])\n", + " Downloading googleapis_common_protos-1.74.0-py3-none-any.whl.metadata (9.2 kB)\n", + "Collecting grpcio<2.0.0,>=1.63.2 (from opentelemetry-exporter-otlp-proto-grpc==1.40.0->opentelemetry-exporter-otlp>=1.27.0->vllm[audio])\n", + " Downloading grpcio-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.metadata (3.8 kB)\n", + "Collecting opentelemetry-exporter-otlp-proto-common==1.40.0 (from opentelemetry-exporter-otlp-proto-grpc==1.40.0->opentelemetry-exporter-otlp>=1.27.0->vllm[audio])\n", + " Downloading opentelemetry_exporter_otlp_proto_common-1.40.0-py3-none-any.whl.metadata (1.9 kB)\n", + "Collecting opentelemetry-proto==1.40.0 (from opentelemetry-exporter-otlp-proto-grpc==1.40.0->opentelemetry-exporter-otlp>=1.27.0->vllm[audio])\n", + " Downloading opentelemetry_proto-1.40.0-py3-none-any.whl.metadata (2.4 kB)\n", + "Collecting protobuf!=6.30.*,!=6.31.*,!=6.32.*,!=6.33.0.*,!=6.33.1.*,!=6.33.2.*,!=6.33.3.*,!=6.33.4.*,>=5.29.6 (from vllm[audio])\n", + " Downloading protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl.metadata (593 bytes)\n", + "Collecting opentelemetry-semantic-conventions==0.61b0 (from opentelemetry-sdk>=1.27.0->vllm[audio])\n", + " Downloading opentelemetry_semantic_conventions-0.61b0-py3-none-any.whl.metadata (2.5 kB)\n", + "Collecting starlette>=0.46.0 (from fastapi>=0.115.0->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached starlette-0.52.1-py3-none-any.whl.metadata (6.3 kB)\n", + "Collecting annotated-types>=0.6.0 (from pydantic>=2.12.0->vllm[audio])\n", + " Using cached annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)\n", + "Collecting pydantic-core==2.41.5 (from pydantic>=2.12.0->vllm[audio])\n", + " Using cached pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (7.3 kB)\n", + "Collecting torch-c-dlpack-ext (from quack-kernels>=0.2.7->vllm[audio])\n", + " Downloading torch_c_dlpack_ext-0.1.5-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl.metadata (14 kB)\n", + "Requirement already satisfied: charset_normalizer<4,>=2 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from requests>=2.26.0->vllm[audio]) (3.4.7)\n", + "Requirement already satisfied: idna<4,>=2.5 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from requests>=2.26.0->vllm[audio]) (3.11)\n", + "Requirement already satisfied: urllib3<3,>=1.26 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from requests>=2.26.0->vllm[audio]) (2.6.3)\n", + "Requirement already satisfied: certifi>=2023.5.7 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from requests>=2.26.0->vllm[audio]) (2026.2.25)\n", + "Collecting huggingface-hub<2.0,>=0.16.4 (from tokenizers>=0.21.1->vllm[audio])\n", + " Downloading huggingface_hub-1.9.0-py3-none-any.whl.metadata (14 kB)\n", + " Using cached huggingface_hub-0.36.2-py3-none-any.whl.metadata (15 kB)\n", + "Collecting safetensors>=0.4.3 (from transformers<5,>=4.56.0->vllm[audio])\n", + " Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.1 kB)\n", + "Collecting httpx-sse>=0.4 (from mcp->vllm[audio])\n", + " Using cached httpx_sse-0.4.3-py3-none-any.whl.metadata (9.7 kB)\n", + "Collecting pyjwt>=2.10.1 (from pyjwt[crypto]>=2.10.1->mcp->vllm[audio])\n", + " Downloading pyjwt-2.12.1-py3-none-any.whl.metadata (4.1 kB)\n", + "Collecting sse-starlette>=1.6.1 (from mcp->vllm[audio])\n", + " Downloading sse_starlette-3.3.4-py3-none-any.whl.metadata (14 kB)\n", + "Requirement already satisfied: cffi>=1.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from soundfile->vllm[audio]) (2.0.0)\n", + "Requirement already satisfied: pycparser in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from cffi>=1.0->soundfile->vllm[audio]) (3.0)\n", + "Collecting dnspython>=2.0.0 (from email-validator>=2.0.0->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading dnspython-2.8.0-py3-none-any.whl.metadata (5.7 kB)\n", + "Collecting typer>=0.16.0 (from fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading typer-0.24.1-py3-none-any.whl.metadata (16 kB)\n", + "Collecting rich-toolkit>=0.14.8 (from fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading rich_toolkit-0.19.7-py3-none-any.whl.metadata (1.0 kB)\n", + "Collecting fastapi-cloud-cli>=0.1.1 (from fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading fastapi_cloud_cli-0.15.1-py3-none-any.whl.metadata (3.3 kB)\n", + "Requirement already satisfied: httpcore==1.* in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from httpx<1,>=0.25.0->anthropic>=0.71.0->vllm[audio]) (1.0.9)\n", + "Requirement already satisfied: h11>=0.16 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from httpcore==1.*->httpx<1,>=0.25.0->anthropic>=0.71.0->vllm[audio]) (0.16.0)\n", + "Collecting hf-xet<2.0.0,>=1.1.3 (from huggingface-hub<2.0,>=0.16.4->tokenizers>=0.21.1->vllm[audio])\n", + " Downloading hf_xet-1.4.3-cp37-abi3-manylinux_2_28_aarch64.whl.metadata (4.9 kB)\n", + "Collecting zipp>=3.20 (from importlib-metadata<8.8.0,>=6.0->opentelemetry-api>=1.27.0->vllm[audio])\n", + " Using cached zipp-3.23.0-py3-none-any.whl.metadata (3.6 kB)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from jinja2->torch==2.10.0->vllm[audio]) (3.0.3)\n", + "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.10.0->mistral_common[image]>=1.10.0->vllm[audio]) (2025.9.1)\n", + "Requirement already satisfied: referencing>=0.28.4 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.10.0->mistral_common[image]>=1.10.0->vllm[audio]) (0.37.0)\n", + "Requirement already satisfied: rpds-py>=0.25.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from jsonschema>=4.21.1->mistral_common>=1.10.0->mistral_common[image]>=1.10.0->vllm[audio]) (0.30.0)\n", + "Collecting pycountry>=23 (from pydantic-extra-types[pycountry]>=2.10.5->mistral_common>=1.10.0->mistral_common[image]>=1.10.0->vllm[audio])\n", + " Downloading pycountry-26.2.16-py3-none-any.whl.metadata (12 kB)\n", + "Collecting python-dotenv>=0.21.0 (from pydantic-settings>=2.0.0->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading python_dotenv-1.2.2-py3-none-any.whl.metadata (27 kB)\n", + "Collecting cryptography>=3.4.0 (from pyjwt[crypto]>=2.10.1->mcp->vllm[audio])\n", + " Downloading cryptography-46.0.6-cp311-abi3-manylinux_2_34_aarch64.whl.metadata (5.7 kB)\n", + "Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch==2.10.0->vllm[audio])\n", + " Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)\n", + "Collecting httptools>=0.6.3 (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading httptools-0.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (3.5 kB)\n", + "Collecting uvloop>=0.15.1 (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading uvloop-0.22.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (4.9 kB)\n", + "Collecting websockets>=10.4 (from uvicorn[standard]>=0.12.0; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading websockets-16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl.metadata (6.8 kB)\n", + "Collecting soxr>=0.5.0 (from mistral_common>=1.10.0->mistral_common[image]>=1.10.0->vllm[audio])\n", + " Downloading soxr-1.0.0-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl.metadata (5.6 kB)\n", + "Collecting cuda-bindings~=13.2.0 (from cuda-python>=12.8->nvidia-cutlass-dsl-libs-base==4.4.2->nvidia-cutlass-dsl>=4.4.0.dev1->vllm[audio])\n", + " Downloading cuda_bindings-13.2.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl.metadata (2.3 kB)\n", + "Collecting cuda-pathfinder~=1.1 (from cuda-python>=12.8->nvidia-cutlass-dsl-libs-base==4.4.2->nvidia-cutlass-dsl>=4.4.0.dev1->vllm[audio])\n", + " Downloading cuda_pathfinder-1.5.1-py3-none-any.whl.metadata (1.9 kB)\n", + "Collecting rignore>=0.5.1 (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading rignore-0.7.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.2 kB)\n", + "Collecting sentry-sdk>=2.20.0 (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading sentry_sdk-2.57.0-py2.py3-none-any.whl.metadata (10 kB)\n", + "Collecting fastar>=0.8.0 (from fastapi-cloud-cli>=0.1.1->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Downloading fastar-0.9.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.0 kB)\n", + "Collecting rich>=13.7.1 (from rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached rich-14.3.3-py3-none-any.whl.metadata (18 kB)\n", + "Collecting shellingham>=1.3.0 (from typer>=0.16.0->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)\n", + "Collecting markdown-it-py>=2.2.0 (from rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached markdown_it_py-4.0.0-py3-none-any.whl.metadata (7.3 kB)\n", + "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/x1/code/external/cookbook/venv/lib/python3.12/site-packages (from rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio]) (2.20.0)\n", + "Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=13.7.1->rich-toolkit>=0.14.8->fastapi-cli>=0.0.8->fastapi-cli[standard]>=0.0.8; extra == \"standard\"->fastapi[standard]>=0.115.0->vllm[audio])\n", + " Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)\n", + "Downloading compressed_tensors-0.14.0.1-py3-none-any.whl (196 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.5/196.5 kB\u001b[0m \u001b[31m65.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:02\u001b[0m\n", + "\u001b[?25hDownloading depyf-0.20.0-py3-none-any.whl (39 kB)\n", + "Using cached diskcache-5.6.3-py3-none-any.whl (45 kB)\n", + "Downloading flashinfer_cubin-0.6.6-py3-none-any.whl (267.7 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m267.7/267.7 MB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:02\u001b[0mm\n", + "\u001b[?25hDownloading flashinfer_python-0.6.6-py3-none-any.whl (7.8 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached lark-1.2.2-py3-none-any.whl (111 kB)\n", + "Downloading lm_format_enforcer-0.11.3-py3-none-any.whl (45 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.4/45.4 kB\u001b[0m \u001b[31m393.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached numba-0.61.2-cp312-cp312-manylinux_2_28_aarch64.whl (3.6 MB)\n", + "Downloading outlines_core-0.2.11-cp312-cp312-manylinux_2_28_aarch64.whl (2.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hUsing cached torch-2.10.0-cp312-cp312-manylinux_2_28_aarch64.whl (146.0 MB)\n", + "Downloading torchaudio-2.10.0-cp312-cp312-manylinux_2_28_aarch64.whl (391 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m392.0/392.0 kB\u001b[0m \u001b[31m610.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mMB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0mm\n", + "\u001b[?25hDownloading torchvision-0.25.0-cp312-cp312-manylinux_2_28_aarch64.whl (2.3 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading aiohttp-3.13.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (1.7 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m00:07\u001b[0m\n", + "\u001b[?25hDownloading anthropic-0.89.0-py3-none-any.whl (478 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m478.8/478.8 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading fastapi-0.135.3-py3-none-any.whl (117 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m117.7/117.7 kB\u001b[0m \u001b[31m349.8 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading filelock-3.25.2-py3-none-any.whl (26 kB)\n", + "Downloading gguf-0.18.0-py3-none-any.whl (114 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.2/114.2 kB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mMB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading llguidance-1.3.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.8 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading mistral_common-1.11.0-py3-none-any.whl (6.5 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.5/6.5 MB\u001b[0m \u001b[31m706.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading model_hosting_container_standards-0.1.14-py3-none-any.whl (121 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.4/121.4 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mMB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached numpy-2.2.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.0 MB)\n", + "Downloading nvidia_cudnn_frontend-1.18.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (2.0 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m763.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m769.9 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cutlass_dsl-4.4.2-py3-none-any.whl (10 kB)\n", + "Downloading nvidia_cutlass_dsl_libs_base-4.4.2-cp312-cp312-manylinux_2_28_aarch64.whl (75.5 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.5/75.5 MB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m0m\n", + "\u001b[?25hDownloading openai-2.30.0-py3-none-any.whl (1.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m0:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading openai_harmony-0.0.8-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.9 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.9/2.9 MB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[36m0:00:01\u001b[0mm eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading opencv_python_headless-4.13.0.92-cp37-abi3-manylinux_2_28_aarch64.whl (35.0 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m35.0/35.0 MB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading opentelemetry_api-1.40.0-py3-none-any.whl (68 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m68.7/68.7 kB\u001b[0m \u001b[31m624.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m5.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading opentelemetry_exporter_otlp-1.40.0-py3-none-any.whl (7.0 kB)\n", + "Downloading opentelemetry_exporter_otlp_proto_grpc-1.40.0-py3-none-any.whl (20 kB)\n", + "Downloading opentelemetry_exporter_otlp_proto_http-1.40.0-py3-none-any.whl (19 kB)\n", + "Downloading opentelemetry_exporter_otlp_proto_common-1.40.0-py3-none-any.whl (18 kB)\n", + "Downloading opentelemetry_proto-1.40.0-py3-none-any.whl (72 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m72.1/72.1 kB\u001b[0m \u001b[31m809.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m726.5 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading opentelemetry_sdk-1.40.0-py3-none-any.whl (141 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.0/142.0 kB\u001b[0m \u001b[31m832.8 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m807.4 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading opentelemetry_semantic_conventions-0.61b0-py3-none-any.whl (231 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m231.6/231.6 kB\u001b[0m \u001b[31m940.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading opentelemetry_semantic_conventions_ai-0.5.1-py3-none-any.whl (11 kB)\n", + "Downloading pillow-12.2.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (6.4 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.4/6.4 MB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached prometheus_fastapi_instrumentator-7.1.0-py3-none-any.whl (19 kB)\n", + "Downloading protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl (324 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m324.6/324.6 kB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hUsing cached pydantic-2.12.5-py3-none-any.whl (463 kB)\n", + "Using cached pydantic_core-2.41.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.9 MB)\n", + "Downloading quack_kernels-0.3.8-py3-none-any.whl (217 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m217.7/217.7 kB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading setuptools-80.10.2-py3-none-any.whl (1.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hUsing cached tiktoken-0.12.0-cp312-cp312-manylinux_2_28_aarch64.whl (1.1 MB)\n", + "Downloading regex-2026.4.4-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (796 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m796.7/796.7 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.3 MB)\n", + "Using cached tqdm-4.67.3-py3-none-any.whl (78 kB)\n", + "Downloading transformers-4.57.6-py3-none-any.whl (12.0 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.0/12.0 MB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m0:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading xgrammar-0.1.33-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (42.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.1/42.1 MB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading av-17.0.0-cp311-abi3-manylinux_2_28_aarch64.whl (34.0 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m34.0/34.0 MB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading blake3-1.0.8-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (371 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m371.3/371.3 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading cachetools-7.0.5-py3-none-any.whl (13 kB)\n", + "Downloading cbor2-5.9.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (287 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m287.9/287.9 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached cloudpickle-3.1.2-py3-none-any.whl (22 kB)\n", + "Downloading einops-0.8.2-py3-none-any.whl (65 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.6/65.6 kB\u001b[0m \u001b[31m582.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m6.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading ijson-3.5.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (149 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m149.8/149.8 kB\u001b[0m \u001b[31m313.9 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading mcp-1.27.0-py3-none-any.whl (215 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m216.0/216.0 kB\u001b[0m \u001b[31m156.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m136.9 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading msgspec-0.20.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (218 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m218.9/218.9 kB\u001b[0m \u001b[31m175.9 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m164.4 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached ninja-1.13.0-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (177 kB)\n", + "Downloading partial_json_parser-0.2.1.1.post7-py3-none-any.whl (10 kB)\n", + "Using cached py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)\n", + "Downloading pybase64-1.4.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (59 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m59.9/59.9 kB\u001b[0m \u001b[31m421.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m449.7 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading resampy-0.4.3-py3-none-any.whl (3.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.1/3.1 MB\u001b[0m \u001b[31m672.3 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[36m0:00:01\u001b[0mm eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached scipy-1.17.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (32.9 MB)\n", + "Using cached sentencepiece-0.2.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (1.3 MB)\n", + "Downloading setproctitle-1.3.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (33 kB)\n", + "Using cached soundfile-0.13.1-py2.py3-none-manylinux_2_28_aarch64.whl (1.2 MB)\n", + "Downloading vllm-0.19.0-cp38-abi3-manylinux_2_31_aarch64.whl (384.7 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m384.7/384.7 MB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:02\u001b[0mm\n", + "\u001b[?25hDownloading watchfiles-1.1.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (449 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m449.4/449.4 kB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[36m0:00:01\u001b[0mm eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached aiohappyeyeballs-2.6.1-py3-none-any.whl (15 kB)\n", + "Using cached aiosignal-1.4.0-py3-none-any.whl (7.5 kB)\n", + "Using cached annotated_doc-0.0.4-py3-none-any.whl (5.3 kB)\n", + "Using cached annotated_types-0.7.0-py3-none-any.whl (13 kB)\n", + "Downloading apache_tvm_ffi-0.1.9-cp312-abi3-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl (2.1 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.1/2.1 MB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hUsing cached distro-1.9.0-py3-none-any.whl (20 kB)\n", + "Using cached docstring_parser-0.17.0-py3-none-any.whl (36 kB)\n", + "Downloading email_validator-2.3.0-py3-none-any.whl (35 kB)\n", + "Downloading fastapi_cli-0.0.24-py3-none-any.whl (12 kB)\n", + "Using cached frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (243 kB)\n", + "Downloading fsspec-2026.3.0-py3-none-any.whl (202 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m202.6/202.6 kB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached httpx_sse-0.4.3-py3-none-any.whl (9.0 kB)\n", + "Using cached huggingface_hub-0.36.2-py3-none-any.whl (566 kB)\n", + "Using cached importlib_metadata-8.7.1-py3-none-any.whl (27 kB)\n", + "Using cached interegular-0.3.3-py37-none-any.whl (23 kB)\n", + "Using cached jiter-0.13.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (348 kB)\n", + "Using cached llvmlite-0.44.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl (41.2 MB)\n", + "Using cached multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (258 kB)\n", + "Using cached networkx-3.6.1-py3-none-any.whl (2.1 MB)\n", + "Using cached propcache-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (225 kB)\n", + "Downloading pydantic_extra_types-2.11.1-py3-none-any.whl (79 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m79.5/79.5 kB\u001b[0m \u001b[31m551.9 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m6.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached pydantic_settings-2.13.1-py3-none-any.whl (58 kB)\n", + "Downloading pyjwt-2.12.1-py3-none-any.whl (29 kB)\n", + "Using cached python_multipart-0.0.22-py3-none-any.whl (24 kB)\n", + "Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (491 kB)\n", + "Downloading sse_starlette-3.3.4-py3-none-any.whl (14 kB)\n", + "Using cached starlette-0.52.1-py3-none-any.whl (74 kB)\n", + "Downloading supervisor-4.3.0-py2.py3-none-any.whl (320 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m320.7/320.7 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached sympy-1.14.0-py3-none-any.whl (6.3 MB)\n", + "Using cached typing_inspection-0.4.2-py3-none-any.whl (14 kB)\n", + "Downloading uvicorn-0.43.0-py3-none-any.whl (68 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m68.6/68.6 kB\u001b[0m \u001b[31m602.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m16.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading click-8.3.2-py3-none-any.whl (108 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m108.4/108.4 kB\u001b[0m \u001b[31m724.6 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hUsing cached yarl-1.23.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (97 kB)\n", + "Using cached astor-0.8.1-py2.py3-none-any.whl (27 kB)\n", + "Downloading dill-0.4.1-py3-none-any.whl (120 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m120.0/120.0 kB\u001b[0m \u001b[31m850.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m792.8 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading jmespath-1.1.0-py3-none-any.whl (20 kB)\n", + "Downloading loguru-0.7.3-py3-none-any.whl (61 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.6/61.6 kB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading nvidia_ml_py-13.595.45-py3-none-any.whl (51 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.8/51.8 kB\u001b[0m \u001b[31m906.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\n", + "\u001b[?25hUsing cached sniffio-1.3.1-py3-none-any.whl (10 kB)\n", + "Downloading tabulate-0.10.0-py3-none-any.whl (39 kB)\n", + "Downloading torch_c_dlpack_ext-0.1.5-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl (443 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m443.1/443.1 kB\u001b[0m \u001b[31m789.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m785.5 kB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading cryptography-46.0.6-cp311-abi3-manylinux_2_34_aarch64.whl (4.3 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.3/4.3 MB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading cuda_python-13.2.0-py3-none-any.whl (8.1 kB)\n", + "Downloading dnspython-2.8.0-py3-none-any.whl (331 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m331.1/331.1 kB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading fastapi_cloud_cli-0.15.1-py3-none-any.whl (32 kB)\n", + "Downloading googleapis_common_protos-1.74.0-py3-none-any.whl (300 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m300.7/300.7 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading grpcio-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (6.6 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading hf_xet-1.4.3-cp37-abi3-manylinux_2_28_aarch64.whl (4.0 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.0/4.0 MB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading httptools-0.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (511 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m511.4/511.4 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached mpmath-1.3.0-py3-none-any.whl (536 kB)\n", + "Downloading pycountry-26.2.16-py3-none-any.whl (8.0 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.0/8.0 MB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m0:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hDownloading python_dotenv-1.2.2-py3-none-any.whl (22 kB)\n", + "Downloading rich_toolkit-0.19.7-py3-none-any.whl (32 kB)\n", + "Downloading soxr-1.0.0-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl (204 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m204.7/204.7 kB\u001b[0m \u001b[31m59.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading typer-0.24.1-py3-none-any.whl (56 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.1/56.1 kB\u001b[0m \u001b[31m499.6 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading uvloop-0.22.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (4.3 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.3/4.3 MB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading websockets-16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl (186 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m186.2/186.2 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached zipp-3.23.0-py3-none-any.whl (10 kB)\n", + "Downloading cuda_bindings-13.2.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl (5.7 MB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mm eta \u001b[36m0:00:01\u001b[0m[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading cuda_pathfinder-1.5.1-py3-none-any.whl (49 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.9/49.9 kB\u001b[0m \u001b[31m448.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m1m3.6 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading fastar-0.9.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (760 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m760.6/760.6 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m:01\u001b[0m\n", + "\u001b[?25hUsing cached rich-14.3.3-py3-none-any.whl (310 kB)\n", + "Downloading rignore-0.7.6-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (897 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m897.8/897.8 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading sentry_sdk-2.57.0-py2.py3-none-any.whl (456 kB)\n", + "\u001b[2K \u001b[38;2;114;156;31m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m456.5/456.5 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m[36m0:00:01\u001b[0mm eta \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hUsing cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)\n", + "Using cached markdown_it_py-4.0.0-py3-none-any.whl (87 kB)\n", + "Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)\n", + "Installing collected packages: supervisor, py-cpuinfo, nvidia-ml-py, mpmath, zipp, websockets, uvloop, typing-inspection, tqdm, tabulate, sympy, sniffio, shellingham, setuptools, setproctitle, sentry-sdk, sentencepiece, safetensors, rignore, regex, python-multipart, python-dotenv, pyjwt, pydantic-core, pycountry, pybase64, protobuf, propcache, pillow, partial-json-parser, outlines_core, nvidia-cudnn-frontend, numpy, ninja, networkx, multidict, msgspec, mdurl, loguru, llvmlite, llguidance, lark, jmespath, jiter, interegular, ijson, httpx-sse, httptools, hf-xet, grpcio, fsspec, frozenlist, flashinfer-cubin, filelock, fastar, einops, docstring-parser, dnspython, distro, diskcache, dill, cuda-pathfinder, cloudpickle, click, cbor2, cachetools, blake3, av, astor, apache-tvm-ffi, annotated-types, annotated-doc, aiohappyeyeballs, yarl, watchfiles, uvicorn, torch, tiktoken, starlette, soxr, soundfile, scipy, pydantic, opentelemetry-proto, opencv-python-headless, numba, markdown-it-py, importlib-metadata, huggingface-hub, googleapis-common-protos, gguf, email-validator, depyf, cuda-bindings, cryptography, aiosignal, torchvision, torchaudio, torch-c-dlpack-ext, tokenizers, sse-starlette, rich, resampy, pydantic-settings, pydantic-extra-types, prometheus-fastapi-instrumentator, opentelemetry-exporter-otlp-proto-common, opentelemetry-api, openai-harmony, openai, lm-format-enforcer, fastapi, cuda-python, anthropic, aiohttp, typer, transformers, rich-toolkit, opentelemetry-semantic-conventions, nvidia-cutlass-dsl-libs-base, model-hosting-container-standards, mcp, xgrammar, opentelemetry-sdk, nvidia-cutlass-dsl, mistral_common, fastapi-cloud-cli, fastapi-cli, compressed-tensors, quack-kernels, opentelemetry-semantic-conventions-ai, opentelemetry-exporter-otlp-proto-http, opentelemetry-exporter-otlp-proto-grpc, flashinfer-python, opentelemetry-exporter-otlp, vllm\n", + " Attempting uninstall: setuptools\n", + " Found existing installation: setuptools 82.0.1\n", + " Uninstalling setuptools-82.0.1:\n", + " Successfully uninstalled setuptools-82.0.1\n", + " Attempting uninstall: lark\n", + " Found existing installation: lark 1.3.1\n", + " Uninstalling lark-1.3.1:\n", + " Successfully uninstalled lark-1.3.1\n", + "Successfully installed aiohappyeyeballs-2.6.1 aiohttp-3.13.5 aiosignal-1.4.0 annotated-doc-0.0.4 annotated-types-0.7.0 anthropic-0.89.0 apache-tvm-ffi-0.1.9 astor-0.8.1 av-17.0.0 blake3-1.0.8 cachetools-7.0.5 cbor2-5.9.0 click-8.3.2 cloudpickle-3.1.2 compressed-tensors-0.14.0.1 cryptography-46.0.6 cuda-bindings-13.2.0 cuda-pathfinder-1.5.1 cuda-python-13.2.0 depyf-0.20.0 dill-0.4.1 diskcache-5.6.3 distro-1.9.0 dnspython-2.8.0 docstring-parser-0.17.0 einops-0.8.2 email-validator-2.3.0 fastapi-0.135.3 fastapi-cli-0.0.24 fastapi-cloud-cli-0.15.1 fastar-0.9.0 filelock-3.25.2 flashinfer-cubin-0.6.6 flashinfer-python-0.6.6 frozenlist-1.8.0 fsspec-2026.3.0 gguf-0.18.0 googleapis-common-protos-1.74.0 grpcio-1.80.0 hf-xet-1.4.3 httptools-0.7.1 httpx-sse-0.4.3 huggingface-hub-0.36.2 ijson-3.5.0 importlib-metadata-8.7.1 interegular-0.3.3 jiter-0.13.0 jmespath-1.1.0 lark-1.2.2 llguidance-1.3.0 llvmlite-0.44.0 lm-format-enforcer-0.11.3 loguru-0.7.3 markdown-it-py-4.0.0 mcp-1.27.0 mdurl-0.1.2 mistral_common-1.11.0 model-hosting-container-standards-0.1.14 mpmath-1.3.0 msgspec-0.20.0 multidict-6.7.1 networkx-3.6.1 ninja-1.13.0 numba-0.61.2 numpy-2.2.6 nvidia-cudnn-frontend-1.18.0 nvidia-cutlass-dsl-4.4.2 nvidia-cutlass-dsl-libs-base-4.4.2 nvidia-ml-py-13.595.45 openai-2.30.0 openai-harmony-0.0.8 opencv-python-headless-4.13.0.92 opentelemetry-api-1.40.0 opentelemetry-exporter-otlp-1.40.0 opentelemetry-exporter-otlp-proto-common-1.40.0 opentelemetry-exporter-otlp-proto-grpc-1.40.0 opentelemetry-exporter-otlp-proto-http-1.40.0 opentelemetry-proto-1.40.0 opentelemetry-sdk-1.40.0 opentelemetry-semantic-conventions-0.61b0 opentelemetry-semantic-conventions-ai-0.5.1 outlines_core-0.2.11 partial-json-parser-0.2.1.1.post7 pillow-12.2.0 prometheus-fastapi-instrumentator-7.1.0 propcache-0.4.1 protobuf-6.33.6 py-cpuinfo-9.0.0 pybase64-1.4.3 pycountry-26.2.16 pydantic-2.12.5 pydantic-core-2.41.5 pydantic-extra-types-2.11.1 pydantic-settings-2.13.1 pyjwt-2.12.1 python-dotenv-1.2.2 python-multipart-0.0.22 quack-kernels-0.3.8 regex-2026.4.4 resampy-0.4.3 rich-14.3.3 rich-toolkit-0.19.7 rignore-0.7.6 safetensors-0.7.0 scipy-1.17.1 sentencepiece-0.2.1 sentry-sdk-2.57.0 setproctitle-1.3.7 setuptools-80.10.2 shellingham-1.5.4 sniffio-1.3.1 soundfile-0.13.1 soxr-1.0.0 sse-starlette-3.3.4 starlette-0.52.1 supervisor-4.3.0 sympy-1.14.0 tabulate-0.10.0 tiktoken-0.12.0 tokenizers-0.22.2 torch-2.10.0 torch-c-dlpack-ext-0.1.5 torchaudio-2.10.0 torchvision-0.25.0 tqdm-4.67.3 transformers-4.57.6 typer-0.24.1 typing-inspection-0.4.2 uvicorn-0.43.0 uvloop-0.22.1 vllm-0.19.0 watchfiles-1.1.1 websockets-16.0 xgrammar-0.1.33 yarl-1.23.0 zipp-3.23.0\n", + "\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m26.0.1\u001b[0m\n", + "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n" + ] + } + ], + "source": [ + "!UV_HTTP_TIMEOUT=300 uv pip install -U vllm[audio] --pre \\\n", + " --extra-index-url https://wheels.vllm.ai/nightly/cu130 \\\n", + " --extra-index-url https://download.pytorch.org/whl/cu130 \\\n", + " --index-strategy unsafe-best-match --extra-index-url https://pypi.nvidia.com\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!uv pip install transformers==5.5.0" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[32mINFO\u001b[0m \u001b[90m04-04 15:18:26\u001b[0m \u001b[90m[importing.py:68]\u001b[0m Triton not installed or not compatible; certain GPU-related functions will not be available.\n", + "\u001b[33mWARNING\u001b[0m \u001b[90m04-04 15:18:26\u001b[0m \u001b[90m[interface.py:229]\u001b[0m Failed to import from vllm._C: ImportError('libcudart.so.12: cannot open shared object file: No such file or directory')\n", + "\u001b[31mERROR\u001b[0m \u001b[90m04-04 15:18:26\u001b[0m \u001b[90m[config.py:29]\u001b[0m Failed to import Triton kernels. Please make sure your triton version is compatible. Error: No module named 'triton'\n", + "\u001b[31mERROR\u001b[0m \u001b[90m04-04 15:18:26\u001b[0m \u001b[90m[gpt_oss_triton_kernels_moe.py:64]\u001b[0m Failed to import Triton kernels. Please make sure your triton version is compatible. Error: No module named 'triton'\n", + "Traceback (most recent call last):\n", + " File \"/home/x1/code/external/cookbook/venv/bin/vllm\", line 8, in \n", + " sys.exit(main())\n", + " ^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py\", line 68, in main\n", + " cmd.subparser_init(subparsers).set_defaults(dispatch_function=cmd.cmd)\n", + " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py\", line 138, in subparser_init\n", + " serve_parser = make_arg_parser(serve_parser)\n", + " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/cli_args.py\", line 349, in make_arg_parser\n", + " parser = AsyncEngineArgs.add_cli_args(parser)\n", + " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py\", line 2264, in add_cli_args\n", + " parser = EngineArgs.add_cli_args(parser)\n", + " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py\", line 1294, in add_cli_args\n", + " vllm_kwargs = get_kwargs(VllmConfig)\n", + " ^^^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py\", line 369, in get_kwargs\n", + " return copy.deepcopy(_compute_kwargs(cls))\n", + " ^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/engine/arg_utils.py\", line 280, in _compute_kwargs\n", + " default = default.default_factory() # type: ignore[call-arg]\n", + " ^^^^^^^^^^^^^^^^^^^^^^^^^\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/pydantic/_internal/_dataclasses.py\", line 121, in __init__\n", + " s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)\n", + " File \"/home/x1/code/external/cookbook/venv/lib/python3.12/site-packages/vllm/config/device.py\", line 56, in __post_init__\n", + " raise RuntimeError(\n", + "RuntimeError: Failed to infer device type, please set the environment variable `VLLM_LOGGING_LEVEL=DEBUG` to turn on verbose logging to help debug the issue.\n" + ] + } + ], + "source": [ + "!vllm serve google/gemma-4-E2B-it --max-model-len 8192" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!vllm serve google/gemma-4-E2B-it \\\n", + " --served-model-name gemma4 \\\n", + " --host 0.0.0.0 \\\n", + " --port 8000 \\\n", + " --gpu-memory-utilization 0.90 \\\n", + " --tensor-parallel-size 1 \\\n", + " --max-model-len 8192 \\ \n", + " --max-num-seqs 16 \\\n", + " --enable-chunked-prefill \\\n", + " --enable-prefix-caching \\\n", + " --generation-config auto \\\n", + " --enforce-eager \\\n", + " --chat-template-content-format openai \\\n", + " --trust-remote-code \\\n", + " --mm-encoder-tp-mode data \\ \n", + " --limit-mm-per-prompt image=2,video=1,audio=1 \\\n", + " --reasoning-parser gemma4 \\\n", + " --enable-auto-tool-choice \\\n", + " --tool-call-parser gemma4\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a_KJh4F1Rjav" + }, + "source": [ + "# Gemma 4\n", + "\n", + "Google has released Gemma-4, a successor to its earlier Gemma models, focusing on open-source accessibility and multimodal vision capabilities. The model is designed to handle complex tasks with extended context lengths, support for multiple languages, and vision capabilities, making it suitable for a wide range of applications.\n", + "\n", + "Gemma-4 has gained attention for outperforming notable models like DeepSeek V3, o3-mini, and Llama 3 405B, with extended context lengths, multilingual support, and multimodal capabilities.\n", + "\n", + "- Extended context length from 32K to 128K\n", + "- Available in 2B, 4B, 21B, and 31B sizes .\n", + "- Supports an extensive range of 140 global languages.\n", + "- Additional Features: Supports function calling, where the model can generate code or instructions to call external functions, and structured outputs, enabling it to produce data in specific formats like JSON or tables." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import base64\n", + "from openai import OpenAI\n", + "import os\n", + "# Initialize the client pointing to your vLLM server\n", + "client = OpenAI(\n", + " base_url=\"http://localhost:8000/v1\", \n", + " api_key=\"token-is-ignored-by-vllm\"\n", + ")\n", + "\n", + "def encode_media(path):\n", + " \"\"\"Helper to encode local files to base64 if not using URLs.\"\"\"\n", + " with open(path, \"rb\") as f:\n", + " return base64.b64encode(f.read()).decode(\"utf-8\")\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "text_response = client.chat.completions.create(\n", + " model=\"gemma4\",\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": \"text\", \"text\": \"What is your name ?\"}\n", + " ]\n", + " }]\n", + ")\n", + "\n", + "print(f\"Text Analysis: {text_response.choices[0].message.content}\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "\n", + "# 1. Processing Text + Image\n", + "image_response = client.chat.completions.create(\n", + " model=\"gemma4\",\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": \"text\", \"text\": \"What is depicted in this image?\"},\n", + " {\n", + " \"type\": \"image_url\",\n", + " \"image_url\": {\"url\": f\"data:image/jpeg;base64,{encode_media('media/image.jpg')}\"}\n", + " }\n", + " ]\n", + " }]\n", + ")\n", + "\n", + "\n", + "print(f\"Image Analysis: {image_response.choices[0].message.content}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\n", + "# 2. Processing Text + Audio (Transcription/Reasoning)\n", + "audio_response = client.chat.completions.create(\n", + " model=\"gemma4\",\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": \"text\", \"text\": \"Listen to this clip and provide a summary.\"},\n", + " {\n", + " \"type\": \"input_audio\",\n", + " \"input_audio\": {\n", + " \"data\": encode_media(\"media/kannada_sample.wav\"),\n", + " \"format\": \"wav\"\n", + " }\n", + " }\n", + " ]\n", + " }]\n", + ")\n", + "\n", + "\n", + "print(f\"Audio Summary: {audio_response.choices[0].message.content}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# 3. Processing Text + Video\n", + "# Note: vLLM expects video via 'video_url' or as a series of frames \n", + "# depending on your specific vLLM version's implementation of the OpenAI spec.\n", + "video_response = client.chat.completions.create(\n", + " model=\"gemma4\",\n", + " messages=[{\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": \"text\", \"text\": \"Describe the movement in this video.\"},\n", + " {\n", + " \"type\": \"video_url\",\n", + " \"video_url\": {\"url\": \"https://example.com/sample_video.mp4\"}\n", + " }\n", + " ]\n", + " }]\n", + ")\n", + "\n", + "\n", + "print(f\"Video Description: {video_response.choices[0].message.content}\")\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "i47OYekFdM4V" + }, + "source": [ + "## Function Calling Gemma\n", + "\n", + "Function calling is a feature that allows large language models to interact with external systems, APIs, and tools in a structured way. This feature enables LLMs to not just generate text responses but to invoke specific functions when appropriate based on user queries. At its core, function calling involves describing functions to the LLM and having it intelligently return the function to call along with necessary arguments for particular tasks. For example, an LLM can be configured to retrieve stock price data by calling a specific function when a user asks about the current price of a particular stock." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "UgU2fCoycyhu" + }, + "outputs": [], + "source": [ + "from duckduckgo_search import DDGS\n", + "\n", + "def search(query:str):\n", + " \"\"\"\n", + " search results to the user query\n", + "\n", + " Args:\n", + " query: user prompt to fetch search results\n", + " \"\"\"\n", + " req = DDGS()\n", + " response = req.text(query,max_results=4)\n", + " context = \"\"\n", + " for result in response:\n", + " context += result['body']\n", + " return context" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CQiUJlCBdb8Y" + }, + "outputs": [], + "source": [ + "tools = [search]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "R3z_9TbvufF3" + }, + "outputs": [], + "source": [ + "query = \"which teams played Cricket ICC Champions Trophy 2025 finals and who won?\" # this is a realtime query, this event occured on March 9th 2025" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jHQrvzvRR7n8" + }, + "source": [ + "## Define prompt for function calling\n", + "\n", + "Function Calling follows these steps:\n", + "\n", + "- Your application sends a prompt to the LLM along with function definitions\n", + "- The LLM analyzes the prompt and decides whether to respond directly or use defined functions\n", + "- If using functions, the LLM generates structured arguments for the function call\n", + "- Your application receives the function call details and executes the actual function\n", + "- The function results are sent back to the LLM\n", + "- The LLM provides a final response incorporating the function results" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "j4LiCaOodjyn" + }, + "outputs": [], + "source": [ + "conversation = [\n", + " {\n", + " \"role\": \"system\",\n", + " \"content\": [{\"type\": \"text\", \"text\": \"\"\"\n", + " You are an expert search assistant. Use `search` to get the most accurate details.\n", + " At each turn, if you decide to invoke any of the function(s), it should be wrapped with ```tool_code```. The python methods described below are imported and available,\n", + " you can only use defined methods. The generated code should be readable and efficient. The response to a method will be wrapped in ```tool_output``` use it to call more tools or generate a helpful, friendly response.\n", + " When using a ```tool_call``` think step by step why and how it should be used.\n", + "\n", + " The following Python methods are available:\n", + " \\`\\`\\`python\n", + " def search(query:str):\n", + " \"\n", + " search results to the user query\n", + "\n", + " Args:\n", + " query: user prompt to fetch search results\n", + " \"\n", + " \\`\\`\\`\n", + "\n", + " User: \\{user_message\\}\n", + " \"\"\"}]\n", + " },\n", + " {\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": \"text\", \"text\": query}\n", + " ]\n", + " }\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "nh51Dr6Xde9r" + }, + "outputs": [], + "source": [ + "inputs = processor.apply_chat_template(\n", + " conversation,\n", + " tools=tools,\n", + " add_generation_prompt=True,\n", + " return_dict=True,\n", + " tokenize=True,\n", + " return_tensors=\"pt\",\n", + ").to(model.device, dtype=torch.bfloat16)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xNO25997ebXH" + }, + "outputs": [], + "source": [ + "outputs = model.generate(**inputs, max_new_tokens=512)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "XN32I1J2ltjN" + }, + "outputs": [], + "source": [ + "output = processor.decode(outputs[0], skip_special_tokens=True)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "moOkqbxAo0TT" + }, + "outputs": [], + "source": [ + "response = output.split(\"model\")[-1]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bGF5oJyntu7C" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "```tool_code\n", + "search(query=\"Which teams played in the Cricket ICC Champions Trophy 2025 final and who won?\")\n", + "```\n" + ] + } + ], + "source": [ + "print(response)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pldp3z7USS5P" + }, + "source": [ + "## Execute the function calling code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ni57IK_OpIya" + }, + "outputs": [], + "source": [ + "def extract_tool_call(text):\n", + " import io\n", + " from contextlib import redirect_stdout\n", + "\n", + " pattern = r\"```tool_code\\s*(.*?)\\s*```\"\n", + " match = re.search(pattern, text, re.DOTALL)\n", + " if match:\n", + " code = match.group(1).strip()\n", + " # Capture stdout in a string buffer\n", + " f = io.StringIO()\n", + " with redirect_stdout(f):\n", + " result = eval(code)\n", + " output = f.getvalue()\n", + " r = result if output == '' else output\n", + " return f'```tool_output\\n{r}\\n```'''\n", + " return None" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "SRYVbLKdosaN" + }, + "outputs": [], + "source": [ + "keyword = extract_tool_call(response)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "eznPlrIBpddw" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, + "text/plain": [ + "\"```tool_output\\nThe ninth edition of the Champions Trophy 2025 saw India being crowned as the winners on 9th March 2025 after they overcame New Zealand in the final. Several exceptional performers lit up the tournament with the bat and ball. The best of them made it to the Team of the Tournament. Here's what the side looks like: 1. Rachin Ravindra (New Zealand)The 2025 ICC Champions Trophy was the ninth edition of the ICC Champions Trophy, a quadrennial ODI cricket tournament organised by the International Cricket Council (ICC). In November 2021 as part of the 2024-2031 ICC men's hosts cycle, the ICC announced that the 2025 Champions Trophy would be played in Pakistan. [4]On 19 December 2024, following an agreement between Board of Control for ...Official ICC Champions Trophy, 2025 Cricket website - live matches, scores, news, highlights, commentary, rankings, videos and fixtures from the International Cricket Council. ... 2025. ICC announces Champions Trophy 2025 Team of the Tournament ... Kohli stars as India march into final | Champions Trophy 2025. 4 March, 2025. ICC Champions ...In a thrilling ICC Champions Trophy final at the Dubai International Cricket Stadium on March 9, 2025, India emerged victorious over New Zealand, clinching the title in a closely contested match ...\\n```\"" + ] + }, + "execution_count": 177, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "keyword" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Ek9wlj3xuOnc" + }, + "outputs": [], + "source": [ + "prompt = [{\n", + " \"role\": \"user\",\n", + " \"content\": [\n", + " {\"type\": \"text\", \"text\": f\"Results from the search:{keyword} User_query : {query}\"}\n", + " ]\n", + " }\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "QB331y_TvN7K" + }, + "outputs": [], + "source": [ + "final_prompt = processor.apply_chat_template(\n", + " prompt,\n", + " tools=tools,\n", + " add_generation_prompt=True,\n", + " return_dict=True,\n", + " tokenize=True,\n", + " return_tensors=\"pt\",\n", + ").to(model.device, dtype=torch.bfloat16)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "rcjE05aSvHEI" + }, + "outputs": [], + "source": [ + "response = model.generate(**final_prompt, max_new_tokens=512)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aBcgbJjEvlW4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "user\n", + "Results from the search:```tool_output\n", + "The ninth edition of the Champions Trophy 2025 saw India being crowned as the winners on 9th March 2025 after they overcame New Zealand in the final. Several exceptional performers lit up the tournament with the bat and ball. The best of them made it to the Team of the Tournament. Here's what the side looks like: 1. Rachin Ravindra (New Zealand)The 2025 ICC Champions Trophy was the ninth edition of the ICC Champions Trophy, a quadrennial ODI cricket tournament organised by the International Cricket Council (ICC). In November 2021 as part of the 2024-2031 ICC men's hosts cycle, the ICC announced that the 2025 Champions Trophy would be played in Pakistan. [4]On 19 December 2024, following an agreement between Board of Control for ...Official ICC Champions Trophy, 2025 Cricket website - live matches, scores, news, highlights, commentary, rankings, videos and fixtures from the International Cricket Council. ... 2025. ICC announces Champions Trophy 2025 Team of the Tournament ... Kohli stars as India march into final | Champions Trophy 2025. 4 March, 2025. ICC Champions ...In a thrilling ICC Champions Trophy final at the Dubai International Cricket Stadium on March 9, 2025, India emerged victorious over New Zealand, clinching the title in a closely contested match ...\n", + "``` User_query : which teams played Cricket ICC Champions Trophy 2025 finals and who won?\n", + "model\n", + "According to the search results:\n", + "\n", + "* **Teams:** India and New Zealand\n", + "* **Winner:** India won the final on March 9, 2025.\n" + ] + } + ], + "source": [ + "print(processor.decode(response[0], skip_special_tokens=True))" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "[Gemma_3]Function_Calling_with_HF.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.4" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}