Serve Models on Jobs

Serve Models on Jobs · Hugging Face Hub documentation Serve Models on Jobs Hub 🏡 View all docsAWS Trainium & InferentiaAccelerateArgillaAutoTrainBitsandbytesCLIChat UIDataset viewerDatasetsDeploying on AWSDiffusersDistilabelEvaluateGoogle CloudGoogle TPUsGradioHubHub Python LibraryHuggingface.jsInference Endpoints (dedicated)Inference ProvidersKernelsLeRobotLeaderboardsLightevalMicrosoft AzureOpenEnvOptimumPEFTReachy MiniSafetensorsSentence TransformersTRLTasksText Embeddings InferenceText Generation InferenceTokenizersTrackioTransformersTransformers.jsXetsmolagentstimm Search documentation EN Join the Hugging Face community and get access to the augmented documentation experience Collaborate on models, datasets and Spaces Faster examples with accelerated inference Switch between documentation themes to get started Copy page Serve Models on Jobs With exposed ports, a Job can act as a temporary inference server: start vLLM on a GPU flavor, point any OpenAI-compatible client at the job’s URL, and cancel the job when you are done. You pay per minute while the job runs, and the endpoint disappears with the job. This is a good fit when the endpoint is a means rather than the product: an evaluation run, a data labelling session, iterating on prompts against a hot model, or a demo that only needs to live for an afternoon. If you want a more permanent endpoint that won’t disappear, you want Inference Endpoints, which provides managed infrastructure with autoscaling, monitoring, and stable URLs. Start a vLLM server The`vllm/vllm-openai` image has everything pre-installed. One command starts an OpenAI-compatible server: Copied ```bash >>> hf jobs run --detach --expose 8000 --flavor a10g-small -s HF_TOKEN \ ... vllm/vllm-openai \ ... vllm serve LiquidAI/LFM2.5-8B-A1B --max-model-len 8192 ✓ Job started id: 6a2b137a59bbdade52d4a58c url: https://huggingface.co/jobs/davanstrien/6a2b137a59bbdade52d4a58c Hint: Exposed ports are reachable at (requi