Modal pushes Auto Endpoints: Put inference optimization into a single command

A One-Line Command to Use Inference Parameters Others Spent Months Tuning

Modal has just rolled out something new called Auto Endpoints. Simply put, it wraps production-grade inference optimizations for vLLM / SGLang — including speculative decoding, quantization, engine patches, KV cache configuration — all behind a single CLI command. You just run modal endpoint create and it handles the rest.

It looks like yet another “plug-and-play inference service,” but a closer look shows that Modal’s approach differs from hosted APIs like Together, Fireworks, and Replicate: it exposes what is usually hidden in a black box, letting you own your entire inference stack. Their tagline is interesting — "Inference you actually own". In 2026, when open-source models have matched closed-source ones in quality but self-deployment still scares off many teams, this is a notable product positioning.

How the One Command Works

Anyone familiar with Modal knows its Python SDK decorator system. Auto Endpoints takes this abstraction a step higher and turns it into a CLI subcommand:

# Deploy the NVFP4 version of Kimi K2.6
modal endpoint create kimi-k2-6-nvfp4 \
  --model nvidia/Kimi-K2.6-NVFP4

# Deploy Qwen3.6 35B A3B (MoE with 3B activated)
modal endpoint create qwen3-6-35b-a3b \
  --model Qwen/Qwen3.6-35B-A3B

# Deploy Gemma 4 E4B IT
modal endpoint create gemma-4-e4b-it \
  --model google/gemma-4-E4B-it

After execution, you get an OpenAI-compatible HTTPS endpoint — backed by a vLLM engine tuned by Modal, GPU pools (H100/H200/B200) that autoscale with your traffic, and a per-replica metrics panel.

The model library currently covers mainstream open-source players — NVIDIA Nemotron 3 Super 120B A12B NVFP4, GPT-OSS 120B, Qwen3.5 397B A17B FP8, the Gemma 4 series, Kimi K2.6 NVFP4 — with new models generally added within days. You can also pull your own weights from Hugging Face or mount private weights from a Modal Volume.

What’s Actually Worth Talking About — the “Exposure”

If it were just “one command to deploy an open-source model,” the market is already crowded. The real difference is: it hands over knobs that hosted providers usually don’t let you touch.

Configurable Speculative Decoding

Speculative decoding has been a big weapon in open-source inference performance — using a small draft model to guess tokens ahead, with the large model only verifying, boosting throughput by 1.5–3×. But it’s extremely sensitive to model pairing, temperature, and acceptance rate, and if not tuned, can be slower.

Most hosted APIs either don’t enable it or enable it by default without letting you control the draft model. Auto Endpoints exposes draft model selection, verification strategy, and fallback logic — letting you tailor setups for different workloads (e.g., code completion vs long-form generation).

Engine Patches Not a Black Box

Modal has its own patches on vLLM / SGLang — and they’ve never hidden that, with code public on GitHub. With Auto Endpoints, you can choose to run the official version, Modal’s patched version, or stack your own patches. This matters for teams heavily customizing vLLM — switching providers often means rewriting your patches.

Per-Replica Metrics

This one’s for SREs. Hosted providers usually give endpoint-level P50/P99 metrics — nice-looking but useless for pinpointing issues. Auto Endpoints gives per-replica TTFT, TPOT, batch size, KV cache hit rate, and memory usage, so when you see latency jitter in the tail, you can directly check if a replica got hit by an extremely long context request.

This level of transparency used to require running Kubernetes + vLLM + Prometheus yourself.

Competition and Differentiation

Placing Auto Endpoints on the industry map, the competitors fall into three categories:

First: pure hosted APIs — Together, Fireworks, Groq, DeepInfra. Advantages: cheap (per-token billing), zero operations. But you don’t get replica metrics, can’t tune speculative decoding, and model availability is at the provider’s discretion. Modal’s positioning is clear: not aiming for this market, but targeting those who “need their own inference stack but don’t want to set up K8s.”

Second: enterprise LLM platforms — Red Hat AI Inference Server, Alibaba Cloud PAI-EAS, AWS SageMaker. These shine in compliance, hybrid cloud, enterprise IT integration. But DevEx suffers — going from pulling an image to getting an endpoint can take days. Modal’s advantage is serverless development speed, with second-level cold starts and a CLI experience, enabling much faster iteration.

Third: self-hosting — vLLM + Ray Serve + self-purchased or rented H100s. This offers maximum performance potential but requires a team skilled in inference optimization. Modal’s pitch to these teams: tuning is pre-packed, yet all knobs are yours to turn — like hiring a 24/7 inference engineer.

Details Worth Noting

• Cold Start: Modal continues pushing down cold start times — initial load for 120B-class models is now down to tens of seconds, thanks to weight shard preheating and GPU pool reservation. Auto Endpoints inherits this, making it friendlier for tidal traffic — scaling to 0 overnight to save cost.
• NVFP4 / FP8 Priority: The catalog is heavy on NVFP4 and FP8 quantized versions, aligned with NVIDIA’s FP4 inference push on Blackwell this year. For teams chasing cost-per-token, this is a clear benefit.
• OpenAI Compatibility: All Auto Endpoints produce OpenAI-format interfaces, meaning your current SDKs, agent frameworks, and evaluation pipelines don’t need changes.
• Data Stays in Account: Important for B2B — inference runs in your own Modal workspace, unlike calling Together API where data passes through their gateway.

A Take

The 2026 LLM inference market is clearly stratified:

• Bottom layer — chips and engines (NVIDIA, vLLM, SGLang, TensorRT-LLM)
• Middle layer — tuning and deployment (Modal, Anyscale, BentoML)
• Top layer — API aggregation (OpenAI Hub-type platforms with one key to call all models — GPT, Claude, Gemini, DeepSeek — aggregated in OpenAI-compatible format, with domestic direct links)

Auto Endpoints is Modal moving one notch up from the middle layer — previously, Modal was “giving you GPU and a Python SDK, you write inference logic yourself.” Now it’s “giving you a one-line command to produce a production-grade endpoint.” The significance: it lowers the self-hosting barrier for open-source models from ‘needs an inference engineer’ to ‘needs a backend who can use CLI’.

For businesses needing to control their inference stack (compliance, custom patches, training-inference integration, RL rollout scenarios) but not wanting to maintain infra teams, this is a strong product. For pure API calls, aggregation platforms remain easier — the two approaches don’t conflict, akin to the “self-hosted vs SaaS” split in the web era.

The next move to watch: whether Modal will extend Auto Endpoints to multimodal inference (VLM, TTS, video generation). Their solutions page already mentions a case of 3× latency optimization for VLM document parsing, so this direction will likely continue.

References

• Modal LLM Solutions Official Page — Auto Endpoints model catalog and CLI usage
• Red Hat AI Inference Server Technical Deep Dive — Enterprise vLLM deployment comparison