Introduction

Deploying large language models (LLMs) at scale often involves a complex struggle between maximizing GPU utilization and maintaining low latency. For developers who have previously relied on the ray-llm repository, the ecosystem has evolved. The functionality of the archived ray-llm project has been upstreamed into the core Ray framework, transitioning to the native ray.serve.llm and ray.data.llm APIs. This shift allows developers to leverage a unified AI compute engine to handle both online serving and offline batch inference with enterprise-grade scalability.

What Is Ray LLM?

Ray LLM refers to the suite of native APIs within the Ray framework that enable the deployment, scaling, and management of Large Language Models. Specifically, it is split into two primary modules: ray.serve.llm for production-grade online inference and ray.data.llm for high-throughput batch processing. These tools are built on top of Ray’s distributed runtime, allowing Python developers to scale models across multiple GPUs and nodes without rewriting their core logic.

Maintained by the Ray project and Anyscale, these APIs are released under the Apache 2.0 license. They provide first-class integration with high-performance inference engines like vLLM and SGLang, effectively replacing the standalone ray-llm repository which served as the initial incubator for these capabilities.

Why Ray LLM Matters

Before the introduction of native LLM APIs, developers often had to write significant amounts of boilerplate code to integrate LLM engines with Ray’s distributed scaling. This created a friction point where users had to manually manage actor placement, GPU memory, and request routing to avoid bottlenecks.

Ray LLM matters because it abstracts the distributed systems complexity. By providing a standardized way to deploy vLLM or OpenAI-compatible endpoints, it allows teams to move from a local prototype to a multi-node cluster in minutes. The integration of ray.data.llm specifically solves the “batch inference pain”—the inefficient process of querying an online server for millions of rows of data, which is often slow and costly.

With the transition to native APIs, the community now benefits from tighter integration with Ray’s core autoscaling and fault tolerance mechanisms, ensuring that production LLM workloads remain resilient even as cluster sizes grow.

Key Features

Online Serving (ray.serve.llm)

• OpenAI-Compatible API: Provides a standardized interface that aligns with vLLM’s server, allowing users to switch between vllm serve and Ray Serve LLM with zero code changes to the client.
• Advanced Parallelism: Seamlessly combines tensor parallelism, pipeline parallelism, and expert parallelism to serve massive models that span multiple GPUs or nodes.
• Prefill-Decode Disaggregation: Optimizes resource utilization by separating the prefill (initial prompt processing) and decode (token generation) phases, reducing latency and increasing throughput.
• Multi-LoRA Support: Allows multiple Low-Rank Adaptation (LoRA) adapters to be served using a shared base model, maximizing GPU memory efficiency.
• Automatic Scaling: Leverages Ray Serve’s built-in autoscaling to dynamically adjust the number of model replicas based on incoming request volume.

Offline Batch Inference (ray.data.llm)

• High-Throughput Processing: Integrates LLM inference directly into Ray Data pipelines, eliminating the need for external proxying or load balancing for batch jobs.
• Streaming Execution: Processes datasets that exceed the aggregate RAM of the cluster by streaming data through the LLM processor, preventing out-of-memory errors.
• Engine-Agnostic Architecture: Supports multiple inference engines, including vLLM and SGLang, as well as querying hosted OpenAI-compatible endpoints.
• Fault Tolerance: Built-in retry semantics and automatic sharding ensure that batch jobs complete even if individual nodes in the cluster fail.

How Ray LLM Compares

When evaluating Ray LLM, it is important to distinguish between the general-purpose distributed compute of Ray and specialized serving frameworks. While tools like vLLM are excellent engines, Ray LLM provides the orchestration layer that allows those engines to scale across a cluster.

The primary differentiator for Ray LLM is its ability to unify online and offline workloads. Most frameworks force a choice: either you use a serving tool for real-time API requests or a batch processing tool for data labeling. Ray LLM allows you to use the same model configuration and the same cluster to perform both tasks, significantly reducing operational overhead.

Compared to Kubernetes-native tools like KServe, Ray provides a more “Python-first” experience. While KServe is powerful for platform engineers, Ray allows the ML engineer to define the deployment logic in Python code, making the iteration cycle much faster.

Getting Started: Installation

To use the native LLM APIs, you must install Ray with the appropriate extensions. The installation process varies depending on whether you are targeting online serving or batch processing.

Using pip

For general LLM support, including Ray Serve and the LLM modules, use the following command:

pip install -U "ray[serve,llm]"

For Ray Data LLM specifically

If your primary focus is batch inference, ensure you have a version of Ray greater than or equal to 2.44.1 to access the ray.data.llm module:

pip install -U "ray[data,llm]>=2.53.0"

Prerequisites

Because most LLM inference engines (like vLLM) are GPU-accelerated, a CUDA-compatible GPU is required for most production deployments. Ensure your NVIDIA drivers and CUDA toolkit are correctly installed on all nodes in your cluster.

How to Use Ray LLM

The workflow for Ray LLM depends on whether you are deploying for online access or processing a dataset.

Online Serving Workflow

For online serving, you define an LLMConfig and use the build_openai_app helper to create a deployment that mimics the OpenAI API. This deployment is then run via serve.run(), which handles the distribution of the model across the cluster.

Batch Inference Workflow

For batch processing, you use build_llm_processor. You create a Ray Data dataset, load it into the cluster, and then apply the LLM processor to the dataset. The processor handles the batching, request routing, and tokenization internally, allowing you to treat the LLM as a functional transformation of your data.

Code Examples

Example 1: Deploying an OpenAI-Compatible Endpoint

This example shows how to deploy a small model using ray.serve.llm to create a production-ready API endpoint.

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app

# Configure the model and engine
llm_config = LLMConfig(
    model_loading_config=dict(
        model_id="qwen-0.5b",
        model_source="Qwen/Qwen2.5-0.5B-Instruct",
    ),
    engine_kwargs=dict(
        # Example: set pooler_config for embeddings
        pooler_config=dict(task="embed"),
    ),
)

# Build the OpenAI-compatible app
app = build_openai_app({"llm_configs": [llm_config]})

# Deploy to the cluster
serve.run(app, blocking=True)

Example 2: High-Throughput Batch Inference

This example demonstrates how to use ray.data.llm to run inference on a dataset of prompts.

import ray
from ray.data.llm import vLLMEngineProcessorConfig, build_processor

# Initialize Ray
ray.init()

# Create a simple dataset of prompts
ds = ray.data.from_items([
    {"prompt": "What is machine learning?"},
    {"prompt": "Explain neural networks in one sentence."},
])

# Configure the vLLM engine for batch processing
config = vLLMEngineProcessorConfig(
    model="meta-llama/Llama-3.1-8B-Instruct",
    # You can adjust concurrency to maximize GPU utilization
    concurrency=4,
)

# Build the processor
processor = build_processor(config)

# Run inference on the dataset
results = processor(ds)

# Show the results
results.show()

Real-World Use Cases

Ray LLM is particularly effective in scenarios where the infrastructure must scale dynamically based on the workload type.

• Enterprise RAG Pipelines: A company can use ray.serve.llm to power a customer-facing chatbot and ray.data.llm to periodically re-index their entire knowledge base by generating embeddings for millions of documents.
• Synthetic Data Generation: ML teams can use ray.data.llm to generate thousands of high-quality synthetic training examples for a smaller, specialized model, leveraging the full throughput of a GPU cluster.
• Multi-Model A/B Testing: Using Ray Serve’s advanced routing, teams can route a percentage of traffic to a new model version (e.g., Llama 3.1 vs Llama 3.2) to compare performance in real-time without downtime.
• Large-Scale Content Classification: A social media platform can use batch inference to classify the sentiment or toxicity of millions of posts per hour, using the lazy execution of Ray Data to ensure the cluster doesn’t crash from memory overflow.

Contributing to Ray LLM

Since the LLM APIs are now part of the core Ray project, all contributions should be directed to the main Ray GitHub repository. There is no longer a separate ray-llm repository for active development.

To contribute, developers should start by browsing the “Issues” tab in the ray-project/ray repository and looking for labels like “good first issue” or “llm” to find areas for improvement. Pull requests should follow the Ray project’s standard contribution guidelines and Code of Conduct, ensuring that all changes are tested against the distributed runtime.

Community and Support

The Ray ecosystem is supported by a massive community of AI engineers. Official support channels include the Ray GitHub Discussions forum, GitHub Issues for bug reports, and the lawide-reaching Ray Slack community where developers can and discuss real-time implementation details.

For detailed technical guidance, the official Ray documentation site is the primary resource. It contains comprehensive user guides for both ray.serve.llm and ray.data.llm, specifically focusing on vLLM compatibility and production deployment strategies.

Conclusion

The transition from the standalone ray-llm repository to native ray.serve.llm and ray.data.llm APIs represents a maturation of the LLM deployment landscape. By integrating these capabilities directly into the Ray core, the project has eliminated the redundant boilerplate and provided a professional, unified approach to scaling LLMs.

For developers who need to move beyond single-node inference and into multi-node, production-grade serving, Ray LLM is the right choice. It provides the necessary orchestration to maximize GPU utilization while maintaining the flexibility of a Python-native environment.

Star the main Ray repository, try the quickstart for ray.serve.llm, and join the community to start scaling your AI workloads.

Resources

• Ray Main GitHub Repository
• Ray Serve LLM Documentation
• Ray Data LLM Documentation
• Anyscale Blog for LLM Scaling Strategies