Introduction

Deploying generative AI models into production requires more than just accuracy; it requires a deep understanding of how the model performs under real-world load. For developers and ML engineers, the challenge is often measuring the exact latency and throughput of Large Language Models (LLMs) without the noise of infrastructure overhead. GenAI-Perf is a specialized command-line tool from NVIDIA, designed to measure the throughput and latency of generative AI models served through an inference server, providing the critical token-level metrics needed to optimize deployment costs and user experience.

What Is GenAI-Perf?

GenAI-Perf is a command-line tool that measures the throughput and latency of generative AI models as served through an inference server. It is part of the Triton Inference Server ecosystem and is specifically optimized for Large Language Models (LLMs), Vision Language Models, Embedding Models, and Ranking Models. The tool allows users to simulate various load patterns—such as concurrent requests or specific request rates—to determine how an inference server handles traffic.

Maintained by NVIDIA, the tool is designed to work with servers that are OpenAI API-compatible or running on the Triton Inference Server. It provides a standardized way to gather performance data, which can then be exported to CSV or JSON formats for deeper analysis and visualization.

Why GenAI-Perf Matters

In the world of LLM deployment, traditional benchmarking tools often fail because they measure the entire request-response cycle as a single block. However, LLMs generate tokens sequentially (streaming), which means the user experience is defined by when the first token appears and how quickly subsequent tokens follow. GenAI-Perf fills this gap by providing granular, token-level metrics that are invisible to standard HTTP benchmarking tools.

By accurately measuring Time to First Token (TTFT) and Inter-Token Latency (ITL), engineers can identify whether a performance bottleneck is occurring during the prefill stage (processing the prompt) or the decoding stage (generating the response). This distinction is crucial for capacity planning, resource allocation, and choosing the right hardware—such as deciding between different NVIDIA GPU generations or adjusting batch sizes to balance throughput and responsiveness.

Key Features

• Token-Level Metrics: Measures critical LLM indicators including Time to First Token (TTFT), Inter-Token Latency (ITL), and output token throughput.
• Diverse Model Support: Capable of benchmarking Large Language Models, Vision Language Models, Embedding Models, Ranking Models, and setups using multiple LoRA adapters.
• Flexible Load Generation: Supports both concurrency mode (maintaining a fixed number of simultaneous requests) and request rate mode (sending requests at a specific frequency).
• OpenAI API Compatibility: Works not only with Triton Inference Server but also with any inference service that conforms to the OpenAI API specification.
• Synthetic and Dataset-Driven Inputs: Users can generate synthetic prompts based on mean and standard deviation of token counts or use real-world datasets like OpenOrca or CNN/DailyMail.
• Detailed Result Logging: Automatically logs all performance results into CSV and JSON files, enabling the creation of custom plots and external visualizations.
• Warm-up Request Support: Allows the specification of warm-up requests to ensure the model is fully loaded into GPU memory and the cache is primed before measurement begins.

How GenAI-Perf Compares

When choosing a benchmarking tool, it is important to distinguish between system-level performance testing and model-specific benchmarking. While tools like Apache JMeter or Locust measure the overall capacity of the server infrastructure, GenAI-Perf focuses on the internal efficiency of the inference engine.

The primary differentiator for GenAI-Perf is its precision. By utilizing a sliding window technique and excluding warm-up periods, it provides a cleaner measurement of the model’s actual performance. In contrast, framework-specific scripts (like those provided by vLLM) are excellent for quick experiments but may lack the rigorous load generation and reporting features required for enterprise-grade capacity planning.

It is worth noting that NVIDIA has recently begun phasing out GenAI-Perf in favor of AIPerf, which offers enhanced features and continued support. For new projects, developers should evaluate AIPerf, though GenAI-Perf remains a stable and widely used tool for Triton-based deployments.

Getting Started: Installation

The most reliable way to install GenAI-Perf is through the Triton Server SDK container, which comes pre-configured with all necessary dependencies, including the Perf Analyzer binary.

Triton SDK Container Method

export RELEASE="24.06"
docker run -it --net=host --gpus=all nvcr.io/nvidia/tritonserver:${RELEASE}-py3-sdk

# Verify installation
genai-perf --help

From Source Installation

Installing from source requires CUDA 12 and Python 3.10+. You must first install the Perf Analyzer binary as GenAI-Perf depends on it.

pip install tritonclient
apt update && apt install -y --no-install-recommends libb64-0d libcurl4

# Clone and install
git clone https://github.com/triton-inference-server/perf_analyzer.git
cd perf_analyzer/genai-perf
pip install -e .

How to Use GenAI-Perf

To use GenAI-Perf, your inference server must already be running. The tool acts as a client that sends requests and measures the time it takes for the server to respond. The basic workflow involves defining the model, the server URL, and the load parameters.

The most common command is the profile command, which executes the benchmark. You specify the --model to target and the --url of your server. To simulate real users, you can use --concurrency to set how many requests are happening simultaneously.

For the most accurate results, always include a --warmup-request-count. This ensures that the GPU kernels are initialized and the model is fully resident in memory before the tool starts recording metrics.

Code Examples

The following examples demonstrate how to profile a model using different backends and API types.

Example 1: Profiling GPT-2 on Triton with TensorRT-LLM

This example uses synthetic data to measure the performance of a GPT-2 model served via the TensorRT-LLM backend.

genai-perf profile \
  -m gpt2 \
  --service-kind triton \
  --backend tensorrtllm \
  --num-prompts 100 \
  --synthetic-input-tokens-mean 200 \
  --output-tokens-mean 100 \
  --streaming \
  --url localhost:8001

Example 2: Profiling an OpenAI-Compatible API

This example targets a server (such as vLLM or NVIDIA NIM) that uses the OpenAI API specification.

genai-perf profile \
  -m llama-3-8b \
  --service-kind openai \
  --url http://localhost:8000/v1 \
  --concurrency 10 \
  --num-prompts 500

Example 3: Using a Custom Dataset

Instead of synthetic data, you can use a predefined dataset to simulate more realistic prompt lengths and distributions.

genai-perf profile \
  -m llama-3 \
  --input-dataset openorca \
  --num-prompts 100 \
  --url localhost:8001

Advanced Configuration

For complex benchmarking scenarios, GenAI-Perf supports the use of YAML configuration files. This allows you to version-control your benchmark settings and ensure reproducibility across different environments.

You can run a benchmark using a config file with the following command: genai-perf config -f benchmark_config.yaml.

Key configuration options include measurement-interval, which defines the time window for stabilization, and tokenizer, which allows you to specify a Hugging Face tokenizer to accurately calculate token counts for prompts and responses.

Real-World Use Cases

GenAI-Perf is particularly effective in the following scenarios:

• Hardware Right-Sizing: An ML engineer compares the TTFT of a Llama-3 model on an NVIDIA A100 vs. an H100 to determine if the hardware upgrade provides a sufficient user-experience improvement to justify the cost.
• Backend Optimization: A developer tests the throughput of the same model served via vLLM vs. TensorRT-LLM to decide which backend provides the best tokens-per-second for their specific workload.
• SLA Verification: A company defines a Service Level Agreement (SLA) that requires the first token to be delivered within 200ms. They use GenAI-Perf to verify that their current deployment can maintain this SLA under a concurrency of 50 simultaneous users.
• LoRA Adapter Performance: A team deploying multiple fine-tuned adapters for different clients uses GenAI-Perf to measure if switching between adapters introduces significant latency spikes.

Contributing to GenAI-Perf

GenAI-Perf is part of the perf_analyzer repository. Contributions are welcome and typically follow the standard GitHub flow: opening an issue to discuss a feature or reporting a bug via the repository’s issue tracker.

If you are submitting a Pull Request, ensure that your changes are consistent with the current CLI options and that you have tested your benchmarks against a running Triton Inference Server. For significant enhancements, it is recommended to discuss the project maintainers via GitHub Discussions.

Community and Support

Since GenAI-Perf is an official NVIDIA tool, the primary support channels are the NVIDIA Developer Forums and the GitHub repository for the Triton Inference Server. Users can find detailed documentation on the NVIDIA Triton documentation site.

The community is highly active, consisting of ML engineers and DevOps professionals who optimize high-scale AI deployments. Support is available through GitHub Issues for bug reports and technical queries regarding the tool’s installation and usage.

Conclusion

GenAI-Perf is an essential tool for any team moving from a prototype LLM to a production-ready service. By providing granular token-level metrics like TTFT and ITL, it removes the guesswork from performance tuning. Whether you are optimizing for cost, user experience, laency, or throughput, GenAI-Perf provides the factual data needed to make informed infrastructure decisions.

While the tool is being transitioned to AIPerf, GenAI-Perf remains a robust and reliable choice for benchmarking Triton-based and OpenAI-compatible services. We recommend starting with the Triton SDK container for the fastest setup and the most stable experience.

Star the repo, try the quickstart, and join the community to start optimizing your generative AI deployments.

Resources

• Triton Inference Server GitHub Repository
• GenAI-Perf / Perf Analyzer Repository
• GenAI-Perf Official Tutorial