Introduction

The efficiency of running Large Language Models (LLMs) on local hardware has historically been limited by the overhead of high-level frameworks and cross-platform abstractions. Mac users, despite having access to powerful Unified Memory architectures, often find themselves hitting performance ceilings with existing runtimes that weren’t built exclusively for the Metal API. BaseRT is a from-scratch LLM inference runtime for Apple Silicon that addresses this gap by writing directly to the Metal GPU API with zero dependencies on PyTorch, MLX, or CoreML. Designed to extract every cycle of performance from M-series chips, BaseRT has recently gained traction after reports showed it delivering up to 6.4x the throughput of industry standards like llama.cpp on M5 hardware. This post explores the architecture, benchmarks, and implementation details that make BaseRT the new performance ceiling for on-device intelligence.

What Is BaseRT?

BaseRT is a high-performance LLM inference engine that primary functions as a native Metal runtime for [target user] developers and AI researchers looking to deploy local models on Apple Silicon. Developed by Base Compute, the project is a C++ implementation that targets the hardware’s unified memory topology directly. Unlike other runtimes that act as wrappers or utilize lazy-evaluated array frameworks, BaseRT features hand-written Metal kernels and a streamlined decode loop optimized for the specific execution model of Apple’s M-series GPUs. It supports an OpenAI-compatible API, allowing for immediate integration with existing tools while offering a specialized .base model format for optimized weight handling.

The project consists of an open-source CLI, format specifications, and language bindings under the Apache-2.0 license, while the core inference engine is distributed as a proprietary prebuilt binary. This hybrid approach allows BaseRT to maintain deep, chip-specific optimizations that are often difficult to sustain in generic open-source forks. By focusing exclusively on macOS 14+ and M1-or-later hardware, the engine avoids the “abstraction tax” of cross-platform compatibility, resulting in significantly lower latency for interactive applications like coding assistants and real-time voice interfaces.

Why BaseRT Matters

The primary hurdle for local LLM adoption has been the “latency gap” between local execution and high-cost cloud APIs. For professional workflows—such as local RAG (Retrieval-Augmented Generation) or autonomous coding agents—the speed at which a model can process long prompts (prefill) and generate tokens (decode) is the decisive factor in usability. BaseRT matters because it proves that standard framework abstractions leave substantial performance on the table. By bypassing the scheduling layers of MLX or the CPU-first logic of llama.cpp, BaseRT recovers hardware-level performance that was previously inaccessible to developers.

Furthermore, the arrival of the Apple M5 generation has introduced dedicated Neural Accelerators and Metal 4 tensor APIs that standard runtimes are only beginning to explore. BaseRT is one of the first runtimes to publish verified benchmarks (arXiv:2607.00501) showing how hand-written kernels can exploit these tensor cores to push prefill throughput to over 6x that of previous records. For developers building on the edge, this means the ability to run larger, more capable models (up to 35B parameters) on consumer hardware with the snappy, interactive feel usually reserved for sub-1B models. This shift effectively democratizes frontier-grade performance for any user with a modern Mac.

Key Features

• Direct Metal API Implementation: Written from the ground up against Metal with no dependencies on intermediate frameworks like MLX or PyTorch to ensure zero abstraction overhead.
• M5 Neural Accelerator Optimization: Specifically tuned to utilize the tensor-core kernels in Apple’s latest chip architecture for massive prefill throughput gains.
• OpenAI-Compatible API: Supports standard endpoints for chat, completions, embeddings, and tool-calling, making it a drop-in replacement for cloud providers.
• Hand-Written Metal Kernels: Includes optimized implementations for dense GEMM, Mixture-of-Experts (MoE) GEMM, and flash-attention prefill.
• Specialized.base Format: A custom model format supporting affine quantization from Q2 through Q8, along with optional AWQ (Activation-aware Weight Quantization) calibration.
• Unified Memory Topology Awareness: Designed to respect Apple Silicon’s unique memory architecture, reducing unnecessary data copies between the CPU and GPU.
• Multimodal and Tool Support: Beyond text, the server handles vision and audio inputs on supported models, as well as complex tool-calling schemas.
• Cross-Language Bindings: Ships with official stable C interfaces and bindings for Python, Node.js, Rust, and Swift.
• Prefix Caching and Paged KV Cache: Implements advanced memory management to speed up multi-turn conversations and handle long contexts efficiently.

How BaseRT Compares

In the ecosystem of local runtimes, BaseRT positions itself as the “performance-first” option for Apple Silicon. While llama.cpp is the gold standard for cross-platform compatibility and MLX provides a flexible research framework, BaseRT targets the production-grade interactive workload. The following table highlights the throughput differences measured on an Apple M5 Pro device across various model families.

The critical takeaway from these comparisons is the margin of lead in prefill throughput, particularly for Mixture-of-Experts (MoE) models where matrix multiplication dominates. By utilizing the M5 Neural Accelerators, BaseRT achieves 3.9x higher prefill than MLX. While llama.cpp remains superior for developers who need to support Intel Macs or non-Apple hardware, BaseRT provides a massive performance ceiling for those committed to the M-series ecosystem. The tradeoff for this speed is a narrower range of supported architectures and the use of a proprietary engine core, which may be a consideration for teams with strict open-source auditing requirements.

Getting Started: Installation

BaseRT requires an Apple Silicon Mac (M1 or newer) running macOS 14 or later. The installation is handled via a single CLI tool that manages both model downloads and server lifecycle.

Step 1: Install the CLI

The easiest way to install BaseRT is to download the prebuilt binary from the repository or use a one-line installer if provided by the documentation.

curl -fsSL https://basecompute.co/getbasert | sh

Step 2: Verify Installation

Once installed, you can verify that the basert command is available in your terminal by checking the version:

basert --version

Prerequisites

Ensure you have sufficient disk space for the models you intend to run. Since BaseRT uses a custom .base format, it will often convert or pull pre-converted weights from Hugging Face, which typically range from 2GB to 20GB depending on the quantization level (Q2-Q8).

How to Use BaseRT

Using BaseRT is designed to be a zero-migration experience for developers already familiar with the OpenAI API or tools like Ollama. The workflow involves pulling a model and then starting the serve process.

First, you pull a model from the Base Compute registry or Hugging Face. The basert pull command handles the conversion to the optimized .base format automatically. Following the pull, you launch the server with basert serve, which exposes a local endpoint (typically at localhost:8080). You can then point any OpenAI-compatible client—such as LibreChat, Open WebUI, or a custom Python script—to this local server to start generating tokens.

Code Examples

Here is how to pull and run a model using the BaseRT CLI. This example uses the Gemma model family, which is highly optimized for the runtime.

CLI: Pull and Serve

# Download and convert the modelnbasert pull basecompute/gemma-4-E4B-itnn# Start the OpenAI-compatible servernbasert serve basecompute/gemma-4-E4B-it --port 8080

Python: OpenAI-Compatible Request

Once the server is running, you can interact with it using the standard openai library. This makes testing performance a 10-minute experiment.

from openai import OpenAInnclient = OpenAI(base_url="http://localhost:8080/v1", api_key="keyless")nnresponse = client.chat.completions.create(n model="gemma-4-E4B-it",n messages=[{"role": "user", "content": "Explain flash-attention in one sentence."}]n)nnprint(response.choices.message.content)

Advanced Configuration

BaseRT supports several advanced flags to tune the engine for specific hardware or workloads. One of the most impactful settings is --prefix-caching, which should be enabled for multi-turn conversations or agentic loops where the system prompt remains constant. For vision-language models, the server automatically detects multimodal inputs if the model supports it. You can also configure AWQ calibration during the conversion process if you are working with custom weights to maintain high accuracy at low bit-rates (Q4). Additionally, the engine supports tool-calling by default, which can be configured via the standard OpenAI function-calling schema in your API requests.

Real-World Use Cases

• Autonomous Coding Agents: The 6.4x prefill throughput lead makes BaseRT the ideal backend for agents like Claude Code or local IDE plugins that need to ingest large codebases as context without waiting seconds for the first token.
• Local Voice Assistants: The low-latency decode path allows for near-instant speech-to-text and text-to-speech loops on device, providing a more natural conversational experience.
• Privacy-Preserving RAG: Enterprises can index and query sensitive internal documentation using local embeddings and LLMs without data ever leaving the Mac, benefiting from the paged KV cache for long document processing.
• Game Development NPC Logic: Game engines can use the Swift or C++ bindings to integrate dynamic, local AI dialogue that runs on the player’s GPU without consuming the entire VRAM budget, thanks to Q2/Q4 quantization support.

Contributing to BaseRT

While the core engine is proprietary, the BaseRT ecosystem is built on open specifications and bindings. Developers can contribute to the official repository by improving the language bindings (Python, Node, Rust, Swift), updating documentation, or submitting performance benchmarks. According to the repository guidelines, the team is particularly interested in community-contributed .base model recipes and architecture descriptors for new model families. If you find a bug in the CLI or the API integration, you can open an issue on the GitHub repository. Pull requests for new features in the open-source CLI are welcome following the project’s coding standards.

Community and Support

The BaseRT community is active on Product Hunt and X (Twitter), where the team frequently shares engine updates and benchmark methodology. For technical support, the GitHub Discussions tab is the primary channel for asking questions about model compatibility or hardware requirements. The project also maintains a dedicated documentation site at docs.basecompute.co, which includes detailed reference guides for the C API and format specifications. Researchers can refer to the arXiv paper “BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal” for deep dives into the hand-written kernels and M5 performance analysis.

Conclusion

BaseRT represents a paradigm shift in how we approach LLM inference on Apple hardware. By prioritizing direct Metal implementation over the safety of general-purpose frameworks, it achieves performance figures that fundamentally change the utility of on-device AI. Whether you are building an interactive agent that needs sub-second response times or a local RAG system processing massive document contexts, BaseRT offers the throughput required for professional deployment. Its seamless integration with the OpenAI ecosystem means there is virtually no cost to testing it against your current setup.

If you are running an M-series Mac and looking for the fastest possible local inference, we recommend starting with the basert pull of a Gemma or Qwen model today. The performance gains on M3, M4, and especially M5 chips make it a compelling choice for any Mac-based developer. Star the repository, benchmark your own workloads, and join the community of engineers pushing the boundaries of what’s possible on Apple Silicon.

Resources

• BaseRT GitHub Repository
• Technical Paper: BaseRT Benchmark Methodology
• Official BaseRT Documentation
• BaseRT Pre-Converted Models (Hugging Face)