Llama Cpp

llama.cpp -- C/C++ LLM Inference Framework Guide

Official Documentation

What is llama.cpp?

llama.cpp is a pure C/C++ LLM inference engine with minimal dependencies, designed for high-performance local inference across CPUs and GPUs. Key properties:

• MIT licensed, extremely active development (~daily releases, currently b8766+)
• Widest hardware support: NVIDIA (CUDA), AMD (ROCm/Vulkan), Apple (Metal), Intel (SYCL/Vulkan), Qualcomm (OpenCL), ARM, WebGPU
• GGUF model format: single-file, mmap-compatible, 40+ quantization types from 1.5-bit to 16-bit
• Built-in HTTP server: OpenAI-compatible API, Anthropic Messages API, streaming, function calling, multimodal
• Language bindings: Python (llama-cpp-python), Go, Rust, C#, Node.js, Java, Swift, and more

Quick Start

Run a model via server (fastest path)

bashCopy code
# Install
brew install llama.cpp    # macOS/Linux
winget install llama.cpp  # Windows

# Start server with a HuggingFace model
llama-server -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M -ngl 99

# Query via curl
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}],"temperature":0.8}'

Embed in a C++ project (library usage)

cppCopy code
#include "llama.h"

// 1. Init backends and load model
ggml_backend_load_all();
auto params = llama_model_default_params();
params.n_gpu_layers = 99;
llama_model * model = llama_model_load_from_file("model.gguf", params);

// 2. Create context
auto ctx_params = llama_context_default_params();
ctx_params.n_ctx = 4096;
llama_context * ctx = llama_init_from_model(model, ctx_params);

// 3. Tokenize, decode, sample (see C API reference for full pattern)

Build from source with GPU

bashCopy code
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON      # or GGML_VULKAN=ON, GGML_METAL=ON
cmake --build build --config Release -j

Core Architecture

Copy code
llama.cpp/
  include/
    llama.h          # C API (primary interface)
    llama-cpp.h      # C++ RAII wrappers (unique_ptr aliases)
    ggml.h           # Tensor computation library
  common/
    common.h         # High-level convenience layer
  tools/
    server/          # HTTP server (llama-server)
    quantize/        # Model quantization tool
  examples/
    simple/          # Minimal inference example
    simple-chat/     # Multi-turn chat example

Inference pipeline:

Copy code
Model (GGUF file)
  -> llama_model_load_from_file()  [load + mmap weights]
  -> llama_init_from_model()       [create context with KV cache]
  -> llama_tokenize()              [text -> tokens]
  -> llama_decode()                [run transformer, fill KV cache]
  -> llama_get_logits()            [get output probabilities]
  -> llama_sampler_sample()        [select next token]
  -> llama_token_to_piece()        [token -> text]
  -> repeat decode/sample loop until EOS

Quantization Quick Reference

bashCopy code
# Quantize a model
llama-quantize model-f16.gguf model-q4km.gguf Q4_K_M

# Convert from HuggingFace
python convert_hf_to_gguf.py /path/to/hf_model --outfile model.gguf

GPU Backends

GPU offloading (-ngl 99) is the single most impactful performance setting.

Server API

The built-in server provides OpenAI-compatible endpoints:

Features: function calling (--jinja), grammar constraints (--grammar/--json-schema), multimodal (vision+audio), parallel decoding, speculative decoding, router mode, LoRA hot-swap, built-in web UI.

CMake Integration

cmakeCopy code
# Method 1: Subdirectory (embedding)
add_subdirectory(vendor/llama.cpp)
target_link_libraries(myapp PRIVATE llama ggml)

# Method 2: Installed package
find_package(llama REQUIRED)
target_link_libraries(myapp PRIVATE llama)

Detailed Reference Documents

• C/C++ API reference: See references/c-api-reference.md for complete llama.h function signatures, types, enums, structs, sampling chain API, and full working examples
• Build system & integration: See references/build-and-integration.md for all CMake options, GPU backend builds, library linking methods, Docker, and package managers
• Server REST API: See references/server-api.md for all HTTP endpoints, CLI flags, environment variables, curl examples, function calling, grammar constraints, and Python client usage
• GGUF & quantization: See references/quantization-guide.md for GGUF format spec, all 40+ quantization types, imatrix generation, model conversion, hardware requirements, and supported architectures
• Performance & GPU backends: See references/performance-and-backends.md for backend comparison, CUDA/Vulkan/Metal optimization, memory management, speculative decoding, and hardware recommendations
• Unreal Engine integration: See references/unreal-engine-integration.md for Llama-Unreal plugin, custom C++ integration with Build.cs, HTTP server approach, performance in-game, and common UE pitfalls