Llama Cpp

llama.cpp

Purpose

• Use this skill to run quantized GGUF models on laptops, workstations, and edge systems.
• Prefer it when you need portable local inference without a heavyweight serving stack.
• llama.cpp is especially useful for CPU-first deployments, low-cost GPU offload, and offline workflows.
• Python users typically access it through llama-cpp-python.

Install

• Fastest path for Python users:

bashCopy code
pip install llama-cpp-python

• Build from source when you need custom acceleration backends or tighter platform control.
• Source builds are common for CUDA, Metal, ROCm, Vulkan, and CPU-tuned environments.
• Confirm the package imports successfully:

bashCopy code
python -c "from llama_cpp import Llama; print('ok')"

Model Format

• llama.cpp primarily uses the GGUF model format.
• GGUF packages tokenizer metadata, architecture settings, and quantized weights in one artifact.
• Choose a GGUF variant that matches your hardware budget and quality target.

Download GGUF Models

• Hugging Face is the standard source for GGUF checkpoints.

• Common repos include:

• bartowski/*

• TheBloke/*

• Typical examples:

• bartowski/Llama-3.1-8B-Instruct-GGUF

• TheBloke/Mistral-7B-Instruct-v0.2-GGUF

• bartowski/Qwen2.5-7B-Instruct-GGUF

• Store the downloaded file locally, for example:

• models/llama-3.1-8b-instruct-q4_k_m.gguf

Quantization Levels

• Q4_K_M: best balance for many local deployments

• Q5_K_M: more quality, more RAM or VRAM

• Q8_0: highest quality among common quantized options

• Q2_K: very small, but quality drops sharply

• Start with Q4_K_M unless you already know the task is quality-sensitive.

• Move to Q5_K_M or Q8_0 for coding, reasoning, or long-form generation where quality matters more.

• Use Q2_K only for extreme memory constraints or experiments.

Basic Python Usage

pythonCopy code
from llama_cpp import Llama

llm = Llama(
    model_path="models/llama-3.1-8b-instruct-q4_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=0,
)

output = llm(
    "Explain why GGUF is useful for local inference.",
    max_tokens=256,
    temperature=0.7,
)

print(output["choices"][0]["text"])

Important Init Parameters

• model_path: path to the .gguf file on disk

• n_gpu_layers: number of transformer layers to offload to GPU

• n_ctx: context size, constrained by the model and available memory

• n_threads: CPU worker threads for prompt processing and generation

• These four parameters are the first tuning knobs to adjust for almost every deployment.

Initialization Guidance

• Keep model_path on a local SSD when possible.
• Set n_threads close to the number of performant CPU cores, not necessarily total logical threads.
• Increase n_ctx only after checking memory pressure.
• Increase n_gpu_layers gradually if the model fails to load or performance is unstable.

Generation Call

• Basic call shape:

pythonCopy code
result = llm(
    "Write a short checklist for running a local LLM service.",
    max_tokens=256,
    temperature=0.7,
)

• Access the text with:

pythonCopy code
print(result["choices"][0]["text"])

• Keep max_tokens bounded for interactive usage.
• Use lower temperatures for summarization, extraction, and tool-style tasks.

Chat Format

• For instruction-tuned models, prefer the chat API:

pythonCopy code
from llama_cpp import Llama

llm = Llama(
    model_path="models/qwen2.5-7b-instruct-q4_k_m.gguf",
    n_ctx=8192,
    n_gpu_layers=20,
    n_threads=8,
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are concise and technical."},
        {"role": "user", "content": "List three tradeoffs of 4-bit quantization."},
    ],
    max_tokens=256,
    temperature=0.4,
)

print(response["choices"][0]["message"]["content"])

• Use the chat API when the model card says the checkpoint is chat-tuned or instruct-tuned.
• Keep prompt templates aligned with the model family if outputs seem malformed.

Streaming Output

• Stream tokens for responsive CLI or web applications:

pythonCopy code
stream = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are concise."},
        {"role": "user", "content": "Describe GPU offload in llama.cpp."},
    ],
    max_tokens=128,
    temperature=0.3,
    stream=True,
)

for chunk in stream:
    delta = chunk["choices"][0].get("delta", {})
    text = delta.get("content")
    if text:
        print(text, end="", flush=True)

• Streaming is useful for chat UIs, terminals, and server-sent event bridges.

GPU Offload

• n_gpu_layers=-1 means full GPU offload when supported by the backend and hardware.
• You can also offload only the first N layers:

pythonCopy code
llm = Llama(
    model_path="models/mistral-7b-instruct-q5_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    n_gpu_layers=-1,
)

• If full offload fails, try a partial value like 20, 30, or 40.
• Partial offload is common on consumer GPUs with limited VRAM.

CPU-Only Example

pythonCopy code
from llama_cpp import Llama

llm = Llama(
    model_path="models/phi-3-mini-q4_k_m.gguf",
    n_ctx=4096,
    n_threads=12,
    n_gpu_layers=0,
)

• CPU-only mode is viable for smaller models and latency-tolerant tasks.
• It is a good fit for offline assistants, batch summarization, and test environments.

Context Size

• n_ctx controls the context window in tokens.
• Larger values increase RAM or VRAM usage.
• The effective maximum depends on the model architecture, quantization, and rope scaling setup.
• Do not assume every GGUF file supports the same long context as the original FP16 checkpoint.

Context Sizing Rule of Thumb

• Start with 4096 or 8192.
• Increase only after verifying memory headroom and prompt quality.
• Very large n_ctx values can degrade throughput significantly.

OpenAI-Compatible Server

• llama-cpp-python includes a server mode:

bashCopy code
python -m llama_cpp.server --model model.gguf

• More realistic example:

bashCopy code
python -m llama_cpp.server \
  --model models/llama-3.1-8b-instruct-q4_k_m.gguf \
  --host 0.0.0.0 \
  --port 8000 \
  --n_ctx 8192

• This is useful for local OpenAI-style integrations, prototypes, and thin service wrappers.
• It is not as throughput-optimized as vLLM, but it is easy to run and distribute.

OpenAI Client Compatibility

• Many local clients can target the server with an OpenAI-compatible base URL:

pythonCopy code
from openai import OpenAI

client = OpenAI(
    api_key="dummy",
    base_url="http://localhost:8000/v1",
)

resp = client.chat.completions.create(
    model="local-model",
    messages=[
        {"role": "system", "content": "You are concise."},
        {"role": "user", "content": "Summarize Q4_K_M vs Q8_0."},
    ],
    max_tokens=128,
)

print(resp.choices[0].message.content)

Build From Source

• Build from source when:

• you need CUDA acceleration not available in your wheel

• you want Metal on macOS

• you need a specific compiler or backend flag

• you are packaging for a controlled deployment target

• Source builds take more effort but often deliver better hardware utilization.

Model Families Commonly Used With GGUF

• Llama

• Qwen

• Mistral

• Phi

• Gemma

• Check the prompt format and tokenizer notes for each family before deploying.

Operational Tips

• Keep a naming convention that encodes model, size, and quantization.
• Store models outside the repo if they are large.
• Benchmark both prompt evaluation speed and generation speed.
• Use the smallest model that meets quality targets.
• Prefer chat-tuned checkpoints for agentic or assistant workloads.

Common Errors

• Model will not load:

• verify model_path

• verify the file is a GGUF checkpoint

• verify the quantization is supported by your build

• Very slow generation:

• increase n_threads

• enable GPU offload

• reduce n_ctx

• use a smaller model

• Out-of-memory:

• reduce n_ctx

• choose Q4_K_M instead of Q8_0

• reduce n_gpu_layers or use CPU-only mode

• Bad chat formatting:

• use create_chat_completion

• verify the checkpoint is instruct-tuned

• check whether the model expects a specific chat template

When To Use This Skill

• You need fully local inference with minimal infrastructure.
• You want a portable inference path for laptops or edge devices.
• You are testing GGUF quantizations before wider deployment.
• You need CPU inference or partial GPU offload instead of a GPU-only server.

Quick Reference

• Install: pip install llama-cpp-python
• Source build: use when you need custom acceleration
• Load model: from llama_cpp import Llama
• Key params: model_path, n_gpu_layers, n_ctx, n_threads
• Generate: llm("prompt", max_tokens=256, temperature=0.7)
• Chat: llm.create_chat_completion(messages=[...])
• Server: python -m llama_cpp.server --model model.gguf
• Best balance quant: Q4_K_M
• Full GPU offload: n_gpu_layers=-1