How to self-host Llama-3.1-Nemotron with Hugging Face TGI as private AI

Open nvidia/Llama-3.1-Nemotron-70B-Instruct-HF on Hugging Face. The model ID is the part after huggingface.co/, and you can copy it with the button next to the name. The text-generation-inference tag means the model works with TGI.

Llama-3.1-Nemotron is open to everyone, so you don't need a Hugging Face account or token to download it.

On your GPU server (4 × 80 GB GPUs or more), run:

docker run -d --rm --name tgi --gpus all --shm-size 1g \
  -p 127.0.0.1:8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id nvidia/Llama-3.1-Nemotron-70B-Instruct-HF \
  --num-shard 4 \
  --cors-allow-origin https://www.typingmind.com

• -d keeps TGI running after you close the terminal.
• -p 127.0.0.1:8080:80 serves TGI on port 8080 of your server, for connections from the server itself only. The SSH tunnel in step III is one of them.
• -v $PWD/data:/data keeps the downloaded model, so the next start is fast.
• --num-shard splits the model across your GPUs.
• --cors-allow-origin https://www.typingmind.com lets only the TypingMind web app call TGI from a browser, not other websites you visit.

The first start downloads 141.1 GB of model weights, so it can take a while. Follow along with:

docker logs -f tgi

TGI is ready when the log prints Connected. Press Ctrl+C to stop following the log, then check it with:

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "tgi", "messages": [{"role": "user", "content": "Say hi"}]}'

You should get a JSON reply with the model's answer.

TypingMind runs on https://, so your browser blocks calls to a plain http:// server, unless the server is at localhost. An SSH tunnel makes TGI show up at localhost:8080 on your computer. On your computer, run:

ssh -N -L 8080:localhost:8080 user@your-gpu-server

Replace user@your-gpu-server with how you log in to the server, and keep this terminal open while you chat. Skip this step if TGI runs on the computer you use for TypingMind.

Open www.typingmind.com in your browser. Use the web version: the TypingMind Mac app and the Setapp version block plain http:// addresses like TGI's.

Click Models in the left sidebar, then Add custom model.

Fill in the form:

Llama-3.1-Nemotron-70B-Instruct-HF (TGI)nvidia/Llama-3.1-Nemotron-70B-Instruct-HF32768http://localhost:8080/v1/chat/completions

Open the Capabilities tab and turn off Auto check capabilities on save, so TypingMind keeps the features you set below.

Then set the features for Llama-3.1-Nemotron:

• Support System Role: on
• Support Streaming Output: on
• Support Plugins: on
• Support Image Input: off
• Support PDF Input: off
• Support External Image URLs: off
• Support Thinking: off
• Support Prompt Caching: off

Click Test & Save. If your browser asks to let www.typingmind.com access other apps and services on this device, click Allow. This lets the page reach TGI at localhost.

Pick Llama-3.1-Nemotron-70B-Instruct-HF (TGI) in the model menu under the chat box and say hi. Replies come straight from your own GPU server.

Llama-3.1-Nemotron-70B-Instruct-HF (TGI)

Use plugins with Llama-3.1-Nemotron

Llama-3.1-Nemotron supports tool calling, so it can use TypingMind plugins. Open the plugin menu next to the model picker and turn on the ones you want, like web search, image generation, or Mermaid diagrams. The model calls them when it needs them.

Llama-3.1-Nemotron-70B-Instruct-HF (TGI)

You can now use your private AI with TypingMind's chat features: prompts, AI agents, projects, and switching between your self-hosted model and cloud models in the same chat.