How to self-host MedGemma with Hugging Face TGI as private AI

Open google/medgemma-4b-it on Hugging Face. The model ID is the part after huggingface.co/, and you can copy it with the button next to the name. The text-generation-inference tag means the model works with TGI.

MedGemma needs access approval. Log in to Hugging Face, open the model page, and accept the license terms (Health AI Developer Foundations). Wait until the page says you have access.

Then create a token with Read access at huggingface.co/settings/tokens. TGI uses it to download the model.

On your GPU server (1 × 16 GB GPU or more), run:

docker run -d --rm --name tgi --gpus all --shm-size 1g \
  -p 127.0.0.1:8080:80 \
  -v $PWD/data:/data \
  -e HF_TOKEN=hf_your_token \
  ghcr.io/huggingface/text-generation-inference:3.3.5 \
  --model-id google/medgemma-4b-it \
  --num-shard 1 \
  --cors-allow-origin https://www.typingmind.com

• Replace hf_your_token with your Hugging Face token.
• -d keeps TGI running after you close the terminal.
• -p 127.0.0.1:8080:80 serves TGI on port 8080 of your server, for connections from the server itself only. The SSH tunnel in step III is one of them.
• -v $PWD/data:/data keeps the downloaded model, so the next start is fast.
• --num-shard 1 keeps the model on one GPU, even if your server has more.
• --cors-allow-origin https://www.typingmind.com lets only the TypingMind web app call TGI from a browser, not other websites you visit.

The first start downloads 8.6 GB of model weights, so it can take a while. Follow along with:

docker logs -f tgi

TGI is ready when the log prints Connected. Press Ctrl+C to stop following the log, then check it with:

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "tgi", "messages": [{"role": "user", "content": "Say hi"}]}'

You should get a JSON reply with the model's answer.

TypingMind runs on https://, so your browser blocks calls to a plain http:// server, unless the server is at localhost. An SSH tunnel makes TGI show up at localhost:8080 on your computer. On your computer, run:

ssh -N -L 8080:localhost:8080 user@your-gpu-server

Replace user@your-gpu-server with how you log in to the server, and keep this terminal open while you chat. Skip this step if TGI runs on the computer you use for TypingMind.

Open www.typingmind.com in your browser. Use the web version: the TypingMind Mac app and the Setapp version block plain http:// addresses like TGI's.

Click Models in the left sidebar, then Add custom model.

Fill in the form:

medgemma-4b-it (TGI)google/medgemma-4b-it32768http://localhost:8080/v1/chat/completions

Open the Capabilities tab and turn off Auto check capabilities on save, so TypingMind keeps the features you set below.

Then set the features for MedGemma:

• Support System Role: on
• Support Streaming Output: on
• Support Image Input: on
• Support Plugins: off
• Support PDF Input: off
• Support External Image URLs: off
• Support Thinking: off
• Support Prompt Caching: off

Click Test & Save. If your browser asks to let www.typingmind.com access other apps and services on this device, click Allow. This lets the page reach TGI at localhost.

Pick medgemma-4b-it (TGI) in the model menu under the chat box and say hi. Replies come straight from your own GPU server.

medgemma-4b-it (TGI)

Send images to MedGemma

MedGemma can read images. Click the paperclip under the chat box, attach a photo or screenshot, and ask about it. The image stays on your own GPU server.

You can now use your private AI with TypingMind's chat features: prompts, AI agents, projects, and switching between your self-hosted model and cloud models in the same chat.