Litert Lm

This skill provides step-by-step guidance for building an Android app that runs an open text LLM through the LiteRT-LM Kotlin API (com.google.ai.edge.litertlm; guide: https://ai.google.dev/edge/litert-lm/android, sources: https://github.com/google-ai-edge/LiteRT-LM/blob/main/README.md) on the CPU or the GPU. The model is one .litertlm file. Images, audio, tool calling and NPU backends are not covered. For vision and audio models, use the litert-runtime skill: https://github.com/google-ai-edge/litert-samples/blob/main/skills/litert-runtime/SKILL.md (LiteRT: https://github.com/google-ai-edge/litert/blob/main/README.md).

Prerequisites

• A Kotlin Android project (Android Studio's Empty Activity template is enough). The litertlm-android 0.18.0 AAR declares minSdk 24.
• The dependency in the app-level build.gradle.kts: implementation("com.google.ai.edge.litertlm:litertlm-android:0.18.0") from Google Maven, which lists the newer releases.
• For the GPU backend, both lines inside <application> in AndroidManifest.xml, and <uses-permission android:name="android.permission.INTERNET"/> if the app downloads the model:

xmlCopy code
<uses-native-library android:name="libvndksupport.so" android:required="false"/>
<uses-native-library android:name="libOpenCL.so" android:required="false"/>

• The model file lives on the device's filesystem (hundreds of MB to a few GB), never in assets or in the APK.

Detailed steps

1. Pick a model from litert-community

Choose a .litertlm file at https://huggingface.co/litert-community. Start with litert-community/gemma-4-E2B-it-litert-lm (the file gemma-4-E2B-it.litertlm, 2.6 GB) or the smaller litert-community/Qwen3-0.6B (Qwen3-0.6B.litertlm, 0.6 GB); both files run on the CPU and the GPU. A SoC suffix such as _Google_Tensor_G5 or .mediatek.mt6993 marks an NPU build (not covered here). The model card names the tested backends and the size. Converting a model yourself is a separate step: get a model.

To check a file on a Mac before it goes on the phone, run it with the LiteRT-LM version the app uses: uvx litert-lm@0.18.0 run --from-huggingface-repo=litert-community/Qwen3-0.6B Qwen3-0.6B.litertlm --prompt="What is the capital of France?" (needs uv). Keep the version in the command: with litert-lm 0.17.0, the same command answers a different question and still exits with 0.

2. Put the model file on the device

For the first run, copy the file into the app's private storage with adb: adb push model.litertlm /data/local/tmp/, then adb shell run-as <package> cp /data/local/tmp/model.litertlm files/ (a debuggable build). For users, download it inside the app, with your HTTP client into filesDir or with DownloadManager into getExternalFilesDir(null) (it cannot write to internal storage), show the progress, and check the size against the model card before loading. The engine takes the absolute path.

3. Initialize the engine off the main thread

Engine(config).initialize(), with EngineConfig(modelPath = …, backend = …, cacheDir = …) as below, loads the weights and blocks for seconds, so it runs on a background thread. Backend.CPU(), the default of EngineConfig, needs nothing from the device; Backend.GPU() needs the two manifest lines: without them the engine initializes, and the first reply ends in the error Can not find OpenCL library on this device. cacheDir speeds up the second load. Engine and Conversation are AutoCloseable. A second close() on either throws IllegalStateException, and so does close() on an engine whose initialize() threw: that engine holds nothing, so the code below stores it only after initialize() returns.

4. Conversation, streaming and the screen

kotlinCopy code
data class ChatState(val ready: Boolean = false, val busy: Boolean = false, val reply: String = "", val error: String? = null)

class ChatViewModel(app: Application) : AndroidViewModel(app) {
    private val executor = Executors.newSingleThreadExecutor()
    private val scope = CoroutineScope(SupervisorJob() + executor.asCoroutineDispatcher())
    private val mutex = Mutex()
    private var engine: Engine? = null
    private var conversation: Conversation? = null
    private val _state = MutableStateFlow(ChatState())
    val state: StateFlow<ChatState> = _state.asStateFlow()

    fun load(modelPath: String, backend: Backend = Backend.CPU()) {
        scope.launch {
            mutex.withLock {
                if (engine?.engineConfig?.modelPath == modelPath) return@launch
                release()
                try {
                    val config = EngineConfig(modelPath = modelPath, backend = backend, cacheDir = getApplication<Application>().cacheDir.path)
                    engine = Engine(config).also { it.initialize() }
                    conversation = checkNotNull(engine).createConversation(ConversationConfig(systemInstruction = Contents.of("You are a helpful assistant.")))
                    _state.value = ChatState(ready = true)
                } catch (e: Exception) {
                    release()
                    _state.value = ChatState(error = e.message)
                }
            }
        }
    }

    fun send(text: String) {
        scope.launch {
            if (!_state.value.ready || _state.value.busy) return@launch
            _state.update { it.copy(busy = true, reply = "", error = null) }
            try {
                checkNotNull(conversation).sendMessageAsync(text).collect { message -> _state.update { it.copy(reply = it.reply + message) } }
            } catch (e: CancellationException) {
                throw e
            } catch (e: Exception) {
                _state.update { it.copy(error = e.message) }
            } finally {
                _state.update { it.copy(busy = false) }
            }
        }
    }

    private suspend fun release() {
        _state.update { it.copy(ready = false) }
        while (_state.value.busy) {
            conversation?.cancelProcess()
            delay(100)
        }
        conversation?.close()
        conversation = null
        engine?.close()
        engine = null
        _state.value = ChatState()
    }

    override fun onCleared() {
        scope.launch {
            mutex.withLock { release() }
            executor.shutdown()
        }.invokeOnCompletion { scope.cancel() }
    }
}

sendMessageAsync(text) returns a Flow<Message> of chunks (toString() gives a chunk's text); sendMessage(text) blocks and returns the whole reply. The conversation keeps its history, so the next send() continues the chat, and busy keeps the button disabled until the flow completes. SamplerConfig(topK = 10, topP = 0.95, temperature = 0.8) in ConversationConfig sets the sampling (all three are required; topP and temperature are Double). load() with the model path that is already loaded does nothing, whatever backend it asks for, so the screen can call it again after a rotation, also once the app has fallen back to another backend; send() runs only when the state is ready and not busy (the button's own test): a tap that lands while release() is stopping a reply does nothing, and one queued behind a load() reaches the model that load() loaded when no reply was streaming. The screen (viewModel() and collectAsStateWithLifecycle() come from androidx.lifecycle:lifecycle-viewmodel-compose and androidx.lifecycle:lifecycle-runtime-compose, 2.10.0; every import the code blocks need is listed in imports):

kotlinCopy code
@Composable
fun ChatScreen(modelPath: String, viewModel: ChatViewModel = viewModel()) {
    val state by viewModel.state.collectAsStateWithLifecycle()
    var input by remember { mutableStateOf("") }
    LaunchedEffect(modelPath) { viewModel.load(modelPath, Backend.GPU()) }
    Column {
        Text(state.reply, Modifier.weight(1f).verticalScroll(rememberScrollState()))
        state.error?.let { Text(it) }
        TextField(value = input, onValueChange = { input = it })
        Button(onClick = { viewModel.send(input); input = "" }, enabled = state.ready && !state.busy) { Text("Send") }
    }
}

5. Lifecycle

• Keep the engine in the ViewModel for as long as its screen lives, or in an application-scoped holder when several screens share it. The ViewModel keeps its own thread and scope: viewModelScope is cancelled before onCleared() runs, so a close launched there would not run, and onCleared() cancels the scope once the close is done. release() turns ready off, calls cancelProcess() until the streaming reply's flow has ended (at the latest when the reply ends on its own), then closes the conversation and the engine, each once; onCleared() and a load() that replaces the model both go through it under the Mutex, one at a time, because release() suspends while a reply stops.
• Create a new conversation to start a fresh chat. The reference app for this API is Google AI Edge Gallery: https://github.com/google-ai-edge/gallery/blob/main/README.md

Troubleshooting

• First reply is slow: initialize() ran on first use; load at app start and set cacheDir.