Mlkit Agent Api

Targeting AICore Developer Preview. This skill provides authoritative execution patterns and failure-prevention rules for on-device agentic workflows in Android apps.

1. Prerequisites and setup

Gradle dependencies

Copy code
// app/build.gradle.kts
dependencies {
    // Core ML Kit GenAI Agent API
    implementation("com.google.mlkit:genai-agent:1.0.0-alpha1")

    // Android AppFunctions dependency for on-device tool declarations
    implementation("androidx.appfunctions:appfunctions:1.0.0-alpha13")
}

[!IMPORTANT] Always use official SDK interfaces from com.google.mlkit.genai.agent.*, androidx.appfunctions.*, com.google.mlkit.genai.prompt.*, and com.google.mlkit.genai.common.*. Never declare local placeholder interfaces or fallback mocks.

2. Minimal agent and runner initialization

Configure an Agent with Agent.builder and wrap execution in AgentRunner:

Copy code
import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner

val agent = Agent.builder()
    .setName("OnDeviceAssistant")
    .setInstructions("You are a helpful on-device assistant.")
    .build()

val runner = AgentRunner(agent)

• Agent.builder defaults to maxTurns = 10 automatically to prevent infinite tool loops.
• AgentRunner defaults to greedy decoding (temperature = 0.0f) for deterministic tool selection and schema compliance.
• AgentRunner(agent) automatically resolves application context using MlKitContext for AppFunction execution in Android apps. An explicit AgentRunner(context, agent) overload is available for headless tests or custom context scopes.

3. Multi-turn chat and lifecycle (ViewModel)

AgentSession preserves conversational history in memory and is not Parcelable . To preserve dialogue history across Android configuration changes (such as screen rotation), retain session inside an AndroidX ViewModel.

When executing single-turn ephemeral queries, managing concurrent multi-sessions, or requiring explicit RAM cleanup, you must follow the guidelines in Session lifecycle.

Copy code
package com.example.app

import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner
import com.google.mlkit.genai.agent.AgentSession
import com.google.mlkit.genai.agent.textTokens
import com.google.mlkit.genai.common.GenAiException
import com.google.mlkit.genai.prompt.Content
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch

class ChatViewModel : ViewModel() {
    private val agent = Agent.builder()
        .setInstructions("You are a helpful on-device assistant.")
        .build()
    private val runner = AgentRunner(agent)
    private val session: AgentSession =
        runner.getOrCreateSession("chat_session")

    private val _uiState = MutableStateFlow<ChatUiState>(ChatUiState.Idle)
    val uiState: StateFlow<ChatUiState> = _uiState.asStateFlow()

    fun sendMessage(prompt: String) {
        viewModelScope.launch {
            _uiState.value = ChatUiState.Loading("")
            try {
                // Content.builder() required; runAsync takes Content
                val content = Content.builder().text(prompt).build()
                var fullResponse = ""
                session.runAsync(content)
                    // Extension function requiring textTokens import
                    .textTokens()
                    .collect { token ->
                        fullResponse += token
                        _uiState.value = ChatUiState.Loading(fullResponse)
                    }
                _uiState.value = ChatUiState.Success(fullResponse)
            } catch (e: GenAiException) {
                // Catch GenAiException to prevent app crashes on AI errors
                val err = e.localizedMessage ?: "AI error (${e.errorCode})"
                _uiState.value = ChatUiState.Error(err)
            }
        }
    }

    override fun onCleared() {
        super.onCleared()
        // Evict conversation session from RAM when ViewModel is destroyed
        session.close()
    }
}

Essential lifecycle and coroutine rules

1. Content object : session.runAsync and runner.runSingleTurnAsync require a Content object. Always build it using Content.builder().text(userPrompt).build().

2. textTokens extension : textTokens is a Kotlin extension function on Flow<AgentEvent>. Always include import com.google.mlkit.genai.agent.textTokens.

3. Session cleanup : Always invoke session.close in ViewModel.onCleared to evict conversational RAM and cancel pending tasks.

4. AgentEvent stream : When collecting Flow<AgentEvent> directly instead of textTokens, always include import com.google.mlkit.genai.agent.AgentEvent. Match against AgentEvent.TextMessage(text, isPartial), AgentEvent.FunctionCall(name, args), and AgentEvent.FunctionResponse(name, response), and always include an else -> {} branch in when (event) expressions (AgentEvent is non-exhaustive for forward compatibility):

   import com.google.mlkit.genai.agent.AgentEvent

   session.runAsync(content).collect { event -> when (event) { is AgentEvent.FunctionCall -> { _uiState.update { it.copy(activeToolStatus = "Tool started: ${event.name}") } } is AgentEvent.FunctionResponse -> { _uiState.update { it.copy(activeToolStatus = "Tool completed: ${event.name}") } } is AgentEvent.TextMessage -> { _uiState.update { it.copy(responseText = it.responseText + event.text) } } else -> {} // Required: AgentEvent is non-exhaustive for forward compat } }

4. Tool calling recipe

All tools used with Agent originate from Android AppFunctions (@AppFunctionDeclaration), registered using the declarative path (enableAppFunctions(true) or addAppFunctionIds(...)) or advanced discovery (AppFunctionAgentTools.discover(...)).

When you need to restrict caller access levels for private in-app tools or wrap discovered AppFunctions with human-in-the-loop (HITL) confirmation decorators, you must consult Tools and functions.

Option A: Declarative AppFunctions path

Define tools declaratively in your app using AndroidX AppFunctions:

Copy code
package com.example.app.tools

import androidx.appfunctions.AppFunctionDeclaration
import androidx.appfunctions.AppFunctionSerializable

@AppFunctionSerializable
data class CalendarEvents(val date: String, val events: List<String>)

class CalendarTools {
    /**
     * Fetches scheduled calendar appointments for a specific date in
     * YYYY-MM-DD format.
     *
     * @param date Date string in YYYY-MM-DD format
     */
    @AppFunctionDeclaration(isDescribedByKDoc = true)
    fun fetch_calendar(
        date: String
    ): CalendarEvents {
        return CalendarEvents(
            date = date,
            events = listOf(
                "Team Standup at 10:00 AM",
                "Product Review at 2:00 PM"
            )
        )
    }
}

Enable AppFunctions on Agent.Builder and initialize AgentRunner:

Copy code
import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner

val agent = Agent.builder()
    .setName("CalendarAssistant")
    .setInstructions("You assist with calendar queries.")
    // Automatically discovers AppFunctions in your app package
    .enableAppFunctions(true)
    // Optional: scope to specific function IDs or method names
    .addAppFunctionIds("fetch_calendar")
    .build()

val runner = AgentRunner(agent)

• AppFunction Contract : Tool methods take domain parameters directly (no framework context parameter required). Returns must be primitives or @AppFunctionSerializable data classes (Map<String, Any?> isn't supported for AppFunctions).
• Restricting Callers with ACCESS_LEVEL_SELF : AppFunctions don't have to be public to external callers. To restrict execution strictly to your app's internal agent (blocking external system assistants and IPC callers), annotate with @OptIn(ExperimentalAppFunctionsApi::class) and @AppFunctionAccessLevel(level = AppFunctionMetadata.ACCESS_LEVEL_SELF, isCompatEnforcementEnabled = true).

Option B: Advanced discovery and decorator pattern

For inspecting, filtering, or decorating discovered AppFunctions (such as Human-in-the-Loop confirmation wrappers using AgentTool(delegate)) before registering with the agent:

Copy code
import com.google.mlkit.genai.agent.Agent
import com.google.mlkit.genai.agent.AgentRunner
import com.google.mlkit.genai.agent.AgentTool
import com.google.mlkit.genai.agent.AppFunctionAgentTools

// Discover tools (optionally scoped to function IDs)
val discoveredTools = AppFunctionAgentTools.discover(
    enabledAppFunctionIds = listOf("fetch_calendar")
)

// Optional: Wrap sensitive tools using AgentTool(delegate) decorator
class HitlConfirmationTool(
    delegate: AgentTool,
    private val onConfirm: suspend (
        toolName: String,
        args: Map<String, Any>
    ) -> Boolean
) : AgentTool(delegate) {
    override suspend fun execute(args: Map<String, Any>): Map<String, Any?> {
        if (!onConfirm(name, args)) {
            return mapOf("status" to "CANCELLED_BY_USER")
        }
        return checkNotNull(delegate).execute(args)
    }
}

val agent = Agent.builder()
    .setName("CalendarAssistant")
    .setInstructions("You assist with calendar queries.")
    .addTools(discoveredTools)
    .build()

val runner = AgentRunner(agent)

[!WARNING] Mutual exclusivity : Configure AppFunctions using either the declarative path (enableAppFunctions(true) on Agent.Builder) or the advanced path (AppFunctionAgentTools.discover passed to addTools), but not both. Using both simultaneously triggers a warning and registers duplicate tool schemas.

Deprecated annotations (@Tool and @Param)

The legacy @Tool and @Param annotations in com.google.mlkit.genai.schema.annotations.*, along with AgentTool.fromAnnotatedObject, Agent.Builder.addTools(annotatedObject: Any), and standalone FunctionTool, have been deprecated or removed. All tools added to Agent must originate from @androidx.appfunctions.AppFunctionDeclaration using Agent.Builder.enableAppFunctions(true) or AppFunctionAgentTools.discover.

5. Ephemeral single-turn execution

For one-off tasks where conversation history isn't needed (such as summarizing text, formatting data, or answering standalone questions), invoke runner.runSingleTurnAsync(content) directly:

Copy code
import com.google.mlkit.genai.agent.AgentRunner
import com.google.mlkit.genai.agent.textTokens
import com.google.mlkit.genai.prompt.Content
import kotlinx.coroutines.flow.Flow

// Executes a single-turn query ephemerally without session persistence:
fun askQuickQuestion(runner: AgentRunner, question: String): Flow<String> {
    val content = Content.builder().text(question).build()
    return runner.runSingleTurnAsync(content).textTokens()
}

• runner.runSingleTurnAsync(content) automatically creates and evicts an ephemeral session upon Flow completion.
• Cleanup reference :
  • session.close: Releases session resources and cancels coroutines (call in ViewModel.onCleared).
  • session.delete or runner.deleteSession(sessionId): Permanently deletes the session from the runner.

6. Error handling and GenAiException reference

Runtime failures during on-device execution (such as model unavailability or exceeding maximum turn limits) are emitted or thrown as GenAiException:

Copy code
import com.google.mlkit.genai.common.GenAiException

Key error codes

• GenAiException.ErrorCode.MAX_TURNS_EXCEEDED (-112): The agent exceeded maxTurns (default 10) without completing the reasoning loop.
• GenAiException.ErrorCode.NOT_AVAILABLE (8): The Gemini Nano model or AICore service isn't available on this device.
• GenAiException.ErrorCode.BUSY (9): System or AICore is busy, or battery or thermal quota has been reached.