Private agents that run on the phone.

Why on-device first

Frontier cloud models are great for hard reasoning. They're also slow to call (RTT + queue + decode), expensive per token, and they exfiltrate the user's data. For 80% of mobile agent work — classification, summarisation, intent routing, on-text editing, voice command parsing — a 1.7B on-device model on neural cores is faster, free, and private. For the other 20%, cloud is one fallback away.

One routing primitive

ModelRoute.preferOnDevice([...]) walks a list of providers and picks the first one whose capabilities cover the request (tool calling? streaming? vision? thinking?). If none cover it, it falls back to the cloud provider you supply. Your app code doesn't branch.

final agent = AgentSpec(
  name: 'offline-helper',
  instructions: 'You are a private on-device assistant.',
  model: ModelRoute.preferOnDevice(
    onDevice: [
      // iOS 19+ — uses Apple Intelligence's on-device foundation model.
      FoundationModelsProvider(),

      // Pixel 8+, Samsung S24+ — uses Google AICore.
      GeminiNanoProvider(modelVariant: 'nano-v3'),

      // Any device — llama.cpp via Fllama, any GGUF you ship.
      FllamaProvider(
        llm: fllamaAdapter,
        modelName: 'Qwen3-1.7B-Instruct',
      ),
    ],
    fallback: AnthropicProvider(apiKey: const String.fromEnvironment('ANTHROPIC_API_KEY')),
  ),
);

Foundation Models (Apple)

FoundationModelsProvider is the iOS 19+ Apple Intelligence on-device model. agentlib wires it through Apple's native Tool protocol, so structured tool calls work end-to-end. Available on iPhone 15 Pro+, iPad with M-series chips, and Apple Silicon Macs.

Gemini Nano (Google)

GeminiNanoProvider uses Google's AICore — the on-device runtime for Pixel and select Samsung devices. Tool calling works on nano-v3. agentlib detects device support at startup; you can list it in a preferOnDevice chain and it'll silently no-op on unsupported devices.

MediaPipe (cross-vendor Android)

MediaPipeProvider runs models via Google AI Edge LlmInference — Gemma 3, Phi, Llama 3.2, Qwen — on almost any Android 12+ device. The bundled MediaPipe runtime handles the GPU/CPU dispatch. Bring your own model file as .task in your app bundle or downloaded on first launch.

Fllama (llama.cpp)

FllamaProvider wraps llama.cpp via the Fllama Flutter package. Any GGUF works. agentlib ships family-aware tool-call parsers for Qwen, Llama 3.2, Phi-4, and Gemma 3 — they format tool calls slightly differently and agentlib normalises them into the same ToolUseContent shape the cloud providers use.

Capability gating

Each provider exposes a capabilities getter — does it stream? Does it support tools? Vision? Thinking? ModelRoute uses this to skip providers that can't handle the current request. So a vision agent will skip Gemini Nano (text-only) and fall through to Fllama with a vision-capable GGUF or to a cloud provider.

Battery-aware re-routing

The OnLowBattery hook fires when the device drops below a threshold. A common pattern: route to a smaller on-device model below 20%, or refuse heavy work and ask the user to plug in.

hooks.register(OnLowBattery((event, ctx) async {
  if (event.percent < 20) {
    return HookOutcome.replace(
      modelOverride: FllamaProvider(llm: tinyGgufAdapter, modelName: 'Qwen3-0.6B'),
    );
  }
  return HookOutcome.carryOn();
}));