The big agent SDKs all assume cloud. They were built when on-device LLMs were a research curiosity. They’re not anymore. Apple Foundation Models ships in iOS 19. Gemini Nano ships in AICore. llama.cpp runs Qwen3-1.7B at 30+ tok/s on a 2024 iPhone. MediaPipe runs Gemma 3 on most modern Android phones. agentlib treats them as first-class.

This article walks through the four on-device providers, the routing primitive that ties them together, and the patterns that make on-device-first feasible in production.

The four providers

FoundationModelsProvider (iOS 19+). Wraps Apple’s on-device foundation model — the one that powers Apple Intelligence. It supports native tool calling through Apple’s Tool protocol. Available on iPhone 15 Pro and newer, M-series iPads, and Apple Silicon Macs running macOS 26+.

final fm = FoundationModelsProvider();
// Auto-detects availability. Falls through silently when used in a ModelRoute.

GeminiNanoProvider (Pixel 8+, S24+). Wraps Google AICore. Tool calling works on nano-v3. agentlib detects device support at startup; on unsupported devices, it returns capabilities indicating it can’t serve the request, and ModelRoute moves on.

final nano = GeminiNanoProvider(modelVariant: 'nano-v3');

MediaPipeProvider (Android 12+). Uses Google AI Edge LlmInference for Android. Supports a growing list of models — Gemma 3, Phi-4, Llama 3.2, Qwen — that you ship as a .task file in your app bundle or download on first launch. Cross-vendor: it runs on most modern Androids, not just Pixel.

final mp = MediaPipeProvider(
  modelAssetPath: 'assets/models/gemma-3-1b-it.task',
);

FllamaProvider (any device). Wraps Fllama, the Flutter package around llama.cpp. Any GGUF works. agentlib ships family-aware tool-call parsers for Qwen, Llama 3.2, Phi-4, and Gemma 3 — each formats tool calls slightly differently and agentlib normalises them.

final llama = FllamaProvider(
  llm: fllamaAdapter,
  modelName: 'Qwen3-1.7B-Instruct',
);

One routing call

The on-device picture is dramatically simpler than it looks because ModelRoute handles dispatch:

final agent = AgentSpec(
  name: 'offline-helper',
  instructions: 'You are a private on-device assistant.',
  model: ModelRoute.preferOnDevice(
    onDevice: [
      FoundationModelsProvider(),                   // iOS 19+
      GeminiNanoProvider(modelVariant: 'nano-v3'),  // Pixel 8+, S24+
      FllamaProvider(llm: fllamaAdapter, modelName: 'Qwen3-1.7B-Instruct'),
    ],
    fallback: AnthropicProvider(apiKey: '<KEY>'),
  ),
);

ModelRoute queries each provider’s capabilities getter — does it stream? Does it support tools? Vision? Thinking? — and picks the first provider that can serve the current request. If none can, it falls back to the cloud provider you specified. The agent code never branches.

When to prefer on-device

You almost always want on-device first for these workloads:

  • Classification (is this email urgent? is this photo a receipt?). 1.7B model is great at this.
  • Summarisation of short-to-medium documents (under ~2 K tokens of context).
  • Intent routing (which subagent should handle this prompt?).
  • On-text editing (rewrite this sentence in formal English).
  • Voice command parsing (extract intent + slots from a spoken sentence).
  • Anything privacy-sensitive where the user’s data shouldn’t leave the device.

You probably still want cloud for these:

  • Long-context reasoning (multi-document analysis, long codebases).
  • Vision tasks beyond what the local model handles (Foundation Models has vision; Gemini Nano doesn’t).
  • Multi-step tool use that needs sophisticated planning (frontier cloud models still plan better than 1.7B locals).
  • First-class structured output with complex schemas. (On-device is catching up here, but still patchy.)

Patterns that work

Pattern 1: tier on-device → cloud. Most apps end up here. Smaller on-device for cheap routine work; cloud for the hard 20%.

Pattern 2: on-device only for sensitive flows. Health, finance, kids, legal. Use ModelRoute.onDeviceOnly([...]) — it throws rather than silently falling back to cloud.

final medicalAgent = AgentSpec(
  name: 'medical-helper',
  instructions: 'You are an on-device medical assistant. No PHI leaves the device.',
  model: ModelRoute.onDeviceOnly([
    FoundationModelsProvider(),
    GeminiNanoProvider(),
    FllamaProvider(llm: fllamaAdapter, modelName: 'Qwen3-1.7B-Instruct'),
  ]),
);

Pattern 3: battery-aware re-routing. Below 20%, switch to a tinier model. The OnLowBattery hook handles this in five lines.

hooks.register(OnLowBattery((event, ctx) async {
  if (event.percent < 20) {
    return HookOutcome.replace(
      modelOverride: FllamaProvider(llm: tinyAdapter, modelName: 'Qwen3-0.6B'),
    );
  }
  return HookOutcome.carryOn();
}));

Pattern 4: network-aware re-routing. On a flaky cell signal, prefer on-device.

hooks.register(OnNetworkChange((event, ctx) async {
  if (!event.isReachable) {
    // Drop cloud from the route entirely.
    return HookOutcome.replace(modelOverride: ctx.config.model.withoutCloud());
  }
  return HookOutcome.carryOn();
}));

Capability gating in detail

Each provider exposes a capabilities getter that returns a ModelCapabilities value:

class ModelCapabilities {
  final bool streaming;
  final bool toolCalling;
  final bool vision;
  final bool thinking;
  final int maxContextTokens;
}

ModelRoute matches the current request’s needs against this. So a vision agent will skip Gemini Nano (text-only) and try Fllama (if you’ve configured a vision GGUF) or fall through to cloud. A simple text-summarisation request will use whichever on-device provider’s available first.

Bring your own runtime

If none of the four providers fits, implement ModelProvider yourself:

class MyRuntimeProvider implements ModelProvider {
  @override
  ModelCapabilities get capabilities => const ModelCapabilities(
    streaming: true,
    toolCalling: true,
    vision: false,
    thinking: false,
    maxContextTokens: 8192,
  );

  @override
  Stream<ProviderEvent> stream(ProviderRequest req) async* {
    // Wrap your runtime's HTTP or FFI API here.
  }
}

That’s the full contract. Two methods. Wrap any ONNX runtime, any local HTTP API, any FFI binding — agentlib’s agent loop will treat it the same as a built-in provider.

What changes in your UI

Nothing, ideally. The streaming event API is the same regardless of which provider serves the request. Your StreamBuilder<RunEvent> works the same whether the model is in your bundle or in Anthropic’s data centre.

What you can add, optionally:

  • A small “running on-device” badge in your UI when ctx.activeProvider.kind == ProviderKind.onDevice. Users like knowing.
  • Per-provider analytics so you can see how often the cloud fallback kicks in.
  • A user setting: “Prefer on-device even if slower” → flip to onDeviceOnly mode.

Performance reference points

These are rough numbers on consumer 2024–2025 devices. Your mileage varies.

ProviderDeviceTokens/secCold start
FoundationModelsiPhone 15 Pro~40~200 ms
Gemini Nano (v3)Pixel 8 Pro~30~150 ms
MediaPipe + Gemma 3 1BPixel 7~22~500 ms
Fllama + Qwen3 1.7BiPhone 15 Pro~28~1.2 s
Fllama + Qwen3 0.6BiPhone XR (2019)~14~2.4 s

In every case the time-to-first-token is much better than a cloud round trip, especially on cell networks.

Bottom line

Running models on the device used to be a science project. It isn’t anymore. Foundation Models and Gemini Nano are ready for production today; Fllama covers everything else with a 1–2 GB ship-cost; MediaPipe gives you the Android breadth. agentlib treats them uniformly so your agent code doesn’t care which one ran.

Install with one pubspec line, plug a preferOnDevice([...]) into your AgentSpec, and ship a private agent that works on a plane.