Does agentlib do speech-to-text itself?

It wraps platform STT (iOS Speech framework, Android RecognitionService) behind a voice CLI in /bin/voice. You can swap in Whisper.cpp via Fllama or a cloud STT if needed. The agent invokes /bin/voice.listen and gets back text.

How does vision work without a vision-capable on-device model?

agentlib's /bin/vision CLI dispatches to whatever vision provider is available — Apple's Vision framework for OCR + scene description on iOS, ML Kit on Android, or a vision-capable LLM (Foundation Models, cloud) for harder tasks. Same agent code; different backend per device.

What about accessibility automation — is that App Store-safe?

Yes for accessibility-service-based automation (Android) and AppIntents/SiriKit on iOS, when used with user consent. The PermissionRequestHook surfaces consent before any /apps action. For drive-other-app workflows, ship a clear in-app explainer; reviewers care more about transparency than the mechanism.

Voice in, vision out, device automation: building a pocket assistant with agentlib

The product everyone wants — a real on-phone assistant that listens, sees, and acts — is finally buildable. agentlib makes the wiring straightforward.

What we’re building

A demo “pocket assistant” agent that can:

Listen to a spoken command.
See the user’s screen if context is needed.
Act by driving other apps through accessibility / AppIntents.
Speak the result back.

All on-device where possible. Cloud as a fallback. Same code on iPhone and Pixel.

The CLIs

agentlib ships first-party CLIs for each pillar:

/bin/voice.listen — start STT, return transcript.
/bin/voice.speak — TTS, plays back text.
/bin/vision.describe — describe an image or current screen.
/bin/vision.ocr — extract text from an image.
/bin/apps.list — list installed apps the assistant can drive.
/bin/apps.<bundle>.<action> — perform an action in another app (accessibility-driven on Android, AppIntents/SiriKit on iOS).

The orchestrator

final assistant = AgentSpec(
  name: 'pocket',
  instructions: '''
You are a private on-device assistant. When the user speaks:
1. Transcribe with `/bin/voice.listen`.
2. Understand the intent.
3. If you need to see the screen, call `/bin/vision.describe`.
4. Carry out the request through `/bin/apps.*` or other CLIs.
5. Speak the result with `/bin/voice.speak`.

Prefer on-device tools. Ask consent before any /apps.* action.
''',
  model: ModelRoute.preferOnDevice(
    onDevice: [FoundationModelsProvider(), GeminiNanoProvider(), FllamaProvider(llm: a, modelName: 'Qwen3-1.7B-Instruct')],
    fallback: AnthropicProvider(apiKey: '<KEY>'),
  ),
  tools: [...virtualFilesystemTools(), shTool, voiceTool, visionTool, appsTool],
);

A turn end-to-end

User: “Tell my partner I’ll be late.”

Inside agentlib, the model emits an Sh pipeline:

Sh("voice listen | jq -r .text > /scratch/intent.txt")
// → /scratch/intent.txt now contains "Tell my partner I'll be late."

Then the model interprets intent and dispatches:

Sh("contacts find 'partner' | jq -r .number | xargs whatsapp send --message 'Running late, see you soon'")

The PermissionRequestHook surfaces a consent sheet for WhatsApp. The user confirms. agentlib snapshots before dispatch (because whatsapp.send is highImpact: true). The send happens. The model emits:

Sh("voice speak --text 'Done, told them you\\'re running late.'")

Everything on-device, except the consent dialog. One Flutter app. Five tool calls.

Vision when needed

User: “What’s on the screen right now?”

Sh("vision describe --screen | jq -r .summary")
// → "A QR code with the URL https://example.com/event"

/bin/vision.describe --screen captures the current foreground screen (iOS Screen Capture API or Android MediaProjection — both require one-time consent), runs it through Foundation Models’ vision capability on iOS 19+ or through Google ML Kit + an LLM caption pass on Android, and returns a structured description.

For dense text, /bin/vision.ocr runs Apple Vision or ML Kit OCR and returns string output. The model can then run regex / jq over it.

Driving other apps

Through Android’s accessibility service or iOS’s AppIntents + SiriKit, the assistant can perform actions in other apps. agentlib normalises this behind /bin/apps.*:

Sh("apps gmail compose --to 'partner@example.com' --subject 'Late' --body 'Be there in 30'")

This works because agentlib registers a per-app CLI that wraps the appropriate native API. On Android, that’s an accessibility-service walk; on iOS, that’s an AppIntent invocation. The model sees one consistent CLI.

Battery + privacy

Wire the lifecycle hooks once and forget about them:

hooks
  ..register(OnLowBattery((e, _) async {
    if (e.percent < 15) {
      return HookOutcome.replace(modelOverride: FllamaProvider(...tiny));
    }
    return HookOutcome.carryOn();
  }))
  ..register(PermissionRequestHook((req, ctx) async {
    final ok = await showConsent(ctx, req);
    return ok ? HookOutcome.carryOn() : HookOutcome.deny('user denied');
  }))
  ..register(PostToolUse((call, result, ctx) async {
    await audit.log(call: call, result: result);
    return HookOutcome.carryOn();
  }));

What it feels like to use

Latency is the main concern. End-to-end target on an iPhone 15 Pro: ~1.5 s from end-of-speech to start-of-TTS for a simple “send message” intent. That’s two on-device model calls (intent classification + planning) plus a native API dispatch. Tight enough to feel snappy.

Patterns to steal

Audio routing. Use flutter_sound or record to capture audio. Pass the buffer to /bin/voice.listen rather than recording inside the CLI — the CLI shouldn’t own the mic stream.

TTS interruption. Wire a “user is speaking again” hook that cancels in-flight TTS. Otherwise the assistant will talk over the user.

Wake-word. Optional but nice: ship a tiny on-device wake-word model (e.g. Porcupine) that triggers voice.listen only when called. Saves battery and respects the no-always-listening promise.