ENGINEERING CASE STUDY · IN DEVELOPMENT
Chitragupta
The model that reads the label can't reason. The one that reasons never sees.
HYBRID VISION PIPELINE · SELF-WRITTEN VISION BRIEFS · GROQ QWEN3.6 + DEEPSEEK V4
A hands-free camera assistant for when your hands are busy. It watches through a phone camera, tracks what is in flight — steps, substitutions, timers — and talks back. The user is listening, not reading, which turns out to drive almost every design decision in the system.
The category is not new — Ray-Ban Meta and Gemini Live got here first, with budgets to match. The approach is: two models that cannot see what the other sees, and a protocol between them designed so this runs continuously on a hard cost ceiling instead of always-on inference. Everything below is measured from one real session on a phone, not from a scripted demo.
infrastructure cost — free tiers end to end, which is the constraint the architecture is built around
vision-token ceiling. In this split the vision model does nothing but look at pictures, so images are the whole budget
measured cost of a close-up frame against a normal one — 3,424 tokens versus roughly 1,400
camera ticks in one live session that correctly produced no speech at all
Watch it work, in its own words
Three excerpts from one 40-minute session. Every line is copied out of the export — the user's words, both models' traffic, the client's resolution switch, and one wrong answer left in. Camera frames are excluded for privacy; the text pipeline is the point.
The three tabs are deliberately different shapes of problem: an open brief that reads small print, a rigid one-line detection contract, and a turn where the camera supplied nothing usable at all.
One coordinator, and tools it can reach for
There is no orchestrator and no pipeline of stages. There is one model acting as a coordinator — it holds what the job is, what has already happened, and what is still outstanding, and it reaches for a tool when it needs something it does not have. Eyes. A clock. A fact it was never trained on.
Nothing in it is task-specific. The same loop is meant to cover cooking, ironing a shirt, a supermarket aisle, or a wheel change on a verge. Below is one task end to end — what happens at each step, and, more usefully, why that step exists at all.
- 01
You say what you are doing
reasoning“I need to change a tyre.” It breaks the job into steps and writes them to a document on disk.
whyThe task list is state, not conversation. It is written to a file and injected into every later prompt, so the model never has to remember — or ask.
- 02
It looks up what it doesn't know
toolIf the procedure isn't something it can be trusted on, it searches the web and reads a page before planning.
whyVideoLLM-online needed instruction data purpose-built for every task it could assist with. Searching at runtime means this can attempt a task nobody prepared data for — which is what makes “any hands-on task” plausible instead of a slogan.
- 03
It decides it needs to see, and opens the camera itself
reasoningNo button. The model calls a tool that turns the camera on, because the next step is one it cannot advise on blind.
whyYour hands are on a wheel brace. Any design that requires you to operate a phone at this moment has already failed at the thing it exists for.
- 04
It writes a brief for eyes it does not have
reasoning“Report which tools are in frame.” Or, on a step where technique matters: “Watch their hands and tell me what they are actually doing.”
whyThis is the only channel between the two models. The reasoning model knows the task but cannot see; the vision model can see but knows nothing. The brief asks for observations and never for a verdict — a model asked whether something is safe will answer whether or not the frame supports it.
- 05
The frame becomes about forty words of text
vision“FOUND: wheel brace on the ground, left of frame.” The pixels are discarded at this line.
whyHosted APIs cannot reuse image tokens between calls, so keeping a visual history is impossible. Keeping a text history is nearly free — which is what lets it answer “where did I put the brace?” an hour later.
- 06
It decides whether that is worth saying out loud
reasoningNothing new, so the entire reply is the literal string [SILENT] and you hear nothing. Something changed, so it speaks.
whyAn assistant that narrates every frame is both unusable and unaffordable. Silence is the default and has to be earned out of.
- 07
It closes the camera — or deliberately keeps it open
browserA “find the jack” goal ends the moment the jack is found. A “watch me tighten these” step stays open until the step is done.
whyThe camera is the only part of this that costs real money. Whether it stays on is a spending decision, and the model makes it — which is also exactly where the cost control has failed before.
- 08
When the task has to wait, it sets an alarm for itself
toolTorque the bolts after lowering the car. Potatoes on for twelve minutes. The timer carries the reason it exists, not just a duration.
whyTracking elapsed time is the thing VideoLLM-online is weakest at — a streaming model watching frames has no reliable clock. A wall-clock timer sidesteps it entirely: it stores a start and a duration, survives a restart, and checking it is subtraction, not inference.
Steps 4 to 7 are the loop that repeats. Everything else happens once. And the two tools that look least impressive on a list — web_search and start_timer — are the two doing the most work against the paper this is built on: one removes the need for purpose-built training data, the other removes the need for the model to track time at all.
Why two models, and not one
The original design ran one vision-language model for everything. It could not survive its own provider's per-minute cap: 8,000 tokens covering input and output together, shared across the image, the conversation history, the tool schemas and the task list. The completion budget alone exceeded it before a single prompt token was counted.
The fix was not a smaller prompt. It was splitting the job so that only the small, image-only request has to fit under that ceiling. One model looks at the picture and writes about forty words of text. A second model — on a provider with no comparable cap — does all the reasoning and every tool call, and never sees a pixel in its life.
Running the vision model alone for everything was tried first, and it lost on both counts: the reasoning was noticeably worse, and it ran out of tokens doing it. The split is not only a cost dodge — it lets each side be chosen for what it is good at. It also makes a text-only model behave like a multimodal one, which is the part I find most reusable: the reasoning provider's caching and thinking budget stay available, and vision is bolted on as a text-producing service in front of it.
One frame in, one short paragraph out. No conversation, no tools, no memory of the previous call.
All reasoning, all ten tools, all persistent state. Receives the frame only as text, and is never told it saw anything.
The protocol: one model briefs the other
This is the part I would defend hardest. The reasoning model holds the entire conversation but cannot see. The vision model is looking straight at the answer but has no idea what anyone asked. Neither problem is fixable by writing a better prompt for one of them.
So the reasoning model writes its own brief — a watch_for field it attaches to a task step — and the vision stage answers only that. An earlier version inferred the brief from the task text, which meant the system could only ever look for things I had anticipated in advance. Below is what it produced unprompted, when the user said their German was bad.
Read every German label clearly. Report the product name and all ingredients. Flag if you see: Rindfleisch, Rind, Rinder-, Kalb, Kalbs-, or any other beef-related word. Also check for gelatin (Gelatine).
The product is identified as Surimi containing "Fish Meat" as the primary ingredient. There is no presence of beef, Rind, Gelatine, or any related terms in the visible ingredients list.
Every architectural decision traces back to a number
The free tier allows 200,000 vision tokens per day, total. In this architecture the vision model does nothing but look at pictures, so image tokens are the entire budget — and image cost scales with resolution, not file size. Compressing the JPEG harder buys nothing at all, which took measurement to establish rather than assumption.
So close-up mode is opt-in per step, and the resolution change has to reach the browser before the next capture: resolution thrown away on the client can never be recovered on the server.
server → frame_detail: "fine" client ← frame detail → fine (live ticks now 1024px) groq ← Groq vision usage: Requested 3424
The model switched close-up mode on, never switched it back, and hit the daily ceiling at 198,310 of 200,000 tokens. Its cost was bounded by scope — finishing the step ends it — and the step never finished.
A cost control that depends on the model's judgement does not hold. Instructions are a hint; the budget needs arithmetic.
A paper I couldn't implement, and what I did instead
The design started from VideoLLM-online, which I presented for my master's seminar. It solves streaming video dialogue — crucially, when a model should speak and when it should stay quiet — using a cached frame history and a trained per-frame decision. Almost none of that survives contact with a hosted API, so the interesting work was re-deriving the same behaviours without the mechanisms that produce them.
| The paper does | Why I can't | What I do instead |
|---|---|---|
| KV-cached frame history | Hosted APIs cannot reuse image tokens across calls | Convert each frame to one short text fact, discard the pixels, accumulate only the text |
| Trained speak / stay-silent decision | No training budget | A silence protocol at prompt level, plus triggers that are pure arithmetic and cost nothing |
| Inference on every frame | 200,000 tokens per day, total | A perceptual diff gate in the browser — an unchanged scene never becomes a request at all |
| DeepStack fine-detail attention | Fixed resolution per call | A coarse/fine tier the reasoning model opts into per step, when it needs to read something small |
| A CLIP encoder, pooled to a handful of tokens per frame | Too little context to answer a real question about a frame | A vision-language model that reads the scene and writes prose about it — which also turns a text-only reasoning model into a multimodal one |
| Instruction data built specifically for the tasks it assists with | No dataset, no budget to build one | It searches the web at runtime, so it can attempt a task nobody prepared data for |
| Tracking elapsed time from the frame stream | The paper's weakest point, and it does not get better on a sampled feed | A wall-clock timer the model sets for itself, carrying the reason it exists. Checking it is subtraction, not inference |
Qwen3-VL is the natural next step — a model built for exactly this — and is a planned extension, not a shipped feature.
Go deeper
Early, and specific about it
- The two-model pipeline end to end, on a phone, over a real 40-minute session
- Reading a foreign-language ingredient list on demand, at a resolution the model chose itself
- Refusing to answer off a frame too blurred to read, on a dietary-restriction question
- Staying silent on 21 of 42 camera ticks with an active goal
- Persisted task state surviving restarts, read back without being prompted
- Continuous conversation. The mic is push-to-talk; the camera loop is the only thing that runs on its own, and making that solid is the current work
- Timers and scheduling — implemented, never exercised in a real run
- A second architecture that passes a 27-check smoke test and has never seen live traffic
- No automated test suite; verification is throwaway harnesses plus reading session exports
- Two known failure modes still open, both documented in the failure log
The second architecture is deliberately not deployed. Running two unproven systems at once makes it impossible to tell which one is lying to you — so it waits until the first is genuinely trustworthy.
What this demonstrates
The one I would point at first: I once spent a session unable to explain why the assistant lost its memory — so I shipped instrumentation instead of a fix, because “the model ignored the task list” and “the task list never reached the prompt” look identical from a transcript and need opposite repairs.
Where the async is actually load-bearing
A framework name in a list proves nothing. This one runs a single worker with a camera firing every four seconds, so concurrency is a correctness problem, not a résumé line.