CHITRAGUPTA · HOW IT WORKS

Two models, one wall, and a written brief across it

Every diagram on this page exists because of a number. The shape of the system is not a preference — it is what fits inside 200,000 vision tokens a day.

What actually happens over a session

The reasoning model is the centre band: every turn routes through it, and it holds every piece of state. The camera is a peripheral it switches on when a question needs eyes — nobody presses a button for that — and switches off again once the question is answered.

WHAT IS NOT TRUE OF THIS DIAGRAM

The centre band runs the full height because everything routes through it — not because it is listening. There is no open microphone. Voice input is push-to-talk (continuous = false), so a turn only begins when one of exactly three things happens:

You ask

typed, or one press of the mic button

A camera tick

the only part that runs on its own

A timer fires

checked by arithmetic on a 15 s poll

So the live part is the camera loop, and only the camera loop. Getting that right with the model is the current work. A tick-driven conversation — the assistant holding a thread on its own between your questions — is the thing after it, and is not built.

The figure keeps VideoLLM-online's band grammar — one model everything feeds into, a sequence interleaved into it in temporal order — but runs the clock downward, so it survives a phone. Read against that paper, the substitutions are the argument.

The two arrows crossing the right gutter are the whole protocol. Amber going out is a brief the reasoning model wrote for eyes it does not have; blue coming back is what those eyes reported. Note that the two briefs are shaped completely differently — a rigid one-line detection contract for a search that runs on every frame, and an open reading instruction when the answer has to be right once.

Two questions, one session. The camera opens and closes twice — the gap between the blue blocks is the part that costs nothing at all. Inside the first window, one frame never left the browser, two produced [SILENT], and one was worth interrupting for. The row after it is the user's correction, which this system still has no way to act on: case 07 in the failure log. The second question then opens the camera again at a higher resolution, because reading a label is not the same job as spotting a bag.

The paper's frozen encoder runs on every frame

Here the vision band exists only while the camera is open, and closes itself the moment the question is answered.

The paper trains an EOS head to decide when to speak

Here that decision is split between a browser-side diff gate and a prompt-level silence protocol. No training budget, so it moved somewhere cheaper.

The paper caches frame history in the model

Here each frame becomes one caption, in text. Hosted APIs cannot reuse image tokens across calls, so continuity has to arrive in words.

The sample interval is user-set from 2 to 15 seconds and defaults to 4. It is a cost dial as much as a responsiveness one: at the coarse frame rate, halving the interval halves how long a day's token budget lasts.

Inside one tick: pixels travel one hop, then become words

A camera frame never reaches the model that answers you. It is converted to a short paragraph of text by a separate model, and the pixels are discarded at that line. Everything downstream — the reasoning, the tools, the memory — is text.

Two decisions are made before any money is spent. The diff gate runs in the browser, so an unchanged scene never becomes a network request at all — it is the single largest cost control in the system. And a tick with nothing to say returns the literal string [SILENT], which is stripped server-side, so a watching assistant is quiet by default rather than narrating.

One model is blind; the other knows nothing

The reasoning model holds the entire conversation but cannot see. The vision model is looking straight at the answer but has no idea what anyone asked. Neither can be fixed by a better prompt to one of them — so they write to each other.

The brief is written by the model, not by me. An earlier version inferred what to look for from the task text, which meant the system could only ever look for things I had anticipated. Now the reasoning model states its own question, and the vision stage answers only that.

Briefs request observations, never judgement. A vision model asked for a verdict will give you one whether or not the frame supports it.

What one minute of watching costs

The free tier allows 200,000 vision tokens per day, total. In this architecture the vision model does nothing but look at pictures, so image tokens are the entire budget — and image cost scales with resolution, not with file size. Compressing the JPEG harder buys nothing at all.

ticks fired15

gate dropped− 8

frames sent7

at 640 px9,800

at 1,024 px23,968

Extrapolated from the session's own measured numbers: a 4-second tick, ~1,400 tokens per coarse frame and 3,424 per fine one, and a gate that skipped roughly half the ticks. At the coarse rate a day's budget is about two and a half hours of continuous watching. At the fine rate it is under one.

So close-up mode is opt-in, per step. The reasoning model turns it on when it needs to read something small, and the resolution change has to reach the browser before the next capture — resolution thrown away on the client can never be recovered on the server.

server →  frame_detail: "fine"
client ←  frame detail → fine (live ticks now 1024px)
groq   ←  Groq vision usage: Requested 3424

That number is also how a session died: the model switched close-up mode on, never switched it back, and hit the ceiling at 198,310 of 200,000 tokens.

A cost control that depends on the model's judgement does not hold.

Everything the model needs is already in front of it

There is no retrieval step and no memory tool. Each reasoning call is assembled from blocks that are always present, because a model that has to ask for its state will reliably forget to ask.

[Camera feed]The vision model's caption for this frame, as text. The only trace a picture leaves.

[Timers]Each running timer and its remaining time. Computed by subtracting two wall-clock numbers — no model is involved, so this block is free.

[Task list]Every item with its status, its note, up to five observations, and its standing watch_for brief. Persisted to disk, so it survives a server restart.

[Silence rule]Added only on a camera tick with an active goal: if nothing is new, reply with exactly [SILENT]. Never applied to something the user actually said.

historyThe last ten turns. Camera ticks are excluded — 42 machine observations would otherwise crowd out the conversation.

Timers store a start time and a duration rather than sleeping, so a restart loses nothing and checking them costs no inference. The task list is written by the model as a full-list replace — it rewrites the whole document each time, which removes an entire class of partial-update bugs at the price of a slightly larger payload.

Ten tools, and one rule learned the hard way

Tool	What it does
update_task_list	Replaces the whole document — items, statuses, notes, briefs, detail tier
log_observation	Appends a fact to an item. Two separate flags decide whether it interrupts the user and whether it ends a goal
request_camera	Asks for a frame when none is attached
request_live_search	Registers a “find X” goal and starts watching for it
start_timer · cancel_timer	Wall-clock, persisted; cancelling deletes rather than marking fired, so it does not announce the timer you just cancelled
web_search · fetch_page	A provider chain, because a single scraped endpoint fails in ways that look like success
calculate · get_time	The cheap deterministic things a language model should not be doing in its head

One flag, one consequence.

A single boolean on log_observation once did three unrelated jobs at once: it guaranteed the user was told, it marked the step complete, and it switched the camera off. The model set it on ordinary progress notes — correctly, by the description it had been given — and the camera kept shutting off mid-task for reasons nobody could see. Splitting it into two flags with one effect each fixed it permanently.

Three designs I rejected

Considered	Rejected because
An orchestrator	A controller routing between specialists has to anticipate every situation in advance. Letting the model reason and call tools in one chain means the thinking is the routing
Multiple agents	Every check-in would pay for a coordination call to achieve what one model reading a shared document already does
A cheap relevance pre-filter	A yes/no call before the real one, to skip irrelevant frames. Its “no” was indistinguishable from legitimate silence, so a frame showing the thing you asked for could vanish leaving no trace. Removed with no replacement

That last one is the rule I reach for most often outside this project: never add a filter whose rejection is indistinguishable from a normal quiet outcome. Failures have to be visible or they are not failures, they are behaviour.

What broke →

Eight failures, each with the symptom, the root cause, and the rule it produced.