Picking a Language Model Small Enough to Live on Your Phone

Gratin has one promise: your recipes never leave your device. The obvious next features all cut against it. Estimate the nutrition of this recipe. Auto-tag it. Clean up this messy paste. Every one is an AI feature, and the easy way to build an AI feature is to call a frontier API, which means sending your dinner to a datacenter. That breaks the promise.

So I asked a smaller question. Can I run a language model on the device instead, and would it be good enough to ship?

The tasks help. Nutrition estimates, tags, and extraction are background jobs. Nobody is watching a cursor blink. I can take three seconds, or thirty, to fill in a nutrition estimate after an import. The outputs are tiny, a JSON object with seven numbers or a few tags. That matters more than it sounds. A small KV cache and no need for interactive speed are exactly the two things that make on-device models painful, and this workload needs neither.

But "the tasks are forgiving" isn't the same as "it works." I had three open questions, and I wanted each one settled with measurement rather than assumption. How big a model actually fits on a phone. Which model is best at my specific tasks. And, the one that turned into a research project of its own, what I grade a nutrition model against when there's no such thing as the true nutrition of a recipe.

How big a model fits on a phone

There are a lot of confident, wrong answers to this online, so I didn't want to rely on any of them. I ran it as an automated research pass instead: a fan-out of agents pulled sixteen sources and extracted twenty-five specific claims, then a second set of adversarial agents tried to refute each claim against primary documentation. Eighteen survived, seven were refuted, and some of the seven were things I'd have built against.

The big one: iOS has no fixed per-app memory limit. People design against a number, 1.5GB or 3GB, whatever they read once. There isn't one. The limit moves with the device and the moment, and when you cross it the kernel kills your app. The only reliable way to know your budget is to ask the device at runtime with os_proc_available_memory(). In practice, on an 8GB iPhone you get about 4 to 5GB before you're playing with fire.

What fits under that budget comes from real quantized model files, not the back-of-envelope math. The usual "half a byte per parameter" rule of thumb is wrong: real 4-bit weights are closer to 0.6, and runtime memory runs higher still because of the cache and compute buffers.

Two more findings settled early. Small models aren't bad at JSON. With grammar-constrained decoding the output is guaranteed to parse and match the schema, whatever the model. What varies is whether the numbers inside are right. And the engine picked itself: llama.cpp. Metal on iOS, Vulkan on Android, one core across the desktop and mobile builds, and grammar constraints built in. The engine ships in the binary; the weights download afterwards to a folder that doesn't get backed up.

So the target is 3 to 4B. That's the answer to "how big," but it says nothing about which model. "Newest is best" is a hypothesis, not a result, and I wanted to validate it against the actual tasks.

The harness

The obvious shortcut is to trust a public leaderboard. MMLU, IFEval, the Hugging Face rankings. None of them measure "estimate the fat in a Danish soup from its ingredient list," which is the job. So I wrote a small test harness that runs candidate models against Gratin's actual tasks and scores four things: is the answer correct, is it valid structured JSON, how fast is it, and does it fit in the device budget.

Two decisions in the harness mattered later. First, it's honest about what it can't measure. Speed runs on my Mac, so it's a proxy for phone speed and the reports say so; correctness and validity are the same on any hardware, so those are real. Second, it's resumable. A full run is every model times every prompt times every task times every example, which is thousands of generations and hours of wall time. So it writes each result to disk the moment it finishes, and on restart it skips everything that's already done. That sounds like a nicety. It's the difference between a benchmark that finishes and one you give up on at hour three when the SSH connection drops. A model with no weights downloaded yet just gets skipped, and you fill it in later.

Now I had a grader. Which exposed the harder problem: what do you grade it against?

There is no true nutrition number

This took more time than anything else in the project. There is no ground truth for recipe nutrition. Not "it's hard to find." It doesn't exist. A recipe is a list of ingredients, and its nutrition is a calculation over portion sizes and food-database matches, and every professional source makes different calls at every step. Scoring a model's estimate means first deciding what the right answer is, and that decision was the real work.

The approach that held up was to stop trusting any single source: compute the nutrition several independent ways, and keep the number only where they agree.

Three sources vote on every recipe

Making the sources disagree surfaces errors that a "just scrape the author's number" pipeline swallows whole. A bulgogi whose published nutrition was for skin-on chicken, when the recipe called for skinless. A Danish soup whose published fat counted the stock meat, the meat the recipe strains out and discards. These aren't typos. A single source can't catch them; two independent sources disagreeing can.

Scaling past the initial 182-recipe pilot meant a public dataset of 14,000, which was filthy the way scraped data always is. The first filter is the Atwater check. A recipe's calories should be about 4·protein + 4·carbs + 9·fat; if the published calories are far from what the macros imply, the row is internally inconsistent and gets dropped. It runs on the row's own numbers, with no external lookup. 94.8% survived.

Two professional nutrition engines, given the same recipe, routinely disagree by 15 to 25%. There is no single true number. The nutrition of a recipe is a range, not a value, and a benchmark that treats it as a value is really measuring how closely a model reproduces one source's arbitrary choices.

That spread also sets the scoring tolerance. Because the sources themselves disagree by up to 25%, a prediction counts as correct when it lands within ±25% of the reference. Scoring tighter than the reference data's own disagreement would just be measuring noise.

So the reference set, the gold that every model is scored against, is tiered by confidence instead of collapsed to a single answer. A row is dropped only when two independent sources agree it's wrong; one source is never enough. Dropping on a single source would delete fatty-meat recipes across the board, since that is where published numbers run low, and bias the benchmark against its highest-variance category.

Which model wins

With the corpus and harness in place, the model comparison could run. I tested nine models in the 2 to 4.5B range, in two passes: a broad first pass over all of them, then a deeper second pass over the top five at 100 examples per task for a ranking I trust.

Bigger doesn't help

A wider sweep, from 0.5B up to 7.6B, shows the shape plainly: accuracy climbs to about 3B and then flattens. A 3.8B model tied the 7.6B model outright, on a budget it fits comfortably in. Two independent reasons not to ship 7B, then. It doesn't fit, and it wouldn't have helped if it did.

The prompt barely matters

Sweeping several system prompts per task, baseline, detailed, strict, and a dietitian persona for nutrition, moved the score by 2 to 3 points. Swapping the model moved it by over 40. So the effort went into model selection, and prompt tuning stayed a footnote.

The model does almost none of the work

Nutrition reads like a language task, which points at the best small language model. It doesn't work out that way.

RAG is the obvious next step: give the model real USDA numbers instead of making it recall nutrition from training. Two versions got tested. In the naive one, the model gets the reference numbers and does the math itself. That was unsurprisingly a disaster. Hand a small model a wall of per-ingredient numbers and ask it to multiply each by a portion, scale by servings, and add them up, and it drowns. It's never going to be as good as a calculator at arithmetic. One model dropped from 40% to 24%. In a gentler version, a calculator does the math and the model only reviews the result and corrects obvious errors. That rescued the weak models, but did almost nothing for the strong one at the top.

Either way, feeding reference data into the model was a dead end. The fix points the other way: take the arithmetic out of the model, not put more into it.

Compose the number

The alternative is to ask a model for one number per recipe. That can work, but it's the less reliable option: the same recipe can come back with slightly different totals on different runs, and there's no way to see which ingredient drove the result.

Composing the number is steadier. Nutrition as a sum of per-ingredient contributions is deterministic (the same recipe always sums to the same total), explainable (every number traces to an ingredient), and it updates cleanly on an edit: change one ingredient and one term in the sum changes. That reliability, more than any single accuracy number, is why the pipeline composes the answer instead of asking for it.

The model is the smallest component

So the deterministic calculator does the math, and it's already the better answer. The compositional pipeline edges past the best bare model on accuracy, and unlike the model it updates the instant you edit a recipe.

That's the base to build on, not the finish line. Because the pipeline is deterministic, its errors can be read off field by field against the corpus.

Fixing the fat tail

About 59% of calorie estimates land within ±25%, and the tail is heavy. A recipe app that's wrong on two recipes in five isn't shippable.

The tail isn't a smooth haze. It's four specific bugs.

The fix follows straight from the diagnosis. Three of the four buckets are judgment calls, not arithmetic: is this the whole-milk row or the powdered-milk row, is the frying oil actually eaten, is "1 jar marinara" a real ingredient or a non-food token. So I let the model make exactly those judgments, per ingredient, and kept the arithmetic deterministic. Confident matches skip the model entirely. Only the ambiguous lines, about a fifth of them, get a bounded question: which variant, how much is eaten, or a per-100g estimate for a genuine gap.

I call it LLM-resolve. On a 40-recipe sample, half known catastrophes and half random, it lifted calories within ±25% from 68% to 75%, and all four macros from 25% to 40%. The model touched about a fifth of the ingredient lines.

The ceiling is the gold, not the model

Seventy-five percent sounds low, but there's no true answer to measure against. Two professional nutrition engines only agree with each other about 83% of the time. Against a gold that noisy, 75% is close to the ceiling, and the gap that's left is a mix of real pipeline error and the gold disagreeing with itself.

Pushing the gold cleaner further would mean paying for a service like Edamam, and that's not an obvious win: it's another professional engine carrying the same 15 to 25% spread as everything else, not a source of truth worth being certain of yet. So the only judge on hand is the small on-device model, and it's a biased one. It called 81% of the gold "too high," because it undercounts. Taken at face value, that would delete a fifth of the corpus. So only its "way too low" verdicts counted, and only when the deterministic pipeline independently agreed the gold was off by 1.5x or more. That confirmed twelve genuinely bad rows out of a 2,000-recipe sample. By the measures available without an independent source, the gold is about 99% clean. The three-way consensus, back when Edamam was in it, put the suspect share nearer 7%, but confirming which of those are really wrong, rather than the model being blind to portions, needs an independent second opinion.

The one lever left is a second independent professional source, an Edamam or a Nutritionix: a better ingredient-to-food matcher than token matching, and a second opinion to clean the gold further. But it's a paid dependency whose own numbers carry the same 15 to 25% spread, so it's a considered option, not a certain fix.

So no, these numbers won't impress on their own, and they don't need to. Nutrition has no precise answer to reach: the sources that define it disagree by 15 to 25%, so beyond a point, chasing accuracy just fits the model to one source's noise. What ships is an honest estimate, not a promise. It lands in the right range, every number traces back to an ingredient, and any line a user disagrees with can be corrected. For a background figure that fills itself in after an import, that clears the bar. And it's a floor rather than a finish: the harness lifts it every time a better model lands.

Why the harness sticks around

It would have been quicker to grab a model off a leaderboard and move on. The harness is worth keeping for three reasons, and they're the same three reasons it'll keep running after the current answer is out of date.

New models keep arriving. When the next Qwen or Gemma or Phi lands, checking whether it beats the current pick on these tasks is a one-command re-run against the real workload.

Smallest-that-works is a moving target. The harness tells me the smallest model I can ship for a given accuracy, and every gigabyte I don't spend on weights is headroom against the memory limit and a faster download.

Prompts are part of the model. The same run that ranks the models also surfaces which prompt each one does best with, per task.

Underneath the recipe specifics, one method carried the whole thing: never trust a single source, and require independent agreement before acting on a number. Adversarial verification on the feasibility claims. Three estimators voting on every gold value. A consistency check that rejects internally broken rows before any external lookup. The counterintuitive results, that bigger didn't help and that RAG hurt more than it helped, came straight out of that discipline.

The most useful output of the gold-data work wasn't the gold. It was the finding that there is no single true nutrition number, which changes the product from "compute the right answer" to "compute a number you can defend, explain, and edit." And the language model kept shrinking as the design got more accurate. It started as the whole point and ended as a judge for the one-in-five ingredients the database can't resolve on its own, with plain arithmetic doing the real work. That isn't the model failing. It's what putting a model on a phone looks like when you account for what the phone can hold, what the model is good at, and what the app has to do on an edit.

This is the engineering behind the AI features in Gratin, a local-first recipe app I build and maintain.