Your Device Is Not Waiting For The Cloud Anymore
A world where we're beholden to American datacentres for AI is just temporary.
This article was reviewed and very heavily hand-edited, using curated sources I hand-picked, drafted by AI using personalised style guidance. I find it useful to keep in touch with the dozens of X.com bookmarks I come across every few days. Maybe you do too.
Three signals this week that local AI crossed from hobbyist to actually useful
Maziyar Panahi stood in the street, downloaded a medical model onto his iPhone over 5G, and got sub-60ms inference from a device in his hand with no backend involved. (source)
Demos like this used to invite a shrug. This week produced three signals, each hitting a different axis of the local-AI story:
distribution,
multimodal pipelines, and
mobile on-device.
The mesh: Hyperspace Pods
Take Varun Mathur’s Pods: a small trusted group installs a CLI, one person creates a pod, and invite links pull other machines into a mesh.
Models that exceed any single laptop’s memory get sharded across devices. Layers split proportionally. Inference pipelines through the ring.
The concept isn’t new. Petals tried this two years ago for BLOOM. What’s changed is the hardware floor and the model ceiling. When we covered MiniMax M2.7 running frontier coding benchmarks on a single MacBook Pro, the implication was: one machine is enough. Pods inverts that. If one machine runs a 32B, four machines run the thing you actually want.
The skeptical reading: pipelined inference across consumer wifi will feel nothing like a cloud API. Inter-device latency on a home network dominates once a forward pass traverses three hops. Mathur’s demo talks about forming the mesh, not the tokens-per-second you get once it’s formed. I’d want to see that number under a load that isn’t one user.
So who runs a pod? The pitch targets a family or startup pooling hardware.
Here’s my direct experience: I think it becomes an issue of capacity.
Home is massively underpowered and overloaded. Do you think my two boys will appreciate their Roblox / War Thunder ping rates being hammered by some random process Dad installed on their machines? Will Pod even run on an iPad?
Startup? Same issue here: capacity. If I have 4 team members that have local compute capacity to run a distributed model, I’d probably want to know what they were doing (or not).
Let’s call it “mesh compute”. Nice idea; let the market figure out future potential niches.
“What’s going on in this video?” Now answered locally.
The Google Gemma post looks like a video tracking demo, which undersells what’s actually happening underneath.
Gemma 4 takes video frames and describes what it sees. Falcon Perception then takes those descriptions, segments the objects, and tracks them across frames. All running locally.
Multimodal pipelines used to be the cloud’s strongest differentiator. Vision model here, language model there, segmentation API over there, orchestrated through sequential HTTP calls. Latency budgets for real-time video processing ruled local deployment out by default. And honestly it’ll still be essential for any actual commercial services.
But two things changed. The models got much smarter and small enough to co-locate, and the glue got good enough that chaining them doesn’t require a research lab. Gemma + Falcon is the first version of what could well become a default pattern: a vision-language model as the router, specialist models as tools, all inside one process.
Building on our coverage of Docker treating models as standard containers, this is what that container abstraction was always pointing at. Pipelines composed of local models, each with a clean interface, chained without a network boundary between them.
The pocket: sub-60ms on a phone
Panahi’s iPhone demo is the most visceral of the three and also the thinnest on substance. Sub-60ms inference is impressive if you believe the number. What he didn’t say: which OpenMed model, what quantisation, what benchmark accuracy compared to the full-size version.
The broader signal still holds. Mobile on-device inference was a 2024 demo category. This year it’s shifting to the “actually use it for something” phase. Medical specialist models are a good canary: accuracy matters, privacy matters more, offline availability is a real clinical requirement.
Local is really compelling for a number of reasons. Phones deliver 15-30W of compute. Sub-60ms at that envelope on a specialist model means dozens of inference events per minute inside a workflow, on battery, without the user noticing. At that latency the model stops being a feature you invoke and becomes part of the interface itself. And they can do it even when the wifi is down and you are in another of Vodafone’s 5G reception dead zones.
Three problems, one assumption
The shared premise across Pods, Gemma+Falcon, and the OpenMed phone demo is that the cloud is no longer the obvious default for the next generation of inference work.
A year ago, local AI’s pitch was privacy. This now expands to:
latency for multimodal (Gemma),
capacity via distribution (Pods),
accessibility on hardware people already own (OpenMed).
Privacy is now one reason among several.
Now to DevUX: I’ve been doing a whole bunch in this space. I’ll have to say that coding agents like Claude Code and Codex have vastly lubricated those wheels: let the agent figure out the connections and APIs while you apply judgement on the quality aspects: security, maintainability, and performance.
🔮 Prediction
Within twelve months, a local-first agent framework ships that treats multi-device meshes (Pods-style) as a first-class target, not an afterthought. The first production users will be professional services firms with regulatory reasons to keep inference inside the office: law, healthcare, accounting. Firms of four to ten workstations, not consumers or hobbyists, who would otherwise pay for a cloud tenant.
The consulting tier we covered in our $35,000 Llama install story will find their rates under pressure. The infrastructure question becomes “plug in the laptops” rather than “deploy a server.” Watch the consulting day-rates for that shift.
Back to the terminal.