PrismML just pulled off a technical stunt that could change how mobile devices — and by extension, on-device crypto tools and dApps — interact with large language models.
What happened
- PrismML released Bonsai 27B, a 27-billion-parameter model compressed small enough to run on a smartphone. A full-precision 27B model normally needs roughly 54 GB of memory; PrismML’s binary build is 3.9 GB and its ternary build is 5.9 GB.
- The binary variant runs at about 11 tokens/sec on an iPhone 17 Pro Max; the ternary build hits ~26 tokens/sec on an M5 Pro laptop. Both are available free under the Apache 2.0 license.
How they shrank it
- The compression is based on Caltech IP and collapses each weight from 16-bit floating point down to a single sign in the binary build (+1 or −1), or to three values (−1, 0, +1) in the ternary build.
- Groups of 128 weights share a 16-bit scaling factor. That yields an average of 1.125 bits per weight for the binary model (about 14× smaller than the full-precision original) and 1.71 bits per weight for the ternary version.
- Unlike many low-bit quantizations, PrismML compresses everything end-to-end — embeddings, attention, and the LM head — rather than leaving “sensitive” layers in higher precision.
Why this matters
- This isn’t just a smaller model; it’s large-scale behavior (27B) squeezed into a footprint that can live on consumer hardware. That scale is where long chain-of-thought reasoning, consistent tool use, and multi-step agentic behavior start to show up reliably — capabilities smaller models often miss.
- The model uses a hybrid attention backbone with ~75% linear layers instead of full quadratic attention, making a 262k-token context window practical on-device — something a conventional attention stack would make prohibitively expensive on a phone.
Performance snapshot
- PrismML previously shipped Bonsai 8B (1.15 GB) in March, proving the 1-bit architecture at smaller scale. The 27B jump is the more significant test.
- On 15 “thinking mode” benchmarks run on NVIDIA H100s (covering knowledge, math, coding, tool use), Ternary Bonsai 27B averaged 80.49 — about 94.6% of the full-precision model. The 1-bit (binary) variant scored 76.11.
- Against some bigger models, Bonsai holds its own for its size: on AIME-style math tests the ternary model hits 93.7% vs 95.3% for Qwen 3.6B; coding scores are 86 vs 88; general knowledge 77% vs 83. The team says overall bench results compare favorably to Gemma 4 and Qwen 3.6 on a capacity-for-size basis.
Real-world use and developer tools
- PrismML bundles DSpark, a speculative decoding layer that drafts candidate token blocks and verifies them in a single forward pass. On H100 GPUs DSpark gave a 1.37× throughput boost with no quality loss; Apple Silicon support is coming but not yet default.
- PrismML claims the model is practical for tasks like iterative coding and creative writing on-device. Independent testing by the article’s authors produced useful results: a multiplayer-leaning browser game “Zombie Type” was built with two rounds of prompts (collision, scoring, graphics), and creative writing showed consistent internal logic and pacing — not necessarily groundbreaking, but solid for a fully local model.
Commercial and ecosystem angles
- PrismML CEO Babak Hassibi told CNBC the company is in early conversations with Apple about the compression tech; a compressed Gemma model is reportedly next, followed by larger frontier builds.
- Availability: 1-bit Bonsai 27B is free to download now under Apache 2.0. PrismML also published prior Bonsai builds and developer tooling; instructions and a local-run primer are available from community guides.
Why crypto folks should care
- On-device LLMs at this scale open clear possibilities for crypto and web3:
- Private, offline assistants for wallet management, local signing guidance, or transaction explanation without sending sensitive data to cloud APIs.
- Improved UX for mobile dApps and wallets with local, low-latency natural language interfaces.
- New workflows for decentralized applications that want to avoid central API dependencies and reduce operating costs.
- These are not out-of-the-box crypto products, but the technical direction — powerful models that run locally with tiny memory footprints — removes a major friction point for private, self-custodial tooling and edge-native intelligence.
Bottom line
Bonsai 27B isn’t just another quantized model: it’s a demonstration that higher-parameter LLMs can be compressed end-to-end to fit consumer hardware without catastrophic quality loss. For developers and crypto projects focused on privacy, self-custody, and on-device intelligence, this is a meaningful advance — and it’s being released under a permissive license, so experimentation can start today.
Read more AI-generated news on: undefined/news
