video — AI 101

Colibrì vs llama.cpp

About this Resource

<h2>Colibrì vs llama.cpp: Running DeepSeek V4 284B on a CPU</h2>Running a massive AI model usually requires a lot of RAM or expensive GPUs. But projects like Colibrì are challenging that assumption by showing that models far larger than available system memory can still run on ordinary consumer hardware. A key example is DeepSeek V4, a 284-billion-parameter Mixture-of-Experts (MoE) model. The experiment compared Colibrì with the popular llama.cpp framework to find out whether Colibrì actually enables something new or simply makes an existing capability easier to use. <h2>How Can a 284B Model Run on a Normal PC?</h2>DeepSeek V4 contains around 284 billion parameters, but it does not use all of them for every token. The model contains 43 layers, with hundreds of expert networks in each layer. For each token, a router selects only a small number of experts to process it. The important part is that the entire model does not need to sit in RAM at once. Think of the model as a huge library stored on an SSD. Instead of moving the entire library into your house, the system can retrieve only the books it needs. Colibrì takes this approach directly. It manages which model components are needed and keeps frequently used experts in memory while retrieving others from storage when necessary. <h2>The Hardware Test</h2>The comparison was performed on a consumer desktop with:<ul><li>AMD Ryzen 5 5600X</li><li>61 GB of RAM</li><li>RTX 3060 with 12 GB VRAM</li><li>NVMe SSD</li><li>DeepSeek V4 model weighing roughly 162–167 GB</li></ul> In other words, the model was more than twice the size of the available RAM. Colibrì was able to run the model despite this limitation. It produced its first token in about 25 seconds and completed the test in roughly 69 seconds. It also tracked which experts were already available in memory. During the test, it achieved around a 71% expert cache hit rate, meaning most requested experts were already available without having to be loaded again from storage. <h2>What Happened With llama.cpp?</h2>The initial llama.cpp test failed because it attempted to allocate around 147 GB of memory, far more than the machine's 61 GB of RAM. The reason was an optimization called repacking. Before inference, llama.cpp can reorganize model weights into a format optimized for CPU processing. While this can improve performance, the transformed data needs additional memory. After disabling repacking, llama.cpp was also able to run the 284B model. This was an important discovery, Colibrì had not created the fundamental ability to run a model larger than RAM. llama.cpp could already do it under the right configuration. <h2>Which One Was Faster?</h2>For the short test, Colibrì finished faster overall.Colibrì Approx. total time - 69 seconds  Token generation ~0.73 tokens/sec     llama.cpp   Approx. total time - 142 secondsToken generation ~1 token/sec  This looks confusing at first, Colibrì finished sooner but generated tokens more slowly. The difference comes from startup time. llama.cpp spent roughly 100 seconds loading and preparing the model, while Colibrì began producing output much earlier. <h2>Where llama.cpp Pulls Ahead</h2>The picture changes dramatically when the prompt becomes longer. With a prompt of around 3,000 tokens, llama.cpp processed the prompt at approximately 8.1 tokens per second, while Colibrì managed only about 0.8 tokens per second. The reason is batching. Llama.cpp can process multiple prompt tokens together and reuse the same expert weights efficiently. Colibrì processes the prompt much more sequentially, requesting the required experts for each token individually. As a result, Colibrì's efficient memory management comes with a major performance trade-off when processing longer prompts. <h2>So, Is Colibrì Revolutionary?</h2>The experiment's conclusion is more nuanced than the hype suggests. Colibrì works. It can run a 284-billion-parameter model on a consumer desktop with significantly less RAM than the model's total size. However, it does not fundamentally unlock a capability that llama.cpp could never provide. With the correct configuration, llama.cpp can also run the model. Where Colibrì stands out is in its approach. It treats models larger than available memory as a normal situation, explicitly manages its memory budget, caches frequently used experts, and avoids the large upfront memory requirement that can cause llama.cpp to fail by default. The trade-off is speed. Colibrì can get to the first response much sooner, but llama.cpp's mature batching and optimized CPU kernels make it substantially faster when processing larger prompts. <h2>The Bigger Picture</h2>The most interesting lesson is that running huge AI models locally isn't always about having enough RAM to store the entire model. Modern MoE models only activate a fraction of their parameters for each token. That creates an opportunity to keep the model on an SSD and move only the pieces needed for the current computation. Colibrì demonstrates how far this idea can go on ordinary hardware. At the same time, llama.cpp shows the advantage of years of optimization around batching and CPU inference. For people building affordable local AI systems, the takeaway is simple, massive models do not necessarily require massive amounts of RAM or an expensive GPU but there can be a significant performance cost for making them fit. Colibrì therefore isn't a replacement for llama.cpp so much as an interesting alternative approach to making extremely large models accessible on consumer hardware.