Getting good efficiency out of fashions that don’t slot in VRAM is all the time a problem. Nevertheless, Gemma 4 has, courtesy of its “combination of consultants” design, a function to spice up efficiency. LM Studio exposes this function via a setting presently tagged as experimental. You’ll be able to select what number of layers of the mannequin to “power MoE [Mixture of Experts] weights onto the CPU,” which conserves VRAM and might velocity up inference.
The MoE (combination of consultants) experimental setting in LM Studio. For fashions that use an MoE design, this setting forces the weights for that facet of the mannequin to be run on the CPU as an alternative of the GPU. With Gemma 4, this resulted in a serious velocity increase for fashions too large to slot in reminiscence.
Foundry
With out the MoE forcing, the general inference time and token era velocity cratered; the mannequin might barely handle a mean of 1.5 tokens per second even for easy queries. With MoE forcing turned on (with the utmost variety of layers supported, 30), token era velocity jumped to anyplace from 5 to 13 tokens per second, relying on the remainder of the system’s load. That’s nonetheless a far cry from the velocity of the smaller fashions, however much more workable.
For quicker time-to-first-token outcomes, you may disable considering, on the potential value of much less sturdy output. For the code-generation question, Gemma 4 spent 6 minutes 26 seconds considering, and over 8 minutes producing the response (5,013 tokens, 9.55 tokens per second). The ensuing code and rationalization was not considerably extra superior or detailed than the non-thinking model.
