The best way to Replace LLM Weights with No Downtime

Think about making an attempt to renovate the muse of a towering skyscraper with out asking its occupants to go away or pause their work. That’s precisely what MoonshotAI’s Checkpoint Engine does for AI fashions. It permits huge language fashions to replace their brains, the weights, whereas nonetheless working, so there’s no downtime. This breakthrough lets builders enhance their AI shortly and effectively, even on fashions with over a trillion parameters working on hundreds of GPUs. It’s quick, dependable, and designed to maintain AI techniques working easily whereas evolving in real-time, making it an important instrument for cutting-edge AI purposes. This text goes over what it’s, the way it works, and why it issues for the way forward for large-scale AI techniques.

What’s Moonshot AI’s Checkpoint engine?

Moonshot AI’s Checkpoint Engine is a specialised middleware designed to replace the weights of giant language fashions (LLMs) in real-time throughout inference with out interrupting ongoing operations. This functionality is important in Reinforcement studying eventualities the place mannequin weights have to be up to date ceaselessly. The Checkpoint Engine presently integrates seamlessly with vLLM inference frameworks and affords optimized efficiency by pipelining and reminiscence administration strategies. It additionally gives options like reusing weights from present cases to cut back overhead in scaling eventualities.

Structure 

The core of the Checkpoint is the ParameterServer class, which handles the load replace logic and orchestrates the information circulate.

1. H2D(Host to Machine): Strikes up to date weights from CPU reminiscence or storage to GPU reminiscence, utilizing optimized switch pipelines.
2. Broadcast: Distributes the load throughout all inference engine cases effectively, leveraging CUDA IPC buffers for shared reminiscence communication.
3. Reload: Every inference engine then selectively reloads related weight shards from the broadcasted information in keeping with its sharding sample.

These three-stage pipelines guarantee environment friendly, overlapping communication and copying for velocity.

When GPU reminiscence is proscribed, the system can fall again to serial execution to take care of reliability.

Strategies Used

The Checkpoint Engine makes use of two fundamental strategies to replace mannequin weights throughout inference.

1. Broadcast Methodology: That is the quickest and the default strategy. That is ultimate when numerous inference cases have to be up to date concurrently. It broadcasts the up to date weights from CPU reminiscence to all inference GPUs synchronously, making certain all cases keep completely in sync with minimal delay. 
2. P2P (Peer-to-Peer) Methodology: It’s used when inference cases are added or eliminated dynamically throughout runtime. It avoids disrupting present inference workloads by sending weights immediately from CPUs in present cases to GPUs in new cases by a peer-to-peer switch system, permitting easy and versatile updates.

Working 

The Checkpoint Engine orchestrates the complete switch course of. It first gathers needed metadata to create a plan, together with deciding the right bucket measurement for information switch. Then, it executes the switch, controlling the inference engine by a ZeroMQ socket to maximise efficiency. It organizes information switch into pipelines with overlapped communication and replica, enabling quick and environment friendly weight updates even underneath heavy workload.

By implementing the above-mentioned strategies and structure, the Checkpoint Engine allows dwell weight updates for LLMs throughout hundreds of GPUs with minimal latency and repair disruption.

Set up and Utilization

Set up

To make use of the quickest broadcast 

Use Code:

pip set up checkpoint-engine

To make use of the versatile P2P implementation:

Use Code:

pip set up 'checkpoint-engine[p2p]'

This can set up mooncake-transfer-engine to assist RDMA switch between completely different ranks.

Instance Use case

Step 1:

Put together an H800 or H20 machine with 8 GPUs with the most recent vLLM. Make sure you embrace /collective_rpc API endpoint commit (obtainable in the principle department) since checkpoint-engine will use this endpoint to replace weights.

Step 2:

set up checkpoint-engine

Code:

uv pip set up 'checkpoint-engine[p2p]'

Step 3:

For our use case, we’re gonna use Qwen/Qwen3-235B-A22B-Instruct-2507 because the check mannequin.

Code:

hf obtain Qwen/Qwen3-235B-A22B-Instruct-2507 --local-dir /decide/fashions/Qwen/Qwen3-235B-A22B-Instruct-2507/

Step 4:

Begin vLLM in dev mode and set –load-format dummy. Be sure that to set –worker-extension-cls=checkpoint_engine.employee.VllmColocateWorkerExtension

Code:

VLLM_SERVER_DEV_MODE=1 python3 -m vllm.entrypoints.openai.api_server --host 0.0.0.0 --port 19730 --trust-remote-code 

    --tensor-parallel-size=8 --max-model-len 4096 --load-format dummy 

    --served-model-name checkpoint-engine-demo --model /decide/fashions/Qwen/Qwen3-235B-A22B-Instruct-2507/ 

    --worker-extension-cls checkpoint_engine.employee.VllmColocateWorkerExtension

To replace weights by checkpoint-engine. No want to attend for vLLM to prepare. Use the code under.

Code:

torchrun --nproc-per-node 8 examples/replace.py --update-method all --checkpoint-path /decide/fashions/Qwen/Qwen3-235B-A22B-Instruct-2507/

To reuse weights from present cases

New checkpoint-engine cases can be a part of present cases and reuse their weights.

Utilizing the strategy under:

Step 1: Begin the prevailing occasion with –save-metas-file global_metas.pkl to save lots of world metas to a file.

Step 2: Use –sleep-time 300 to verify they keep alive.

Code:

torchrun --nproc-per-node 8 examples/replace.py --checkpoint-path $MODEL_PATH 

    --sleep-time 300 --save-metas-file global_metas.pkl

Step 3: After a checkpoint is registered, new cases can get hold of a replica of the checkpoint by setting –load-metas-file global_metas.pkl

Code:

torchrun --nproc-per-node 8 examples/replace.py --load-metas-file global_metas.pkl

FP8 quantization

Presently, FP8 quantization doesn’t work in vLLM when updating weights. It makes use of a easy patch in patches/vllm_fp8.patch to deal with the proper weight replace. Additionally ,this patch is barely examined in DeepSeek-V3.1 and Kimi-K2. So there are probabilities of having some compatibility points with different fashions.

Check

Run a easy correctness check for checkpoint_engine

Code:

torchrun --nproc-per-node 8 exams/test_update.py

Benchmark

Insights

Listed below are a couple of observations that I’ve made:

1. The printed methodology usually affords the quickest replace time, optimized for synchronous weight updates throughout many inference cases.
2. The P2P methodology takes longer however allows dynamic updates when cases be a part of or go away throughout runtime.
3. These benchmark exhibits the scalability of Checkpoint Engine, dealing with a trillion parameter fashions effectively on clusters starting from 8 to 256 GPUs

Limitations of Checkpoint Engine

Whereas Checkpoint Engine is a robust resolution for dwell weight updates in LLMs, it presently has some limitations.

• Works Finest with vLLM for Now: The engine is especially examined with the vLLM framework. If you happen to’re hoping to make use of it with different AI frameworks or customized setups, you would possibly want some additional work to get it working easily.
• Pipeline Nonetheless Bettering: The perfect seamless pipeline that overlaps information strikes completely isn’t totally completed but. This implies there’s nonetheless potential to make the updates even sooner.
• P2P Replace Might Be Smoother: The peer-to-peer methodology sends information by a bottleneck at one fundamental node earlier than sharing it with others, which may sluggish issues down when you’ve gotten a number of GPUs.
• Wants Additional GPU Reminiscence: The intelligent broadcast system makes use of extra GPU reminiscence to hurry issues up. On machines with much less reminiscence, it switches to a slower, much less environment friendly course of.
• Restricted Assist for FP8 Fashions: If you happen to’re working with the newer FP8 quantized fashions, you’ll want some experimental patches. And even then, not all fashions play properly, but past a few examined ones.

Conclusion

Moonshot AI’s Checkpoint Engine is a game-changer for updating enormous AI fashions with out stopping them. It retains the whole lot working easily, even whereas the mannequin’s “mind” is getting smarter in real-time. Whereas it nonetheless has a couple of areas to enhance, the potential is big. If you happen to’re working with giant AI techniques, this instrument is unquestionably value watching. It’s serving to make the way forward for AI sooner and extra environment friendly, with none downtime.

Continuously Requested Questions

I’m a Knowledge Science Trainee at Analytics Vidhya, passionately engaged on the event of superior AI options corresponding to Generative AI purposes, Massive Language Fashions, and cutting-edge AI instruments that push the boundaries of know-how. My function additionally entails creating participating academic content material for Analytics Vidhya’s YouTube channels, growing complete programs that cowl the complete spectrum of machine studying to generative AI, and authoring technical blogs that join foundational ideas with the most recent improvements in AI. By way of this, I intention to contribute to constructing clever techniques and share data that evokes and empowers the AI neighborhood.