mistralai

/

Devstral-Small-2-24B-Instruct-2512

like 457

Follow

Mistral AI_ 14.3k

Hey @cpatonn ,

The model was trained natively in FP8 so these are the "original weights".
You can retrieve BF16 quite easily by doing the same as is shown here:
https://huggingface.co/mistralai/Ministral-3-14B-Instruct-2512#transformers-bf16

(you could then run model.save_pretrained(...) and you should have correct BF16 weights :-)

Thank you for sharing with me :)

After convert it to BF16 format with transformers and model.save_pretrained(...), the output is model.safetensors.index.json not consolidated.safetensors.index.json. vLLM can't serve it anymore even with nightly build.

Related: https://github.com/vllm-project/vllm/issues/19953

@YourFriendSky in this case you should load with

--config-format hf --load-format hf --tokenizer_mode hf

so that it loads from Transformers format and not ours.

Thanks for bringing this up, we plan to improve integrations with Transformers and making it easier to change the format of the checkpoints.