Sophon-Realtime-32B-Flash-BF16

The BF16 thinker of Sophon-Realtime-32B-Flash, not quantized. The FP8 thinker in the Flash release was quantized from exactly these weights; the talker and voices are the same files as in the Flash release.

Use this repository when your hardware cannot run the FP8 thinker, for example Ascend NPUs or GPUs without FP8 support, or when you want the unquantized weights for your own quantization or research. On one 80 GB NVIDIA GPU, use the FP8 Flash release instead: the BF16 thinker alone needs about 64 GB, so it usually runs with tensor parallelism over two devices.

  • Layout: config.json · thinker/ (BF16, ~63 GB) · talker/ · voices/, the same as the Flash release, so the flash/ code serves it with a config that gives the thinker enough memory.
  • Community guide for Ascend 910C (two devices, TP=2): PR #7.
  • Version 1.4.1 · GitHub · changelog

License: Apache-2.0; see NOTICE for the sources of the bundled weights.

Downloads last month
93
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including adventists-ai/Sophon-Realtime-32B-Flash-BF16