aws-neuron/optimum-neuron-cache
Updated
ā¢
27
AI & ML interests
None defined yet.
Recent Activity
View all activity
dacorvoĀ
updated
a
model
2 days ago
Post
356
Building high-performance, reproducible kernels forĀ AMD ROCmĀ just got a lot easier.
I've put together a guide on building, testing, and sharingĀ ROCm-compatible kernelsĀ using the Hugging FaceĀ
Learn how to:
- Use Nix for reproducible builds
- Integrate kernels as native PyTorch operators
- Share your kernels on the Hub for anyone to use withĀ
We use the š award-winning RadeonFlow GEMM kernel as a practical example.
š Check out the full guide here : https://huggingface.co/blog/build-rocm-kernels
I've put together a guide on building, testing, and sharingĀ ROCm-compatible kernelsĀ using the Hugging FaceĀ
kernel-builder and kernelsĀ libraries; so you can focus onĀ optimizing performanceĀ rather than spending time on setup.Learn how to:
- Use Nix for reproducible builds
- Integrate kernels as native PyTorch operators
- Share your kernels on the Hub for anyone to use withĀ
kernels.get_kernel()We use the š award-winning RadeonFlow GEMM kernel as a practical example.
š Check out the full guide here : https://huggingface.co/blog/build-rocm-kernels
Post
2785
š Big news for AI builders!
Weāre thrilled to announce that the Qwen3-VL family of vision-language models is now available on Azure AI Foundry, thanks to our collaboration with Microsoft.
We bring open-source innovation to enterprise-grade AI infrastructure, making it easier than ever for enterprise to deploy and scale the latest and greatest from models from hugging Face securely within Azure.
š Highlights:
- Deploy Qwen3-VL instantly via managed endpoints
- Built-in governance, telemetry, and lifecycle management
- True multimodal reasoning ā vision, language, and code understanding
- State-of-the-art performance, outperforming closed-source models like Gemini 2.5 Pro and GPT-5
- Available in both *Instruct* and *Thinking* modes, across 24 model sizes
š Get started today: search for Qwen3-VL in the Hugging Face Collection on Azure AI Foundry.
Weāre thrilled to announce that the Qwen3-VL family of vision-language models is now available on Azure AI Foundry, thanks to our collaboration with Microsoft.
We bring open-source innovation to enterprise-grade AI infrastructure, making it easier than ever for enterprise to deploy and scale the latest and greatest from models from hugging Face securely within Azure.
š Highlights:
- Deploy Qwen3-VL instantly via managed endpoints
- Built-in governance, telemetry, and lifecycle management
- True multimodal reasoning ā vision, language, and code understanding
- State-of-the-art performance, outperforming closed-source models like Gemini 2.5 Pro and GPT-5
- Available in both *Instruct* and *Thinking* modes, across 24 model sizes
š Get started today: search for Qwen3-VL in the Hugging Face Collection on Azure AI Foundry.
- 1 reply
Post
843
Whatās your biggest headache deploying Hugging Face models to the cloudāand how can we fix it for you?
-
- 8 replies
Post
483
Post
3899
š¤ Collaborating with AMD to ensure Hugging Face Transformers runs smoothly on AMD GPUs!
We run daily CI on AMD MI325 to track the health of the most important model architectures and weāve just made our internal dashboard public.
By making this easily accessible, we hope to spark community contributions and improve support for everyone!
We run daily CI on AMD MI325 to track the health of the most important model architectures and weāve just made our internal dashboard public.
By making this easily accessible, we hope to spark community contributions and improve support for everyone!
-
- 2 replies
Post
430
š Optimum libraries keep growing, and Optimum v2 is just around the corner!
I recently added ONNX export support for a bunch of new models in the optimum-onnx library, including: DeepSeek-V3, Cohere, Nemotron, Arcee, StableLM ā¦ and more!
ā” With ONNX export, you can run your favorite models faster and more efficiently across different hardware backends, making deployment and experimentation much smoother.
š” Have a model youād love to see supported? Contributions are super welcome ā letās make Optimum even better together!
#ONNX #Optimum #HuggingFace #OpenSource #AI
I recently added ONNX export support for a bunch of new models in the optimum-onnx library, including: DeepSeek-V3, Cohere, Nemotron, Arcee, StableLM ā¦ and more!
ā” With ONNX export, you can run your favorite models faster and more efficiently across different hardware backends, making deployment and experimentation much smoother.
š” Have a model youād love to see supported? Contributions are super welcome ā letās make Optimum even better together!
#ONNX #Optimum #HuggingFace #OpenSource #AI
jeffboudierĀ
posted
an
update
3 months ago
Post
3057
Quick 30s demo of the new Hub > Azure AI integration to deploy HF models in your own Azure account. Now with Py and CLI!
GG @alvarobartt @kramp @pagezyhf
GG @alvarobartt @kramp @pagezyhf
Post
3216
We've improved the Deploy button on Hugging Face model pages for Microsoft Azure
1/ no more long waits before seeing model support status
2/ ready-to-use CLI and Python snippets
3/ redirection to Azure AI Foundry rather than Azure ML
ā if you see any bugs or have feedback, open an issue on our repo:
https://github.com/huggingface/Microsoft-Azure
1/ no more long waits before seeing model support status
2/ ready-to-use CLI and Python snippets
3/ redirection to Azure AI Foundry rather than Azure ML
ā if you see any bugs or have feedback, open an issue on our repo:
https://github.com/huggingface/Microsoft-Azure
Post
3192
Is there a "one-size-fits-all" recipe for quantizing Large Language Models? š¤
As part of my ongoing work in mixed-precision quantization, I've been exploring this question by measuring layer-by-layer sensitivity. The goal is to see if we can find universal rules for which layers can be quantized aggressively without impacting performance.The results are fascinating and reveal two key insights:
1ļøā£ Sensitivity profiles are like architectural "fingerprints." Models from the same family share strikingly similar sensitivity patterns. As you can see in the charts below for the Gemma and SmolLM families, the ranking and relative sensitivity of the layers remain remarkably consistent. This suggests that the underlying architecture is a primary driver of a model's quantization behavior.
2ļøā£ A "universal" mixed-precision quantization strategy is challenging. While models within a family are similar, these "fingerprints" change dramatically when comparing different architectures like LLaMA, Qwen, and StableLM. This highlights the difficulty in creating a generalized mixed-precision configuration that works optimally across all model families.
However, there is one near-universal truth we uncovered: the mlp.down_proj layer consistently emerges as one of the most sensitive components across all models studied.
This finding strongly resonates with the work in "The Super Weight in Large Language Models" (by Mengxia Yu et al.). The paper identifies that functionally critical parameters, or "super weights," are concentrated in these down_proj layers. Our empirical results provide clear validation for this theory, showing these layers are highly intolerant to precision loss.
In short, while every architecture has a unique sensitivity profile, a fingerprint shaped not only by its core design but also by its specific training dataset and optimization approach, some components remain universally critical!
What are your thoughts?
As part of my ongoing work in mixed-precision quantization, I've been exploring this question by measuring layer-by-layer sensitivity. The goal is to see if we can find universal rules for which layers can be quantized aggressively without impacting performance.The results are fascinating and reveal two key insights:
1ļøā£ Sensitivity profiles are like architectural "fingerprints." Models from the same family share strikingly similar sensitivity patterns. As you can see in the charts below for the Gemma and SmolLM families, the ranking and relative sensitivity of the layers remain remarkably consistent. This suggests that the underlying architecture is a primary driver of a model's quantization behavior.
2ļøā£ A "universal" mixed-precision quantization strategy is challenging. While models within a family are similar, these "fingerprints" change dramatically when comparing different architectures like LLaMA, Qwen, and StableLM. This highlights the difficulty in creating a generalized mixed-precision configuration that works optimally across all model families.
However, there is one near-universal truth we uncovered: the mlp.down_proj layer consistently emerges as one of the most sensitive components across all models studied.
This finding strongly resonates with the work in "The Super Weight in Large Language Models" (by Mengxia Yu et al.). The paper identifies that functionally critical parameters, or "super weights," are concentrated in these down_proj layers. Our empirical results provide clear validation for this theory, showing these layers are highly intolerant to precision loss.
In short, while every architecture has a unique sensitivity profile, a fingerprint shaped not only by its core design but also by its specific training dataset and optimization approach, some components remain universally critical!
What are your thoughts?
- 4 replies
Post
2189
Deploy GPT OSS models with Hugging Face on Azure AI!
Weāre thrilled to enable OpenAI GPT OSS models on Azure AI Model Catalog for Azure users to try the model securely the day of its release.
In our official launch blogpost, thereās a section on how to deploy the model to your Azure AI Hub. Get started today!
https://huggingface.co/blog/welcome-openai-gpt-oss#azure
Weāre thrilled to enable OpenAI GPT OSS models on Azure AI Model Catalog for Azure users to try the model securely the day of its release.
In our official launch blogpost, thereās a section on how to deploy the model to your Azure AI Hub. Get started today!
https://huggingface.co/blog/welcome-openai-gpt-oss#azure
Post
278
We now have the newest Open AI models available on the Dell Enterprise Hub!
We built the Dell Enterprise Hub to provide access to the latest and greatest model from the Hugging Face community to our on-prem customers. Weāre happy to give secure access to this amazing contribution from Open AI on the day of its launch!
https://dell.huggingface.co/
We built the Dell Enterprise Hub to provide access to the latest and greatest model from the Hugging Face community to our on-prem customers. Weāre happy to give secure access to this amazing contribution from Open AI on the day of its launch!
https://dell.huggingface.co/
Post
359
šŖ Qwen/Qwen3ā235BāA22BāInstructā2507āFP8 is now available in Microsoft Azure for oneāclick deployment! š
Check out their blogpost: https://qwenlm.github.io/blog/qwen3/
You can now find it in the Hugging Face Collection in Azure ML or Azure AI Foundry, along with 10k other Hugging Face models š¤š¤
Qwen/Qwen3-235B-A22B-Instruct-2507-FP8
Bear with us for the nonāquantized version.
Check out their blogpost: https://qwenlm.github.io/blog/qwen3/
You can now find it in the Hugging Face Collection in Azure ML or Azure AI Foundry, along with 10k other Hugging Face models š¤š¤
Qwen/Qwen3-235B-A22B-Instruct-2507-FP8
Bear with us for the nonāquantized version.
Post
1565
In our recent push to make more models available on Azure, we recently added SmolLM v3 in the catalog! š
@juanjucm wrote a really detailed guide on how to deploy on Azure AI š¤
https://huggingface.co/docs/microsoft-azure/azure-ai/examples/deploy-smollm3
If you want to see other models, please let us know
@juanjucm wrote a really detailed guide on how to deploy on Azure AI š¤
https://huggingface.co/docs/microsoft-azure/azure-ai/examples/deploy-smollm3
If you want to see other models, please let us know
- 1 reply
Post
212
š New in Azure Model Catalog: NVIDIA Parakeet TDT 0.6B V2
We're excited to welcome Parakeet TDT 0.6B V2āa state-of-the-art English speech-to-text modelāto the Azure Foundry Model Catalog.
What is it?
A powerful ASR model built on the FastConformer-TDT architecture, offering:
š Word-level timestamps
āļø Automatic punctuation & capitalization
š Strong performance across noisy and real-world audio
It runs with NeMo, NVIDIAās optimized inference engine.
Want to give it a try? š§ You can test it with your own audio (up to 3 hours) on Hugging Face Spaces before deploying.If it fits your need, deploy easily from the Hugging Face Hub or Azure ML Studio with secure, scalable infrastructure!
š Learn more by following this guide written by @alvarobartt
https://huggingface.co/docs/microsoft-azure/azure-ai/examples/deploy-nvidia-parakeet-asr
We're excited to welcome Parakeet TDT 0.6B V2āa state-of-the-art English speech-to-text modelāto the Azure Foundry Model Catalog.
What is it?
A powerful ASR model built on the FastConformer-TDT architecture, offering:
š Word-level timestamps
āļø Automatic punctuation & capitalization
š Strong performance across noisy and real-world audio
It runs with NeMo, NVIDIAās optimized inference engine.
Want to give it a try? š§ You can test it with your own audio (up to 3 hours) on Hugging Face Spaces before deploying.If it fits your need, deploy easily from the Hugging Face Hub or Azure ML Studio with secure, scalable infrastructure!
š Learn more by following this guide written by @alvarobartt
https://huggingface.co/docs/microsoft-azure/azure-ai/examples/deploy-nvidia-parakeet-asr
Post
1272
If you want to dive into how the HF team worked with
@seungrokj
at
@AMD
to optimize kernels on MI300, you should give a read to our latest blog!
Such a great educational material for anyone curious about the world of optimizing low level ML.
https://huggingface.co/blog/mi300kernels
to optimize kernels on MI300, you should give a read to our latest blog!
Such a great educational material for anyone curious about the world of optimizing low level ML.
https://huggingface.co/blog/mi300kernels
badaouiĀ
authored
a
paper
5 months ago
Post
1640
In case you missed it, Hugging Face expanded its collaboration with Azure a few weeks ago with a curated catalog of 10,000 models, accessible from Azure AI Foundry and Azure ML!
@alvarobartt cooked during these last days to prepare the one and only documentation you need, if you wanted to deploy Hugging Face models on Azure. It comes with an FAQ, great guides and examples on how to deploy VLMs, LLMs, smolagents and more to come very soon.
We need your feedback: come help us and let us know what else you want to see, which model we should add to the collection, which model task we should prioritize adding, what else we should build a tutorial for. Youāre just an issue away on our GitHub repo!
https://huggingface.co/docs/microsoft-azure/index
@alvarobartt cooked during these last days to prepare the one and only documentation you need, if you wanted to deploy Hugging Face models on Azure. It comes with an FAQ, great guides and examples on how to deploy VLMs, LLMs, smolagents and more to come very soon.
We need your feedback: come help us and let us know what else you want to see, which model we should add to the collection, which model task we should prioritize adding, what else we should build a tutorial for. Youāre just an issue away on our GitHub repo!
https://huggingface.co/docs/microsoft-azure/index
jeffboudierĀ
posted
an
update
5 months ago
Post
551
AMD summer hackathons are here!
A chance to get hands-on with MI300X GPUs and accelerate models.
š«š· Paris - Station F - July 5-6
š®š³ Mumbai - July 12-13
š®š³ Bengaluru - July 19-20
Hugging Face and GPU Mode will be on site and on July 6 in Paris @ror will share lessons learned while building new kernels to accelerate Llama 3.1 405B on ROCm
Register to Paris event: https://lu.ma/fmvdjmur?tk=KeAbiP
All dates: https://lu.ma/calendar/cal-3sxhD5FdxWsMDIz
A chance to get hands-on with MI300X GPUs and accelerate models.
š«š· Paris - Station F - July 5-6
š®š³ Mumbai - July 12-13
š®š³ Bengaluru - July 19-20
Hugging Face and GPU Mode will be on site and on July 6 in Paris @ror will share lessons learned while building new kernels to accelerate Llama 3.1 405B on ROCm
Register to Paris event: https://lu.ma/fmvdjmur?tk=KeAbiP
All dates: https://lu.ma/calendar/cal-3sxhD5FdxWsMDIz
Post
3251
Hackathons in Paris on July 5th and 6th!
Hugging Face just wrapped 4 months of deep work with AMD to push kernel-level optimization on their MI300X GPUs. Now, it's time to share everything we learned.
Join us in Paris at STATION F for a hands-on weekend of workshops and a hackathon focused on making open-source LLMs faster and more efficient on AMD.
Prizes, amazing host speakers, ... if you want more details, navigate to https://lu.ma/fmvdjmur!
Hugging Face just wrapped 4 months of deep work with AMD to push kernel-level optimization on their MI300X GPUs. Now, it's time to share everything we learned.
Join us in Paris at STATION F for a hands-on weekend of workshops and a hackathon focused on making open-source LLMs faster and more efficient on AMD.
Prizes, amazing host speakers, ... if you want more details, navigate to https://lu.ma/fmvdjmur!
-
- 2 replies