📖CUDA Learn Notes with PyTorch: fp32、fp16/bf16、fp8/int8、flash_attn、sgemm、sgemv、warp/block reduce、dot prod、elementwise、softmax、layernorm、rmsnorm、hist etc. 👉News: Most of my time now is focused on LLM/VLM/Diffusion Inference. Please check 📖Awesome-LLM-Inference , 📖Awesome-SD-Inference and 📖CUDA-Learn-Notes for more details.
- / = not supported now.
- ✔️ = known work and already supported now.
- ❔ = in my plan, but not coming soon, maybe a few weeks later.
- workflow: custom CUDA kernel impl -> Torch python binding -> Run tests.
👉TIPS: * means using Tensor Cores(MMA PTX), otherwise, using CUDA Cores by default.
💡说明: 大佬们写的文章实在是太棒了,学到了很多东西。欢迎大家提PR推荐更多优秀的文章!
GNU General Public License v3.0
Welcome to 🌟👆🏻star & submit a PR to this repo!