--- dated: '2020-10-01' title: 'Dense Linear Algebra on AMD GPUs' blurb: 'TL;DR: we were able to get 73% faster time-to-solution for solvers, and up to 60x faster for BLAS operations.' image: ./magma-paper.webp --- I published my first paper **Design, Optimization, and Benchmarking of Dense Linear Algebra Algorithms on AMD GPUs**, in [IEEE HPEC 2020](https://ieee-hpec.org/). This was a major project of mine at [ICL](https://icl.utk.edu) that aimed to take the existing [MAGMA](https://icl.utk.edu/magma/index.html) linear algebra library which ran on NVIDIA GPUs, and port it to AMD GPUs. Specifically, it needed to work on the [Frontier supercomputer at ORNL](https://en.wikipedia.org/wiki/Frontier_(supercomputer)), which only has AMD GPUs. Interestingly, my [co-author won the Turing award](https://en.wikipedia.org/wiki/Jack_Dongarra). He is a legend in the HPC community, and I was very honored to work with him. I'd also like to thank [Stan Tomov](https://icl.utk.edu/~tomov/) for mentoring me and helping me get this research job. **TL;DR: we were able to get 73% faster time-to-solution for solvers, and up to 60x faster for BLAS operations.** We tuned various algorithms (matrix multiplication, rank-k updates) that are common building blocks of machine learning, physical simulations, and other applications. There were a number of obstacles to extracting good performance out of the new hardware; specifically the change in warp size (32 -> 64), new compiler (NVIDIA's nvcc -> AMD's ROCm/hipcc), and overall higher peak performance of newer hardware. So much of the HPC ecosystem was built around CUDA, and luckily AMD's HIP was very similar conceptually. You can access it at these places:d * [at ICL's PDF](https://www.icl.utk.edu/files/publications/2020/icl-utk-1405-2020.pdf) * [at Semantic Scholar](https://www.semanticscholar.org/paper/Design%2C-Optimization%2C-and-Benchmarking-of-Dense-on-Brown-Abdelfattah/d233df73ce91141aad90c956b4c3ddf6cd2b3847) * [at HGPU](https://hgpu.org/?p=22887) * [at OS hackathon's resources](https://www.oshackathon.org/resources#h.2swl4wedmt8y) Also, I have a local archive copy [here](/posts/magma-paper/dense-linear-algebra-amd-gpus-hpec-2020.pdf):