I filmed over 6+ Hours of footage and hand wrote 85+ pages of notes to teach absolute beginners about AI Inference.
Over the last 2 months, whilst working as a full-time SWE intern, I was grinding after hours to create this video series. I learned to code, amongst many other things, through YouTube videos aimed at beginners. Now, with the rise of AI, these videos tend to show up less, so I decided I wanted to change that.
This is . A lot of thinking and consideration went into creating this, and all of my notes are written by me. For transparency almost all code was also written by me with exception of small bug-fixes, readme updates and stuff like that. There is also extra notes pertaining to each video which are AI generated, but main notes are all me.
The series walks you through building a small Llama 3.2 1B inference engine in C++ and CUDA, starting with what happens when you send an LLM a prompt.
I cover:
How text becomes tokens, and tokens become embeddings.
Loading model weights into GPU memory.
CUDA basics, writing your own GPU kernels, and matrix multiplication with cuBLAS.
The transformer step by step: RMSNorm, RoPE, attention, and the SwiGLU feed-forward network.
Prefill and decode: processing your prompt and generating a response one token at a time.
KV caching and how it saves repeated computation.
Static and continuous batching to handle multiple requests.
Paged KV caching and paged attention to manage and read cached data on the GPU.
Here is the github repo as well as a link to my youtube:
Over the last 2 months, whilst working as a full-time SWE intern, I was grinding after hours to create this video series. I learned to code, amongst many other things, through YouTube videos aimed at beginners. Now, with the rise of AI, these videos tend to show up less, so I decided I wanted to change that.
This is . A lot of thinking and consideration went into creating this, and all of my notes are written by me. For transparency almost all code was also written by me with exception of small bug-fixes, readme updates and stuff like that. There is also extra notes pertaining to each video which are AI generated, but main notes are all me.
The series walks you through building a small Llama 3.2 1B inference engine in C++ and CUDA, starting with what happens when you send an LLM a prompt.
I cover: How text becomes tokens, and tokens become embeddings. Loading model weights into GPU memory. CUDA basics, writing your own GPU kernels, and matrix multiplication with cuBLAS. The transformer step by step: RMSNorm, RoPE, attention, and the SwiGLU feed-forward network. Prefill and decode: processing your prompt and generating a response one token at a time. KV caching and how it saves repeated computation. Static and continuous batching to handle multiple requests. Paged KV caching and paged attention to manage and read cached data on the GPU.
Here is the github repo as well as a link to my youtube:
https://github.com/samir-sarwar/samir-vllm https://www.youtube.com/@samir-sarwar
(if anyone hiring for summer 2027 interns focused on inference or mle hmu please!! )