aiefs vol1 foundations

View aiefs vol1 foundations flipbook.

VOL_001 — REFERENCE MANUAL © 2026 · OPEN SOURCE · MIT LICENSE AI ENGINEERING FROM SCRATCH. Foundations: Math, Tooling, and Classical Machine Learning VOLUME I / VI · 52 CHAPTERS · PHASES 00 · 01 · 02 EDITION 2026.10 · A SNAPSHOT OF A LIVING COURSE · NEWEST BUILD ALWAYS AT THE LINK BELOW aiengineeringfromscratch.com github.com/rohitg00/ai-engineering-from-scratch

Contents About This Volume 1 How to Use This Book 2 Part I — Setup & Tooling 3 Dev Environment 4 Git & Collaboration 9 GPU Setup & Cloud 12 APIs & Keys 15 Jupyter Notebooks 19 Python Environments 24 Docker for AI 29 Editor Setup 37 Data Management 42 Terminal & Shell 48 Linux for AI 55 Debugging and Profiling 62 Part II — Math Foundations 70 Linear Algebra Intuition 71 Vectors, Matrices & Operations 80 Matrix Transformations 87 Calculus for Machine Learning 95 Chain Rule & Automatic Differentiation 108 Probability and Distributions 119 Bayes’ Theorem 128 Optimization 136 Information Theory 144 AIENGINEERINGFROMSCRATCH.COM i

CONTENTS Dimensionality Reduction 153 Singular Value Decomposition 162 Tensor Operations 174 Numerical Stability 182 Norms and Distances 194 Statistics for Machine Learning 205 Sampling Methods 215 Linear Systems 229 Convex Optimization 241 Complex Numbers for AI 253 The Fourier Transform 262 Graph Theory for Machine Learning 272 Stochastic Processes 282 Part III — ML Fundamentals 292 What Is Machine Learning 293 Linear Regression 302 Logistic Regression 314 Decision Trees and Random Forests 325 Support Vector Machines 335 K-Nearest Neighbors and Distances 343 Unsupervised Learning 351 Feature Engineering & Selection 362 Model Evaluation 374 Bias-Variance Tradeoff 388 Ensemble Methods 398 Hyperparameter Tuning 406 ML Pipelines 418 Naive Bayes 426 Time Series Fundamentals 436 Anomaly Detection 446 AIENGINEERINGFROMSCRATCH.COM ii

CONTENTS Handling Imbalanced Data 456 Feature Selection 467 AIENGINEERINGFROMSCRATCH.COM iii

About This Volume This is Volume 1 of AI Engineering from Scratch, a six-volume compilation of the open course of the same name. Each volume stands alone; cross-references cite course phase numbers, which map to volumes like this: Vol Title Course phases 1 Foundations — Math, Tooling, and Classical Machine Learning 00, 01, 02 2 Deep Learning — Networks, Vision, and Speech 03, 04, 06 3 Language — NLP Foundations and the Transformer 05, 07 4 Large Language Models — Generation, Reinforcement, Pretraining, and Engineering 08, 09, 10, 11 5 Agents — Multimodality, Protocols, Autonomy, and Swarms 12, 13, 14, 15, 16 6 Production — Infrastructure, Safety, and Capstones 17, 18, 19 The chapters in this volume come from course phases 00, 01, 02. Chapter prerequisites name phases, not volumes; use the table above to translate. AIENGINEERINGFROMSCRATCH.COM 1

How to Use This Book This volume is one loop of a larger machine, and it works best when you run the whole loop: 1. Read the chapter here. The prose, the derivations, and the code walkthroughs are complete on the page. 2. Run the code from the repository. Every chapter has a code/ directory with a working implementation you can run and break: https://github.com/rohitg00/ai-engineering- from-scratch 3. Open the web edition for what paper cannot do. Animated figures you can watch and drag, and a quiz per chapter that grades itself: https://aiengineeringfromscratch.com The repository is the living edition. Lessons are updated as the field moves; the book is a snapshot with a version number. When they disagree, the repo is right. Learning with an AI This course is built to be read by agents as well as people. The machine-readable index of every lesson lives at https://aiengineeringfromscratch.com/llms.txt. If you learn with an AI assistant, paste this and go: I am working through AI Engineering from Scratch, Volume 1: Foundations. Fetch https://aiengineeringfromscratch.com/llms.txt, find the lesson I name, and act as my tutor: quiz me on its Key Terms, review my solutions to its Exercises, and walk me through its code from the repository. AIENGINEERINGFROMSCRATCH.COM 2

Part I — Setup & Tooling Course phase 00. Live edition with animated figures and quizzes: https://aiengineeringfroms cratch.com/catalog.html AIENGINEERINGFROMSCRATCH.COM 3

Dev Environment Your tools shape your thinking. Set them up once, set them up right. Type: Build Languages: Python, Node.js, Rust Prerequisites: None Time: ~45 minutes Learning Objectives • Set up Python 3.11+, Node.js 20+, and Rust toolchains from scratch • Configure virtual environments and package managers for reproducible builds • Verify GPU access with CUDA/MPS and run a test tensor operation • Understand the four-layer stack: system, packages, runtimes, AI libraries The Problem You’re about to learn AI engineering across 500+ lessons using Python, TypeScript, Rust, and Julia. If your environment is broken, every single lesson becomes a fight against tooling instead of learning. Most people skip environment setup. Then they spend hours debugging import errors, version conflicts, and missing CUDA drivers. We’re going to do this once, properly. The Concept An AI engineering environment has four layers: Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/01-dev-environment We install bottom-up. Each layer depends on the one below it. Interactive figure: s0-env-stack. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/01-dev-environment Build It Step 1: System Foundation Check your system and install the basics. # macOS xcode-select --install brew install git curl wget # Ubuntu/Debian AIENGINEERINGFROMSCRATCH.COM 4

DEV ENVIRONMENT sudo apt update && sudo apt install -y build-essential git curl wget unzip # Windows (use WSL2) wsl --install -d Ubuntu-24.04 Step 2: Python with uv We use uv — it’s 10-100x faster than pip and handles virtual environments automatically. curl -LsSf https://astral.sh/uv/install.sh | sh uv python install 3.12 uv venv source .venv/bin/activate # or .venv\Scripts\activate on Windows uv pip install numpy matplotlib jupyter Verify: import sys print(f"Python {sys.version}") import numpy as np print(f"NumPy {np.__version__}") a = np.array([1, 2, 3]) print(f"Vector: {a}, dot product with itself: {np.dot(a, a)}") Step 3: Node.js with pnpm For TypeScript lessons (agents, MCP servers, web apps). curl -fsSL https://fnm.vercel.app/install | bash fnm install 22 fnm use 22 npm install -g pnpm node -e "console.log('Node', process.version)" The fnm installer checks for unzip first and exits with Not installing fnm due to missin g dependencies. when it is absent: on Linux it unpacks a zip archive, on macOS it installs through Homebrew. macOS ships unzip; Ubuntu, Debian, and WSL2 get it from the Step 1 apt line (sudo apt install -y unzip if you skipped that step). macOS / Apple Silicon (M1/M2/M3/M4): If the installer stops with Error: Cannot install under Rosetta 2 in ARM default prefix (/opt/homebrew), your terminal is running under Rosetta 2 (arch prints i386) while Homebrew is a native arm64 build. Install fnm forcing arm64, wire it into your shell, then rerun the commands above from fnm install 22: arch -arm64 brew install fnm echo 'eval "$(fnm env --use-on-cd)"' >> ~/.zshrc source ~/.zshrc Step 4: Rust For performance-critical lessons (inference, systems). AIENGINEERINGFROMSCRATCH.COM 5

DEV ENVIRONMENT curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh rustc --version cargo --version Step 5: Julia (Optional) For math-heavy lessons where Julia shines. curl -fsSL https://install.julialang.org | sh julia -e 'println("Julia ", VERSION)' Step 6: GPU Setup (If You Have One) NVIDIA (Linux / Windows): nvidia-smi # Install PyTorch with CUDA uv pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124↪ macOS / Apple Silicon (M1/M2/M3/M4): There is no CUDA on a Mac — that’s expected, not a failure. Do not pass --index-url .../cuXXX (those wheels are Linux/Windows only, so the install fails). Install the plain build, which includes Apple’s MPS (Metal) GPU backend: uv pip install torch torchvision torchaudio Verify (works on any platform): import torch print(f"CUDA available: {torch.cuda.is_available()}") # False on macOS — expected↪ print(f"MPS available: {torch.backends.mps.is_available()}") # True on Apple Silicon↪ if torch.cuda.is_available(): print(f"GPU: {torch.cuda.get_device_name(0)}") No GPU? No problem. Most lessons work on CPU. For training-heavy lessons, use Google Colab or cloud GPUs. Step 7: Verify the route you want to start Run every command in this lesson from the repository root, the directory that contains README .md and phases/. The preflight checks only what you need to start the selected route. It skips later tools by default so a new learner sees one clear answer instead of a wall of warnings. Start the full beginner sequence: python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route beginner↪ Or check only the route you want: python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route ml-foundations↪ python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route llm-engineering↪ AIENGINEERINGFROMSCRATCH.COM 6

DEV ENVIRONMENT python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route agents↪ python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route mcp python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route agent-skills↪ python3 phases/00-setup-and-tooling/01-dev-environment/code/verify.py --route certification↪ Add --show-later when you want the same preflight to inspect optional tools and dependen- cies used by later lessons. A missing later tool never blocks the selected route. Each failed required check includes the detected path or import error and an exact corrective command. The Agent Skills and certification routes also show manual host checks because a Python script cannot prove that an AI host has discovered a skill or that your chosen skill scope is writable. When the beginner preflight passes, it prints the exact first runnable lesson: Ready to start Beginner course. Next: python3 phases/01-math-foundations/01-linear-algebra-intuition/code/vectors.py↪ Use It Your environment is ready to start the route you checked. Install later tools when a lesson asks for them instead of blocking your first lesson on the whole stack. Here is what you will use across the curriculum: Language Used In Package Manager Python Phases 1-12 (ML, DL, NLP, Vision, Audio, LLMs) uv TypeScript Phases 13-17 (Tools, Agents, Swarms, Infra) pnpm Rust Phases 12, 15-17 (Per formance-critical systems) cargo Julia Phase 1 (Math foundations) Pkg This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment/code 1. Run the verification script and fix any failures 2. Create a Python virtual environment for this course and install PyTorch 3. Write a “hello world” in all four languages and run each one AIENGINEERINGFROMSCRATCH.COM 7

DEV ENVIRONMENT Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/01-dev-environment • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/01-dev-environment/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/01-dev-environment The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 8

Git & Collaboration Version control is not optional. Every experiment, every model, every lesson you build here gets tracked. Type: Learn Languages: – Prerequisites: Phase 0, Lesson 01 Time: ~30 minutes Learning Objectives • Configure git identity and use the daily workflow of add, commit, and push • Create and merge branches for isolated experiments without breaking main • Write a .gitignore that excludes model checkpoints and large binary files • Navigate the commit history with git log to understand project evolution The Problem You’re about to write hundreds of code files across 20 phases. Without version control you will lose work, break things you can’t undo, and have no way to collaborate with others. Git is the tool. GitHub is where the code lives. This lesson covers what you need for this course and nothing more. The Concept Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/02-git-and-collaboration Three things to remember: 1. Save often (git commit) 2. Push to remote (git push) 3. Branch for experiments (git checkout -b experiment) Interactive figure: s0-commit-dag. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/02-git-and-collaboration Build It Step 1: Configure git git config --global user.name "Your Name" git config --global user.email "you@example.com" Step 2: The daily workflow git status git add file.py AIENGINEERINGFROMSCRATCH.COM 9

GIT & COLLABORATION git commit -m "Add perceptron implementation" git push origin main Step 3: Branching for experiments git checkout -b experiment/new-optimizer # ... make changes, commit ... git checkout main git merge experiment/new-optimizer Step 4: Working with this course repo You can’t push to the course repo itself — only maintainers have write access. Fork it on GitHub first (the Fork button, top right) so origin points at your own copy: git clone https://github.com/YOUR-USERNAME/ai-engineering-from-scratch.git cd ai-engineering-from-scratch git checkout -b my-progress # work through lessons, commit your code git push origin my-progress Use It For this course, you need exactly these commands: Command When git clone Get the course repo git add + git commit Save your work git push Back it up to GitHub git checkout -b Try something without breaking main git log --oneline See what you’ve done That’s it. You don’t need rebase, cherry-pick, or submodules for this course. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/00-setup-and-tooling/02-git-and-collaboration/code 1. Fork this repo, clone your fork, create a branch called my-progress, make a file, commit it, push it 2. Create a .gitignore that excludes model checkpoint files (.pt, .pth, .safetensors) 3. Look at the commit history of this repo with git log --oneline and read how lessons were added AIENGINEERINGFROMSCRATCH.COM 10

GIT & COLLABORATION Key Terms Term What people say What it actually means Commit “Saving” A snapshot of your entire project at a point in time Branch “A copy” A pointer to a commit that moves forward as you work Merge “Combining code” Taking changes from one branch and applying them to another Remote “The cloud” A copy of your repo hosted somewhere else (GitHub, GitLab) Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/02-git-and-collaboration • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/02-git-and-collaboration/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/02-git-and-collaboration The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 11

GPU Setup & Cloud Training on CPU is fine for learning. Training for real needs a GPU. Type: Build Languages: Python Prerequisites: Phase 0, Lesson 01 Time: ~45 minutes Learning Objectives • Verify local GPU availability using nvidia-smi and PyTorch’s CUDA API • Configure Google Colab with a T4 GPU for free cloud-based experiments • Benchmark matrix multiplication on CPU vs GPU and measure the speedup • Estimate the largest model that fits in your VRAM using the fp16 rule of thumb The Problem Most lessons in phases 1-3 run fine on CPU. But once you start training CNNs, transformers, or LLMs (phases 4+), you need GPU acceleration. A training run that takes 8 hours on CPU takes 10 minutes on GPU. You have three options: local GPU, cloud GPU, or Google Colab (free). The Concept Your options: 1. Local NVIDIA GPU Cost: $0 (you already have it) Setup: Install CUDA + cuDNN Best for: Regular use, large datasets 2. Google Colab (free tier) Cost: $0 Setup: None Best for: Quick experiments, no GPU at home 3. Cloud GPU (Lambda, RunPod, Vast.ai) Cost: $0.20-2.00/hr Setup: SSH + install Best for: Serious training, large models Interactive figure: s0-gpu-dispatch. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/03-gpu-setup-and-cloud AIENGINEERINGFROMSCRATCH.COM 12

GPU SETUP & CLOUD Build It Option 1: Local NVIDIA GPU Check if you have one: nvidia-smi Install PyTorch with CUDA: import torch print(f"CUDA available: {torch.cuda.is_available()}") print(f"CUDA version: {torch.version.cuda}") if torch.cuda.is_available(): print(f"GPU: {torch.cuda.get_device_name(0)}") print(f"Memory: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")↪ Option 2: Google Colab 1. Go to colab.research.google.com 2. Runtime > Change runtime type > T4 GPU 3. Run !nvidia-smi to verify Upload notebooks from this course directly to Colab. Option 3: Cloud GPU For Lambda Labs, RunPod, or Vast.ai: ssh user@your-gpu-instance pip install torch torchvision torchaudio python -c "import torch; print(torch.cuda.get_device_name(0))" No GPU? No problem. Most lessons work on CPU. The ones that need GPU will say so and include Colab links. device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print(f"Using: {device}") Build It: GPU vs CPU benchmark import torch import time size = 5000 a_cpu = torch.randn(size, size) b_cpu = torch.randn(size, size) start = time.time() c_cpu = a_cpu @ b_cpu AIENGINEERINGFROMSCRATCH.COM 13

GPU SETUP & CLOUD cpu_time = time.time() - start print(f"CPU: {cpu_time:.3f}s") if torch.cuda.is_available(): a_gpu = a_cpu.to("cuda") b_gpu = b_cpu.to("cuda") torch.cuda.synchronize() start = time.time() c_gpu = a_gpu @ b_gpu torch.cuda.synchronize() gpu_time = time.time() - start print(f"GPU: {gpu_time:.3f}s") print(f"Speedup: {cpu_time / gpu_time:.0f}x") Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/00-setup-and-tooling/03-gpu-setup-and-cloud/code 1. Run the benchmark above and compare CPU vs GPU times 2. If you don’t have a GPU, run it on Google Colab and compare 3. Check how much GPU memory you have and estimate the largest model you can fit (rule of thumb: 2 bytes per parameter for fp16) Key Terms Term What people say What it actually means CUDA “GPU programming” NVIDIA’s parallel computing platform that lets you run code on the GPU VRAM “GPU memory” Video RAM on the GPU, separate from system RAM. Limits model size. fp16 “Half precision” 16-bit floating point, uses half the memory of fp32 with minimal accuracy loss Tensor Core “Fast matrix hardware” Specialized GPU cores for matrix multiplication, 4-8x faster than regular cores Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/03-gpu-setup-and-cloud • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/03-gpu-setup-and-cloud/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/03-gpu-setup-and-cloud The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 14

APIs & Keys Every AI API works the same way: send a request, get a response. The details change, the pattern doesn’t. Type: Build Languages: Python, TypeScript Prerequisites: Phase 0, Lesson 01 Time: ~30 minutes Learning Objectives • Store API keys securely using environment variables and .env files • Make an LLM API call using both the Anthropic Python SDK and raw HTTP • Compare SDK-based and raw HTTP request/response formats for debugging • Identify and handle common API errors including authentication and rate limits The Problem Starting from Phase 11, you’ll call LLM APIs (Anthropic, OpenAI, Google). In Phase 13-16 you’ll build agents that use these APIs in loops. You need to know how API keys work, how to store them safely, and how to make your first API call. The Concept Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/04-apis-and-keys Every API call has: 1. An endpoint (URL) 2. An API key (authentication) 3. A request body (what you want) 4. A response body (what you get back) Interactive figure: s0-secret-inject. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00- setup-and-tooling/04-apis-and-keys Build It Step 1: Store API keys safely Never put API keys in code. Use environment variables. export ANTHROPIC_API_KEY="sk-ant-..." export OPENAI_API_KEY="sk-..." Or use a .env file (add it to .gitignore): ANTHROPIC_API_KEY=sk-ant-... OPENAI_API_KEY=sk-... AIENGINEERINGFROMSCRATCH.COM 15

APIS & KEYS Step 2: First API call (Python) import os import anthropic client = anthropic.Anthropic() MODEL = os.environ.get("LLM_MODEL", "claude-sonnet-5") response = client.messages.create( model=MODEL, max_tokens=256, messages=[{"role": "user", "content": "What is a neural network in one sentence?"}]↪ ) print(response.content[0].text) LLM_MODEL selects the Anthropic model id, and the default is the un-dated Sonnet alias. Other providers (OpenAI, Google, and others) follow the same pattern of a key plus a model id, but each has its own SDK, endpoint, and request/response schema. Step 3: First API call (TypeScript) import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic(); const MODEL = process.env.LLM_MODEL ?? "claude-sonnet-5"; const response = await client.messages.create({ model: MODEL, max_tokens: 256, messages: [{ role: "user", content: "What is a neural network in one sentence?" }],↪ }); console.log(response.content[0].text); Step 4: Raw HTTP (no SDK) import os import urllib.request import json url = "https://api.anthropic.com/v1/messages" headers = { "Content-Type": "application/json", "x-api-key": os.environ["ANTHROPIC_API_KEY"], "anthropic-version": "2023-06-01", } body = json.dumps({ "model": os.environ.get("LLM_MODEL", "claude-sonnet-5"), AIENGINEERINGFROMSCRATCH.COM 16

APIS & KEYS "max_tokens": 256, "messages": [{"role": "user", "content": "What is a neural network in one sentence?"}],↪ }).encode() req = urllib.request.Request(url, data=body, headers=headers, method="POST") with urllib.request.urlopen(req) as resp: result = json.loads(resp.read()) print(result["content"][0]["text"]) This is what the SDKs do under the hood. Understanding the raw HTTP call helps when debugging. Use It For this course: API When you need it Free tier Anthropic (Claude) Phases 11-16 (agents, tools) $5 credit on signup OpenAI Phase 11 (comparison) $5 credit on signup Hugging Face Phases 4-10 (models, datasets) Free You don’t need all of them right now. Set them up when the lesson requires it. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/04-apis-and-keys Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/04-apis-and-keys/code 1. Get an Anthropic API key and make your first API call 2. Try the raw HTTP version and compare the response format to the SDK version 3. Intentionally use a wrong API key and read the error message Key Terms Term What people say What it actually means API key “Password for the API” A unique string that identifies your account and authorizes requests Rate limit “They’re throttling me” Maximum requests per minute/hour to prevent abuse and ensure fair usage Token “A word” (in API context) A billing unit: input and output tokens are counted and charged separately AIENGINEERINGFROMSCRATCH.COM 17

APIS & KEYS Term What people say What it actually means Streaming “Real-time responses” Getting the response word by word instead of waiting for the full response Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/04-apis-and-keys • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/04-apis-and-keys/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/04-apis-and-keys The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 18

Jupyter Notebooks Notebooks are the lab bench of AI engineering. You prototype here, then move what works into production. Type: Build Languages: Python Prerequisites: Phase 0, Lesson 01 Time: ~30 minutes Learning Objectives • Install and launch JupyterLab, Jupyter Notebook, or VS Code with the Jupyter extension • Use magic commands (%timeit, %%time, %matplotlib inline) to benchmark and visualize inline • Distinguish when to use notebooks vs scripts and apply the “explore in notebooks, ship in scripts” workflow • Identify and avoid common notebook traps: out-of-order execution, hidden state, and memory leaks The Problem Every AI paper, tutorial, and Kaggle competition uses Jupyter notebooks. They let you run code in pieces, see outputs inline, mix code with explanations, and iterate fast. If you try to learn AI without notebooks, you’re doing math homework without scratch paper. But notebooks have real traps. People use them for everything, including things they’re terri- ble at. Knowing when to use a notebook and when to use a script will save you from debugging nightmares later. The Concept A notebook is a list of cells. Each cell is either code or text. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/05-jupyter-notebooks The kernel is a Python process running in the background. When you run a cell, it sends the code to the kernel, which executes it and sends back the result. All cells share the same kernel, so variables persist between cells. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/05-jupyter-notebooks That “whatever order you click” part is both the superpower and the foot-gun. Interactive figure: s0-cell-order. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/05-jupyter-notebooks AIENGINEERINGFROMSCRATCH.COM 19

JUPYTER NOTEBOOKS Build It Step 1: Pick your interface Three options, one format: Interface Install Best for JupyterLab pip install jupyterlab then jupyter lab Full IDE experience, multiple tabs, file browser, terminal Jupyter Notebook pip install notebook then jupyter notebook Simple, lightweight, one notebook at a time VS Code Install “Jupyter” extension Already in your editor, git integration, debugging All three read and write the same .ipynb file. Pick whatever you like. JupyterLab is the most common in AI work. pip install jupyterlab jupyter lab Step 2: Keyboard shortcuts that matter You operate in two modes. Press Escape for command mode (blue bar on the left), Enter for edit mode (green bar). Command mode (most used): Key Action Shift+Enter Run cell, move to next A Insert cell above B Insert cell below DD Delete cell M Convert to markdown Y Convert to code Z Undo cell operation Ctrl+Shift+H Show all shortcuts Edit mode: Key Action Tab Autocomplete Shift+Tab Show function signature Ctrl+/ Toggle comment Shift+Enter is the one you’ll use a thousand times a day. Learn it first. Step 3: Cell types Code cells run Python and show the output: AIENGINEERINGFROMSCRATCH.COM 20

JUPYTER NOTEBOOKS import numpy as np data = np.random.randn(1000) data.mean(), data.std() Output: (0.0032, 0.9987) Markdown cells render formatted text. Use them to document what you’re doing and why. Supports headers, bold, italic, LaTeX math ($E = mc^2$), tables, and images. Step 4: Magic commands These aren’t Python. They’re Jupyter-specific commands that start with % (line magic) or %% (cell magic). Time your code: %timeit np.random.randn(10000) Output: 45.2 us +/- 1.3 us per loop %%time model.fit(X_train, y_train, epochs=10) Output: Wall time: 2.34 s %timeit runs the code many times and averages. %%time runs it once. Use %timeit for mi- crobenchmarks, %%time for training runs. Enable inline plots: %matplotlib inline Every plt.plot() or plt.show() now renders directly in the notebook. Install packages without leaving the notebook: !pip install scikit-learn The ! prefix runs any shell command. Check environment variables: %env CUDA_VISIBLE_DEVICES Step 5: Display rich output inline Notebooks auto-display the last expression in a cell. But you can control it: import pandas as pd df = pd.DataFrame({ "model": ["Linear", "Random Forest", "Neural Net"], "accuracy": [0.72, 0.89, 0.94], "training_time": [0.1, 2.3, 45.6] }) df This renders a formatted HTML table, not a text dump. Same with plots: import matplotlib.pyplot as plt plt.figure(figsize=(8, 4)) plt.plot([1, 2, 3, 4], [1, 4, 2, 3]) AIENGINEERINGFROMSCRATCH.COM 21

JUPYTER NOTEBOOKS plt.title("Inline Plot") plt.show() The plot appears right below the cell. This is why notebooks dominate AI work. You see the data, the plot, and the code together. For images: from IPython.display import Image, display display(Image(filename="architecture.png")) Step 6: Google Colab Colab is a free Jupyter notebook in the cloud. It gives you a GPU, pre-installed libraries, and Google Drive integration. No setup required. 1. Go to colab.research.google.com 2. Upload any .ipynb file from this course 3. Runtime > Change runtime type > T4 GPU (free) Colab differences from local Jupyter: - Files don’t persist between sessions (save to Drive or download) - Pre-installed: numpy, pandas, matplotlib, torch, tensorflow, sklearn - from g oogle.colab import files to upload/download files - from google.colab import drive; drive.mount('/content/drive') for persistent storage - Sessions time out after 90 minutes of inactivity (free tier) Use It Notebooks vs Scripts: When to use which Use notebooks for Use scripts for Exploring a dataset Training pipelines Prototyping a model Reusable utilities Visualizing results Anything with if __name__ Explaining your work Code that runs on a schedule Quick experiments Production code Course exercises Packages and libraries The rule: explore in notebooks, ship in scripts. A common workflow in AI: 1. Explore data in a notebook 2. Prototype your model in the notebook 3. Once it works, move the code to .py files 4. Import those .py files back into the notebook for further experiments Common traps Out-of-order execution. You run cell 5, then cell 2, then cell 7. The notebook works on your machine but breaks when someone runs it top to bottom. Fix: Kernel > Restart & Run All before sharing. Hidden state. You delete a cell but the variable it created is still in memory. The notebook looks clean but depends on a ghost cell. Fix: Restart the kernel regularly. Memory leaks. Loading a 4GB dataset, training a model, loading another dataset. Nothing gets freed. Fix: del variable_name and gc.collect(), or restart the kernel. AIENGINEERINGFROMSCRATCH.COM 22

JUPYTER NOTEBOOKS This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/05-jupyter-notebooks Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/00-setup-and-tooling/05-jupyter-notebooks/code 1. Open JupyterLab, create a notebook, and use %timeit to compare list comprehension vs numpy for creating an array of 100,000 random numbers 2. Create a notebook with both markdown and code cells that loads a CSV, displays a dataframe, and plots a chart. Then run Kernel > Restart & Run All to verify it works top to bottom 3. Take the code from code/notebook_tips.py, paste it into a Colab notebook, and run it with a free GPU Key Terms Term What people say What it actually means Kernel “The thing running my code” A separate Python process that executes cells and keeps variables in memory Cell “A code block” An independently runnable unit in a notebook, either code or markdown Magic command “Jupyter tricks” Special commands prefixed with % or %% that control the notebook environment .ipynb “Notebook file” A JSON file containing cells, outputs, and metadata. Stands for IPython Notebook Further Reading • JupyterLab Docs for the full feature set • Google Colab FAQ for Colab-specific limits and features • 28 Jupyter Notebook Tips for power-user shortcuts Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/05-jupyter-notebooks • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/05-jupyter-notebooks/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/05-jupyter-notebooks The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 23

Python Environments Dependency hell is real. Virtual environments are the cure. Type: Build Languages: Shell Prerequisites: Phase 0, Lesson 01 Time: ~30 minutes Learning Objectives • Create isolated virtual environments using uv, venv, or conda • Write a pyproject.toml with optional dependency groups and generate lockfiles for re- producibility • Diagnose and fix common pitfalls: global installs, pip/conda mixing, CUDA version mis- matches • Implement a per-phase environment strategy for projects with conflicting dependencies The Problem You install PyTorch 2.4 for a fine-tuning project. Next week, a different project needs PyTorch 2.1 because its CUDA build is pinned. You upgrade globally, and the first project breaks. You downgrade, and the second one breaks. This is dependency hell. It happens constantly in AI/ML work because: • PyTorch, JAX, and TensorFlow each ship their own CUDA bindings • Model libraries pin specific framework versions • A global pip install overwrites whatever was there before • CUDA 11.8 builds don’t work with CUDA 12.x drivers (and vice versa) The fix: every project gets its own isolated environment with its own packages. The Concept Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/06-python-environments Interactive figure: s0-env-isolation. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00- setup-and-tooling/06-python-environments Build It Option 1: uv venv (Recommended) uv is the fastest Python package manager (10-100x faster than pip). It handles virtual envi- ronments, Python versions, and dependency resolution in one tool. AIENGINEERINGFROMSCRATCH.COM 24

PYTHON ENVIRONMENTS curl -LsSf https://astral.sh/uv/install.sh | sh uv python install 3.12 cd your-project uv venv source .venv/bin/activate Install packages: uv pip install torch numpy Create a project with pyproject.toml in one step: uv init my-ai-project cd my-ai-project uv add torch numpy matplotlib Option 2: venv (Built-in) If you can’t install uv, Python ships with venv: python3 -m venv .venv source .venv/bin/activate # Linux/macOS .venv\Scripts\activate # Windows pip install torch numpy Slower than uv, but works everywhere Python is installed. Option 3: conda (When You Need It) Conda manages non-Python dependencies like CUDA toolkits, cuDNN, and C libraries. Use it when: • You need a specific CUDA toolkit version without installing it system-wide • You’re on a shared cluster where you can’t install system packages • A library’s install instructions say “use conda” # Install miniconda (not the full Anaconda) curl -LsSf https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda.sh↪ bash miniconda.sh -b conda create -n myproject python=3.12 conda activate myproject conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia One rule: if you use conda for an environment, use conda for all packages in that environment. Mixing pip install into a conda env causes dependency conflicts that are painful to debug. For This Course: Per-Phase Strategy You could create one environment for the whole course. Don’t. Different phases need different (sometimes conflicting) dependencies. Strategy: AIENGINEERINGFROMSCRATCH.COM 25

PYTHON ENVIRONMENTS ai-engineering-from-scratch/ ├── .venv/ <-- shared lightweight env for phases 0-3 ├── phases/ │ ├── 04-neural-networks/ │ │ └── .venv/ <-- PyTorch env │ ├── 05-cnns/ │ │ └── .venv/ <-- same PyTorch env (symlink or shared) │ ├── 08-transformers/ │ │ └── .venv/ <-- might need different transformer versions │ └── 11-llm-apis/ │ └── .venv/ <-- API SDKs, no torch needed The script in code/env_setup.sh creates the base environment for this course. pyproject.toml Basics Every Python project should have a pyproject.toml. It replaces setup.py, setup.cfg, and requirements.txt in one file. [project] name = "ai-engineering-from-scratch" version = "0.1.0" requires-python = ">=3.11" dependencies = [ "numpy>=1.26", "matplotlib>=3.8", "jupyter>=1.0", "scikit-learn>=1.4", ] [project.optional-dependencies] torch = ["torch>=2.3", "torchvision>=0.18"] llm = ["anthropic>=0.39", "openai>=1.50"] Then install: uv pip install -e ".[torch]" # base + PyTorch uv pip install -e ".[llm]" # base + LLM SDKs uv pip install -e ".[torch,llm]" # everything Lockfiles A lockfile pins every dependency (including transitive ones) to exact versions. This guarantees reproducibility: anyone who installs from the lockfile gets exactly the same packages. # uv generates uv.lock automatically when using uv add uv add numpy # pip-tools approach uv pip compile pyproject.toml -o requirements.lock uv pip install -r requirements.lock Commit your lockfile to git. When someone clones the repo, they install from the lockfile and get identical versions. AIENGINEERINGFROMSCRATCH.COM 26

PYTHON ENVIRONMENTS Common Mistakes 1. Installing globally pip install torch # BAD: installs to system Python source .venv/bin/activate pip install torch # GOOD: installs to virtual environment Check where your packages go: which python # should show .venv/bin/python, not /usr/bin/python which pip # should show .venv/bin/pip 2. Mixing pip and conda conda create -n myenv python=3.12 conda activate myenv conda install pytorch -c pytorch pip install some-other-package # BAD: can break conda's dependency tracking conda install some-other-package # GOOD: let conda manage everything If you must use pip inside conda (some packages are pip-only), install all conda packages first, then pip packages last. 3. Forgetting to activate python train.py # uses system Python, missing packages source .venv/bin/activate python train.py # uses project Python, packages found Your shell prompt should show the environment name: (.venv) $ python train.py 4. Committing .venv to git echo ".venv/" >> .gitignore Virtual environments are 200MB-2GB. They’re local, not portable between machines. Commit pyproject.toml and the lockfile instead. 5. CUDA version mismatch nvidia-smi # shows driver CUDA version (e.g., 12.4) python -c "import torch; print(torch.version.cuda)" # shows PyTorch CUDA version # These must be compatible. # PyTorch CUDA version must be <= driver CUDA version. Use It Run the setup script to create your course environment: bash phases/00-setup-and-tooling/06-python-environments/code/env_setup.sh AIENGINEERINGFROMSCRATCH.COM 27

PYTHON ENVIRONMENTS This creates a .venv at the repo root with core dependencies installed and verified. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/00-setup-and-tooling/06-python-environments/code 1. Run env_setup.sh and verify all checks pass 2. Create a second virtual environment, install a different version of numpy in it, and con- firm the two environments are isolated 3. Write a pyproject.toml for a project that needs both PyTorch and the Anthropic SDK 4. Deliberately install a package globally (without activating a venv), notice where it goes, then uninstall it Key Terms Term What people say What it actually means Virtual envi- ronment “A venv” An isolated directory containing a Python interpreter and packages, separate from the system Python Lockfile “Pinned dependencies” A file listing every package and its exact version, guaranteeing identical installs across machines pyproject.toml“The new setup.py” The standard Python project configuration file, replacing setup.py/setup.cfg/requireme nts.txt Transitive depen- dency “A dependency of a dependency” Package B depends on C; if you install A which depends on B, C is a transitive dependency of A CUDA mismatch “My GPU isn’t working” PyTorch was compiled for a different CUDA version than what your GPU driver supports Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/06-python-environments • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/06-python-environments/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/06-python-environments The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 28

Docker for AI Containers make “works on my machine” a thing of the past. Type: Build Languages: Docker Prerequisites: Phase 0, Lessons 01 and 03 Time: ~60 minutes Learning Objectives • Build a GPU-enabled Docker image with CUDA, PyTorch, and AI libraries from a Docker- file • Mount host directories as volumes to persist models, datasets, and code across container rebuilds • Configure the NVIDIA Container Toolkit to expose GPUs inside containers • Orchestrate multi-service AI applications (inference server + vector database) using Docker Compose The Problem You trained a model on your laptop with PyTorch 2.3, CUDA 12.4, and Python 3.12. Your colleague has PyTorch 2.1, CUDA 11.8, and Python 3.10. Your model crashes on their machine. Your Dockerfile works on both. AI projects are dependency nightmares. A typical stack includes Python, PyTorch, CUDA drivers, cuDNN, system-level C libraries, and specialized packages like flash-attn that need exact compiler versions. Docker packages all of this into a single image that runs identically everywhere. The Concept Docker wraps your code, runtime, libraries, and system tools into an isolated unit called a container. Think of it as a lightweight virtual machine, except it shares the host OS kernel instead of running its own, so it starts in seconds instead of minutes. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/07-docker-for-ai Why AI projects need Docker more than most 1. GPU drivers are fragile. CUDA 12.4 code does not run on CUDA 11.8. Docker isolates the CUDA toolkit inside the container while sharing the host GPU driver through the NVIDIA Container Toolkit. 2. Model weights are large. A 7B parameter model is 14 GB in fp16. You do not want to re-download it every time you rebuild. Docker volumes let you mount a models directory from the host. AIENGINEERINGFROMSCRATCH.COM 29

DOCKER FOR AI 3. Multi-service architectures are common. A real AI application is not just a Python script. It is an inference server, a vector database for RAG, maybe a web frontend. Docker Compose orchestrates all of these with one command. Key vocabulary Term What it means Image A read-only template. Your recipe. Built from a Dockerfile. Container A running instance of an image. Your kitchen. Dockerfile Instructions to build an image. Layer by layer. Volume Persistent storage that survives container restarts. docker-compose A tool for defining multi-container applications in YAML. Common container patterns in AI Dev Container Full toolkit. Editor support. Jupyter. Debugging tools. Used during development and experimentation. Training Container Minimal. Just the training script and dependencies. Runs on GPU clusters. No editor, no Jupyter. Inference Container Optimized for serving. Small image. Fast cold start. Runs behind a load balancer in production. Interactive figure: s0-image-layers. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/07-docker-for-ai Build It Step 1: Install Docker # macOS brew install --cask docker open /Applications/Docker.app # Ubuntu curl -fsSL https://get.docker.com | sh sudo usermod -aG docker $USER # Log out and back in for group change to take effect Verify: docker --version docker run hello-world AIENGINEERINGFROMSCRATCH.COM 30

DOCKER FOR AI Step 2: Install NVIDIA Container Toolkit (Linux with NVIDIA GPU) This lets Docker containers access your GPU. macOS and Windows (WSL2) users can skip this; Docker Desktop handles GPU passthrough differently on those platforms. distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg↪ curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-c ⌋ ontainer.list | \ ↪ ↪ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ ↪ ↪ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker Test GPU access inside a container: docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi If you see your GPU info, the toolkit is working. Step 3: Understand base images Choosing the right base image saves hours of debugging. nvidia/cuda:12.4.1-devel-ubuntu22.04 Full CUDA toolkit. Compilers included. Use for: building packages that need nvcc (flash-attn, bitsandbytes) Size: ~4 GB nvidia/cuda:12.4.1-runtime-ubuntu22.04 CUDA runtime only. No compilers. Use for: running pre-built code Size: ~1.5 GB pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime PyTorch pre-installed on top of CUDA. Use for: skipping the PyTorch install step Size: ~6 GB python:3.12-slim No CUDA. CPU only. Use for: inference on CPU, lightweight tools Size: ~150 MB Step 4: Write a Dockerfile for AI development Here is the Dockerfile in code/Dockerfile. Walk through it: FROM --platform=linux/amd64 nvidia/cuda:12.4.1-devel-ubuntu22.04 AIENGINEERINGFROMSCRATCH.COM 31

DOCKER FOR AI ENV DEBIAN_FRONTEND=noninteractive ENV PYTHONUNBUFFERED=1 RUN apt-get update && apt-get install -y --no-install-recommends \ software-properties-common \ git \ curl \ build-essential \ && add-apt-repository -y ppa:deadsnakes/ppa \ && apt-get update && apt-get install -y --no-install-recommends \ python3.12 \ python3.12-venv \ python3.12-dev \ && rm -rf /var/lib/apt/lists/* RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.12 1 RUN curl -sSL https://raw.githubusercontent.com/pypa/get-pip/3b73145063be545b649a ⌋ d9ca83ea8da5fc915a4f/public/get-pip.py -o /tmp/get-pip.py \ ↪ ↪ && echo "a341e1a43e38001c551a1508a73ff23636a11970b61d901d9a1cad2a18f57055 /tmp/get-pip.py" | sha256sum -c - \↪ && python /tmp/get-pip.py \ && rm /tmp/get-pip.py \ && update-alternatives --install /usr/bin/pip pip /usr/local/bin/pip3.12 1 RUN python -m pip install --no-cache-dir --upgrade pip setuptools wheel RUN python -m pip install --no-cache-dir \ torch==2.6.0+cu124 \ torchvision==0.21.0+cu124 \ torchaudio==2.6.0+cu124 \ --index-url https://download.pytorch.org/whl/cu124 RUN python -m pip install --no-cache-dir \ numpy \ pandas \ scikit-learn \ matplotlib \ jupyter \ transformers \ datasets \ accelerate \ safetensors WORKDIR /workspace VOLUME ["/workspace", "/models"] EXPOSE 8888 CMD ["python"] Build it: AIENGINEERINGFROMSCRATCH.COM 32

DOCKER FOR AI docker build -t ai-dev -f phases/00-setup-and-tooling/07-docker-for-ai/code/Dockerfile .↪ This takes a while the first time (downloading CUDA base image + PyTorch). Subsequent builds use cached layers. macOS / Apple Silicon (M1/M2/M3/M4): The --platform=linux/amd64 on the FROM line is what makes this build succeed on a Mac. The CUDA base image also ships an arm64 variant and Docker Desktop picks it automatically on Apple Silicon, but PyTorch publishes its cu124 wheels for x86_64 only, so the pip install torch==2.6.0+cu124 layer fails with No matching distribution found for torch==2.6.0+cu124. Pinning the platform pulls the x86_64 image and runs it under emulation: the build is slower and the container has no GPU (there is no CUDA on a Mac either way). Drop --gpus all from the docker run commands below on a Mac. For GPU work on Apple Silicon, run the lessons natively with the MPS build from Lesson 01 and keep this image for x86_64 Linux hosts with an NVIDIA GPU. Run it: docker run --rm -it --gpus all \ -v $(pwd):/workspace \ -v ~/models:/models \ ai-dev python -c "import torch; print(f'PyTorch {torch.__version__}, CUDA: {torch.cuda.is_available()}')"↪ Run Jupyter inside the container: docker run --rm -it --gpus all \ -v $(pwd):/workspace \ -v ~/models:/models \ -p 8888:8888 \ ai-dev jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root Step 5: Volume mounts for data and models Volume mounts are critical for AI work. Without them, your 14 GB model downloads vanish when the container stops. # Mount your code -v $(pwd):/workspace # Mount a shared models directory -v ~/models:/models # Mount datasets -v ~/datasets:/data Inside your training script, load from the mounted path: from transformers import AutoModel model = AutoModel.from_pretrained("/models/llama-7b") The model lives on your host filesystem. Rebuild the container as often as you want without re-downloading. Step 6: Docker Compose for multi-service AI apps A real RAG application needs an inference server and a vector database. Docker Compose runs both with one command. AIENGINEERINGFROMSCRATCH.COM 33

DOCKER FOR AI See code/docker-compose.yml: services: ai-dev: build: context: . dockerfile: Dockerfile deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] volumes: - ../../../:/workspace - ~/models:/models - ~/datasets:/data ports: - "8888:8888" stdin_open: true tty: true command: jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root qdrant: image: qdrant/qdrant:v1.12.5 ports: - "6333:6333" - "6334:6334" volumes: - qdrant_data:/qdrant/storage volumes: qdrant_data: Start everything: cd phases/00-setup-and-tooling/07-docker-for-ai/code docker compose up -d Now your AI dev container can reach the vector database at http://qdrant:6333 by service name. Docker Compose creates a shared network automatically. Test the connection from inside the AI container: from qdrant_client import QdrantClient client = QdrantClient(host="qdrant", port=6333) print(client.get_collections()) Stop everything: docker compose down Add -v to also delete the qdrant volume: docker compose down -v AIENGINEERINGFROMSCRATCH.COM 34

DOCKER FOR AI Step 7: Useful Docker commands for AI work # List running containers docker ps # List all images and their sizes docker images # Remove unused images (reclaim disk space) docker system prune -a # Check GPU usage inside a running container docker exec -it <container_id> nvidia-smi # Copy a file from container to host docker cp <container_id>:/workspace/results.csv ./results.csv # View container logs docker logs -f <container_id> Use It You now have a reproducible AI development environment. For the rest of this course: • Use docker compose up to start your dev environment and vector database together • Mount your code, models, and data as volumes so nothing is lost between rebuilds • When a lesson requires a new Python package, add it to the Dockerfile and rebuild • Share your Dockerfile with teammates. They get the exact same environment. No GPU? Remove the --gpus all flag and the NVIDIA deploy block. The container still works for CPU- based lessons. PyTorch detects the absence of CUDA and falls back to CPU automatically. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/07-docker-for-ai/code 1. Build the Dockerfile and run python -c "import torch; print(torch.__version__)" inside the container 2. Start the docker-compose stack and verify Qdrant is accessible from the AI container at http://qdrant:6333/collections 3. Add flask to the Dockerfile, rebuild, and run a simple API server on port 5000. Map the port with -p 5000:5000 4. Measure the image size with docker images. Try switching the base image from devel to runtime and compare sizes Key Terms AIENGINEERINGFROMSCRATCH.COM 35

DOCKER FOR AI Term What people say What it actually means Container “Lightweight VM” An isolated process using the host kernel, with its own filesystem and network Image layer “Cached step” Each Dockerfile instruction creates a layer. Unchanged layers are cached, so rebuilds are fast. NVIDIA Container Toolkit “GPU in Docker” A runtime hook that exposes host GPUs to containers via --gpus flag Volume mount “Shared folder” A directory on the host mapped into the container. Changes persist after the container stops. Base image “Starting point” The FROM image your Dockerfile builds on top of. Determines what is pre-installed. Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/07-docker-for-ai • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/07-docker-for-ai/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/07-docker-for-ai The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 36

Editor Setup Your editor is your co-pilot. Configure it once so it stays out of your way and starts pulling its weight. Type: Build Languages: – Prerequisites: Phase 0, Lesson 01 Time: ~20 minutes Learning Objectives • Install VS Code with essential extensions for Python, Jupyter, linting, and remote SSH • Configure format-on-save, type checking, and notebook output scrolling for AI workflows • Set up Remote SSH to edit and debug code on remote GPU machines as if they were local • Evaluate editor alternatives (Cursor, Windsurf, Neovim) and their tradeoffs for AI work The Problem You’ll spend thousands of hours inside your editor writing Python, running notebooks, debug- ging training loops, and SSH-ing into GPU boxes. A misconfigured editor turns every session into friction: no autocomplete, no type hints, no inline errors, manual formatting, and a clunky terminal workflow. The right setup takes 20 minutes. Skipping it costs you 20 minutes every day. The Concept An AI engineering editor setup needs five things: Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/08-editor-setup Interactive figure: s0-lsp-roundtrip. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00- setup-and-tooling/08-editor-setup Build It Step 1: Install VS Code VS Code is the recommended editor. It is free, runs on every OS, has first-class Jupyter notebook support, and the extension ecosystem covers everything you need for AI work. Download it from code.visualstudio.com. Verify from the terminal: code --version AIENGINEERINGFROMSCRATCH.COM 37

EDITOR SETUP If code is not found on macOS, open VS Code, press Cmd+Shift+P, type “Shell Command”, and select “Install ‘code’ command in PATH”. Step 2: Install Essential Extensions Open the integrated terminal in VS Code (Ctrl+` on every platform) and install the extensions that matter for AI work: code --install-extension ms-python.python code --install-extension ms-python.vscode-pylance code --install-extension ms-toolsai.jupyter code --install-extension eamodio.gitlens code --install-extension ms-vscode-remote.remote-ssh code --install-extension ms-python.debugpy code --install-extension ms-python.black-formatter code --install-extension charliermarsh.ruff What each one does: Extension Why Python Language support, virtual env detection, run/debug Pylance Fast type checking, autocomplete, import resolution Jupyter Run notebooks inside VS Code, variable explorer GitLens See who changed what, inline git blame Remote SSH Open a folder on a remote GPU box as if it were local Debugpy Step-through debugging for Python Black Formatter Auto-format on save, consistent style Ruff Fast linting, catches common mistakes The file code/.vscode/extensions.json in this lesson contains the full recommendations list. When you open the project folder, VS Code will prompt you to install them. Step 3: Configure Settings Copy the settings from code/.vscode/settings.json in this lesson, or apply them manually through Settings > Open Settings (JSON). The key settings for AI work: { "python.analysis.typeCheckingMode": "basic", "editor.formatOnSave": true, "editor.rulers": [88, 120], "notebook.output.scrolling": true, "files.autoSave": "afterDelay" } AIENGINEERINGFROMSCRATCH.COM 38

EDITOR SETUP Why these matter: • Type checking on basic: Catches wrong argument types before you run. Saves debug- ging time on tensor shape mismatches and wrong API parameters. • Format on save: Never think about formatting again. Black handles it. • Rulers at 88 and 120: Black wraps at 88. The 120 marker shows when docstrings and comments are getting too long. • Notebook output scrolling: Training loops print thousands of lines. Without scrolling, the output panel explodes. • Auto-save: You will forget to save. Your training script will run stale code. Auto-save prevents that. Step 4: Terminal Integration VS Code’s integrated terminal is where you run training scripts, monitor GPUs, and manage environments. Set it up properly: { "terminal.integrated.defaultProfile.osx": "zsh", "terminal.integrated.defaultProfile.linux": "bash", "terminal.integrated.fontSize": 13, "terminal.integrated.scrollback": 10000 } Useful shortcuts: Action macOS Linux/Windows Toggle terminal Ctrl+` Ctrl+` New terminal Ctrl+Shift+` Ctrl+Shift+` Split terminal Cmd+\ Ctrl+Shift+5 Split terminals are useful: one for running your script, one for monitoring GPU with nvidia- smi -l 1 or watch -n 1 nvidia-smi. Step 5: Remote Development (SSH into GPU Boxes) This is the most important extension for AI work. You will run training on remote machines (cloud VMs, lab servers, Lambda, Vast.ai). Remote SSH lets you open the remote filesystem, edit files, run terminals, and debug as if everything were local. Setup: 1. Install the Remote SSH extension (done in Step 2). 2. Press Ctrl+Shift+P (or Cmd+Shift+P), type “Remote-SSH: Connect to Host”. 3. Enter user@your-gpu-box-ip. 4. VS Code installs its server component on the remote machine automatically. For passwordless access, set up SSH keys: ssh-keygen -t ed25519 -C "your-email@example.com" ssh-copy-id user@your-gpu-box-ip Add the host to ~/.ssh/config for convenience: AIENGINEERINGFROMSCRATCH.COM 39

EDITOR SETUP Host gpu-box HostName 203.0.113.50 User ubuntu IdentityFile ~/.ssh/id_ed25519 ForwardAgent yes Now Remote-SSH: Connect to Host > gpu-box connects instantly. Alternatives Cursor cursor.com is a VS Code fork with built-in AI code generation. It uses the same extension ecosystem and settings format. If you use Cursor, everything in this lesson still applies. Im- port the same settings.json and extensions.json. Windsurf windsurf.com is another AI-first VS Code fork. Same story: same extensions, same settings format, same Remote SSH support. Vim/Neovim If you already use Vim or Neovim and are productive in it, stay there. The minimum setup for AI Python work: • pyright or pylsp for type checking (via Mason or manual install) • nvim-lspconfig for language server integration • jupyter-vim or molten-nvim for notebook-like execution • telescope.nvim for file/symbol search • none-ls.nvim with black and ruff for formatting/linting If you do not already use Vim, do not start now. The learning curve will compete with learning AI engineering. Use VS Code. Use It With this setup, your daily workflow looks like: 1. Open the project folder in VS Code (or connect via Remote SSH to a GPU box). 2. Write Python in the editor with autocomplete, type hints, and inline errors. 3. Run Jupyter notebooks inline with the Jupyter extension. 4. Use the integrated terminal for training scripts, uv pip install, and GPU monitoring. 5. Review changes with GitLens before committing. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/08-editor-setup/code 1. Install VS Code and all extensions listed in Step 2 2. Copy the settings.json from this lesson into your VS Code config 3. Open a Python file and verify that Pylance shows type hints and Black formats on save AIENGINEERINGFROMSCRATCH.COM 40

EDITOR SETUP 4. If you have access to a remote machine, set up Remote SSH and open a folder on it Key Terms Term What people say What it actually means LSP “Autocomplete engine” Language Server Protocol: a standard for editors to get type info, completions, and diagnostics from a language-specific server Pylance “The Python plugin” Microsoft’s Python language server using Pyright for type checking and IntelliSense Remote SSH “Working on the server” VS Code extension that runs a lightweight server on a remote machine and streams the UI to your local editor Format on save “Auto-prettier” The editor runs a formatter (Black, Ruff) every time you save, so code style is always consistent Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/08-editor-setup • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/08-editor-setup/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/08-editor-setup The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 41

Data Management Data is the fuel. How you manage it determines how fast you go. Type: Build Language: Python Prerequisites: Phase 0, Lesson 01 Time: ~45 minutes Learning Objectives • Load, stream, and cache datasets using the Hugging Face datasets library • Convert between CSV, JSON, Parquet, and Arrow formats and explain their tradeoffs • Create reproducible train/validation/test splits with fixed random seeds • Manage large model and dataset files using .gitignore, Git LFS, or DVC The Problem Every AI project starts with data. You need to find datasets, download them, convert be- tween formats, split them for training and evaluation, and version them so experiments are reproducible. Doing this manually every time is slow and error-prone. You need a repeatable workflow. The Concept Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/09-data-management The Hugging Face datasets library is the standard way to load data for AI work. It handles downloading, caching, format conversion, and streaming out of the box. Interactive figure: s0-data-pipeline. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00- setup-and-tooling/09-data-management Build It Step 1: Install the datasets library pip install datasets huggingface_hub Step 2: Load a dataset from datasets import load_dataset dataset = load_dataset("stanfordnlp/imdb") print(dataset) print(dataset["train"][0]) AIENGINEERINGFROMSCRATCH.COM 42

DATA MANAGEMENT This downloads the IMDB movie review dataset. After the first download, it loads from cache at ~/.cache/huggingface/datasets/. Step 3: Stream large datasets Some datasets are too large to fit on disk. Streaming loads them row by row without down- loading the full thing. dataset = load_dataset("wikimedia/wikipedia", "20231101.en", split="train", streaming=True)↪ for i, example in enumerate(dataset): print(example["title"]) if i >= 4: break Streaming gives you an IterableDataset. You process rows as they arrive. Memory usage stays constant regardless of dataset size. Step 4: Dataset formats The datasets library uses Apache Arrow under the hood. You can convert to other formats depending on what your pipeline needs. dataset = load_dataset("stanfordnlp/imdb", split="train") dataset.to_csv("imdb_train.csv") dataset.to_json("imdb_train.json") dataset.to_parquet("imdb_train.parquet") Format comparison: Format Size Read Speed Best For CSV Large Slow Human readability, spreadsheets JSON Large Slow APIs, nested data Parquet Small Fast Analytics, columnar queries Arrow Small Fastest In-memory processing (what datasets uses internally) For AI work, Parquet is the best storage format. Arrow is what you work with in memory. CSV and JSON are for interchange. Step 5: Data splits Every ML project needs three splits: • Train: The model learns from this (typically 80%) • Validation: You check progress during training (typically 10%) • Test: Final evaluation after training is done (typically 10%) Some datasets come pre-split. When they don’t, split them yourself: AIENGINEERINGFROMSCRATCH.COM 43

DATA MANAGEMENT dataset = load_dataset("stanfordnlp/imdb", split="train") split = dataset.train_test_split(test_size=0.2, seed=42) train_val = split["train"].train_test_split(test_size=0.125, seed=42) train_ds = train_val["train"] val_ds = train_val["test"] test_ds = split["test"] print(f"Train: {len(train_ds)}, Val: {len(val_ds)}, Test: {len(test_ds)}") Always set a seed for reproducibility. The same seed produces the same split every time. Step 6: Download and cache models Models are large files. The huggingface_hub library handles downloading and caching. from huggingface_hub import hf_hub_download, snapshot_download model_path = hf_hub_download( repo_id="sentence-transformers/all-MiniLM-L6-v2", filename="config.json" ) print(f"Cached at: {model_path}") model_dir = snapshot_download("sentence-transformers/all-MiniLM-L6-v2") print(f"Full model at: {model_dir}") Models cache to ~/.cache/huggingface/hub/. Once downloaded, they load instantly on sub- sequent runs. Step 7: Handle large files Model weights and large datasets should not go into git. Three options: Option A: .gitignore (simplest) *.bin *.safetensors *.pt *.onnx data/*.parquet data/*.csv models/ Option B: Git LFS (track large files in git) git lfs install git lfs track "*.bin" git lfs track "*.safetensors" git add .gitattributes Git LFS stores pointers in your repo and the actual files on a separate server. GitHub gives you 1 GB free. Option C: DVC (data version control) pip install dvc dvc init AIENGINEERINGFROMSCRATCH.COM 44

DATA MANAGEMENT dvc add data/training_set.parquet git add data/training_set.parquet.dvc data/.gitignore git commit -m "Track training data with DVC" DVC creates small .dvc files that point to your data. The data itself lives in S3, GCS, or another remote storage backend. Approach Complexity Best For .gitignore Low Personal projects, downloaded data you can re-fetch Git LFS Medium Teams sharing model weights via git DVC High Reproducible experiments, large datasets, teams For this course, .gitignore is enough. Use DVC when you need to reproduce exact experi- ments across machines. Step 8: Storage patterns Local storage works for datasets under ~10 GB. The HF cache handles this automatically. Cloud storage is for anything larger or shared across machines: import os local_path = os.path.expanduser("~/.cache/huggingface/datasets/") # s3_path = "s3://my-bucket/datasets/" # gcs_path = "gs://my-bucket/datasets/" DVC integrates with S3 and GCS directly: dvc remote add -d myremote s3://my-bucket/dvc-store dvc push For this course, local storage is sufficient. Cloud storage becomes relevant when you fine-tune on remote GPU instances. Datasets Used in This Course Dataset Lessons Size What It Teaches IMDB Tokenization, classification 84 MB Text classification basics WikiText Language modeling 181 MB Next-token prediction SQuAD QA systems 35 MB Question answering, spans Common Crawl (subset) Embeddings Varies Large-scale text processing MNIST Vision basics 21 MB Image classification fundamentals COCO (subset) Multimodal Varies Image-text pairs AIENGINEERINGFROMSCRATCH.COM 45

DATA MANAGEMENT You do not need to download all of these now. Each lesson specifies what it needs. Use It Run the utility script to verify everything works: python code/data_utils.py This downloads a small dataset, converts it, splits it, and prints a summary. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/09-data-management Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/00-setup-and-tooling/09-data-management/code 1. Load the glue dataset with the mrpc config and inspect the first 5 examples 2. Stream the c4 dataset and count how many examples you can process in 10 seconds 3. Convert a dataset to Parquet and compare the file size to CSV 4. Create a 70/15/15 train/val/test split with a fixed seed and verify the sizes Key Terms Term What people say What it actually means Dataset split “Training data” A named subset (train/val/test) used at different stages of the ML lifecycle Streaming “Load it lazily” Processing data row by row from a remote source without downloading the full dataset Parquet “Compressed CSV” A columnar file format optimized for analytical queries and storage efficiency Arrow “Fast dataframe” An in-memory columnar format used internally by the datasets library for zero-copy reads Git LFS “Git for big files” An extension that stores large files outside the git repo while keeping pointers in version control DVC “Git for data” A version control system for datasets and models that integrates with cloud storage Cache “Already downloaded” A local copy of previously fetched data, stored at ~/.cache/huggingface/ by default Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/09-data-management • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/09-data-management/code AIENGINEERINGFROMSCRATCH.COM 46

DATA MANAGEMENT • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/09-data-management The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 47

Terminal & Shell The terminal is where AI engineers live. Get comfortable here. Type: Learn Languages: – Prerequisites: Phase 0, Lesson 01 Time: ~35 minutes Learning Objectives • Use piping, redirects, and grep to filter and process training logs from the command line • Create persistent tmux sessions with multiple panes for concurrent training and GPU monitoring • Monitor system and GPU resources with htop, nvtop, and nvidia-smi • Transfer files between local and remote machines using SSH, scp, and rsync The Problem You will spend more time in the terminal than in any editor. Training runs, GPU monitoring, log tailing, remote SSH sessions, environment management. Every AI workflow touches the shell. If you’re slow here, you’re slow everywhere. This lesson covers the terminal skills that matter for AI work. No history of Unix. No deep-dive into Bash scripting. Just what you need. The Concept Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/10-terminal-and-shell Three things running at once. One terminal. You can detach, go home, SSH back in, and reattach. The training keeps running. Interactive figure: s0-shell-pipeline. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00- setup-and-tooling/10-terminal-and-shell Build It Step 1: Know your shell Check which shell you’re running: echo $SHELL Most systems use bash or zsh. Both work fine. The commands in this course work in either. Key things to know: AIENGINEERINGFROMSCRATCH.COM 48

TERMINAL & SHELL # Move around cd ~/projects/ai-engineering-from-scratch pwd ls -la # History search (most useful shortcut you'll learn) # Ctrl+R then type part of a previous command # Press Ctrl+R again to cycle through matches # Clear terminal clear # or Ctrl+L # Cancel a running command # Ctrl+C # Suspend a running command (resume with fg) # Ctrl+Z Step 2: Piping and redirects Piping connects commands together. This is how you process logs, filter output, and chain tools. You will use this constantly. # Count how many times "loss" appears in a log cat train.log | grep "loss" | wc -l # Extract just the loss values from training output grep "loss:" train.log | awk '{print $NF}' > losses.txt # Watch a log file update in real time, filtering for errors tail -f train.log | grep --line-buffered "ERROR" # Sort experiments by final accuracy grep "final_accuracy" results/*.log | sort -t= -k2 -n -r # Redirect stdout and stderr to separate files python train.py > output.log 2> errors.log # Redirect both to the same file python train.py > train_full.log 2>&1 The three redirects you need: Symbol What it does > Write stdout to file (overwrite) >> Append stdout to file 2> Write stderr to file 2>&1 Send stderr to same place as stdout \| Send stdout of one command as stdin to the next Step 3: Background processes Training runs take hours. You don’t want to keep your terminal open the whole time. AIENGINEERINGFROMSCRATCH.COM 49

TERMINAL & SHELL # Run in background (output still goes to terminal) python train.py & # Run in background, immune to hangup (closing terminal won't kill it) nohup python train.py > train.log 2>&1 & # Check what's running in background jobs ps aux | grep train.py # Bring a background job to foreground fg %1 # Kill a background process kill %1 # or find its PID and kill that kill $(pgrep -f "train.py") The difference between &, nohup, and screen/tmux: Method Survives terminal close? Can reattach? command & No No nohup command & Yes No (check log file) screen / tmux Yes Yes For anything longer than a few minutes, use tmux. Step 4: tmux tmux lets you create persistent terminal sessions with multiple panes. This is the single most useful tool for managing training runs. # Install # macOS brew install tmux # Ubuntu sudo apt install tmux # Start a named session tmux new -s training # Split horizontally # Ctrl+B then " # Split vertically # Ctrl+B then % # Navigate between panes # Ctrl+B then arrow keys # Detach (session keeps running) # Ctrl+B then d AIENGINEERINGFROMSCRATCH.COM 50

TERMINAL & SHELL # Reattach tmux attach -t training # List sessions tmux ls # Kill a session tmux kill-session -t training A typical AI workflow session: tmux new -s train # Pane 1: start training python train.py --epochs 100 --lr 1e-4 # Ctrl+B, " to split, then run GPU monitor watch -n1 nvidia-smi # Ctrl+B, % to split vertically, tail the logs tail -f logs/experiment.log # Now detach with Ctrl+B, d # SSH out, go get coffee, come back # tmux attach -t train Step 5: Monitoring with htop and nvtop # System processes (better than top) htop # GPU processes (if you have NVIDIA GPU) # Install: sudo apt install nvtop (Ubuntu) or brew install nvtop (macOS) nvtop # Quick GPU check without nvtop nvidia-smi # Watch GPU usage update every second watch -n1 nvidia-smi # See which processes are using the GPU nvidia-smi --query-compute-apps=pid,name,used_memory --format=csv htop keybindings you’ll use: - F6 or > to sort by column (sort by memory to find memory leaks) - F5 to toggle tree view (see child processes) - F9 to kill a process - / to search for a process name Step 6: SSH for remote GPU boxes When you rent a cloud GPU (Lambda, RunPod, Vast.ai), you connect via SSH. # Basic connection ssh user@gpu-box-ip AIENGINEERINGFROMSCRATCH.COM 51

TERMINAL & SHELL # With a specific key ssh -i ~/.ssh/my_gpu_key user@gpu-box-ip # Copy files to remote scp model.pt user@gpu-box-ip:~/models/ # Copy files from remote scp user@gpu-box-ip:~/results/metrics.json ./ # Sync a whole directory (faster for many files) rsync -avz ./data/ user@gpu-box-ip:~/data/ # Port forward (access remote Jupyter/TensorBoard locally) ssh -L 8888:localhost:8888 user@gpu-box-ip # Now open localhost:8888 in your browser # SSH config for convenience # Add to ~/.ssh/config: # Host gpu # HostName 192.168.1.100 # User ubuntu # IdentityFile ~/.ssh/gpu_key # # Then just: # ssh gpu Step 7: Useful aliases for AI work Add these to your ~/.bashrc or ~/.zshrc: source phases/00-setup-and-tooling/10-terminal-and-shell/code/shell_aliases.sh Or copy the ones you want. The key aliases: # GPU status at a glance alias gpu='nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used,memory.t ⌋ otal,temperature.gpu --format=csv,noheader' ↪ ↪ # Kill all Python training processes alias killtraining='pkill -f "python.*train"' # Quick virtual environment activate alias ae='source .venv/bin/activate' # Watch training loss alias watchloss='tail -f logs/*.log | grep --line-buffered "loss"' See code/shell_aliases.sh for the full set. Step 8: Common AI terminal patterns These come up repeatedly in practice: AIENGINEERINGFROMSCRATCH.COM 52

TERMINAL & SHELL # Run training, log everything, notify when done python train.py 2>&1 | tee train.log; echo "DONE" | mail -s "Training complete" you@email.com↪ # Compare two experiment logs side by side diff <(grep "accuracy" exp1.log) <(grep "accuracy" exp2.log) # Find the largest model files (clean up disk space) find . -name "*.pt" -o -name "*.safetensors" | xargs du -h | sort -rh | head -20 # Download a model from Hugging Face wget https://huggingface.co/model/resolve/main/model.safetensors # Untar a dataset tar xzf dataset.tar.gz -C ./data/ # Count lines in all Python files (see how big your project is) find . -name "*.py" | xargs wc -l | tail -1 # Check disk space (training data fills disks fast) df -h du -sh ./data/* # Environment variable check before training env | grep -i cuda env | grep -i torch Use It Here’s when each tool comes into play during this course: Tool When you use it tmux Every training run (Phases 3+) tail -f + grep Monitoring training logs nohup / & Quick background tasks htop / nvtop Debugging slow training, OOM errors SSH + rsync Working on cloud GPUs Piping + redirects Processing experiment results Aliases Saving time on repetitive commands Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/00-setup-and-tooling/10-terminal-and-shell/code 1. Install tmux, create a session with three panes, and run htop in one, watch -n1 date in another, and a Python script in the third. Detach and reattach. 2. Add the aliases from code/shell_aliases.sh to your shell config and reload with source ~/.zshrc (or ~/.bashrc). 3. Create a fake training log with for i in $(seq 1 100); do echo "epoch $i loss: $(echo "scale=4; 1/$i" | bc)"; sleep 0.1; done > fake_train.log and then use grep, tail, AIENGINEERINGFROMSCRATCH.COM 53

TERMINAL & SHELL and awk to extract just the loss values. 4. Set up an SSH config entry for a server you have access to (or use localhost to practice the syntax). Key Terms Term What people say What it actually means Shell “The terminal” The program that interprets your commands (bash, zsh, fish) tmux “Terminal multiplexer” A program that lets you run multiple terminal sessions inside one window, and detach/reattach Pipe “The bar thing” The \| operator that sends one command’s output as input to another PID “Process ID” A unique number assigned to every running process, used to monitor or kill it nohup “No hangup” Runs a command immune to the hangup signal, so closing the terminal won’t kill it SSH “Connecting to the server” Secure Shell, an encrypted protocol for running commands on a remote machine Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/10-terminal-and-shell • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/10-terminal-and-shell/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/10-terminal-and-shell The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 54

Linux for AI Most AI runs on Linux. You need to know enough to not be stuck. Type: Learn Languages: – Prerequisites: Phase 0, Lesson 01 Time: ~30 minutes Learning Objectives • Navigate the Linux file system and perform essential file operations from the command line • Manage file permissions with chmod and chown to resolve “Permission denied” errors • Install system packages with apt and set up a fresh GPU box for AI work • Identify macOS-to-Linux differences that commonly trip up developers working on re- mote machines The Problem You develop on macOS or Windows. But the moment you SSH into a cloud GPU box, rent a Lambda instance, or spin up an EC2 machine, you land in Ubuntu. The terminal is your only interface. There is no Finder, no Explorer, no GUI. If you can’t navigate the file system, install packages, and manage processes from the command line, you’re stuck paying for idle GPU hours while googling “how to unzip a file in Linux.” This is a survival guide. It covers exactly what you need to operate on a remote Linux machine for AI work. Nothing more. File System Layout Linux organizes everything under a single root /. There is no C:\ or /Volumes. The directories you’ll actually touch: Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/11-linux-for-ai Your home directory is ~ or /home/your-username. Almost everything you do happens here. Essential Commands These are the 15 commands that cover 95% of what you’ll do on a remote GPU box. Moving Around pwd # Where am I? ls # What's here? ls -la # What's here, including hidden files with details? AIENGINEERINGFROMSCRATCH.COM 55

LINUX FOR AI cd /path/to/dir # Go there cd ~ # Go home cd .. # Go up one level Files and Directories mkdir my-project # Create a directory mkdir -p a/b/c # Create nested directories in one shot cp file.txt backup.txt # Copy a file cp -r src/ src-backup/ # Copy a directory (recursive) mv old.txt new.txt # Rename a file mv file.txt /tmp/ # Move a file rm file.txt # Delete a file (no trash, it's gone) rm -rf my-dir/ # Delete a directory and everything inside rm -rf is permanent. There is no undo. Double-check the path before hitting enter. Reading Files cat file.txt # Print entire file head -20 file.txt # First 20 lines tail -20 file.txt # Last 20 lines tail -f log.txt # Follow a log file in real time (Ctrl+C to stop) less file.txt # Scroll through a file (q to quit) Searching grep "error" training.log # Find lines containing "error" grep -r "learning_rate" . # Search all files in current directory grep -i "cuda" config.yaml # Case-insensitive search find . -name "*.py" # Find all Python files under current dir find . -name "*.ckpt" -size +1G # Find checkpoint files larger than 1GB Permissions Every file in Linux has an owner and permission bits. You’ll run into this when scripts won’t execute or you can’t write to a directory. ls -l train.py # -rwxr-xr-- 1 user group 2048 Mar 19 10:00 train.py # ^^^ owner permissions: read, write, execute # ^^^ group permissions: read, execute # ^^ everyone else: read only Common fixes: chmod +x train.sh # Make a script executable chmod 755 deploy.sh # Owner: full, others: read+execute chmod 644 config.yaml # Owner: read+write, others: read only AIENGINEERINGFROMSCRATCH.COM 56

LINUX FOR AI chown user:group file.txt # Change who owns a file (needs sudo) When something says “Permission denied,” it’s almost always a permissions issue. chmod +x or sudo will fix most cases. Package Management (apt) Ubuntu uses apt. This is how you install system-level software. sudo apt update # Refresh the package list (always do this first) sudo apt install -y htop # Install a package (-y skips confirmation) sudo apt install -y build-essential # C compiler, make, etc. Needed by many Python packages↪ sudo apt install -y tmux # Terminal multiplexer (keep sessions alive after disconnect)↪ apt list --installed # What's installed? sudo apt remove htop # Uninstall Common packages you’ll install on a fresh GPU box: sudo apt update && sudo apt install -y \ build-essential \ git \ curl \ wget \ tmux \ htop \ unzip \ python3-venv Users and sudo You’re usually logged in as a regular user. Some operations need root (admin) access. whoami # What user am I? sudo command # Run a single command as root sudo su # Become root (exit to go back, use sparingly) On cloud GPU instances, you’re typically the only user and already have sudo access. Don’t run everything as root. Use sudo only when needed. Processes and systemd When your training hangs, or you need to check what’s running: htop # Interactive process viewer (q to quit) ps aux | grep python # Find running Python processes kill 12345 # Gracefully stop process with PID 12345 kill -9 12345 # Force kill (use when graceful doesn't work) nvidia-smi # GPU processes and memory usage systemd manages services (background daemons). You’ll use it if you run inference servers: AIENGINEERINGFROMSCRATCH.COM 57

LINUX FOR AI sudo systemctl start nginx # Start a service sudo systemctl stop nginx # Stop it sudo systemctl restart nginx # Restart it sudo systemctl status nginx # Check if it's running sudo systemctl enable nginx # Start automatically on boot Disk Space GPU boxes often have limited disk space. Models and datasets fill it fast. df -h # Disk usage for all mounted drives df -h /home # Disk usage for /home specifically du -sh * # Size of each item in current directory du -sh ~/.cache # Size of your cache (pip, huggingface models land here) du -sh /data/checkpoints/ # Check how big your checkpoints are # Find the biggest space hogs du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20 Common space savers: # Clear pip cache pip cache purge # Clear apt cache sudo apt clean # Remove old checkpoints you don't need rm -rf checkpoints/epoch_01/ checkpoints/epoch_02/ Networking You’ll download models, transfer files, and hit APIs from the command line. # Download files wget https://example.com/model.bin # Download a file curl -O https://example.com/data.tar.gz # Same thing with curl curl -s https://api.example.com/health | python3 -m json.tool # Hit an API, pretty-print JSON↪ # Transfer files between machines scp model.bin user@remote:/data/ # Copy file to remote machine scp user@remote:/data/results.csv . # Copy file from remote to local scp -r user@remote:/data/checkpoints/ ./local-dir/ # Copy directory # Sync directories (faster than scp for large transfers, resumes on failure) rsync -avz --progress ./data/ user@remote:/data/ rsync -avz --progress user@remote:/results/ ./results/ Use rsync over scp for anything large. It only transfers changed bytes and handles interrupted connections. AIENGINEERINGFROMSCRATCH.COM 58

LINUX FOR AI tmux: Keep Sessions Alive When you SSH into a remote box, closing your laptop kills your training run. tmux prevents this. tmux new -s train # Start a new session named "train" # ... start your training, then: # Ctrl+B, then D # Detach (training keeps running) tmux ls # List sessions tmux attach -t train # Reattach to session # Inside tmux: # Ctrl+B, then % # Split pane vertically # Ctrl+B, then " # Split pane horizontally # Ctrl+B, then arrow keys # Switch between panes Always run long training jobs inside tmux. Always. WSL2 for Windows Users If you’re on Windows, WSL2 gives you a real Linux environment without dual-booting. # In PowerShell (admin) wsl --install -d Ubuntu-24.04 # After restart, open Ubuntu from Start menu sudo apt update && sudo apt upgrade -y WSL2 runs a real Linux kernel. Everything in this lesson works inside it. Your Windows files are at /mnt/c/Users/YourName/ from inside WSL. GPU passthrough works with NVIDIA drivers installed on the Windows side. Install the Win- dows NVIDIA driver (not the Linux one), and CUDA will be available inside WSL2. Gotchas: macOS to Linux Things that will trip you up if you’re coming from macOS: macOS Linux Notes brew install sudo apt install Different package names sometimes. brew install htop vs sudo apt install htop works the same, but brew install readline vs sudo apt install libreadline-dev does not. open file.txt xdg-open file.txt But you won’t have a GUI on a remote box. Use cat or less. pbcopy / pbpaste Not available Pipe to/from clipboard doesn’t exist over SSH. AIENGINEERINGFROMSCRATCH.COM 59

LINUX FOR AI macOS Linux Notes ~/.zshrc ~/.bashrc macOS defaults to zsh. Most Linux servers use bash. /opt/homebrew/ /usr/bin/, /usr/local/bin/ Binaries live in different places. sed -i '' 's/a/b/' file sed -i 's/a/b/' file macOS sed needs an empty string after -i. Linux does not. Case-insensitive filesystem Case-sensitive filesystem Model.py and model.py are two different files on Linux. Line endings \n Line endings \n Same. But Windows uses \r\n, which breaks bash scripts. Run dos2unix to fix. Quick Reference Card Navigation: pwd, ls, cd, find Files: cp, mv, rm, mkdir, cat, head, tail, less Search: grep, find Permissions: chmod, chown, sudo Packages: apt update, apt install Processes: htop, ps, kill, nvidia-smi Services: systemctl start/stop/restart/status Disk: df -h, du -sh Network: curl, wget, scp, rsync Sessions: tmux new/attach/detach Interactive figure: s0-process-fork. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/11-linux-for-ai Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/11-linux-for-ai/code 1. SSH into any Linux machine (or open WSL2) and navigate to your home directory. Create a project folder, create three empty files inside it with touch, then list them with ls -la. 2. Install htop with apt, run it, and identify which process is using the most memory. 3. Start a tmux session, run sleep 300 inside it, detach, list sessions, and reattach. 4. Use df -h to check available disk space, then use du -sh ~/.cache/* to find what’s taking up space in your cache. 5. Transfer a file from your local machine to a remote one using scp, then do the same transfer with rsync and compare the experience. Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/11-linux-for-ai • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/11-linux-for-ai/code AIENGINEERINGFROMSCRATCH.COM 60

LINUX FOR AI • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/11-linux-for-ai The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 61

Debugging and Profiling The worst AI bugs don’t crash. They train silently on garbage and report a beautiful loss curve. Type: Build Language: Python Prerequisites: Lesson 1 (Dev Environment), basic PyTorch familiarity Time: ~60 minutes Learning Objectives • Use conditional breakpoint() and debug_print to inspect tensor shapes, dtypes, and NaN values mid-training • Profile training loops with cProfile, line_profiler, and tracemalloc to find bottlenecks • Detect common AI bugs: shape mismatches, NaN loss, data leakage, and wrong-device tensors • Set up TensorBoard to visualize loss curves, weight histograms, and gradient distribu- tions The Problem AI code fails differently than regular code. A web app crashes with a stack trace. A miscon- figured training loop runs for 8 hours, burns $200 in GPU time, and produces a model that predicts the mean of every input. The code never errored. The bug was a tensor on the wrong device, a forgotten .detach(), or labels leaking into features. You need debugging tools that catch these silent failures before they waste your time and compute. The Concept AI debugging operates at three levels: Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/12-debugging-and-profiling Most people jump straight to level 3 (staring at TensorBoard). But 80% of AI bugs live at levels 1 and 2. Interactive figure: s0-flame-hot. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/00-setup-and- tooling/12-debugging-and-profiling AIENGINEERINGFROMSCRATCH.COM 62

DEBUGGING AND PROFILING Build It Part 1: Print Debugging (Yes, It Works) Print debugging gets dismissed. It shouldn’t. For tensor code, a targeted print statement beats stepping through a debugger because you need to see shapes, dtypes, and value ranges all at once. def debug_print(name, tensor): print(f"{name}: shape={tensor.shape}, dtype={tensor.dtype}, " f"device={tensor.device}, " f"min={tensor.min().item():.4f}, max={tensor.max().item():.4f}, " f"mean={tensor.mean().item():.4f}, " f"has_nan={tensor.isnan().any().item()}") Call this after every suspicious operation. When the bug is found, remove the prints. Simple. Part 2: Python Debugger (pdb and breakpoint) The built-in debugger is underrated for AI work. Drop breakpoint() into your training loop and inspect tensors interactively. def training_step(model, batch, criterion, optimizer): inputs, labels = batch outputs = model(inputs) loss = criterion(outputs, labels) if loss.item() > 100 or torch.isnan(loss): breakpoint() loss.backward() optimizer.step() When the debugger drops you in, useful commands: • p outputs.shape to check shapes • p loss.item() to see the loss value • p torch.isnan(outputs).sum() to count NaNs • p model.fc1.weight.grad to check gradients • c to continue, q to quit This is conditional debugging. You only stop when something looks wrong. For a 10,000-step training run, that matters. Part 3: Python Logging Replace print statements with logging when your debugging goes beyond a quick check. import logging logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s", handlers=[ logging.FileHandler("training.log"), logging.StreamHandler() ] AIENGINEERINGFROMSCRATCH.COM 63

DEBUGGING AND PROFILING ) logger = logging.getLogger(__name__) logger.info("Starting training: lr=%.4f, batch_size=%d", lr, batch_size) logger.warning("Loss spike detected: %.4f at step %d", loss.item(), step) logger.error("NaN loss at step %d, stopping", step) Logging gives you timestamps, severity levels, and file output. When a training run fails at 3 AM, you want a log file, not terminal output that scrolled off screen. Part 4: Timing Code Sections Knowing where time goes is the first step to optimization. import time class Timer: def __init__(self, name=""): self.name = name def __enter__(self): self.start = time.perf_counter() return self def __exit__(self, *args): elapsed = time.perf_counter() - self.start print(f"[{self.name}] {elapsed:.4f}s") with Timer("data loading"): batch = next(dataloader_iter) with Timer("forward pass"): outputs = model(batch) with Timer("backward pass"): loss.backward() Common finding: data loading takes 60% of training time. The fix is num_workers > 0 in your DataLoader, not a faster GPU. Part 5: cProfile and line_profiler When you need more than manual timers: python -m cProfile -s cumtime train.py This shows every function call sorted by cumulative time. For line-by-line profiling: pip install line_profiler @profile def train_step(model, data, target): output = model(data) loss = F.cross_entropy(output, target) loss.backward() return loss AIENGINEERINGFROMSCRATCH.COM 64

DEBUGGING AND PROFILING # Run with: kernprof -l -v train.py Part 6: Memory Profiling CPU Memory with tracemalloc import tracemalloc tracemalloc.start() # your code here model = build_model() data = load_dataset() snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics("lineno") for stat in top_stats[:10]: print(stat) CPU Memory with memory_profiler pip install memory_profiler from memory_profiler import profile @profile def load_data(): raw = read_csv("data.csv") # watch memory jump here processed = preprocess(raw) # and here return processed Run with python -m memory_profiler your_script.py to see line-by-line memory usage. GPU Memory with PyTorch import torch if torch.cuda.is_available(): print(torch.cuda.memory_summary()) print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB") print(f"Cached: {torch.cuda.memory_reserved() / 1e9:.2f} GB") When you hit OOM (Out of Memory): 1. Reduce batch size (first thing to try, always) 2. Use torch.cuda.empty_cache() to free cached memory 3. Use del tensor followed by torch.cuda.empty_cache() for large intermediates 4. Use mixed precision (torch.cuda.amp) to halve memory usage 5. Use gradient checkpointing for very deep models AIENGINEERINGFROMSCRATCH.COM 65

DEBUGGING AND PROFILING Part 7: Common AI Bugs and How to Catch Them Shape Mismatch The most frequent bug. A tensor has shape [batch, features] when the model expects [bat ch, channels, height, width]. def check_shapes(model, sample_input): print(f"Input: {sample_input.shape}") hooks = [] def make_hook(name): def hook(module, inp, out): in_shape = inp[0].shape if isinstance(inp, tuple) else inp.shape out_shape = out.shape if hasattr(out, "shape") else type(out) print(f" {name}: {in_shape} -> {out_shape}") return hook for name, module in model.named_modules(): hooks.append(module.register_forward_hook(make_hook(name))) with torch.no_grad(): model(sample_input) for h in hooks: h.remove() Run this once with a sample batch. It maps every shape transformation in your model. NaN Loss NaN loss means something exploded. Common causes: • Learning rate too high • Division by zero in custom loss • Log of zero or negative number • Exploding gradients in RNNs def detect_nan(model, loss, step): if torch.isnan(loss): print(f"NaN loss at step {step}") for name, param in model.named_parameters(): if param.grad is not None: if torch.isnan(param.grad).any(): print(f" NaN gradient in {name}") if torch.isinf(param.grad).any(): print(f" Inf gradient in {name}") return True return False Data Leakage Your model gets 99% accuracy on the test set. Sounds great. It’s a bug. def check_data_leakage(train_set, test_set, id_column="id"): train_ids = set(train_set[id_column].tolist()) test_ids = set(test_set[id_column].tolist()) AIENGINEERINGFROMSCRATCH.COM 66

DEBUGGING AND PROFILING overlap = train_ids & test_ids if overlap: print(f"DATA LEAKAGE: {len(overlap)} samples in both train and test") return True return False Also check for temporal leakage: using future data to predict the past. Sort by timestamp before splitting. Wrong Device Tensors on different devices (CPU vs GPU) cause runtime errors. But sometimes a tensor silently stays on CPU while everything else is on GPU, and training just runs slowly. def check_devices(model, *tensors): model_device = next(model.parameters()).device print(f"Model device: {model_device}") for i, t in enumerate(tensors): if t.device != model_device: print(f" WARNING: tensor {i} on {t.device}, model on {model_device}") Part 8: TensorBoard Basics TensorBoard shows you what’s happening inside training over time. pip install tensorboard from torch.utils.tensorboard import SummaryWriter writer = SummaryWriter("runs/experiment_1") for step in range(num_steps): loss = train_step(model, batch) writer.add_scalar("loss/train", loss.item(), step) writer.add_scalar("lr", optimizer.param_groups[0]["lr"], step) if step % 100 == 0: for name, param in model.named_parameters(): writer.add_histogram(f"weights/{name}", param, step) if param.grad is not None: writer.add_histogram(f"grads/{name}", param.grad, step) writer.close() Launch it: tensorboard --logdir=runs What to look for: • Loss not decreasing: Learning rate too low, or model architecture issue • Loss oscillating wildly: Learning rate too high • Loss goes to NaN: Numerical instability (see NaN section above) • Train loss decreasing, val loss increasing: Overfitting • Weight histograms collapsing to zero: Vanishing gradients • Gradient histograms exploding: Need gradient clipping AIENGINEERINGFROMSCRATCH.COM 67

DEBUGGING AND PROFILING Part 9: VS Code Debugger For interactive debugging, configure VS Code with a launch.json: { "version": "0.2.0", "configurations": [ { "name": "Debug Training", "type": "debugpy", "request": "launch", "program": "${file}", "console": "integratedTerminal", "justMyCode": false } ] } Set breakpoints by clicking the gutter. Use the Variables pane to inspect tensor properties. The Debug Console lets you run arbitrary Python expressions mid-execution. Useful for stepping through data preprocessing pipelines where you want to see each trans- formation. Use It Here’s the debugging workflow that catches most AI bugs: 1. Before training: Run check_shapes with a sample batch. Verify input and output di- mensions match expectations. 2. First 10 steps: Use debug_print on loss, outputs, and gradients. Confirm nothing is NaN and values are in reasonable ranges. 3. During training: Log loss, learning rate, and gradient norms. Use TensorBoard for visualization. 4. When something breaks: Drop breakpoint() at the failure point. Inspect tensors interactively. 5. For performance: Time your data loading vs forward vs backward pass. Profile memory if you’re near OOM. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/00-setup-and-tooling/12-debugging-and- profiling Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engine ering-from-scratch/tree/main/phases/00-setup-and-tooling/12-debugging-and-profiling/code 1. Run debug_tools.py and read through each section’s output. Modify the dummy model to introduce a NaN (hint: divide by zero in the forward pass) and watch the detector catch it. 2. Profile a training loop with cProfile and identify the slowest function. 3. Use tracemalloc to find which line in your data loading pipeline allocates the most mem- ory. AIENGINEERINGFROMSCRATCH.COM 68

DEBUGGING AND PROFILING 4. Set up TensorBoard for a simple training run and identify whether the model is overfit- ting. 5. Use breakpoint() inside a training loop. Practice inspecting tensor shapes, devices, and gradient values from the debugger prompt. Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/00-setup-and-tooling/12-debugging-and-profiling • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/00-setup-and-tooling/12-debugging-and-profiling/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/00-setup-and-tooling/12-debugging-and-profiling The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 69

Part II — Math Foundations Course phase 01. Live edition with animated figures and quizzes: https://aiengineeringfroms cratch.com/catalog.html AIENGINEERINGFROMSCRATCH.COM 70

Linear Algebra Intuition Every AI model is just matrix math wearing a fancy hat. Type: Learn Languages: Python, Julia Prerequisites: Phase 0 Time: ~60 minutes Learning Objectives • Implement vector and matrix operations (addition, dot product, matrix multiply) from scratch in Python • Explain geometrically what the dot product, projection, and Gram-Schmidt process do • Determine linear independence, rank, and basis of a set of vectors using row reduction • Connect linear algebra concepts to their AI applications: embeddings, attention scores, and LoRA The Problem Open any ML paper. Within the first page, you’ll see vectors, matrices, dot products, and transformations. Without linear algebra intuition, these are just symbols. With it, you can see what a neural network is actually doing – moving points around in space. You don’t need to be a mathematician. You need to see what these operations mean geomet- rically, then code them yourself. The Concept Vectors Are Points (and Directions) A vector is just a list of numbers. But those numbers mean something – they’re coordinates in space. 2D vector [3, 2]: x y Point 3 2 The vector points from origin (0,0) to (3, 2) on the plane The vector has magnitude sqrt(3^2 + 2^2) = sqrt(13) and points up and to the right. In AI, vectors represent everything: - A word → a vector of 768 numbers (its “meaning” in embedding space) - An image → a vector of millions of pixel values - A user → a vector of preferences AIENGINEERINGFROMSCRATCH.COM 71

LINEAR ALGEBRA INTUITION Matrices Are Transformations A matrix transforms one vector into another. It can rotate, scale, stretch, or project. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/01-linear-algebra-intuition In AI, matrices ARE the model: - Neural network weights → matrices that transform input into output - Attention scores → matrices that decide what to focus on - Embeddings → matrices that map words to vectors The Dot Product Measures Similarity The dot product of two vectors tells you how similar they are. a · b = a₁×b₁ + a₂×b₂ + ... + aₙ×bₙ Same direction: a · b > 0 (similar) Perpendicular: a · b = 0 (unrelated) Opposite direction: a · b < 0 (dissimilar) This is literally how search engines, recommendation systems, and RAG work – find vectors with high dot products. Linear Independence Vectors are linearly independent if no vector in the set can be written as a combination of the others. If v1, v2, v3 are independent, they span a 3D space. If one is a combination of the others, they only span a plane. Why it matters for AI: your feature matrix should have linearly independent columns. If two features are perfectly correlated (linearly dependent), the model cannot distinguish their ef- fects. This causes multicollinearity in regression – the weight matrix becomes unstable, and small input changes produce wild output swings. Concrete example: v1 = [1, 0, 0] v2 = [0, 1, 0] v3 = [2, 1, 0] # v3 = 2*v1 + v2 v1 and v2 are independent – neither is a scalar multiple or combination of the other. But v3 = 2*v1 + v2, so {v1, v2, v3} is a dependent set. These three vectors all lie in the xy-plane. No matter how you combine them, you cannot reach [0, 0, 1]. You have three vectors but only two dimensions of freedom. In a dataset: if feature_3 = 2*feature_1 + feature_2, adding feature_3 gives the model zero new information. Worse, it makes the normal equations singular – there is no unique solution for the weights. Basis and Rank A basis is a minimal set of linearly independent vectors that span the entire space. The number of basis vectors is the dimension of the space. The standard basis for 3D space is {[1,0,0], [0,1,0], [0,0,1]}. But any three independent vectors in 3D form a valid basis. The choice of basis is a choice of coordinate system. AIENGINEERINGFROMSCRATCH.COM 72

LINEAR ALGEBRA INTUITION Rank of a matrix = number of linearly independent columns = number of linearly independent rows. If rank < min(rows, cols), the matrix is rank-deficient. This means: - The system has infinitely many solutions (or none) - Information is lost in the transformation - The matrix cannot be inverted Situation Rank What it means for ML Full rank (rank = min(m, n)) Maximum possible Unique least-squares solution exists. Model is well-conditioned. Rank deficient (rank < min(m, n)) Below maximum Features are redundant. Infinitely many weight solutions. Regularization needed. Rank 1 1 Every column is a scaled copy of one vector. All data lies on a line. Near rank-deficient (small singular values) Numerically low Matrix is ill-conditioned. Tiny input noise causes large output changes. Use SVD truncation or ridge regression. Projection Projecting vector a onto vector b gives the component of a in the direction of b: proj_b(a) = (a dot b / b dot b) * b The residual (a - proj_b(a)) is perpendicular to b. This orthogonal decomposition is the foun- dation of least-squares fitting. Projection is everywhere in ML: - Linear regression minimizes the distance from observations to the column space – the solution IS a projection - PCA projects data onto the directions of maximum variance - Attention in transformers computes projections of queries onto keys Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/01-linear-algebra-intuition Example: a = [3, 4], b = [1, 0] proj_b(a) = (31 + 40) / (11 + 00) * [1, 0] = 3 * [1, 0] = [3, 0] The projection drops the y-component. This is dimensionality reduction in its simplest form – throw away the directions you don’t care about. Gram-Schmidt Process Converting any set of independent vectors into an orthonormal basis. Orthonormal means every vector has length 1 and every pair is perpendicular. The algorithm: 1. Take the first vector, normalize it 2. Take the second vector, subtract its projection onto the first, normalize 3. Take the third vector, subtract its projections onto all previous vectors, normalize 4. Repeat for remaining vectors Input: v1, v2, v3, ... (linearly independent) u1 = v1 / |v1| w2 = v2 - (v2 dot u1) * u1 u2 = w2 / |w2| w3 = v3 - (v3 dot u1) * u1 - (v3 dot u2) * u2 u3 = w3 / |w3| AIENGINEERINGFROMSCRATCH.COM 73

LINEAR ALGEBRA INTUITION Output: u1, u2, u3, ... (orthonormal basis) This is how QR decomposition works internally. Q is the orthonormal basis, R captures the projection coefficients. QR decomposition is used in: - Solving linear systems (more stable than Gaussian elimination) - Computing eigenvalues (QR algorithm) - Least-squares regres- sion (the standard numerical method) Interactive figure: eigen-directions. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/01-linear-algebra-intuition Build It Step 1: Vectors from scratch (Python) class Vector: def __init__(self, components): self.components = list(components) self.dim = len(self.components) def __add__(self, other): return Vector([a + b for a, b in zip(self.components, other.components)]) def __sub__(self, other): return Vector([a - b for a, b in zip(self.components, other.components)]) def dot(self, other): return sum(a * b for a, b in zip(self.components, other.components)) def magnitude(self): return sum(x**2 for x in self.components) ** 0.5 def normalize(self): mag = self.magnitude() return Vector([x / mag for x in self.components]) def cosine_similarity(self, other): return self.dot(other) / (self.magnitude() * other.magnitude()) def __repr__(self): return f"Vector({self.components})" a = Vector([1, 2, 3]) b = Vector([4, 5, 6]) print(f"a + b = {a + b}") print(f"a · b = {a.dot(b)}") print(f"|a| = {a.magnitude():.4f}") print(f"cosine similarity = {a.cosine_similarity(b):.4f}") AIENGINEERINGFROMSCRATCH.COM 74

LINEAR ALGEBRA INTUITION Step 2: Matrices from scratch (Python) class Matrix: def __init__(self, rows): self.rows = [list(row) for row in rows] self.shape = (len(self.rows), len(self.rows[0])) def __matmul__(self, other): if isinstance(other, Vector): return Vector([ sum(self.rows[i][j] * other.components[j] for j in range(self.shape[1]))↪ for i in range(self.shape[0]) ]) rows = [] for i in range(self.shape[0]): row = [] for j in range(other.shape[1]): row.append(sum( self.rows[i][k] * other.rows[k][j] for k in range(self.shape[1]) )) rows.append(row) return Matrix(rows) def transpose(self): return Matrix([ [self.rows[j][i] for j in range(self.shape[0])] for i in range(self.shape[1]) ]) def __repr__(self): return f"Matrix({self.rows})" rotation_90 = Matrix([[0, -1], [1, 0]]) point = Vector([3, 1]) rotated = rotation_90 @ point print(f"Original: {point}") print(f"Rotated 90°: {rotated}") Step 3: Why this matters for AI import random random.seed(42) weights = Matrix([[random.gauss(0, 0.1) for _ in range(3)] for _ in range(2)]) input_vector = Vector([1.0, 0.5, -0.3]) output = weights @ input_vector print(f"Input (3D): {input_vector}") print(f"Output (2D): {output}") print("This is what a neural network layer does -- matrix multiplication.") AIENGINEERINGFROMSCRATCH.COM 75

LINEAR ALGEBRA INTUITION Step 4: Julia version a = [1.0, 2.0, 3.0] b = [4.0, 5.0, 6.0] println("a + b = ", a + b) println("a · b = ", a ⋅ b) # Julia supports unicode operators println("|a| = ", √(a ⋅ a)) println("cosine = ", (a ⋅ b) / (√(a ⋅ a) * √(b ⋅ b))) # Matrix-vector multiplication W = [0.1 -0.2 0.3; 0.4 0.5 -0.1] x = [1.0, 0.5, -0.3] println("Wx = ", W * x) println("This is a neural network layer.") Step 5: Linear independence and projection from scratch (Python) def is_linearly_independent(vectors): n = len(vectors) dim = len(vectors[0].components) mat = Matrix([v.components[:] for v in vectors]) rows = [row[:] for row in mat.rows] rank = 0 for col in range(dim): pivot = None for row in range(rank, len(rows)): if abs(rows[row][col]) > 1e-10: pivot = row break if pivot is None: continue rows[rank], rows[pivot] = rows[pivot], rows[rank] scale = rows[rank][col] rows[rank] = [x / scale for x in rows[rank]] for row in range(len(rows)): if row != rank and abs(rows[row][col]) > 1e-10: factor = rows[row][col] rows[row] = [rows[row][j] - factor * rows[rank][j] for j in range(dim)]↪ rank += 1 return rank == n def project(a, b): scalar = a.dot(b) / b.dot(b) return Vector([scalar * x for x in b.components]) def gram_schmidt(vectors): orthonormal = [] for v in vectors: w = v for u in orthonormal: AIENGINEERINGFROMSCRATCH.COM 76

LINEAR ALGEBRA INTUITION proj = project(w, u) w = w - proj if w.magnitude() < 1e-10: continue orthonormal.append(w.normalize()) return orthonormal v1 = Vector([1, 0, 0]) v2 = Vector([1, 1, 0]) v3 = Vector([1, 1, 1]) basis = gram_schmidt([v1, v2, v3]) for i, u in enumerate(basis): print(f"u{i+1} = {u}") print(f" |u{i+1}| = {u.magnitude():.6f}") print(f"u1 · u2 = {basis[0].dot(basis[1]):.6f}") print(f"u1 · u3 = {basis[0].dot(basis[2]):.6f}") print(f"u2 · u3 = {basis[1].dot(basis[2]):.6f}") Use It Now the same thing with NumPy – what you’ll actually use in practice: import numpy as np a = np.array([1, 2, 3], dtype=float) b = np.array([4, 5, 6], dtype=float) print(f"a + b = {a + b}") print(f"a · b = {np.dot(a, b)}") print(f"|a| = {np.linalg.norm(a):.4f}") print(f"cosine = {np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)):.4f}") W = np.random.randn(2, 3) * 0.1 x = np.array([1.0, 0.5, -0.3]) print(f"Wx = {W @ x}") Rank, Projection, and QR with NumPy import numpy as np A = np.array([[1, 2], [2, 4]]) print(f"Rank: {np.linalg.matrix_rank(A)}") a = np.array([3, 4]) b = np.array([1, 0]) proj = (np.dot(a, b) / np.dot(b, b)) * b print(f"Projection of {a} onto {b}: {proj}") Q, R = np.linalg.qr(np.random.randn(3, 3)) print(f"Q is orthogonal: {np.allclose(Q @ Q.T, np.eye(3))}") print(f"R is upper triangular: {np.allclose(R, np.triu(R))}") AIENGINEERINGFROMSCRATCH.COM 77

LINEAR ALGEBRA INTUITION PyTorch – Tensors Are Vectors with Autodiff import torch x = torch.randn(3, requires_grad=True) y = torch.tensor([1.0, 0.0, 0.0]) similarity = torch.dot(x, y) similarity.backward() print(f"x = {x.data}") print(f"y = {y.data}") print(f"dot product = {similarity.item():.4f}") print(f"d(dot)/dx = {x.grad}") The gradient of the dot product with respect to x is just y. PyTorch computed this automatically. Every operation in a neural network is built from operations like this – matrix multiplies, dot products, projections – and autodiff tracks gradients through all of them. You just built from scratch what NumPy does in one line. Now you know what’s happening under the hood. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/01-linear-algebra-intuition Connections Everything in this lesson connects to specific parts of modern AI: Concept Where it shows up Dot product Attention scores in transformers, cosine similarity in RAG Matrix multiply Every neural network layer, every linear transformation Linear independence Feature selection, avoiding multicollinearity Rank Determining if a system is solvable, LoRA (low-rank adaptation) Projection Linear regression (projecting onto column space), PCA Gram-Schmidt / QR Numerical solvers, eigenvalue computation Orthonormal basis Stable numerical computation, whitening transforms LoRA deserves special mention. It fine-tunes large language models by decomposing weight updates into low-rank matrices. Instead of updating a 4096x4096 weight matrix (16M pa- rameters), LoRA updates two matrices of size 4096x16 and 16x4096 (131K parameters). The rank-16 constraint means LoRA assumes the weight update lives in a 16-dimensional subspace of the full 4096-dimensional space. That is linear algebra doing real work. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engine ering-from-scratch/tree/main/phases/01-math-foundations/01-linear-algebra-intuition/code 1. Implement Vector.angle_between(other) that returns the angle in degrees between two vectors AIENGINEERINGFROMSCRATCH.COM 78

LINEAR ALGEBRA INTUITION 2. Create a 2D scaling matrix that doubles the x-coordinate and triples the y-coordinate, then apply it to the vector [1, 1] 3. Given 5 random word-like vectors (dimension 50), find the two most similar using cosine similarity 4. Verify that the Gram-Schmidt output is truly orthonormal: check that every pair has dot product 0 and every vector has magnitude 1 5. Create a 3x3 matrix with rank 2. Verify using the rank() method. Then explain what geometric object the columns span. 6. Project the vector [1, 2, 3] onto [1, 1, 1]. What does the result represent geometrically? Key Terms Term What people say What it actually means Vector “An arrow” A list of numbers representing a point or direction in n-dimensional space Matrix “A table of numbers” A transformation that maps vectors from one space to another Dot product “Multiply and sum” A measure of how aligned two vectors are – the core of similarity search Embedding “Some AI magic” A vector that represents the meaning of something (word, image, user) Linear inde- pendence “They don’t overlap” No vector in the set can be written as a combination of the others Rank “How many dimensions” The number of linearly independent columns (or rows) in a matrix Projection “The shadow” The component of one vector in the direction of another Basis “The coordinate axes” A minimal set of independent vectors that span the space Orthonormal “Perpendicular unit vectors” Vectors that are mutually perpendicular and each have length 1 Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/01-linear-algebra-intuition • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/01-linear-algebra-intuition/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/01-linear-algebra-intuition The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 79

Vectors, Matrices & Operations Every neural network is just matrix multiplication with extra steps. Type: Build Languages: Python, Julia Prerequisites: Phase 1, Lesson 01 (Linear Algebra Intuition) Time: ~60 minutes Learning Objectives • Build a Matrix class with element-wise operations, matrix multiplication, transpose, de- terminant, and inverse • Distinguish element-wise multiplication from matrix multiplication and explain when each applies • Implement a single dense neural network layer (relu(W @ x + b)) using only the from- scratch Matrix class • Explain broadcasting rules and how bias addition works in neural network frameworks The Problem You want to build a neural network. You read the code and see this: output = activation(weights @ input + bias) That @ is matrix multiplication. The weights are a matrix. The input is a vector. If you do not know what those operations do, this line is magic. If you do know, it is the entire forward pass of a layer in three operations. Every image your model processes is a matrix of pixel values. Every word embedding is a vector. Every layer of every neural network is a matrix transformation. You cannot build AI systems without being fluent in matrix operations the same way you cannot write code without understanding variables. This lesson builds that fluency from scratch. The Concept Vectors: ordered lists of numbers A vector is a list of numbers with a direction and magnitude. In AI, vectors represent data points, features, or parameters. v = [3, 4] -- a 2D vector w = [1, 0, -2] -- a 3D vector A 2D vector [3, 4] points to coordinates (3, 4) on a plane. Its length (magnitude) is 5 (the 3-4-5 triangle). AIENGINEERINGFROMSCRATCH.COM 80

VECTORS, MATRICES & OPERATIONS Matrices: grids of numbers A matrix is a 2D grid. Rows and columns. An m x n matrix has m rows and n columns. A = | 1 2 3 | -- 2x3 matrix (2 rows, 3 columns) | 4 5 6 | In neural networks, weight matrices transform input vectors into output vectors. A layer with 784 inputs and 128 outputs uses a 128x784 weight matrix. Why shapes matter Matrix multiplication has a strict rule: (m x n) @ (n x p) = (m x p). The inner dimensions must match. (128 x 784) @ (784 x 1) = (128 x 1) weights input output Inner dimensions: 784 = 784 -- valid If you get a shape mismatch error in PyTorch, this is why. The operations map Operation What it does Neural network use Addition Element-wise combine Adding bias to output Scalar multiply Scale every element Learning rate * gradients Matrix multiply Transform vectors Layer forward pass Transpose Flip rows and columns Backpropagation Determinant Single number summary Checking invertibility Inverse Undo a transformation Solving linear systems Identity Do-nothing matrix Initialization, residual connections Element-wise vs matrix multiplication This distinction trips up beginners constantly. Element-wise: multiply matching positions. Both matrices must be the same shape. | 1 2 | | 5 6 | | 5 12 | | 3 4 | * | 7 8 | = | 21 32 | Matrix multiplication: dot products of rows and columns. Inner dimensions must match. | 1 2 | | 5 6 | | 1*5+2*7 1*6+2*8 | | 19 22 | | 3 4 | @ | 7 8 | = | 3*5+4*7 3*6+4*8 | = | 43 50 | Different operations, different results, different rules. Broadcasting When you add a bias vector to a matrix of outputs, the shapes do not match. Broadcasting stretches the smaller array to fit. | 1 2 3 | + [10, 20, 30] | 4 5 6 | AIENGINEERINGFROMSCRATCH.COM 81

VECTORS, MATRICES & OPERATIONS Broadcasting stretches the vector across rows: | 1 2 3 | | 10 20 30 | | 11 22 33 | | 4 5 6 | + | 10 20 30 | = | 14 25 36 | Every modern framework does this automatically. Understanding it prevents confusion when shapes seem wrong but the code runs. Interactive figure: vector-projection. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/02-vectors-matrices-operations Build It Step 1: Vector class class Vector: def __init__(self, data): self.data = list(data) self.size = len(self.data) def __repr__(self): return f"Vector({self.data})" def __add__(self, other): return Vector([a + b for a, b in zip(self.data, other.data)]) def __sub__(self, other): return Vector([a - b for a, b in zip(self.data, other.data)]) def __mul__(self, scalar): return Vector([x * scalar for x in self.data]) def dot(self, other): return sum(a * b for a, b in zip(self.data, other.data)) def magnitude(self): return sum(x ** 2 for x in self.data) ** 0.5 Step 2: Matrix class with core operations class Matrix: def __init__(self, data): self.data = [list(row) for row in data] self.rows = len(self.data) self.cols = len(self.data[0]) self.shape = (self.rows, self.cols) def __repr__(self): rows_str = "\n ".join(str(row) for row in self.data) return f"Matrix({self.shape}):\n {rows_str}" def __add__(self, other): AIENGINEERINGFROMSCRATCH.COM 82

VECTORS, MATRICES & OPERATIONS return Matrix([ [self.data[i][j] + other.data[i][j] for j in range(self.cols)] for i in range(self.rows) ]) def __sub__(self, other): return Matrix([ [self.data[i][j] - other.data[i][j] for j in range(self.cols)] for i in range(self.rows) ]) def scalar_multiply(self, scalar): return Matrix([ [self.data[i][j] * scalar for j in range(self.cols)] for i in range(self.rows) ]) def element_wise_multiply(self, other): return Matrix([ [self.data[i][j] * other.data[i][j] for j in range(self.cols)] for i in range(self.rows) ]) def matmul(self, other): return Matrix([ [ sum(self.data[i][k] * other.data[k][j] for k in range(self.cols)) for j in range(other.cols) ] for i in range(self.rows) ]) def transpose(self): return Matrix([ [self.data[j][i] for j in range(self.rows)] for i in range(self.cols) ]) def determinant(self): if self.shape == (1, 1): return self.data[0][0] if self.shape == (2, 2): return self.data[0][0] * self.data[1][1] - self.data[0][1] * self.data[1][0]↪ det = 0 for j in range(self.cols): minor = Matrix([ [self.data[i][k] for k in range(self.cols) if k != j] for i in range(1, self.rows) ]) det += ((-1) ** j) * self.data[0][j] * minor.determinant() return det def inverse_2x2(self): AIENGINEERINGFROMSCRATCH.COM 83

VECTORS, MATRICES & OPERATIONS det = self.determinant() if det == 0: raise ValueError("Matrix is singular, no inverse exists") return Matrix([ [self.data[1][1] / det, -self.data[0][1] / det], [-self.data[1][0] / det, self.data[0][0] / det] ]) @staticmethod def identity(n): return Matrix([ [1 if i == j else 0 for j in range(n)] for i in range(n) ]) Step 3: See it work A = Matrix([[1, 2], [3, 4]]) B = Matrix([[5, 6], [7, 8]]) print("A + B =", (A + B).data) print("A @ B =", A.matmul(B).data) print("A^T =", A.transpose().data) print("det(A) =", A.determinant()) print("A^-1 =", A.inverse_2x2().data) I = Matrix.identity(2) print("A @ A^-1 =", A.matmul(A.inverse_2x2()).data) Step 4: Connect to neural networks import random inputs = Matrix([[0.5], [0.8], [0.2]]) weights = Matrix([ [random.uniform(-1, 1) for _ in range(3)] for _ in range(2) ]) bias = Matrix([[0.1], [0.1]]) def relu_matrix(m): return Matrix([[max(0, val) for val in row] for row in m.data]) pre_activation = weights.matmul(inputs) + bias output = relu_matrix(pre_activation) print(f"Input shape: {inputs.shape}") print(f"Weight shape: {weights.shape}") print(f"Output shape: {output.shape}") print(f"Output: {output.data}") This is a single dense layer: output = relu(W @ x + b). Every dense layer in every neural network does exactly this. AIENGINEERINGFROMSCRATCH.COM 84

VECTORS, MATRICES & OPERATIONS Use It NumPy does everything above in fewer lines and orders of magnitude faster. import numpy as np A = np.array([[1, 2], [3, 4]]) B = np.array([[5, 6], [7, 8]]) print("A + B =\n", A + B) print("A * B (element-wise) =\n", A * B) print("A @ B (matrix multiply) =\n", A @ B) print("A^T =\n", A.T) print("det(A) =", np.linalg.det(A)) print("A^-1 =\n", np.linalg.inv(A)) print("I =\n", np.eye(2)) inputs = np.random.randn(3, 1) weights = np.random.randn(2, 3) bias = np.array([[0.1], [0.1]]) output = np.maximum(0, weights @ inputs + bias) print(f"\nNeural network layer: {weights.shape} @ {inputs.shape} = {output.shape}") print(f"Output:\n{output}") The @ operator in Python calls __matmul__. NumPy implements it with optimized BLAS rou- tines written in C and Fortran. Same math, 100x faster. Broadcasting in NumPy: matrix = np.array([[1, 2, 3], [4, 5, 6]]) bias = np.array([10, 20, 30]) print(matrix + bias) NumPy automatically broadcasts the 1D bias across both rows. This is how bias addition works in every neural network framework. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/02-vectors-matrices- operations Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/02-vectors-matrices- operations/code 1. Verify the inverse. Multiply A @ A.inverse_2x2() and confirm you get the identity matrix. Try it with three different 2x2 matrices. What happens when the determinant is zero? 2. Implement 3x3 inverse. Extend the Matrix class to compute inverses for 3x3 matrices using the adjugate method. Test it against NumPy’s np.linalg.inv. 3. Build a two-layer network. Using only your Matrix class (no NumPy), create a two- layer neural network: input (3) -> hidden (4) -> output (2). Initialize random weights, AIENGINEERINGFROMSCRATCH.COM 85

VECTORS, MATRICES & OPERATIONS run a forward pass, and verify all shapes are correct. Key Terms Term What people say What it actually means Vector “An arrow” An ordered list of numbers. In AI: a point in high-dimensional space. Matrix “A table of numbers” A linear transformation. It maps vectors from one space to another. Matrix multiply “Just multiply the numbers” Dot products between every row of the first matrix and every column of the second. Order matters. Transpose “Flip it” Swap rows and columns. Turns an m x n matrix into n x m. Critical in backpropagation. Determinant “Some number from the matrix” Measures how much the matrix scales area (2D) or volume (3D). Zero means the transformation crushes a dimension. Inverse “Undo the matrix” The matrix that reverses the transformation. Only exists when the determinant is not zero. Identity matrix “The boring matrix” The matrix equivalent of multiplying by 1. Used in residual connections (ResNets). Broadcasting “Magic shape fixing” Stretching a smaller array to match a larger one by repeating along missing dimensions. Element- wise “Regular multiplication” Multiply matching positions. Both arrays must have the same shape (or be broadcastable). Further Reading • 3Blue1Brown: Essence of Linear Algebra - visual intuition for every operation covered here • NumPy documentation on broadcasting - the exact rules NumPy follows • Stanford CS229 Linear Algebra Review - concise reference for ML-specific linear algebra Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/02-vectors-matrices-operations • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/02-vectors-matrices-operations/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/02-vectors-matrices-operations The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 86

Matrix Transformations A matrix is a machine that reshapes space. Learn what it does to every point, and you understand the whole transformation. Type: Build Languages: Python, Julia Prerequisites: Phase 1, Lessons 01-02 (Linear Alge- bra Intuition, Vectors & Matrices Operations) Time: ~75 minutes Learning Objectives • Construct rotation, scaling, shearing, and reflection matrices and apply them to 2D and 3D points • Compose multiple transformations by matrix multiplication and verify that order matters • Compute eigenvalues and eigenvectors of 2x2 matrices from the characteristic equation • Explain why eigenvalues determine PCA directions, RNN stability, and spectral cluster- ing behavior The Problem You read about PCA and see “find the eigenvectors of the covariance matrix.” You read about model stability and see “check if all eigenvalues have magnitude less than 1.” You read about data augmentation and see “apply a random rotation.” None of this makes sense until you understand what matrices do to space geometrically. Matrices are not just grids of numbers. They are spatial machines. A rotation matrix spins points. A scaling matrix stretches them. A shearing matrix tilts them. Every transformation a neural network applies to data is one of these operations or a composition of them. This lesson makes those operations concrete. The Concept Transformations as matrices Every linear transformation in 2D can be written as a 2x2 matrix. The matrix tells you exactly where the basis vectors [1, 0] and [0, 1] end up. Everything else follows. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations Rotation A 2D rotation by angle theta keeps distances and angles intact. It moves every point along a circular arc. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations AIENGINEERINGFROMSCRATCH.COM 87

MATRIX TRANSFORMATIONS In 3D, you rotate around an axis. Each axis has its own rotation matrix: Rz(theta) = | cos -sin 0 | Rotate around z-axis | sin cos 0 | (x-y plane spins, z stays) | 0 0 1 | Rx(theta) = | 1 0 0 | Rotate around x-axis | 0 cos -sin | (y-z plane spins, x stays) | 0 sin cos | Ry(theta) = | cos 0 sin | Rotate around y-axis | 0 1 0 | (x-z plane spins, y stays) | -sin 0 cos | Scaling Scaling stretches or compresses along each axis independently. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations Shearing Shearing tilts one axis while keeping the other fixed. It turns rectangles into parallelograms. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations Shear matrices: - Shx = [[1, k], [0, 1]] shifts x by k * y - Shy = [[1, 0], [k, 1]] shifts y by k * x Reflection Reflection mirrors points across an axis or line. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations Reflection matrices: - Reflect across y-axis: [[-1, 0], [0, 1]] - Reflect across x-axis: [[1, 0], [0, -1]] Composition: chaining transformations Applying transformation A then B is the same as multiplying their matrices: result = B @ A @ point. Order matters. Rotate then scale gives different results than scale then rotate. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations Composed: S @ R = [[0, -2], [0.5, 0]] Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations Composed: R @ S = [[0, -0.5], [2, 0]] Different results. Matrix multiplication is not commutative. AIENGINEERINGFROMSCRATCH.COM 88

MATRIX TRANSFORMATIONS Eigenvalues and eigenvectors Most vectors change direction when a matrix hits them. Eigenvectors are special: the matrix only scales them, never rotates them. The scaling factor is the eigenvalue. A @ v = lambda * v v is the eigenvector (direction that survives) lambda is the eigenvalue (how much it stretches) Example: A = | 2 1 | | 1 2 | Eigenvector [1, 1] with eigenvalue 3: A @ [1,1] = [3, 3] = 3 * [1, 1] (same direction, scaled by 3) Eigenvector [1, -1] with eigenvalue 1: A @ [1,-1] = [1, -1] = 1 * [1, -1] (same direction, unchanged) The matrix stretches space by 3x along [1, 1] and keeps [1, -1] unchanged. Every other direction is a mix of these two. Eigendecomposition If a matrix has n linearly independent eigenvectors, it can be decomposed: A = V @ D @ V^(-1) V = matrix whose columns are eigenvectors D = diagonal matrix of eigenvalues V^(-1) = inverse of V This says: rotate into eigenvector coordinates, scale along each axis, rotate back. Why eigenvalues matter PCA. The eigenvectors of the covariance matrix are the principal components. The eigenval- ues tell you how much variance each component captures. Sort by eigenvalue, keep the top k, and you have dimensionality reduction. Stability. In recurrent networks and dynamical systems, eigenvalues with magnitude > 1 cause outputs to explode. Magnitude < 1 causes them to vanish. This is the vanish- ing/exploding gradient problem stated in one sentence. Spectral methods. Graph neural networks use eigenvalues of the adjacency matrix. Spec- tral clustering uses eigenvalues of the Laplacian. The eigenvectors reveal the structure of the graph. Determinant as volume scaling factor The determinant of a transformation matrix tells you how much it scales area (2D) or volume (3D). det = 1: area preserved (rotation) det = 2: area doubled det = 0: space crushed to lower dimension (singular) det = -1: area preserved but orientation flipped (reflection) AIENGINEERINGFROMSCRATCH.COM 89

MATRIX TRANSFORMATIONS | det(Rotation) | = 1 (always) | det(Scale sx, sy) | = sx * sy | det(Shear) | = 1 (area preserved) | det(Reflection) | = -1 (orientation flipped) Interactive figure: matrix-transform. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/03-matrix-transformations Build It Step 1: Transformation matrices from scratch (Python) import math def rotation_2d(theta): c, s = math.cos(theta), math.sin(theta) return [[c, -s], [s, c]] def scaling_2d(sx, sy): return [[sx, 0], [0, sy]] def shearing_2d(kx, ky): return [[1, kx], [ky, 1]] def reflection_x(): return [[1, 0], [0, -1]] def reflection_y(): return [[-1, 0], [0, 1]] def mat_vec_mul(matrix, vector): return [ sum(matrix[i][j] * vector[j] for j in range(len(vector))) for i in range(len(matrix)) ] def mat_mul(a, b): rows_a, cols_b = len(a), len(b[0]) cols_a = len(a[0]) return [ [sum(a[i][k] * b[k][j] for k in range(cols_a)) for j in range(cols_b)] for i in range(rows_a) ] point = [1.0, 0.0] angle = math.pi / 4 rotated = mat_vec_mul(rotation_2d(angle), point) print(f"Rotate (1,0) by 45 deg: ({rotated[0]:.4f}, {rotated[1]:.4f})") scaled = mat_vec_mul(scaling_2d(2, 3), [1.0, 1.0]) AIENGINEERINGFROMSCRATCH.COM 90

MATRIX TRANSFORMATIONS print(f"Scale (1,1) by (2,3): ({scaled[0]:.1f}, {scaled[1]:.1f})") sheared = mat_vec_mul(shearing_2d(1, 0), [1.0, 1.0]) print(f"Shear (1,1) kx=1: ({sheared[0]:.1f}, {sheared[1]:.1f})") reflected = mat_vec_mul(reflection_y(), [2.0, 1.0]) print(f"Reflect (2,1) across y: ({reflected[0]:.1f}, {reflected[1]:.1f})") Step 2: Composition of transformations R = rotation_2d(math.pi / 2) S = scaling_2d(2, 0.5) rotate_then_scale = mat_mul(S, R) scale_then_rotate = mat_mul(R, S) point = [1.0, 0.0] result1 = mat_vec_mul(rotate_then_scale, point) result2 = mat_vec_mul(scale_then_rotate, point) print(f"Rotate 90 then scale: ({result1[0]:.2f}, {result1[1]:.2f})") print(f"Scale then rotate 90: ({result2[0]:.2f}, {result2[1]:.2f})") print(f"Same? {result1 == result2}") Step 3: Eigenvalues from scratch (2x2) For a 2x2 matrix [[a, b], [c, d]], eigenvalues solve the characteristic equation: lambda^2 - (a+d)*lambda + (ad - bc) = 0. def eigenvalues_2x2(matrix): a, b = matrix[0] c, d = matrix[1] trace = a + d det = a * d - b * c discriminant = trace ** 2 - 4 * det if discriminant < 0: real = trace / 2 imag = (-discriminant) ** 0.5 / 2 return (complex(real, imag), complex(real, -imag)) sqrt_disc = discriminant ** 0.5 return ((trace + sqrt_disc) / 2, (trace - sqrt_disc) / 2) def eigenvector_2x2(matrix, eigenvalue): a, b = matrix[0] c, d = matrix[1] if abs(b) > 1e-10: v = [b, eigenvalue - a] elif abs(c) > 1e-10: v = [eigenvalue - d, c] else: if abs(a - eigenvalue) < 1e-10: v = [1, 0] else: v = [0, 1] AIENGINEERINGFROMSCRATCH.COM 91

MATRIX TRANSFORMATIONS mag = (v[0] ** 2 + v[1] ** 2) ** 0.5 return [v[0] / mag, v[1] / mag] A = [[2, 1], [1, 2]] vals = eigenvalues_2x2(A) print(f"Matrix: {A}") print(f"Eigenvalues: {vals[0]:.4f}, {vals[1]:.4f}") for val in vals: vec = eigenvector_2x2(A, val) result = mat_vec_mul(A, vec) scaled = [val * vec[0], val * vec[1]] print(f" lambda={val:.1f}, v={[round(x,4) for x in vec]}") print(f" A@v = {[round(x,4) for x in result]}") print(f" l*v = {[round(x,4) for x in scaled]}") Step 4: Determinant as volume scaling factor def det_2x2(matrix): return matrix[0][0] * matrix[1][1] - matrix[0][1] * matrix[1][0] print(f"det(rotation 45) = {det_2x2(rotation_2d(math.pi/4)):.4f}") print(f"det(scale 2,3) = {det_2x2(scaling_2d(2, 3)):.1f}") print(f"det(shear kx=1) = {det_2x2(shearing_2d(1, 0)):.1f}") print(f"det(reflect y) = {det_2x2(reflection_y()):.1f}") singular = [[1, 2], [2, 4]] print(f"det(singular) = {det_2x2(singular):.1f}") print("Singular: columns are proportional, space collapses to a line.") Use It NumPy handles all of this with optimized routines. import numpy as np theta = np.pi / 4 R = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]]) point = np.array([1.0, 0.0]) print(f"Rotate (1,0) by 45 deg: {R @ point}") S = np.diag([2.0, 3.0]) composed = S @ R print(f"Scale(2,3) after Rotate(45): {composed @ point}") A = np.array([[2, 1], [1, 2]], dtype=float) eigenvalues, eigenvectors = np.linalg.eig(A) print(f"\nEigenvalues: {eigenvalues}") print(f"Eigenvectors (columns):\n{eigenvectors}") for i in range(len(eigenvalues)): AIENGINEERINGFROMSCRATCH.COM 92

MATRIX TRANSFORMATIONS v = eigenvectors[:, i] lam = eigenvalues[i] print(f" A @ v{i} = {A @ v}, lambda * v{i} = {lam * v}") print(f"\ndet(R) = {np.linalg.det(R):.4f}") print(f"det(S) = {np.linalg.det(S):.1f}") B = np.array([[3, 1], [0, 2]], dtype=float) vals, vecs = np.linalg.eig(B) D = np.diag(vals) V = vecs reconstructed = V @ D @ np.linalg.inv(V) print(f"\nEigendecomposition A = V @ D @ V^-1:") print(f"Original:\n{B}") print(f"Reconstructed:\n{reconstructed}") 3D rotations with NumPy def rotation_3d_z(theta): c, s = np.cos(theta), np.sin(theta) return np.array([[c, -s, 0], [s, c, 0], [0, 0, 1]]) def rotation_3d_x(theta): c, s = np.cos(theta), np.sin(theta) return np.array([[1, 0, 0], [0, c, -s], [0, s, c]]) point_3d = np.array([1.0, 0.0, 0.0]) rotated_z = rotation_3d_z(np.pi / 2) @ point_3d rotated_x = rotation_3d_x(np.pi / 2) @ point_3d print(f"\n3D point: {point_3d}") print(f"Rotate 90 around z: {np.round(rotated_z, 4)}") print(f"Rotate 90 around x: {np.round(rotated_x, 4)}") This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/03-matrix-transformations Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engine ering-from-scratch/tree/main/phases/01-math-foundations/03-matrix-transformations/code 1. Apply rotation, scaling, and shearing to a unit square (corners at [0,0], [1,0], [1,1], [0,1]). Print the transformed corners for each. Verify that rotation preserves distances between corners. 2. Find the eigenvalues of the matrix [[4, 2], [1, 3]] by hand using the characteristic equa- tion. Then verify with your from-scratch function and with NumPy. 3. Create a composition of three transformations (rotate 30 degrees, scale by [1.5, 0.8], shear with kx=0.3) and apply it to 8 points arranged in a circle. Print before and after coordinates. Compute the determinant of the composed matrix and verify it equals the product of the individual determinants. AIENGINEERINGFROMSCRATCH.COM 93

MATRIX TRANSFORMATIONS Key Terms Term What people say What it actually means Rotation matrix “Spins things” An orthogonal matrix that moves points along circular arcs while preserving distances and angles. Determinant is always 1. Scaling matrix “Makes things bigger” A diagonal matrix that stretches or compresses independently along each axis. Determinant is the product of scale factors. Shearing matrix “Slants things” A matrix that shifts one coordinate proportionally to another, turning rectangles into parallelograms. Determinant is 1. Reflection “Mirrors things” A matrix that flips space across an axis or plane. Determinant is -1. Composition “Do two things” Multiplying transformation matrices to chain operations. Order matters: B @ A means apply A first, then B. Eigenvector “Special direction” A direction that the matrix only scales, never rotates. The transformation’s fingerprint. Eigenvalue “How much it stretches” The scalar factor by which the matrix scales its eigenvector. Can be negative (flip) or complex (rotation). Eigendecomposition“Break the matrix apart” Writing a matrix as V @ D @ V^(-1), separating it into its fundamental scaling directions and magnitudes. Determinant “A single number from a matrix” The factor by which the transformation scales area (2D) or volume (3D). Zero means the transformation is irreversible. Characteristic equation “Where eigenvalues come from” det(A - lambda * I) = 0. The polynomial whose roots are the eigenvalues. Further Reading • 3Blue1Brown: Linear Transformations – visual intuition for how matrices reshape space • 3Blue1Brown: Eigenvectors and Eigenvalues – the best visual explanation of what eigen- vectors mean geometrically • MIT 18.06 Lecture 21: Eigenvalues and Eigenvectors – Gilbert Strang’s classic treatment Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/03-matrix-transformations • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/03-matrix-transformations/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/03-matrix-transformations The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 94

Calculus for Machine Learning Derivatives tell you which way is downhill. That is all a neural network needs to learn. Type: Learn Language: Python Prerequisites: Phase 1, Lessons 01-03 Time: ~60 minutes Learning Objectives • Compute numerical and analytical derivatives for common ML functions (x^2, sigmoid, cross-entropy) • Implement gradient descent from scratch to minimize a loss function in 1D and 2D • Derive the gradient of a linear regression model and train it via manual weight updates • Explain the Hessian matrix, Taylor series approximations, and their connection to opti- mization methods The Problem You have a neural network with millions of weights. Each weight is a knob. You need to figure out which direction to turn every single knob to make the model slightly less wrong. Calculus gives you that direction. Without calculus, training a neural network would mean trying random changes and hoping for the best. With derivatives, you know exactly how each weight affects the error. You turn every knob the right way, every time. The Concept What is a derivative? A derivative measures the rate of change. For a function y = f(x), the derivative f’(x) tells you: if you nudge x by a tiny amount, how much does y change? Geometrically, the derivative is the slope of the tangent line at a point. f(x) = x^2: x f(x) f’(x) (slope) 0 0 0 (flat, at the bottom) 1 1 2 2 4 4 (tangent line slope at this point) 3 9 6 At x=2, the slope is 4. If you move x a tiny bit to the right, y increases by about 4 times that amount. At x=0, the slope is 0. You are at the bottom of the bowl. AIENGINEERINGFROMSCRATCH.COM 95

CALCULUS FOR MACHINE LEARNING The formal definition: f'(x) = lim f(x + h) - f(x) h->0 ----------------- h In code, you skip the limit and just use a very small h. That is the numerical derivative. Partial derivatives: one variable at a time Real functions have many inputs. A neural network loss depends on thousands of weights. A partial derivative holds all variables constant except one, then takes the derivative with respect to that one. f(x, y) = x^2 + 3xy + y^2 df/dx = 2x + 3y (treat y as a constant) df/dy = 3x + 2y (treat x as a constant) Each partial derivative answers: if I nudge just this one weight, how does the loss change? The gradient: vector of all partial derivatives The gradient collects every partial derivative into one vector. For a function f(x, y, z), the gradient is: grad f = [ df/dx, df/dy, df/dz ] The gradient points in the direction of steepest ascent. To minimize a function, go in the opposite direction. Contour plot of f(x,y) = x^2 + y^2: The function forms a bowl shape with concentric circles as contour lines. The minimum is at (0, 0). Point grad f -grad f (descent direction) (1, 1) [2, 2] (points uphill, away from minimum) [-2, -2] (points downhill, toward minimum) (0, 0) [0, 0] (flat, at the minimum) [0, 0] This is gradient descent in a picture. Compute the gradient, negate it, take a step. The connection to optimization Training a neural network is optimization. You have a loss function L(w1, w2, …, wn) that measures how wrong the model is. You want to minimize it. Gradient descent update rule: w_new = w_old - learning_rate * dL/dw For every weight: 1. Compute the partial derivative of loss with respect to that weight AIENGINEERINGFROMSCRATCH.COM 96

CALCULUS FOR MACHINE LEARNING 2. Subtract a small multiple of it from the weight 3. Repeat The learning rate controls step size. Too big and you overshoot. Too small and you crawl. Loss landscape (1D slice): The loss function L(w) forms a curve with peaks and valleys as the weight w varies. Feature Description Global minimum The lowest point on the entire curve – the best solution Local minimum A valley that is lower than its neighbors but not the lowest overall Slope Gradient descent follows the slope downhill from any starting point Gradient descent follows the slope downhill. It can get stuck in local minima, but in high- dimensional spaces (millions of weights) this is rarely a practical problem. Numerical vs analytical derivatives There are two ways to compute a derivative. Analytical: apply calculus rules by hand. For f(x) = x^2, the derivative is f’(x) = 2x. Exact. Fast. Numerical: approximate using the definition. Compute f(x+h) and f(x-h) for a tiny h, then use the difference. Numerical (central difference): f'(x) ~= f(x + h) - f(x - h) ----------------------- 2h h = 0.0001 works well in practice Numerical derivatives are slower but work for any function. Analytical derivatives are fast but require you to derive the formula. Neural network frameworks use a third approach: automatic differentiation, which computes exact derivatives mechanically. You will see that in Phase 3. Derivatives by hand for simple functions These are the derivatives you will see over and over in ML. Function Derivative Used in -------- ---------- ------- f(x) = x^2 f'(x) = 2x Loss functions (MSE) f(x) = wx + b f'(w) = x Linear layer (gradient w.r.t. weight) f'(b) = 1 Linear layer (gradient w.r.t. bias) f'(x) = w Linear layer (gradient w.r.t. input) f(x) = e^x f'(x) = e^x Softmax, attention f(x) = ln(x) f'(x) = 1/x Cross-entropy loss f(x) = 1/(1+e^-x) f'(x) = f(x)(1-f(x)) Sigmoid activation AIENGINEERINGFROMSCRATCH.COM 97

CALCULUS FOR MACHINE LEARNING For f(x) = x^2: f(x) = x^2 f'(x) = 2x x f(x) f'(x) meaning -2 4 -4 slope tilts left (decreasing) -1 1 -2 slope tilts left (decreasing) 0 0 0 flat (minimum!) 1 1 2 slope tilts right (increasing) 2 4 4 slope tilts right (increasing) For f(w) = wx + b with x=3, b=1: f(w) = 3w + 1 f'(w) = 3 The derivative with respect to w is just x. If x is big, a small change in w causes a big change in output. The chain rule When functions are composed, the chain rule tells you how to differentiate. If y = f(g(x)), then dy/dx = f'(g(x)) * g'(x) Example: y = (3x + 1)^2 outer: f(u) = u^2 f'(u) = 2u inner: g(x) = 3x + 1 g'(x) = 3 dy/dx = 2(3x + 1) * 3 = 6(3x + 1) Neural networks are chains of functions: input -> linear -> activation -> linear -> activation -> loss. Backpropagation is the chain rule applied repeatedly from output to input. That is the entire algorithm. The Hessian Matrix The gradient tells you the slope. The Hessian tells you the curvature. The Hessian is the matrix of second-order partial derivatives. For a function f(x1, x2, …, xn), entry (i, j) of the Hessian is: H[i][j] = d^2f / (dx_i * dx_j) For a 2-variable function f(x, y): H = | d^2f/dx^2 d^2f/dxdy | | d^2f/dydx d^2f/dy^2 | What the Hessian tells you at a critical point (where gradient = 0): Hessian property Meaning Example surface Positive definite (all eigenvalues > 0) Local minimum Bowl pointing up Negative definite (all eigenvalues < 0) Local maximum Bowl pointing down Indefinite (mixed eigenvalues) Saddle point Horse saddle shape Example: f(x, y) = x^2 - y^2 (a saddle function) AIENGINEERINGFROMSCRATCH.COM 98

CALCULUS FOR MACHINE LEARNING df/dx = 2x df/dy = -2y d^2f/dx^2 = 2 d^2f/dy^2 = -2 d^2f/dxdy = 0 H = | 2 0 | | 0 -2 | Eigenvalues: 2 and -2 (one positive, one negative) --> Saddle point at (0, 0) Compare with f(x, y) = x^2 + y^2 (a bowl): H = | 2 0 | | 0 2 | Eigenvalues: 2 and 2 (both positive) --> Local minimum at (0, 0) Why the Hessian matters in ML: Newton’s method uses the Hessian to take better optimization steps than gradient descent. Instead of just following the slope, it accounts for curvature: Newton's update: w_new = w_old - H^(-1) * gradient Gradient descent: w_new = w_old - lr * gradient Newton’s method converges faster because the Hessian “rescales” the gradient – steep direc- tions get smaller steps, flat directions get larger steps. The catch: for a neural network with N parameters, the Hessian is N x N. A model with 1 million parameters would need a 1 trillion-entry matrix. That is why we use approximations. Method What it uses Cost Convergence Gradient descent First derivatives only O(N) per step Slow (linear) Newton’s method Full Hessian O(N^3) per step Fast (quadratic) L-BFGS Approximate Hessian from gradient history O(N) per step Medium (superlinear) Adam Per-parameter adaptive rates (diagonal Hessian approx) O(N) per step Medium Natural gradient Fisher information matrix (statistical Hessian) O(N^2) per step Fast In practice, Adam is the default optimizer for deep learning. It approximates second-order information cheaply by tracking the running mean and variance of gradients per parameter. Taylor Series Approximation Any smooth function can be approximated locally by a polynomial: f(x + h) = f(x) + f'(x)*h + (1/2)*f''(x)*h^2 + (1/6)*f'''(x)*h^3 + ... The more terms you include, the better the approximation – but only near the point x. Why Taylor series matter for ML: AIENGINEERINGFROMSCRATCH.COM 99

CALCULUS FOR MACHINE LEARNING • First-order Taylor = gradient descent. When you use f(x + h) ~ f(x) + f’(x)h, you are making a linear approximation. Gradient descent minimizes this linear model to choose h = -lr f’(x). • Second-order Taylor = Newton’s method. Using f(x + h) ~ f(x) + f’(x)h + (1/2)f’’(x)*h^2, you get a quadratic model. Minimizing it gives h = -f’(x)/f’’(x) – Newton’s step. • Loss function design. MSE and cross-entropy are smooth, which means their Taylor expansions are well-behaved. This is not an accident. Smooth losses make optimization predictable. Approximation order What it captures Optimization method ------------------- ----------------- ------------------- 0th order (constant) Just the value Random search 1st order (linear) Slope Gradient descent 2nd order (quadratic) Curvature Newton's method Higher orders Finer structure Rarely used in ML The key insight: all gradient-based optimization is really about approximating the loss func- tion locally and stepping to the minimum of that approximation. Integrals in ML Derivatives tell you rates of change. Integrals compute accumulations – area under a curve. In ML, you rarely compute integrals by hand, but the concept is everywhere: Probability. For a continuous random variable with density p(x): P(a < X < b) = integral from a to b of p(x) dx The area under the probability density curve between a and b is the probability of landing in that range. Expected value. The average outcome weighted by probability: E[f(X)] = integral of f(x) * p(x) dx The expected loss over a data distribution is an integral. Training minimizes an empirical approximation of this. KL divergence. Measures how different two distributions are: KL(p || q) = integral of p(x) * log(p(x) / q(x)) dx Used in VAEs, knowledge distillation, and Bayesian inference. Normalization constants. In Bayesian inference: p(w | data) = p(data | w) * p(w) / integral of p(data | w) * p(w) dw The denominator is an integral over all possible parameter values. It is often intractable, which is why we use approximations like MCMC and variational inference. Integral concept Where it appears in ML Area under curve Probability from density functions Expected value Loss functions, risk minimization KL divergence VAEs, policy optimization, distillation Normalization Bayesian posteriors, softmax denominator Marginal likelihood Model comparison, evidence lower bound (ELBO) AIENGINEERINGFROMSCRATCH.COM 100

CALCULUS FOR MACHINE LEARNING Multivariable Chain Rule in a Computation Graph The chain rule does not just apply to scalar functions in a line. In a neural network, variables fan out and merge. Here is how derivatives flow through a simple forward pass: Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/04-calculus-for-ml The backward pass computes gradients right to left: Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/04-calculus-for-ml Each arrow multiplies by the local derivative. The gradient for any parameter is the product of all local derivatives along the path from loss to that parameter. When paths branch and merge, you sum the contributions (multivariate chain rule). This is all backpropagation is: the chain rule applied systematically through a computation graph, from output to inputs. The Jacobian matrix When a function maps a vector to a vector (like a neural network layer), its derivative is a matrix. The Jacobian contains every partial derivative of every output with respect to every input. For f: R^n -> R^m, the Jacobian J is an m x n matrix: x1 x2 … xn f1 df1/dx1 df1/dx2 … df1/dxn f2 df2/dx1 df2/dx2 … df2/dxn … … … … … fm dfm/dx1 dfm/dx2 … dfm/dxn You will not compute Jacobians by hand for neural networks. PyTorch handles it. But knowing it exists helps you understand shapes in backpropagation: if a layer maps R^n to R^m, its Jacobian is m x n. The gradient flows backward through the transpose of this matrix. Why this matters for neural networks Every weight in a neural network gets a gradient. The gradient tells you how to adjust that weight to reduce the loss. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/04-calculus-for-ml Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/04-calculus-for-ml Each weight update: - W1 = W1 - lr * dL/dW1 - W2 = W2 - lr * dL/dW2 The forward pass computes the prediction and loss. The backward pass computes the gradient of the loss with respect to every weight. Then every weight takes a small step downhill. Repeat for millions of steps. That is deep learning. Interactive figure: derivative-tangent. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/04-calculus-for-ml AIENGINEERINGFROMSCRATCH.COM 101

CALCULUS FOR MACHINE LEARNING Build It Step 1: Numerical derivative from scratch def numerical_derivative(f, x, h=1e-7): return (f(x + h) - f(x - h)) / (2 * h) def f(x): return x ** 2 for x in [-2, -1, 0, 1, 2]: numerical = numerical_derivative(f, x) analytical = 2 * x print(f"x={x:2d} f'(x) numerical={numerical:.6f} analytical={analytical:.1f}")↪ The numerical derivative matches the analytical one to many decimal places. Step 2: Partial derivatives and gradients def numerical_gradient(f, point, h=1e-7): gradient = [] for i in range(len(point)): point_plus = list(point) point_minus = list(point) point_plus[i] += h point_minus[i] -= h partial = (f(point_plus) - f(point_minus)) / (2 * h) gradient.append(partial) return gradient def f_multi(point): x, y = point return x**2 + 3*x*y + y**2 grad = numerical_gradient(f_multi, [1.0, 2.0]) print(f"Numerical gradient at (1,2): {[f'{g:.4f}' for g in grad]}") print(f"Analytical gradient at (1,2): [2*1+3*2, 3*1+2*2] = [{2*1+3*2}, {3*1+2*2}]") Step 3: Gradient descent to find the minimum of f(x) = x^2 x = 5.0 lr = 0.1 for step in range(20): grad = 2 * x x = x - lr * grad print(f"step {step:2d} x={x:8.4f} f(x)={x**2:10.6f}") Starting at x=5, each step moves closer to x=0 (the minimum). Step 4: Gradient descent on a 2D function def f_2d(point): x, y = point AIENGINEERINGFROMSCRATCH.COM 102

CALCULUS FOR MACHINE LEARNING return x**2 + y**2 point = [4.0, 3.0] lr = 0.1 for step in range(30): grad = numerical_gradient(f_2d, point) point = [p - lr * g for p, g in zip(point, grad)] loss = f_2d(point) if step % 5 == 0 or step == 29: print(f"step {step:2d} point=({point[0]:7.4f}, {point[1]:7.4f}) f={loss:.6f}")↪ Step 5: Comparing numerical and analytical derivatives import math test_functions = [ ("x^2", lambda x: x**2, lambda x: 2*x), ("x^3", lambda x: x**3, lambda x: 3*x**2), ("sin(x)", lambda x: math.sin(x), lambda x: math.cos(x)), ("e^x", lambda x: math.exp(x), lambda x: math.exp(x)), ("1/x", lambda x: 1/x, lambda x: -1/x**2), ] x = 2.0 print(f"{'Function':<12} {'Numerical':>12} {'Analytical':>12} {'Error':>12}") print("-" * 50) for name, f, df in test_functions: num = numerical_derivative(f, x) ana = df(x) err = abs(num - ana) print(f"{name:<12} {num:12.6f} {ana:12.6f} {err:12.2e}") Step 6: Computing the Hessian numerically def hessian_2d(f, x, y, h=1e-5): fxx = (f(x + h, y) - 2 * f(x, y) + f(x - h, y)) / (h ** 2) fyy = (f(x, y + h) - 2 * f(x, y) + f(x, y - h)) / (h ** 2) fxy = (f(x + h, y + h) - f(x + h, y - h) - f(x - h, y + h) + f(x - h, y - h)) / (4 * h ** 2)↪ return [[fxx, fxy], [fxy, fyy]] def saddle(x, y): return x ** 2 - y ** 2 def bowl(x, y): return x ** 2 + y ** 2 H_saddle = hessian_2d(saddle, 0.0, 0.0) H_bowl = hessian_2d(bowl, 0.0, 0.0) print(f"Saddle Hessian: {H_saddle}") # [[2, 0], [0, -2]] -- mixed signs print(f"Bowl Hessian: {H_bowl}") # [[2, 0], [0, 2]] -- both positive The Hessian of the saddle function has eigenvalues 2 and -2 (mixed signs, confirming a saddle AIENGINEERINGFROMSCRATCH.COM 103

CALCULUS FOR MACHINE LEARNING point). The bowl has eigenvalues 2 and 2 (both positive, confirming a minimum). Step 7: Taylor approximation in action import math def taylor_approx(f, f_prime, f_double_prime, x0, h, order=2): result = f(x0) if order >= 1: result += f_prime(x0) * h if order >= 2: result += 0.5 * f_double_prime(x0) * h ** 2 return result x0 = 0.0 for h in [0.1, 0.5, 1.0, 2.0]: true_val = math.sin(h) t1 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), x0, h, order=1) t2 = taylor_approx(math.sin, math.cos, lambda x: -math.sin(x), x0, h, order=2) print(f"h={h:.1f} sin(h)={true_val:.4f} order1={t1:.4f} order2={t2:.4f}") Near x0=0, sin(x) ~ x (first-order Taylor). The approximation is excellent for small h but breaks down for large h. This is why gradient descent works best with small learning rates – each step assumes the linear approximation is accurate. Step 8: Why this matters for a neural network import random random.seed(42) w = random.gauss(0, 1) b = random.gauss(0, 1) lr = 0.01 xs = [1.0, 2.0, 3.0, 4.0, 5.0] ys = [3.0, 5.0, 7.0, 9.0, 11.0] for epoch in range(200): total_loss = 0 dw = 0 db = 0 for x, y in zip(xs, ys): pred = w * x + b error = pred - y total_loss += error ** 2 dw += 2 * error * x db += 2 * error dw /= len(xs) db /= len(xs) total_loss /= len(xs) w -= lr * dw b -= lr * db if epoch % 40 == 0 or epoch == 199: AIENGINEERINGFROMSCRATCH.COM 104

CALCULUS FOR MACHINE LEARNING print(f"epoch {epoch:3d} w={w:.4f} b={b:.4f} loss={total_loss:.6f}") print(f"\nLearned: y = {w:.2f}x + {b:.2f}") print(f"Actual: y = 2x + 1") Every gradient-based training loop follows this pattern: predict, compute loss, compute gra- dients, update weights. Use It With NumPy, the same operations are faster and more concise: import numpy as np x = np.array([1, 2, 3, 4, 5], dtype=float) y = np.array([3, 5, 7, 9, 11], dtype=float) w, b = np.random.randn(), np.random.randn() lr = 0.01 for epoch in range(200): pred = w * x + b error = pred - y loss = np.mean(error ** 2) dw = np.mean(2 * error * x) db = np.mean(2 * error) w -= lr * dw b -= lr * db print(f"Learned: y = {w:.2f}x + {b:.2f}") You just built gradient descent from scratch. PyTorch automates the gradient computation, but the update loop is identical. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/04-calculus-for-ml/code 1. Implement numerical_second_derivative(f, x) using numerical_derivative called twice. Verify that the second derivative of x^3 at x=2 is 12. 2. Use gradient descent to find the minimum of f(x, y) = (x - 3)^2 + (y + 1)^2. Start from (0, 0). The answer should converge to (3, -1). 3. Add momentum to the gradient descent loop: maintain a velocity vector that accumulates past gradients. Compare convergence speed with and without momentum on f(x) = x^4 - 3x^2. Key Terms AIENGINEERINGFROMSCRATCH.COM 105

CALCULUS FOR MACHINE LEARNING Term What people say What it actually means Derivative “The slope” The rate of change of a function at a point. Tells you how much the output changes per unit change in input. Partial derivative “Derivative of one variable” The derivative with respect to one variable while all others are held constant. Gradient “Direction of steepest ascent” A vector of all partial derivatives. Points in the direction that increases the function fastest. Gradient descent “Go downhill” Subtract the gradient (times a learning rate) from the parameters to reduce the loss. The core of neural network training. Learning rate “Step size” A scalar that controls how big each gradient descent step is. Too large: diverge. Too small: converge slowly. Chain rule “Multiply the derivatives” The rule for differentiating composed functions: df/dx = df/dg * dg/dx. The mathematical basis of backpropagation. Jacobian “Matrix of derivatives” When a function maps vectors to vectors, the Jacobian is the matrix of all partial derivatives of outputs with respect to inputs. Numerical derivative “Finite differences” Approximating a derivative by evaluating the function at two nearby points and computing the slope between them. Backpropagation“Reverse-mode autodiff” Computing gradients layer by layer from output to input using the chain rule. How neural networks learn. Hessian “Matrix of second derivatives” The matrix of all second-order partial derivatives. Describes the curvature of a function. Positive definite Hessian at a critical point means local minimum. Taylor series “Polynomial approximation” Approximating a function near a point using its derivatives: f(x+h) ~ f(x) + f’(x)h + (1/2)f’’(x)h^2 + … The basis for understanding why gradient descent and Newton’s method work. Integral “Area under the curve” The accumulation of a quantity over a range. In ML, integrals define probabilities, expected values, and KL divergence. Further Reading • 3Blue1Brown: Essence of Calculus - visual intuition for derivatives, integrals, and the chain rule • Stanford CS231n: Backpropagation - how gradients flow through neural network layers Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/04-calculus-for-ml • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/04-calculus-for-ml/code AIENGINEERINGFROMSCRATCH.COM 106

CALCULUS FOR MACHINE LEARNING • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/04-calculus-for-ml The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 107

Chain Rule & Automatic Differentiation The chain rule is the engine behind every neural network that learns. Type: Build Language: Python Prerequisites: Phase 1, Lesson 04 (Derivatives & Gradients) Time: ~90 minutes Learning Objectives • Build a minimal autograd engine (Value class) that records operations and computes gradients via reverse-mode autodiff • Implement forward and backward passes through a computation graph using topological sort • Construct and train a multi-layer perceptron on XOR using only the from-scratch auto- grad engine • Verify autodiff correctness using gradient checking against numerical finite differences The Problem You can compute derivatives of simple functions. But a neural network is not a simple function. It is hundreds of functions composed together: matrix multiply, add bias, apply activation, matrix multiply again, softmax, cross-entropy loss. The output is a function of a function of a function. To train the network, you need the gradient of the loss with respect to every single weight. Doing this by hand is impossible for millions of parameters. Doing it numerically (finite dif- ferences) is too slow. The chain rule gives you the math. Automatic differentiation gives you the algorithm. To- gether they let you compute exact gradients through arbitrary compositions of functions in time proportional to a single forward pass. This is how PyTorch, TensorFlow, and JAX work. You will build a miniature version from scratch. The Concept The Chain Rule If y = f(g(x)), the derivative of y with respect to x is: dy/dx = dy/dg * dg/dx = f'(g(x)) * g'(x) Multiply the derivatives along the chain. Each link contributes its local derivative. Example: y = sin(x^2) AIENGINEERINGFROMSCRATCH.COM 108

CHAIN RULE & AUTOMATIC DIFFERENTIATION g(x) = x^2 g'(x) = 2x f(g) = sin(g) f'(g) = cos(g) dy/dx = cos(x^2) * 2x For deeper compositions, the chain extends: y = f(g(h(x))) dy/dx = f'(g(h(x))) * g'(h(x)) * h'(x) Every layer in a neural network is one link in this chain. Computational Graphs A computational graph makes the chain rule visual. Every operation becomes a node. Data flows forward through the graph. Gradients flow backward. Forward pass (compute values): Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/05-chain-rule-and-autodiff Backward pass (compute gradients): Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/05-chain-rule-and-autodiff The backward pass applies the chain rule at every node, propagating gradients from output to inputs. Forward Mode vs Reverse Mode There are two ways to apply the chain rule through a graph. Forward mode starts at the inputs and pushes derivatives forward. It computes dx/dx = 1 and propagates through each operation. Good when you have few inputs and many outputs. Forward mode: seed dx/dx = 1, propagate forward x = 2 (dx/dx = 1) a = x^2 (da/dx = 2x = 4) y = sin(a) (dy/dx = cos(a) * da/dx = cos(4) * 4 = -2.615) Reverse mode starts at the output and pulls gradients backward. It computes dy/dy = 1 and propagates through each operation in reverse. Good when you have many inputs and few outputs. Reverse mode: seed dy/dy = 1, propagate backward y = sin(a) (dy/dy = 1) a = x^2 (dy/da = cos(a) = cos(4) = -0.654) x = 2 (dy/dx = dy/da * da/dx = -0.654 * 4 = -2.615) Neural networks have millions of inputs (weights) and one output (loss). Reverse mode com- putes all gradients in one backward pass. This is why backpropagation uses reverse mode. Mode Seed Direction Best when Forward dx_i/dx_i = 1 Input to output Few inputs, many outputs AIENGINEERINGFROMSCRATCH.COM 109

CHAIN RULE & AUTOMATIC DIFFERENTIATION Mode Seed Direction Best when Reverse dy/dy = 1 Output to input Many inputs, few outputs (neural nets) Dual Numbers for Forward Mode Forward mode can be implemented elegantly with dual numbers. A dual number has the form a + b*epsilon where epsilon^2 = 0. Dual number: (value, derivative) (2, 1) means: value is 2, derivative w.r.t. x is 1 Arithmetic rules: (a, a') + (b, b') = (a+b, a'+b') (a, a') * (b, b') = (a*b, a'*b + a*b') sin(a, a') = (sin(a), cos(a)*a') Seed the input variable with derivative 1. The derivative propagates automatically through every operation. Building an Autograd Engine An autograd engine needs three things: 1. Value wrapping. Wrap every number in an object that stores its value and gradient. 2. Graph recording. Every operation records its inputs and the local gradient function. 3. Backward pass. Topological sort the graph, then walk it in reverse, applying the chain rule at each node. This is exactly what PyTorch’s autograd does. The torch.Tensor class wraps values, records operations when requires_grad=True, and computes gradients when you call .backward(). How PyTorch Autograd Works Under the Hood When you write PyTorch code: x = torch.tensor(2.0, requires_grad=True) y = x ** 2 + 3 * x + 1 y.backward() print(x.grad) # 7.0 = 2*x + 3 = 2*2 + 3 PyTorch internally: 1. Creates a Tensor node for x with requires_grad=True 2. Every operation (**, *, +) creates a new node and records the backward function 3. y.backward() triggers reverse-mode autodiff through the recorded graph 4. Each node’s grad_fn computes local gradients and passes them to parent nodes 5. Gradients accumulate in .grad attributes via addition (not replacement) The graph is dynamic (define-by-run). A new graph is built on every forward pass. This is why PyTorch supports control flow (if/else, loops) inside models. Interactive figure: chain-rule. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01-math- foundations/05-chain-rule-and-autodiff AIENGINEERINGFROMSCRATCH.COM 110

CHAIN RULE & AUTOMATIC DIFFERENTIATION Build It Step 1: The Value class class Value: def __init__(self, data, children=(), op=''): self.data = data self.grad = 0.0 self._backward = lambda: None self._prev = set(children) self._op = op def __repr__(self): return f"Value(data={self.data:.4f}, grad={self.grad:.4f})" Every Value stores its numeric data, its gradient (initially zero), a backward function, and pointers to child nodes that produced it. Step 2: Arithmetic operations with gradient tracking def __add__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data + other.data, (self, other), '+') def _backward(): self.grad += out.grad other.grad += out.grad out._backward = _backward return out def __mul__(self, other): other = other if isinstance(other, Value) else Value(other) out = Value(self.data * other.data, (self, other), '*') def _backward(): self.grad += other.data * out.grad other.grad += self.data * out.grad out._backward = _backward return out def relu(self): out = Value(max(0, self.data), (self,), 'relu') def _backward(): self.grad += (1.0 if out.data > 0 else 0.0) * out.grad out._backward = _backward return out Each operation creates a closure that knows how to compute local gradients and multiply by the upstream gradient (out.grad). The += handles the case where a value is used in multiple operations. Step 3: The backward pass def backward(self): topo = [] visited = set() AIENGINEERINGFROMSCRATCH.COM 111

CHAIN RULE & AUTOMATIC DIFFERENTIATION def build_topo(v): if v not in visited: visited.add(v) for child in v._prev: build_topo(child) topo.append(v) build_topo(self) self.grad = 1.0 for v in reversed(topo): v._backward() Topological sort ensures every node’s gradient is fully computed before it propagates to its children. The seed gradient is 1.0 (dy/dy = 1). Step 4: More operations for a complete engine The basic Value class handles addition, multiplication, and relu. A real autograd engine needs more. Here are the operations you need to build neural networks: def __neg__(self): return self * -1 def __sub__(self, other): return self + (-other) def __radd__(self, other): return self + other def __rmul__(self, other): return self * other def __rsub__(self, other): return other + (-self) def __pow__(self, n): out = Value(self.data ** n, (self,), f'**{n}') def _backward(): self.grad += n * (self.data ** (n - 1)) * out.grad out._backward = _backward return out def __truediv__(self, other): return self * (other ** -1) if isinstance(other, Value) else self * (Value(other) ** -1)↪ def exp(self): import math e = math.exp(self.data) out = Value(e, (self,), 'exp') def _backward(): self.grad += e * out.grad out._backward = _backward return out AIENGINEERINGFROMSCRATCH.COM 112

CHAIN RULE & AUTOMATIC DIFFERENTIATION def log(self): import math out = Value(math.log(self.data), (self,), 'log') def _backward(): self.grad += (1.0 / self.data) * out.grad out._backward = _backward return out def tanh(self): import math t = math.tanh(self.data) out = Value(t, (self,), 'tanh') def _backward(): self.grad += (1 - t ** 2) * out.grad out._backward = _backward return out Why each operation matters: Operation Backward rule Used in __sub__ Reuses add + neg Loss computation (pred - target) __pow__ n * x^(n-1) Polynomial activations, MSE (error^2) __truediv__ Reuses mul + pow(-1) Normalization, learning rate scaling exp exp(x) * upstream Softmax, log-likelihood log (1/x) * upstream Cross-entropy loss, log probabilities tanh (1 - tanh^2) * upstream Classic activation function The clever part: __sub__ and __truediv__ are defined in terms of existing operations. They get correct gradients for free because the chain rule composes through the underlying add/mul/pow operations. Step 5: Mini MLP from scratch With a complete Value class, you can build a neural network. No PyTorch. No NumPy. Just Values and the chain rule. import random class Neuron: def __init__(self, n_inputs): self.w = [Value(random.uniform(-1, 1)) for _ in range(n_inputs)] self.b = Value(0.0) def __call__(self, x): act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b) return act.tanh() def parameters(self): AIENGINEERINGFROMSCRATCH.COM 113

CHAIN RULE & AUTOMATIC DIFFERENTIATION return self.w + [self.b] class Layer: def __init__(self, n_inputs, n_outputs): self.neurons = [Neuron(n_inputs) for _ in range(n_outputs)] def __call__(self, x): return [n(x) for n in self.neurons] def parameters(self): return [p for n in self.neurons for p in n.parameters()] class MLP: def __init__(self, sizes): self.layers = [Layer(sizes[i], sizes[i+1]) for i in range(len(sizes)-1)] def __call__(self, x): for layer in self.layers: x = layer(x) return x[0] if len(x) == 1 else x def parameters(self): return [p for layer in self.layers for p in layer.parameters()] A Neuron computes tanh(w1*x1 + w2*x2 + ... + b). A Layer is a list of neurons. An MLP stacks layers. Every weight is a Value, so calling loss.backward() propagates gradients to every parameter. Training on XOR: random.seed(42) model = MLP([2, 4, 1]) # 2 inputs, 4 hidden neurons, 1 output xs = [[0, 0], [0, 1], [1, 0], [1, 1]] ys = [-1, 1, 1, -1] # XOR pattern (using -1/1 for tanh) for step in range(100): preds = [model(x) for x in xs] loss = sum((p - y) ** 2 for p, y in zip(preds, ys)) for p in model.parameters(): p.grad = 0.0 loss.backward() lr = 0.05 for p in model.parameters(): p.data -= lr * p.grad if step % 20 == 0: print(f"step {step:3d} loss = {loss.data:.4f}") print("\nPredictions after training:") for x, y in zip(xs, ys): print(f" input={x} target={y:2d} pred={model(x).data:6.3f}") This is micrograd. A complete neural network training loop in pure Python with automatic AIENGINEERINGFROMSCRATCH.COM 114

CHAIN RULE & AUTOMATIC DIFFERENTIATION differentiation. Every commercial deep learning framework does the same thing at massive scale. Step 6: Gradient checking How do you know your autodiff is correct? Compare it against numerical derivatives. This is gradient checking. def gradient_check(build_expr, x_val, h=1e-7): x = Value(x_val) y = build_expr(x) y.backward() autodiff_grad = x.grad y_plus = build_expr(Value(x_val + h)).data y_minus = build_expr(Value(x_val - h)).data numerical_grad = (y_plus - y_minus) / (2 * h) diff = abs(autodiff_grad - numerical_grad) return autodiff_grad, numerical_grad, diff Test it on a complex expression: def expr(x): return (x ** 3 + x * 2 + 1).tanh() ad, num, diff = gradient_check(expr, 0.5) print(f"Autodiff: {ad:.8f}") print(f"Numerical: {num:.8f}") print(f"Difference: {diff:.2e}") # Difference should be < 1e-5 Gradient checking is essential when implementing new operations. If your backward pass has a bug, the numerical check catches it. Every serious deep learning implementation runs gradient checks during development. When to use gradient checking: Situation Do gradient check? Adding a new operation to your autograd Yes, always Debugging a training loop that won’t converge Yes, check gradients first Production training No, too slow (2x forward passes per parameter) Unit tests for autograd code Yes, automate it Step 7: Verify against manual calculation x1 = Value(2.0) x2 = Value(3.0) a = x1 * x2 # a = 6.0 b = a + Value(1.0) # b = 7.0 y = b.relu() # y = 7.0 y.backward() AIENGINEERINGFROMSCRATCH.COM 115

CHAIN RULE & AUTOMATIC DIFFERENTIATION print(f"y = {y.data}") # 7.0 print(f"dy/dx1 = {x1.grad}") # 3.0 (= x2) print(f"dy/dx2 = {x2.grad}") # 2.0 (= x1) Manual check: y = relu(x1*x2 + 1). Since x1*x2 + 1 = 7 > 0, relu is identity. dy/dx1 = x2 = 3. dy/dx2 = x1 = 2. The engine matches. Use It Verify against PyTorch import torch x1 = torch.tensor(2.0, requires_grad=True) x2 = torch.tensor(3.0, requires_grad=True) a = x1 * x2 b = a + 1.0 y = torch.relu(b) y.backward() print(f"PyTorch dy/dx1 = {x1.grad.item()}") # 3.0 print(f"PyTorch dy/dx2 = {x2.grad.item()}") # 2.0 Same gradients. Your engine computes the same result as PyTorch because the math is the same: reverse-mode autodiff via the chain rule. A more complex expression a = Value(2.0) b = Value(-3.0) c = Value(10.0) f = (a * b + c).relu() # relu(2*(-3) + 10) = relu(4) = 4 f.backward() print(f"df/da = {a.grad}") # -3.0 (= b) print(f"df/db = {b.grad}") # 2.0 (= a) print(f"df/dc = {c.grad}") # 1.0 This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/05-chain-rule-and-autodiff Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engine ering-from-scratch/tree/main/phases/01-math-foundations/05-chain-rule-and-autodiff/code 1. Add __pow__ to the Value class so you can compute x ** n. Verify that d/dx(x^3) at x=2 equals 12.0. 2. Add tanh as an activation function. Verify that tanh'(0) = 1 and tanh'(2) = 0.0707 (approx). AIENGINEERINGFROMSCRATCH.COM 116

CHAIN RULE & AUTOMATIC DIFFERENTIATION 3. Build a computation graph for a single neuron: y = relu(w1*x1 + w2*x2 + b). Compute all five gradients and verify against PyTorch. 4. Implement forward-mode autodiff using dual numbers. Create a Dual class and verify it gives the same derivatives as your reverse-mode engine. Key Terms Term What people say What it actually means Chain rule “Multiply the derivatives” The derivative of composed functions equals the product of each function’s local derivative, evaluated at the right point Computational graph “The network diagram” A directed acyclic graph where nodes are operations and edges carry values (forward) or gradients (backward) Forward mode “Push derivatives forward” Autodiff that propagates derivatives from inputs to outputs. One pass per input variable. Reverse mode “Backpropagation” Autodiff that propagates gradients from outputs to inputs. One pass per output variable. Autograd “Automatic gradients” A system that records operations on values, builds a graph, and computes exact gradients via the chain rule Dual numbers “Value plus derivative” Numbers of the form a + b*epsilon (epsilon^2 = 0) that carry derivative information through arithmetic Topological sort “Dependency order” Ordering graph nodes so every node comes after all its dependencies. Required for correct gradient propagation. Gradient ac- cumulation “Add, don’t replace” When a value feeds into multiple operations, its gradient is the sum of all incoming gradient contributions Dynamic graph “Define by run” A computation graph rebuilt on every forward pass, allowing Python control flow inside models (PyTorch style) Gradient checking “Numerical verification” Comparing autodiff gradients against numerical finite-difference gradients to verify correctness. Essential for debugging. MLP “Multi-layer perceptron” A neural network with one or more hidden layers of neurons. Each neuron computes a weighted sum plus bias, then applies an activation function. Neuron “Weighted sum + activation” The basic unit: output = activation(w1x1 + w2x2 + … + b). The weights and bias are learnable parameters. Further Reading • 3Blue1Brown: Backpropagation calculus – visual explanation of the chain rule in neural networks AIENGINEERINGFROMSCRATCH.COM 117

CHAIN RULE & AUTOMATIC DIFFERENTIATION • PyTorch Autograd mechanics – how the real system works • Baydin et al., Automatic Differentiation in Machine Learning: a Survey – comprehensive reference Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/05-chain-rule-and-autodiff • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/05-chain-rule-and-autodiff/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/05-chain-rule-and-autodiff The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 118

Probability and Distributions Probability is the language AI uses to express uncertainty. Type: Learn Language: Python Prerequisites: Phase 1, Lessons 01-04 Time: ~75 minutes Learning Objectives • Implement PMFs and PDFs from scratch for Bernoulli, categorical, Poisson, uniform, and normal distributions • Compute expected value, variance, and use the Central Limit Theorem to explain why Gaussians dominate • Build softmax and log-softmax functions with the numerical stability trick (subtract max logit) • Calculate cross-entropy loss from logits and connect it to negative log-likelihood The Problem A classifier outputs [0.03, 0.91, 0.06]. A language model picks the next word from 50,000 candidates. A diffusion model generates images by sampling from learned distributions. All of these are probability in action. Every prediction a model makes is a probability distribution. Every loss function measures how far the predicted distribution is from the true one. Every training step adjusts parameters to make one distribution look more like another. Without probability, you cannot read a single ML paper, debug a single model, or understand why your training loss is NaN. The Concept Events, Sample Spaces, and Probability The sample space S is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1. Coin flip: S = {H, T} P(H) = 0.5, P(T) = 0.5 Single die roll: S = {1, 2, 3, 4, 5, 6} P(even) = P({2, 4, 6}) = 3/6 = 0.5 Three axioms define all of probability: 1. P(A) >= 0 for any event A 2. P(S) = 1 (something always happens) 3. P(A or B) = P(A) + P(B) when A and B cannot both occur Everything else (Bayes’ theorem, expectations, distributions) follows from these three rules. AIENGINEERINGFROMSCRATCH.COM 119

PROBABILITY AND DISTRIBUTIONS Conditional Probability and Independence P(A|B) is the probability of A given that B happened. P(A|B) = P(A and B) / P(B) Example: deck of cards P(King | Face card) = P(King and Face card) / P(Face card) = (4/52) / (12/52) = 4/12 = 1/3 Two events are independent when knowing one tells you nothing about the other: Independent: P(A|B) = P(A) Equivalent to: P(A and B) = P(A) * P(B) Coin flips are independent. Drawing cards without replacement is not. Probability Mass Functions vs Probability Density Functions Discrete random variables have a probability mass function (PMF). Each outcome has a spe- cific probability that you can read off directly. PMF: P(X = k) Fair die: P(X = 1) = 1/6 P(X = 2) = 1/6 ... P(X = 6) = 1/6 Sum of all probabilities = 1 Continuous random variables have a probability density function (PDF). The density at a single point is not a probability. Probability comes from integrating the density over an interval. PDF: f(x) P(a <= X <= b) = integral of f(x) from a to b f(x) can be greater than 1 (density, not probability) integral from -inf to +inf of f(x) dx = 1 This distinction matters in ML. Classification outputs are PMFs (discrete choices). VAE latent spaces use PDFs (continuous). Common Distributions Bernoulli: one trial, two outcomes. Models binary classification. P(X = 1) = p P(X = 0) = 1 - p Mean = p, Variance = p(1-p) Categorical: one trial, k outcomes. Models multi-class classification (softmax output). P(X = i) = p_i, where sum of p_i = 1 Example: P(cat) = 0.7, P(dog) = 0.2, P(bird) = 0.1 Uniform: all outcomes equally likely. Used for random initialization. AIENGINEERINGFROMSCRATCH.COM 120

PROBABILITY AND DISTRIBUTIONS Discrete: P(X = k) = 1/n for k in {1, ..., n} Continuous: f(x) = 1/(b-a) for x in [a, b] Normal (Gaussian): the bell curve. Parameterized by mean (mu) and variance (sigma^2). f(x) = (1 / sqrt(2*pi*sigma^2)) * exp(-(x - mu)^2 / (2*sigma^2)) Standard normal: mu = 0, sigma = 1 68% of data within 1 sigma 95% within 2 sigma 99.7% within 3 sigma Poisson: counts of rare events in a fixed interval. Models event rates. P(X = k) = (lambda^k * e^(-lambda)) / k! Mean = lambda, Variance = lambda Expected Value and Variance Expected value is the weighted average outcome. Discrete: E[X] = sum of x_i * P(X = x_i) Continuous: E[X] = integral of x * f(x) dx Variance measures spread around the mean. Var(X) = E[(X - E[X])^2] = E[X^2] - (E[X])^2 Standard deviation = sqrt(Var(X)) In ML, expected value appears as the loss function (average loss over the data distribution). Variance tells you about model stability. High variance in gradients means noisy training. Joint and Marginal Distributions A joint distribution P(X, Y) describes two random variables together. Joint PMF example (X = weather, Y = umbrella): Y=0 (no umbrella) Y=1 (umbrella) Marginal P(X) X=0 (sun) 0.40 0.10 P(X=0) = 0.50 X=1 (rain) 0.05 0.45 P(X=1) = 0.50 Marginal P(Y) P(Y=0) = 0.45 P(Y=1) = 0.55 1.00 The marginal distribution sums out the other variable: P(X = x) = sum over all y of P(X = x, Y = y) The row and column totals in the table above are the marginals. Why the Normal Distribution Shows Up Everywhere The Central Limit Theorem: the sum (or average) of many independent random variables converges to a normal distribution, regardless of the original distribution. Roll 1 die: uniform distribution (flat) Average of 2 dice: triangular (peaked) Average of 30 dice: nearly perfect bell curve AIENGINEERINGFROMSCRATCH.COM 121

PROBABILITY AND DISTRIBUTIONS This works for ANY starting distribution. This is why: - Measurement errors are approximately normal (many small independent sources) - Weight initializations in neural networks use normal distributions - Gradient noise in SGD is approximately normal (sum of many sample gradients) - The normal distribution is the maximum entropy distribution for a given mean and variance Log Probabilities Raw probabilities cause numerical problems. Multiplying many small probabilities together quickly underflows to zero. P(sentence) = P(word1) * P(word2) * ... * P(word_n) = 0.01 * 0.003 * 0.02 * ... -> 0.0 (underflow after ~30 terms) Log probabilities fix this. Multiplications become additions. log P(sentence) = log P(word1) + log P(word2) + ... + log P(word_n) = -4.6 + -5.8 + -3.9 + ... -> finite number (no underflow) Rules: - log(a * b) = log(a) + log(b) - log probabilities are always <= 0 (since 0 < P <= 1) - More negative = less likely - Cross-entropy loss is the negative log probability of the correct class Softmax as a Probability Distribution Neural networks output raw scores (logits). Softmax converts them into a valid probability distribution. softmax(z_i) = exp(z_i) / sum(exp(z_j) for all j) Properties: - All outputs are in (0, 1) - All outputs sum to 1 - Preserves relative ordering of inputs - exp() amplifies differences between logits The softmax trick: subtract the max logit before exponentiating to prevent overflow. z = [100, 101, 102] exp(102) = overflow z_shifted = z - max(z) = [-2, -1, 0] exp(0) = 1 (safe) Same result, no overflow. Log-softmax combines softmax and log for numerical stability. PyTorch uses this internally for cross-entropy loss. Sampling Sampling means drawing random values from a distribution. In ML: - Dropout randomly samples which neurons to zero out - Data augmentation samples random transformations - AIENGINEERINGFROMSCRATCH.COM 122

PROBABILITY AND DISTRIBUTIONS Language models sample the next token from the predicted distribution - Diffusion models sample noise and progressively denoise Sampling from arbitrary distributions requires techniques like inverse transform sampling, rejection sampling, or the reparameterization trick (used in VAEs). Interactive figure: gaussian-pdf. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01-math- foundations/06-probability-and-distributions Build It Step 1: Probability basics import math import random def factorial(n): result = 1 for i in range(2, n + 1): result *= i return result def combinations(n, k): return factorial(n) // (factorial(k) * factorial(n - k)) def conditional_probability(p_a_and_b, p_b): return p_a_and_b / p_b p_king_given_face = conditional_probability(4/52, 12/52) print(f"P(King | Face card) = {p_king_given_face:.4f}") Step 2: PMF and PDF from scratch def bernoulli_pmf(k, p): return p if k == 1 else (1 - p) def categorical_pmf(k, probs): return probs[k] def poisson_pmf(k, lam): return (lam ** k) * math.exp(-lam) / factorial(k) def uniform_pdf(x, a, b): if a <= x <= b: return 1.0 / (b - a) return 0.0 def normal_pdf(x, mu, sigma): coeff = 1.0 / (sigma * math.sqrt(2 * math.pi)) exponent = -0.5 * ((x - mu) / sigma) ** 2 return coeff * math.exp(exponent) AIENGINEERINGFROMSCRATCH.COM 123

PROBABILITY AND DISTRIBUTIONS Step 3: Expected value and variance def expected_value(values, probabilities): return sum(v * p for v, p in zip(values, probabilities)) def variance(values, probabilities): mu = expected_value(values, probabilities) return sum(p * (v - mu) ** 2 for v, p in zip(values, probabilities)) die_values = [1, 2, 3, 4, 5, 6] die_probs = [1/6] * 6 mu = expected_value(die_values, die_probs) var = variance(die_values, die_probs) print(f"Die: E[X] = {mu:.4f}, Var(X) = {var:.4f}, SD = {var**0.5:.4f}") Step 4: Sampling from distributions def sample_bernoulli(p, n=1): return [1 if random.random() < p else 0 for _ in range(n)] def sample_categorical(probs, n=1): cumulative = [] total = 0 for p in probs: total += p cumulative.append(total) samples = [] for _ in range(n): r = random.random() for i, c in enumerate(cumulative): if r <= c: samples.append(i) break return samples def sample_normal_box_muller(mu, sigma, n=1): samples = [] for _ in range(n): u1 = random.random() u2 = random.random() z = math.sqrt(-2 * math.log(u1)) * math.cos(2 * math.pi * u2) samples.append(mu + sigma * z) return samples Step 5: Softmax and log probabilities def softmax(logits): max_logit = max(logits) shifted = [z - max_logit for z in logits] exps = [math.exp(z) for z in shifted] total = sum(exps) return [e / total for e in exps] def log_softmax(logits): AIENGINEERINGFROMSCRATCH.COM 124

PROBABILITY AND DISTRIBUTIONS max_logit = max(logits) shifted = [z - max_logit for z in logits] log_sum_exp = max_logit + math.log(sum(math.exp(z) for z in shifted)) return [z - log_sum_exp for z in logits] def cross_entropy_loss(logits, target_index): log_probs = log_softmax(logits) return -log_probs[target_index] Step 6: Central Limit Theorem demonstration def demonstrate_clt(dist_fn, n_samples, n_averages): averages = [] for _ in range(n_averages): samples = [dist_fn() for _ in range(n_samples)] averages.append(sum(samples) / len(samples)) return averages Step 7: Visualization import matplotlib.pyplot as plt xs = [mu + sigma * (i - 500) / 100 for i in range(1001)] ys = [normal_pdf(x, mu, sigma) for x, mu, sigma in ...] plt.plot(xs, ys) Full implementations with all visualizations are in code/probability.py. Use It With NumPy and SciPy, everything above is one-liners: import numpy as np from scipy import stats normal = stats.norm(loc=0, scale=1) samples = normal.rvs(size=10000) print(f"Mean: {np.mean(samples):.4f}, Std: {np.std(samples):.4f}") print(f"P(X < 1.96) = {normal.cdf(1.96):.4f}") logits = np.array([2.0, 1.0, 0.1]) from scipy.special import softmax, log_softmax probs = softmax(logits) log_probs = log_softmax(logits) print(f"Softmax: {probs}") print(f"Log-softmax: {log_probs}") You built these from scratch. Now you know what the library calls are doing. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/06-probability-and- AIENGINEERINGFROMSCRATCH.COM 125

PROBABILITY AND DISTRIBUTIONS distributions/code 1. Implement inverse transform sampling for the exponential distribution. Verify by sam- pling 10,000 values and comparing the histogram to the true PDF. 2. Build a joint distribution table for two loaded dice. Compute the marginal distributions and check whether the dice are independent. 3. Compute the cross-entropy loss for a 5-class classifier that outputs logits [2.0, 0.5, -1 .0, 3.0, 0.1] when the correct class is index 3. Then verify your answer with PyTorch’s nn.CrossEntropyLoss. 4. Write a function that takes a list of log probabilities and returns the most likely sequence, the total log probability, and the equivalent raw probability. Test it with a sentence of 50 words where each word has probability 0.01. Key Terms Term What people say What it actually means Sample space “All the possibilities” The set S of every possible outcome of an experiment PMF “The probability function” A function that gives the exact probability of each discrete outcome, summing to 1 PDF “The probability curve” A density function for continuous variables. Integrate it over an interval to get probability Conditional probability “Probability given something” P(A|B) = P(A and B) / P(B). The foundation of Bayesian thinking and Bayes’ theorem Independence “They don’t affect each other” P(A and B) = P(A) * P(B). Knowing one event tells you nothing about the other Expected value “The average” The probability-weighted sum of all outcomes. The loss function is an expected value Variance “How spread out” The expected squared deviation from the mean. High variance = noisy, unstable estimates Normal distribution “The bell curve” f(x) = (1/sqrt(2pisigma^2)) * exp(-(x-mu)2/(2*sigma2)). Appears everywhere due to the CLT Central Limit Theorem “Averages become normal” The mean of many independent samples converges to a normal distribution regardless of the source Joint distribution “Two variables together” P(X, Y) describes the probability of every combination of X and Y outcomes Marginal distribution “Sum out the other variable” P(X) = sum_y P(X, Y). Recovers one variable’s distribution from the joint Log probability “Log of the probability” log P(x). Turns products into sums, preventing numerical underflow in long sequences Softmax “Turn scores into probabilities” softmax(z_i) = exp(z_i) / sum(exp(z_j)). Maps real-valued logits to a valid probability distribution AIENGINEERINGFROMSCRATCH.COM 126

PROBABILITY AND DISTRIBUTIONS Term What people say What it actually means Cross- entropy “The loss function” -sum(p_true * log(p_predicted)). Measures how different two distributions are. Lower is better Logits “Raw model outputs” Unnormalized scores before softmax. Named after the logistic function Sampling “Drawing random values” Generating values according to a probability distribution. How models generate output Further Reading • 3Blue1Brown: But what is the Central Limit Theorem? - visual proof of why averages become normal • Stanford CS229 Probability Review - concise reference covering everything here and more • The Log-Sum-Exp Trick - why numerical stability matters and how to achieve it Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/06-probability-and-distributions • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/06-probability-and-distributions/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/06-probability-and-distributions The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 127

Bayes’ Theorem Probability is about what you expect. Bayes’ theorem is about what you learn. Type: Build Language: Python Prerequisites: Phase 1, Lesson 06 (Probability Fundamen- tals) Time: ~75 minutes Learning Objectives • Apply Bayes’ theorem to compute posterior probabilities from priors, likelihoods, and evidence • Build a Naive Bayes text classifier from scratch with Laplace smoothing and log-space computation • Compare MLE and MAP estimation and explain how MAP corresponds to L2 regulariza- tion • Implement sequential Bayesian updating using Beta-Binomial conjugate priors for A/B testing The Problem A medical test is 99% accurate. You test positive. What are the chances you actually have the disease? Most people say 99%. The real answer depends on how rare the disease is. If 1 in 10,000 people have it, a positive result only gives you about a 1% chance of being sick. The other 99% of positive results are false alarms from healthy people. This is not a trick question. It is Bayes’ theorem. Every spam filter, every medical diagnostic, every machine learning model that quantifies uncertainty uses this exact reasoning. You start with a belief. You see evidence. You update. If you build ML systems without understanding this, you will misinterpret model outputs, set bad thresholds, and ship overconfident predictions. The Concept From joint probability to Bayes You already know from Lesson 06 that conditional probability is: P(A|B) = P(A and B) / P(B) And symmetrically: P(B|A) = P(A and B) / P(A) Both expressions share the same numerator: P(A and B). Set them equal and rearrange: AIENGINEERINGFROMSCRATCH.COM 128

BAYES’ THEOREM P(A and B) = P(A|B) * P(B) = P(B|A) * P(A) Therefore: P(A|B) = P(B|A) * P(A) / P(B) That is Bayes’ theorem. Four quantities, one equation. The four parts Part Name What it means P(A|B) Posterior Your updated belief about A after seeing evidence B P(B|A) Likelihood How probable the evidence B is if A is true P(A) Prior Your belief about A before seeing any evidence P(B) Evidence Total probability of seeing B under all possibilities The evidence term P(B) acts as a normalizer. You can expand it using the law of total proba- bility: P(B) = P(B|A) * P(A) + P(B|not A) * P(not A) Medical test example A disease affects 1 in 10,000 people. The test is 99% accurate (catches 99% of sick people, gives false positives 1% of the time). P(sick) = 0.0001 (prior: disease is rare) P(positive|sick) = 0.99 (likelihood: test catches it) P(positive|healthy) = 0.01 (false positive rate) P(positive) = P(positive|sick) * P(sick) + P(positive|healthy) * P(healthy) = 0.99 * 0.0001 + 0.01 * 0.9999 = 0.000099 + 0.009999 = 0.010098 P(sick|positive) = P(positive|sick) * P(sick) / P(positive) = 0.99 * 0.0001 / 0.010098 = 0.0098 = 0.98% Less than 1%. The prior dominates. When a condition is rare, even accurate tests produce mostly false positives. This is why doctors order confirmation tests. Spam filter example You receive an email containing the word “lottery”. Is it spam? P(spam) = 0.3 (30% of email is spam) P("lottery"|spam) = 0.05 (5% of spam emails contain "lottery") P("lottery"|not spam) = 0.001 (0.1% of legitimate emails contain "lottery") P("lottery") = 0.05 * 0.3 + 0.001 * 0.7 AIENGINEERINGFROMSCRATCH.COM 129

BAYES’ THEOREM = 0.015 + 0.0007 = 0.0157 P(spam|"lottery") = 0.05 * 0.3 / 0.0157 = 0.955 = 95.5% One word shifts the probability from 30% to 95.5%. A real spam filter applies Bayes across hundreds of words simultaneously. Naive Bayes: independence assumption Naive Bayes extends this to multiple features by assuming all features are conditionally inde- pendent given the class: P(class | feature_1, feature_2, ..., feature_n) = P(class) * P(feature_1|class) * P(feature_2|class) * ... * P(feature_n|class) / P(feature_1, feature_2, ..., feature_n) The “naive” part is the independence assumption. In text, word occurrences are not inde- pendent (“New” and “York” are correlated). But the assumption works surprisingly well in practice because the classifier only needs to rank classes, not produce calibrated probabili- ties. Since the denominator is the same for all classes, you can skip it and just compare numerators: score(class) = P(class) * product of P(feature_i | class) Pick the class with the highest score. Maximum likelihood estimation (MLE) How do you get P(feature|class) from training data? Count. P("free"|spam) = (number of spam emails containing "free") / (total spam emails) This is MLE: choose the parameter values that make the observed data most likely. You are maximizing the likelihood function, which for discrete counts reduces to relative frequency. Problem: if a word never appears in spam during training, MLE gives it probability zero. One unseen word kills the entire product. Fix this with Laplace smoothing: P(word|class) = (count(word, class) + 1) / (total_words_in_class + vocabulary_size) Adding 1 to every count ensures no probability is ever zero. Maximum a posteriori (MAP) MLE asks: what parameters maximize P(data|parameters)? MAP asks: what parameters maximize P(parameters|data)? By Bayes’ theorem: P(parameters|data) proportional to P(data|parameters) * P(parameters) MAP adds a prior over the parameters themselves. If you believe parameters should be small, you encode that as a prior that penalizes large values. This is identical to L2 regularization in ML. The “ridge” penalty in ridge regression is literally a Gaussian prior on the weights. AIENGINEERINGFROMSCRATCH.COM 130

BAYES’ THEOREM Estimation Optimizes ML equivalent MLE P(data|params) Unregularized training MAP P(data|params) * P(params) L2 / L1 regularization Bayesian vs frequentist: the practical difference Frequentists treat parameters as fixed unknowns. They ask: “If I repeated this experiment many times, what would happen?” Bayesians treat parameters as distributions. They ask: “Given what I have observed, what do I believe about the parameters?” For building ML systems, the practical difference: Aspect Frequentist Bayesian Output Point estimate Distribution over values Uncertainty Confidence intervals (about procedure) Credible intervals (about parameter) Small data Can overfit Prior acts as regularization Computation Usually faster Often requires sampling (MCMC) Most production ML is frequentist (SGD, point estimates). Bayesian methods shine when you need calibrated uncertainty (medical decisions, safety-critical systems) or when data is scarce (few-shot learning, cold start). Why Bayesian thinking matters for ML The connection is deeper than analogy: Priors are regularization. A Gaussian prior on weights is L2 regularization. A Laplace prior is L1. Every time you add a regularization term, you are making a Bayesian statement about what parameter values you expect. Posteriors are uncertainty. A single predicted probability tells you nothing about how confi- dent the model is in that estimate. Bayesian methods give you a distribution: “I think P(spam) is between 0.8 and 0.95.” Bayes updates are online learning. Today’s posterior becomes tomorrow’s prior. When your model sees new data, it updates its beliefs incrementally instead of retraining from scratch. Model comparison is Bayesian. Bayesian information criterion (BIC), marginal likelihood, and Bayes factors all use Bayesian reasoning to choose between models without overfitting. Interactive figure: bayes-update. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01-math- foundations/07-bayes-theorem AIENGINEERINGFROMSCRATCH.COM 131

BAYES’ THEOREM Build It Step 1: Bayes theorem function def bayes(prior, likelihood, false_positive_rate): evidence = likelihood * prior + false_positive_rate * (1 - prior) posterior = likelihood * prior / evidence return posterior result = bayes(prior=0.0001, likelihood=0.99, false_positive_rate=0.01) print(f"P(sick|positive) = {result:.4f}") Step 2: Naive Bayes classifier import math from collections import defaultdict class NaiveBayes: def __init__(self, smoothing=1.0): self.smoothing = smoothing self.class_counts = defaultdict(int) self.word_counts = defaultdict(lambda: defaultdict(int)) self.class_word_totals = defaultdict(int) self.vocab = set() def train(self, documents, labels): for doc, label in zip(documents, labels): self.class_counts[label] += 1 words = doc.lower().split() for word in words: self.word_counts[label][word] += 1 self.class_word_totals[label] += 1 self.vocab.add(word) def predict(self, document): words = document.lower().split() total_docs = sum(self.class_counts.values()) vocab_size = len(self.vocab) best_class = None best_score = float("-inf") for cls in self.class_counts: score = math.log(self.class_counts[cls] / total_docs) for word in words: count = self.word_counts[cls].get(word, 0) total = self.class_word_totals[cls] score += math.log((count + self.smoothing) / (total + self.smoothing * vocab_size))↪ if score > best_score: best_score = score best_class = cls return best_class Log probabilities prevent underflow. Multiplying many small probabilities produces numbers too tiny for floating point. Summing log-probabilities is numerically stable and mathematically AIENGINEERINGFROMSCRATCH.COM 132

BAYES’ THEOREM equivalent. Step 3: Train on spam data train_docs = [ "win free money now", "free lottery ticket winner", "claim your prize today free", "urgent offer free cash", "congratulations you won free", "meeting tomorrow at noon", "project update attached", "can we schedule a call", "quarterly report review", "lunch on thursday sounds good", "team standup notes attached", "please review the pull request", ] train_labels = [ "spam", "spam", "spam", "spam", "spam", "ham", "ham", "ham", "ham", "ham", "ham", "ham", ] classifier = NaiveBayes() classifier.train(train_docs, train_labels) test_messages = [ "free money waiting for you", "meeting rescheduled to friday", "you won a free prize", "please review the attached report", ] for msg in test_messages: print(f" '{msg}' -> {classifier.predict(msg)}") Step 4: Inspect the learned probabilities def show_top_words(classifier, cls, n=5): vocab_size = len(classifier.vocab) total = classifier.class_word_totals[cls] probs = {} for word in classifier.vocab: count = classifier.word_counts[cls].get(word, 0) probs[word] = (count + classifier.smoothing) / (total + classifier.smoothing * vocab_size)↪ sorted_words = sorted(probs.items(), key=lambda x: x[1], reverse=True) for word, prob in sorted_words[:n]: print(f" {word}: {prob:.4f}") print("\nTop spam words:") show_top_words(classifier, "spam") AIENGINEERINGFROMSCRATCH.COM 133

BAYES’ THEOREM print("\nTop ham words:") show_top_words(classifier, "ham") Use It Scikit-learn ships production-ready naive Bayes implementations: from sklearn.feature_extraction.text import CountVectorizer from sklearn.naive_bayes import MultinomialNB from sklearn.metrics import classification_report vectorizer = CountVectorizer() X_train = vectorizer.fit_transform(train_docs) clf = MultinomialNB() clf.fit(X_train, train_labels) X_test = vectorizer.transform(test_messages) predictions = clf.predict(X_test) for msg, pred in zip(test_messages, predictions): print(f" '{msg}' -> {pred}") Same algorithm. CountVectorizer handles tokenization and vocabulary building. Multinomi- alNB handles smoothing and log-probabilities internally. Your from-scratch version does the same thing in 40 lines. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/07-bayes-theorem Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/07-bayes-theorem/code 1. Multiple tests. A patient tests positive twice on independent tests (both 99% accurate, disease prevalence 1 in 10,000). What is P(sick) after both tests? Use the posterior from the first test as the prior for the second. 2. Smoothing impact. Run the spam classifier with smoothing values of 0.01, 0.1, 1.0, and 10.0. How do the top word probabilities change? What happens with smoothing=0 and a word that appears only in ham? 3. Add features. Extend the NaiveBayes class to also use message length (short/long) as a feature alongside word counts. Estimate P(short|spam) and P(short|ham) from the training data and fold it into the prediction score. 4. MAP by hand. Given observed data (7 heads in 10 coin flips), compute the MAP estimate of the bias using a Beta(2,2) prior. Compare it to the MLE estimate (7/10). Key Terms AIENGINEERINGFROMSCRATCH.COM 134

BAYES’ THEOREM Term What people say What it actually means Prior “My initial guess” P(hypothesis) before observing evidence. In ML: the regularization term. Likelihood “How well the data fits” P(evidence|hypothesis). How probable the observed data is under a specific hypothesis. Posterior “My updated belief” P(hypothesis|evidence). The prior multiplied by the likelihood, then normalized. Evidence “The normalizing constant” P(data) across all hypotheses. Ensures the posterior sums to 1. Naive Bayes “That simple text classifier” A classifier that assumes features are independent given the class. Works well despite the false assumption. Laplace smoothing “Add-one smoothing” Adding a small count to every feature to prevent zero probabilities from unseen data. MLE “Just use the frequencies” Choose parameters that maximize P(data|parameters). No prior. Can overfit with small data. MAP “MLE with a prior” Choose parameters that maximize P(data|parameters) * P(parameters). Equivalent to regularized MLE. Log- probability “Work in log space” Using log(P) instead of P to avoid floating-point underflow when multiplying many small numbers. False positive “A wrong alarm” The test says positive, but the true state is negative. Drives the base rate fallacy. Further Reading • 3Blue1Brown: Bayes’ theorem - visual explanation with the medical test example • Stanford CS229: Generative Learning Algorithms - naive Bayes and its connection to discriminative models • Think Bayes - free book, Bayesian statistics with Python code • scikit-learn Naive Bayes - production implementations and when to use each variant Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/07-bayes-theorem • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/07-bayes-theorem/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/07-bayes-theorem The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 135

Optimization Training a neural network is nothing more than finding the bottom of a valley. Type: Build Language: Python Prerequisites: Phase 1, Lessons 04-05 (Derivatives, Gradi- ents) Time: ~75 minutes Learning Objectives • Implement vanilla gradient descent, SGD with momentum, and Adam from scratch • Compare optimizer convergence on the Rosenbrock function and explain why Adam adapts per-weight learning rates • Distinguish convex from non-convex loss landscapes and explain the role of saddle points in high dimensions • Configure learning rate schedules (step decay, cosine annealing, warmup) for training stability The Problem You have a loss function. It tells you how wrong your model is. You have gradients. They tell you which direction makes the loss worse. Now you need a strategy for walking downhill. The naive approach is simple: move opposite the gradient. Scale the step by some number called the learning rate. Repeat. This is gradient descent, and it works. But “works” has caveats. Too large a learning rate and you overshoot the valley entirely, bouncing between walls. Too small and you crawl toward the answer over thousands of unnecessary steps. Hit a saddle point and you stop moving even though you have not found a minimum. Every optimizer in deep learning is an answer to the same question: how do you get to the bottom of the valley faster and more reliably? The Concept What optimization means Optimization is finding the input values that minimize (or maximize) a function. In machine learning, the function is the loss. The inputs are the model’s weights. Training is optimization. minimize L(w) where: L = loss function w = model weights (could be millions of parameters) Gradient descent (vanilla) The simplest optimizer. Compute the gradient of the loss with respect to every weight. Move each weight in the opposite direction of its gradient. Scale the step by the learning rate. AIENGINEERINGFROMSCRATCH.COM 136

OPTIMIZATION w = w - lr * gradient That is the entire algorithm. One line. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/08-optimization Learning rate: the most important hyperparameter The learning rate controls step size. It determines everything about convergence. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/08-optimization There is no formula for the right learning rate. You find it by experiment. Common starting points: 0.001 for Adam, 0.01 for SGD with momentum. SGD vs batch vs mini-batch Vanilla gradient descent computes the gradient over the entire dataset before taking one step. This is called batch gradient descent. It is stable but slow. Stochastic gradient descent (SGD) computes the gradient on a single random sample and steps immediately. It is noisy but fast. Mini-batch gradient descent splits the difference. Compute the gradient over a small batch (32, 64, 128, 256 samples), then step. This is what everyone actually uses. Variant Batch size Gradient quality Speed per step Noise Batch GD Entire dataset Exact Slow None SGD 1 sample Very noisy Fast High Mini-batch 32-256 Good estimate Balanced Moderate The noise in SGD and mini-batch is not a bug. It helps escape shallow local minima and saddle points. Momentum: the ball rolling downhill Vanilla gradient descent only looks at the current gradient. If the gradient zigzags (common in narrow valleys), progress is slow. Momentum fixes this by accumulating past gradients into a velocity term. v = beta * v + gradient w = w - lr * v The analogy: a ball rolling downhill. It does not stop and restart at every bump. It builds speed in consistent directions and dampens oscillations. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/08-optimization beta (typically 0.9) controls how much history to keep. Higher beta means more momentum, smoother paths, but slower response to direction changes. AIENGINEERINGFROMSCRATCH.COM 137

OPTIMIZATION Adam: adaptive learning rates Different weights need different learning rates. A weight that rarely gets large gradients should take bigger steps when it finally does. A weight that gets huge gradients constantly should take smaller steps. Adam (Adaptive Moment Estimation) tracks two things per weight: 1. First moment (m): running average of gradients (like momentum) 2. Second moment (v): running average of squared gradients (gradient magnitude) m = beta1 * m + (1 - beta1) * gradient v = beta2 * v + (1 - beta2) * gradient^2 m_hat = m / (1 - beta1^t) bias correction v_hat = v / (1 - beta2^t) bias correction w = w - lr * m_hat / (sqrt(v_hat) + epsilon) The division by sqrt(v_hat) is the key insight. Weights with large gradients get divided by a large number (small effective step). Weights with small gradients get divided by a small number (large effective step). Each weight gets its own adaptive learning rate. Default hyperparameters: lr=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8. These de- faults work well for most problems. Learning rate schedules A fixed learning rate is a compromise. Early in training, you want large steps to make fast progress. Late in training, you want small steps to fine-tune near the minimum. Common schedules: Schedule Formula Use case Step decay lr = lr * factor every N epochs Simple, manual control Exponential decay lr = lr_0 * decay^t Smooth reduction Cosine annealing lr = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T)) Transformers, modern training Warmup + decay Linear ramp up, then decay Large models, prevents early instability Convex vs non-convex A convex function has one minimum. Gradient descent always finds it. A quadratic like f(x) = x^2 is convex. Neural network loss functions are non-convex. They have many local minima, saddle points, and flat regions. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/08-optimization In practice, local minima in high-dimensional neural networks are rarely a problem. Most local minima have loss values close to the global minimum. Saddle points (flat in some direc- tions, curved in others) are the real obstacle. Momentum and noise from mini-batches help escape them. AIENGINEERINGFROMSCRATCH.COM 138

OPTIMIZATION Loss landscape visualization The loss is a function of all weights. For a model with 1 million weights, the loss landscape lives in 1,000,001-dimensional space. We visualize it by picking two random directions in weight space and plotting the loss along those directions, producing a 2D surface. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/08-optimization Sharp minima generalize poorly. Flat minima generalize well. This is one reason SGD with momentum often outperforms Adam on final test accuracy: its noise prevents settling into sharp minima. Interactive figure: gradient-descent. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/08-optimization Build It Step 1: Define a test function The Rosenbrock function is a classic optimization benchmark. Its minimum is at (1, 1) inside a narrow curved valley that is easy to find but hard to follow. f(x, y) = (1 - x)^2 + 100 * (y - x^2)^2 def rosenbrock(params): x, y = params return (1 - x) ** 2 + 100 * (y - x ** 2) ** 2 def rosenbrock_gradient(params): x, y = params df_dx = -2 * (1 - x) + 200 * (y - x ** 2) * (-2 * x) df_dy = 200 * (y - x ** 2) return [df_dx, df_dy] Step 2: Vanilla gradient descent class GradientDescent: def __init__(self, lr=0.001): self.lr = lr def step(self, params, grads): return [p - self.lr * g for p, g in zip(params, grads)] Step 3: SGD with momentum class SGDMomentum: def __init__(self, lr=0.001, momentum=0.9): self.lr = lr self.momentum = momentum self.velocity = None def step(self, params, grads): if self.velocity is None: AIENGINEERINGFROMSCRATCH.COM 139

OPTIMIZATION self.velocity = [0.0] * len(params) self.velocity = [ self.momentum * v + g for v, g in zip(self.velocity, grads) ] return [p - self.lr * v for p, v in zip(params, self.velocity)] Step 4: Adam class Adam: def __init__(self, lr=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8): self.lr = lr self.beta1 = beta1 self.beta2 = beta2 self.epsilon = epsilon self.m = None self.v = None self.t = 0 def step(self, params, grads): if self.m is None: self.m = [0.0] * len(params) self.v = [0.0] * len(params) self.t += 1 self.m = [ self.beta1 * m + (1 - self.beta1) * g for m, g in zip(self.m, grads) ] self.v = [ self.beta2 * v + (1 - self.beta2) * g ** 2 for v, g in zip(self.v, grads) ] m_hat = [m / (1 - self.beta1 ** self.t) for m in self.m] v_hat = [v / (1 - self.beta2 ** self.t) for v in self.v] return [ p - self.lr * mh / (vh ** 0.5 + self.epsilon) for p, mh, vh in zip(params, m_hat, v_hat) ] Step 5: Run and compare def optimize(optimizer, func, grad_func, start, steps=5000): params = list(start) history = [params[:]] for _ in range(steps): grads = grad_func(params) params = optimizer.step(params, grads) history.append(params[:]) return history AIENGINEERINGFROMSCRATCH.COM 140

OPTIMIZATION start = [-1.0, 1.0] gd_history = optimize(GradientDescent(lr=0.0005), rosenbrock, rosenbrock_gradient, start)↪ sgd_history = optimize(SGDMomentum(lr=0.0001, momentum=0.9), rosenbrock, rosenbrock_gradient, start)↪ adam_history = optimize(Adam(lr=0.01), rosenbrock, rosenbrock_gradient, start) for name, history in [("GD", gd_history), ("SGD+M", sgd_history), ("Adam", adam_history)]:↪ final = history[-1] loss = rosenbrock(final) print(f"{name:6s} -> x={final[0]:.6f}, y={final[1]:.6f}, loss={loss:.8f}") Expected output: Adam converges fastest. SGD with momentum follows a smoother path. Vanilla GD makes slow progress along the narrow valley. Use It In practice, use PyTorch or JAX optimizers. They handle parameter groups, weight decay, gradient clipping, and GPU acceleration. import torch model = torch.nn.Linear(784, 10) sgd = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9) adam = torch.optim.Adam(model.parameters(), lr=0.001) adamw = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01) scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(adam, T_max=100) Rules of thumb: • Start with Adam (lr=0.001). It works for most problems without tuning. • Switch to SGD with momentum (lr=0.01, momentum=0.9) when you need the best final accuracy and can afford more tuning. • Use AdamW (Adam with decoupled weight decay) for transformers. • Always use a learning rate schedule for training runs longer than a few epochs. • If training is unstable, reduce the learning rate. If training is too slow, increase it. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/08-optimization Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/08-optimization/code 1. Learning rate sweep. Run vanilla gradient descent on the Rosenbrock function with learning rates [0.0001, 0.0005, 0.001, 0.005, 0.01]. Plot or print the final loss after 5000 steps for each. Find the largest learning rate that still converges. AIENGINEERINGFROMSCRATCH.COM 141

OPTIMIZATION 2. Momentum comparison. Run SGD with momentum values [0.0, 0.5, 0.9, 0.99] on the Rosenbrock function. Track the loss at every step. Which momentum value converges fastest? Which overshoots? 3. Saddle point escape. Define the function f(x, y) = x^2 - y^2 (a saddle point at the origin). Start at (0.01, 0.01). Compare how vanilla GD, SGD with momentum, and Adam behave. Which escapes the saddle point? 4. Implement learning rate decay. Add an exponential decay schedule to the Gradient- Descent class: lr = lr_0 * 0.999^step. Compare convergence with and without decay on the Rosenbrock function. Key Terms Term What people say What it actually means Gradient descent “Go downhill” Update weights by subtracting the gradient scaled by the learning rate. The most basic optimizer. Learning rate “Step size” A scalar that controls how far each update moves the weights. Too large causes divergence. Too small wastes compute. Momentum “Keep rolling” Accumulate past gradients into a velocity vector. Dampens oscillations and accelerates movement through consistent directions. SGD “Random sampling” Stochastic gradient descent. Compute gradient on a random subset instead of the full dataset. Almost always means mini-batch SGD in practice. Mini-batch “A chunk of data” A small subset of training data (32-256 samples) used to estimate the gradient. Balances speed and gradient accuracy. Adam “The default optimizer” Adaptive Moment Estimation. Tracks per-weight running averages of gradients and squared gradients to give each weight its own learning rate. Bias correction “Fix the cold start” Adam’s first and second moments are initialized to zero. Bias correction divides by (1 - beta^t) to compensate during early steps. Learning rate schedule “Change lr over time” A function that adjusts the learning rate during training. Large steps early, small steps late. Convex function “One valley” A function where any local minimum is the global minimum. Gradient descent always finds it. Neural network losses are not convex. Saddle point “Flat but not a minimum” A point where the gradient is zero but it is a minimum in some directions and a maximum in others. Common in high dimensions. AIENGINEERINGFROMSCRATCH.COM 142

OPTIMIZATION Term What people say What it actually means Loss landscape “The terrain” The loss function plotted over weight space. Visualized by slicing along two random directions. Convergence “Getting there” The optimizer has reached a point where further steps do not meaningfully reduce the loss. Further Reading • Sebastian Ruder: An overview of gradient descent optimization algorithms - comprehen- sive survey of all major optimizers • Why Momentum Really Works (Distill) - interactive visualization of momentum dynamics • Adam: A Method for Stochastic Optimization (Kingma & Ba, 2014) - the original Adam paper, readable and short • Visualizing the Loss Landscape of Neural Nets (Li et al., 2018) - the paper that showed sharp vs flat minima Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/08-optimization • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/08-optimization/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/08-optimization The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 143

Information Theory Information theory measures surprise. Loss functions are built on it. Type: Learn Language: Python Prerequisites: Phase 1, Lesson 06 (Probability) Time: ~60 minutes Learning Objectives • Compute entropy, cross-entropy, and KL divergence from scratch and explain their rela- tionship • Derive why minimizing cross-entropy loss is equivalent to maximizing log-likelihood • Calculate mutual information between features and a target to rank feature importance • Explain perplexity as the effective vocabulary size a language model chooses from The Problem You call CrossEntropyLoss() in every classification model you train. You see “perplexity” in every language model paper. You read about KL divergence in VAEs, distillation, and RLHF. These are not disconnected concepts. They are all the same idea wearing different hats. Information theory gives you the language to reason about uncertainty, compression, and prediction. Claude Shannon invented it in 1948 to solve communication problems. Turns out, training a neural network is a communication problem: the model is trying to transmit the correct label through a noisy channel of learned weights. This lesson builds every formula from scratch so you see where they come from and why they work. The Concept Information Content (Surprise) When something unlikely happens, it carries more information. A coin landing heads? Not surprising. A lottery win? Very surprising. The information content of an event with probability p is: I(x) = -log(p(x)) Using log base 2 gives you bits. Using natural log gives you nats. Same idea, different units. Event Probability Surprise (bits) Fair coin heads 0.5 1.0 Rolling a 6 0.167 2.58 1-in-1000 event 0.001 9.97 Certain event 1.0 0.0 Certain events carry zero information. You already knew they would happen. AIENGINEERINGFROMSCRATCH.COM 144

INFORMATION THEORY Entropy (Average Surprise) Entropy is the expected surprise across all possible outcomes of a distribution. H(P) = -sum( p(x) * log(p(x)) ) for all x A fair coin has maximum entropy for a binary variable: 1 bit. A biased coin (99% heads) has low entropy: 0.08 bits. You already know what will happen, so each flip tells you almost nothing. Fair coin: H = -(0.5 * log2(0.5) + 0.5 * log2(0.5)) = 1.0 bit Biased coin: H = -(0.99 * log2(0.99) + 0.01 * log2(0.01)) = 0.08 bits Entropy measures the irreducible uncertainty in a distribution. You cannot compress below it. Cross-Entropy (The Loss Function You Use Every Day) Cross-entropy measures the average surprise when you use distribution Q to encode events that actually come from distribution P. H(P, Q) = -sum( p(x) * log(q(x)) ) for all x P is the true distribution (the labels). Q is your model’s predictions. If Q matches P perfectly, cross-entropy equals entropy. Any mismatch makes it larger. In classification, P is a one-hot vector (the true class has probability 1, everything else 0). This simplifies cross-entropy to: H(P, Q) = -log(q(true_class)) That is the entire cross-entropy loss formula for classification. Maximize the predicted prob- ability of the correct class. KL Divergence (Distance Between Distributions) KL divergence measures how much extra surprise you get from using Q instead of P. D_KL(P || Q) = sum( p(x) * log(p(x) / q(x)) ) for all x = H(P, Q) - H(P) Cross-entropy is entropy plus KL divergence. Since entropy of the true distribution is constant during training, minimizing cross-entropy is the same as minimizing KL divergence. You are pushing your model’s distribution toward the true distribution. KL divergence is not symmetric: D_KL(P || Q) != D_KL(Q || P). It is not a true distance metric. Mutual Information Mutual information measures how much knowing one variable tells you about another. I(X; Y) = H(X) - H(X|Y) = H(X) + H(Y) - H(X, Y) If X and Y are independent, mutual information is zero. Knowing one tells you nothing about the other. If they are perfectly correlated, mutual information equals the entropy of either variable. In feature selection, high mutual information between a feature and the target means the feature is useful. Low mutual information means it is noise. AIENGINEERINGFROMSCRATCH.COM 145

INFORMATION THEORY Conditional Entropy H(Y|X) measures how much uncertainty remains about Y after you observe X. H(Y|X) = H(X,Y) - H(X) Two extremes: - If X completely determines Y, then H(Y|X) = 0. Knowing X eliminates all uncertainty about Y. Example: X = temperature in Celsius, Y = temperature in Fahrenheit. - If X tells you nothing about Y, then H(Y|X) = H(Y). Knowing X does not reduce your uncertainty at all. Example: X = coin flip, Y = tomorrow’s weather. Conditional entropy is always non-negative and never exceeds H(Y): 0 <= H(Y|X) <= H(Y) In machine learning, conditional entropy appears in decision trees. At each split, the algo- rithm picks the feature X that minimizes H(Y|X) – the feature that removes the most uncer- tainty about the label Y. Joint Entropy H(X,Y) is the entropy of the joint distribution of X and Y together. H(X,Y) = -sum sum p(x,y) * log(p(x,y)) for all x, y Key property: H(X,Y) <= H(X) + H(Y) Equality holds when X and Y are independent. If they share information, the joint entropy is less than the sum of individual entropies. The “missing” entropy is exactly the mutual information. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/09-information-theory The relationships: - H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y) - I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) - H(X,Y) = H(X) + H(Y) - I(X;Y) Mutual Information (Deep Dive) Mutual information I(X;Y) quantifies how much knowing one variable reduces uncertainty about the other. I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) = H(X) + H(Y) - H(X,Y) = sum sum p(x,y) * log(p(x,y) / (p(x) * p(y))) Properties: - I(X;Y) >= 0 always. You never lose information by observing something. - I(X;Y) = 0 if and only if X and Y are independent. - I(X;Y) = I(Y;X). It is symmetric, unlike KL diver- gence. - I(X;X) = H(X). A variable shares all its information with itself. Mutual information for feature selection. In ML, you want features that are informative about the target. Mutual information gives you a principled way to rank features: 1. For each feature X_i, compute I(X_i; Y) where Y is the target variable. 2. Rank features by MI score. 3. Keep the top k features. AIENGINEERINGFROMSCRATCH.COM 146

INFORMATION THEORY This works for any relationship between feature and target – linear, nonlinear, monotonic, or not. Correlation only catches linear relationships. MI catches everything. Method Detects Computational cost Handles categorical? Pearson correlation Linear relationships O(n) No Spearman correlation Monotonic relationships O(n log n) No Mutual information Any statistical dependency O(n log n) with binning Yes Label Smoothing and Cross-Entropy Standard classification uses hard targets: [0, 0, 1, 0]. The true class gets probability 1, ev- erything else gets 0. Label smoothing replaces these with soft targets: soft_target = (1 - epsilon) * hard_target + epsilon / num_classes With epsilon = 0.1 and 4 classes: - Hard target: [0, 0, 1, 0] - Soft target: [0.025, 0.025, 0.925, 0.025] From an information theory perspective, label smoothing increases the entropy of the target distribution. Hard one-hot targets have entropy 0 – there is no uncertainty. Soft targets have positive entropy. Why this helps: - Prevents the model from driving logits to extreme values (infinite logits would be needed to perfectly match a one-hot target under cross-entropy) - Acts as regular- ization: the model cannot be 100% confident - Improves calibration: predicted probabilities better reflect true uncertainty - Reduces the gap between training and inference behavior The cross-entropy loss with label smoothing becomes: L = (1 - epsilon) * CE(hard_target, prediction) + epsilon * H_uniform(prediction) The second term penalizes predictions that are far from uniform – a direct regularization on confidence. Why Cross-Entropy Is THE Classification Loss Three perspectives, same conclusion. Information theory view. Cross-entropy measures how many bits you waste by using your model’s distribution instead of the true distribution. Minimizing it makes your model the most efficient encoder of reality. Maximum likelihood view. For N training samples with true classes y_i: Likelihood = product( q(y_i) ) Log-likelihood = sum( log(q(y_i)) ) Negative log-likelihood = -sum( log(q(y_i)) ) That last line is cross-entropy loss. Minimizing cross-entropy = maximizing the likelihood of the training data under your model. Gradient view. The gradient of cross-entropy with respect to the logits is simply (predicted - true). Clean, stable, and fast to compute. This is why it pairs perfectly with softmax. AIENGINEERINGFROMSCRATCH.COM 147

INFORMATION THEORY Bits vs Nats The only difference is the log base. log base 2 -> bits (information theory tradition) log base e -> nats (machine learning convention) log base 10 -> hartleys (rarely used) 1 nat = 1/ln(2) bits = 1.4427 bits. PyTorch and TensorFlow use natural log (nats) by default. Perplexity Perplexity is the exponential of cross-entropy. It tells you the effective number of equally likely choices the model is uncertain between. Perplexity = 2^H(P,Q) (if using bits) Perplexity = e^H(P,Q) (if using nats) A language model with perplexity 50 is, on average, as confused as if it had to pick uniformly from 50 possible next tokens. Lower is better. GPT-2 achieved perplexity ~30 on common benchmarks. Modern models are in the single digits for well-represented domains. Interactive figure: entropy-kl. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01-math- foundations/09-information-theory Build It Step 1: Information content and entropy import math def information_content(p, base=2): if p <= 0 or p > 1: return float('inf') if p <= 0 else 0.0 return -math.log(p) / math.log(base) def entropy(probs, base=2): return sum( p * information_content(p, base) for p in probs if p > 0 ) fair_coin = [0.5, 0.5] biased_coin = [0.99, 0.01] fair_die = [1/6] * 6 print(f"Fair coin entropy: {entropy(fair_coin):.4f} bits") print(f"Biased coin entropy: {entropy(biased_coin):.4f} bits") print(f"Fair die entropy: {entropy(fair_die):.4f} bits") AIENGINEERINGFROMSCRATCH.COM 148

INFORMATION THEORY Step 2: Cross-entropy and KL divergence def cross_entropy(p, q, base=2): total = 0.0 for pi, qi in zip(p, q): if pi > 0: if qi <= 0: return float('inf') total += pi * (-math.log(qi) / math.log(base)) return total def kl_divergence(p, q, base=2): return cross_entropy(p, q, base) - entropy(p, base) true_dist = [0.7, 0.2, 0.1] good_model = [0.6, 0.25, 0.15] bad_model = [0.1, 0.1, 0.8] print(f"Entropy of true dist: {entropy(true_dist):.4f} bits") print(f"CE (good model): {cross_entropy(true_dist, good_model):.4f} bits") print(f"CE (bad model): {cross_entropy(true_dist, bad_model):.4f} bits") print(f"KL divergence (good): {kl_divergence(true_dist, good_model):.4f} bits") print(f"KL divergence (bad): {kl_divergence(true_dist, bad_model):.4f} bits") Step 3: Cross-entropy as classification loss def softmax(logits): max_logit = max(logits) exps = [math.exp(z - max_logit) for z in logits] total = sum(exps) return [e / total for e in exps] def cross_entropy_loss(true_class, logits): probs = softmax(logits) return -math.log(probs[true_class]) logits = [2.0, 1.0, 0.1] true_class = 0 probs = softmax(logits) loss = cross_entropy_loss(true_class, logits) print(f"Logits: {logits}") print(f"Softmax: {[f'{p:.4f}' for p in probs]}") print(f"True class: {true_class}") print(f"Loss: {loss:.4f} nats") print(f"Perplexity: {math.exp(loss):.2f}") Step 4: Cross-entropy equals negative log-likelihood import random random.seed(42) AIENGINEERINGFROMSCRATCH.COM 149

INFORMATION THEORY n_samples = 1000 n_classes = 3 true_labels = [random.randint(0, n_classes - 1) for _ in range(n_samples)] model_logits = [[random.gauss(0, 1) for _ in range(n_classes)] for _ in range(n_samples)]↪ ce_loss = sum( cross_entropy_loss(label, logits) for label, logits in zip(true_labels, model_logits) ) / n_samples nll = -sum( math.log(softmax(logits)[label]) for label, logits in zip(true_labels, model_logits) ) / n_samples print(f"Cross-entropy loss: {ce_loss:.6f}") print(f"Negative log-likelihood: {nll:.6f}") print(f"Difference: {abs(ce_loss - nll):.2e}") Step 5: Mutual information def mutual_information(joint_probs, base=2): rows = len(joint_probs) cols = len(joint_probs[0]) margin_x = [sum(joint_probs[i][j] for j in range(cols)) for i in range(rows)] margin_y = [sum(joint_probs[i][j] for i in range(rows)) for j in range(cols)] mi = 0.0 for i in range(rows): for j in range(cols): pxy = joint_probs[i][j] if pxy > 0: mi += pxy * math.log(pxy / (margin_x[i] * margin_y[j])) / math.log(base)↪ return mi independent = [[0.25, 0.25], [0.25, 0.25]] dependent = [[0.45, 0.05], [0.05, 0.45]] print(f"MI (independent): {mutual_information(independent):.4f} bits") print(f"MI (dependent): {mutual_information(dependent):.4f} bits") Use It The same concepts using NumPy, the way you will use them in practice: import numpy as np def np_entropy(p): p = np.asarray(p, dtype=float) mask = p > 0 AIENGINEERINGFROMSCRATCH.COM 150

INFORMATION THEORY result = np.zeros_like(p) result[mask] = p[mask] * np.log(p[mask]) return -result.sum() def np_cross_entropy(p, q): p, q = np.asarray(p, dtype=float), np.asarray(q, dtype=float) mask = p > 0 return -(p[mask] * np.log(q[mask])).sum() def np_kl_divergence(p, q): return np_cross_entropy(p, q) - np_entropy(p) true = np.array([0.7, 0.2, 0.1]) pred = np.array([0.6, 0.25, 0.15]) print(f"Entropy: {np_entropy(true):.4f} nats") print(f"Cross-ent: {np_cross_entropy(true, pred):.4f} nats") print(f"KL div: {np_kl_divergence(true, pred):.4f} nats") You built from scratch what torch.nn.CrossEntropyLoss() does internally. Now you know why the loss goes down during training: your model’s predicted distribution is getting closer to the true distribution, measured in nats of wasted information. Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/01-math-foundations/09-information-theory/code 1. Compute the entropy of the English alphabet assuming uniform distribution (26 letters). Then estimate it using actual letter frequencies. Which is higher and why? 2. A model outputs logits [5.0, 2.0, 0.5] for a sample with true class 1. Compute the cross- entropy loss by hand, then verify with your cross_entropy_loss function. What logits would give zero loss? 3. Show that KL divergence is not symmetric. Pick two distributions P and Q and compute D_KL(P || Q) and D_KL(Q || P). Explain why they differ. 4. Build a function that computes perplexity for a sequence of token predictions. Given a list of (true_token_index, predicted_logits) pairs, return the perplexity of the sequence. Key Terms Term What people say What it actually means Information content “Surprise” The number of bits (or nats) needed to encode an event: -log(p) Entropy “Randomness” The average surprise across all outcomes of a distribution. Measures irreducible uncertainty. Cross- entropy “The loss function” Average surprise when using model distribution Q to encode events from true distribution P. AIENGINEERINGFROMSCRATCH.COM 151

INFORMATION THEORY Term What people say What it actually means KL divergence “Distance between distributions” Extra bits wasted by using Q instead of P. Equals cross-entropy minus entropy. Not symmetric. Mutual information “How related are X and Y” Reduction in uncertainty about X from knowing Y. Zero means independent. Softmax “Turn logits into probabilities” Exponentiate and normalize. Maps any real-valued vector to a valid probability distribution. Perplexity “How confused the model is” Exponential of cross-entropy. The effective vocabulary size the model is choosing from at each step. Bits “Shannon’s unit” Information measured with log base 2. One bit resolves one fair coin flip. Nats “ML’s unit” Information measured with natural log. Used by PyTorch and TensorFlow by default. Negative log- likelihood “NLL loss” Identical to cross-entropy loss for one-hot labels. Minimizing it maximizes the probability of correct predictions. Further Reading • Shannon 1948: A Mathematical Theory of Communication - the original paper, still read- able • Visual Information Theory (Chris Olah) - best visual explanation of entropy and KL diver- gence • PyTorch CrossEntropyLoss docs - how the framework implements what you just built Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/09-information-theory • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/09-information-theory/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/09-information-theory The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 152

Dimensionality Reduction High-dimensional data has structure. You find it by looking from the right angle. Type: Build Language: Python Prerequisites: Phase 1, Lessons 01 (Linear Algebra Intu- ition), 02 (Vectors, Matrices & Operations), 03 (Eigenvalues & Eigenvectors), 06 (Probability & Distributions) Time: ~90 minutes Learning Objectives • Implement PCA from scratch: center data, compute the covariance matrix, eigendecom- pose, and project • Use explained variance ratio and the elbow method to choose the number of principal components • Compare PCA, t-SNE, and UMAP for visualizing MNIST digits in 2D and explain their tradeoffs • Apply kernel PCA with an RBF kernel to separate nonlinear data structures that standard PCA cannot handle The Problem You have a dataset with 784 features per sample. Maybe it is pixel values of handwritten digits. Maybe it is gene expression levels. Maybe it is user behavior signals. You cannot visualize 784 dimensions. You cannot plot them. You cannot even think about them. But most of those 784 features are redundant. The actual information lives on a much smaller surface. A handwritten “7” does not need 784 independent numbers to describe it. It needs a few: the angle of the stroke, the length of the crossbar, how much it leans. The rest is noise. Dimensionality reduction finds that smaller surface. It takes your 784-dimensional data and compresses it to 2, 10, or 50 dimensions while keeping the structure that matters. The Concept The curse of dimensionality High-dimensional spaces are unintuitive. Three things break as dimensions grow. Distance becomes meaningless. In high dimensions, the distance between any two random points converges to the same value. If every point is roughly the same distance from every other point, nearest-neighbor search stops working. Dimension Avg distance ratio (max/min between random points) 2 ~5.0 10 ~1.8 100 ~1.2 1000 ~1.02 AIENGINEERINGFROMSCRATCH.COM 153

DIMENSIONALITY REDUCTION Volume concentrates in corners. A unit hypercube in d dimensions has 2^d corners. In 100 dimensions, nearly all the volume is in the corners, far from the center. Data points spread to the edges and your models starve for data in the interior. You need exponentially more data. To maintain the same density of samples in a space, going from 2D to 20D means you need 10^18 times more data. You never have enough. Reducing dimensions brings the data density back to something workable. PCA: find the directions that matter Principal Component Analysis (PCA) finds the axes along which your data varies the most. It rotates your coordinate system so the first axis captures the most variance, the second captures the next most, and so on. The algorithm: 1. Center the data (subtract the mean from each feature) 2. Compute covariance (how features move together) 3. Eigendecomposition (find the principal directions) 4. Sort by eigenvalue (biggest variance first) 5. Project (keep top k eigenvectors, drop the rest) Why eigendecomposition? The covariance matrix is symmetric and positive semi-definite. Its eigenvectors are orthogonal directions in feature space. The eigenvalues tell you how much variance each direction captures. The eigenvector with the largest eigenvalue points along the direction of maximum variance. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/10-dimensionality-reduction • Before PCA: Data cloud is spread diagonally across both x and y axes • After PCA: Coordinate system is rotated so PC1 aligns with the direction of maximum variance (elongated spread) and PC2 aligns with the direction of minimum variance (nar- row spread) • Dimensionality reduction: Dropping PC2 projects the data onto PC1, losing very little information Explained variance ratio Each principal component captures a fraction of the total variance. The explained variance ratio tells you how much. Component Eigenvalue Explained ratio Cumulative PC1 4.73 0.473 0.473 PC2 2.51 0.251 0.724 PC3 1.12 0.112 0.836 PC4 0.89 0.089 0.925 ... When the cumulative explained variance reaches 0.95, you know that many components cap- ture 95% of the information. Everything after that is mostly noise. Choosing the number of components Three strategies: 1. Threshold. Keep enough components to explain 90-95% of the variance. 2. Elbow method. Plot explained variance per component. Look for a sharp drop-off. AIENGINEERINGFROMSCRATCH.COM 154

DIMENSIONALITY REDUCTION 3. Downstream performance. Use PCA as preprocessing. Sweep k and measure your model’s accuracy. The best k is wherever accuracy plateaus. t-SNE: preserve neighborhoods t-Distributed Stochastic Neighbor Embedding (t-SNE) is designed for visualization. It maps high-dimensional data to 2D (or 3D) while preserving which points are near each other. The intuition: in the original space, compute a probability distribution over pairs of points based on their distances. Near points get high probability. Far points get low probability. Then find a 2D arrangement where the same probability distribution holds. Points that were neighbors in 784 dimensions stay neighbors in 2D. Key properties of t-SNE: - Non-linear. It can unfold complex manifolds that PCA cannot. - Stochastic. Different runs produce different layouts. - Perplexity parameter controls how many neighbors to consider (typical range: 5-50). - Distances between clusters in the output are not meaningful. Only the clusters themselves are. - Slow on large datasets. O(n^2) by default. UMAP: faster, better global structure Uniform Manifold Approximation and Projection (UMAP) works similarly to t-SNE but with two advantages: - Faster. It uses approximate nearest-neighbor graphs instead of computing all pairwise distances. - Better global structure. The relative positions of clusters in the output tend to be more meaningful than in t-SNE. UMAP builds a weighted graph in high-dimensional space (the “fuzzy topological representa- tion”) and then finds a low-dimensional layout that preserves this graph as well as possible. Key parameters: - n_neighbors: how many neighbors define local structure (similar to per- plexity). Higher values preserve more global structure. - min_dist: how tightly points pack together in the output. Lower values create denser clusters. When to use which Method Use case Preserves Speed PCA Preprocessing before training Global variance Fast (exact), works on millions of samples PCA Quick exploratory visualization Linear structure Fast t-SNE Publication-quality 2D plots Local neighborhoods Slow (< 10k samples ideal) UMAP 2D visualization at scale Local + some global structure Medium (handles millions) PCA Feature reduction for models Variance-ranked features Fast t-SNE / UMAP Understanding cluster structure Cluster separation Medium to slow Rule of thumb: use PCA for preprocessing and data compression. Use t-SNE or UMAP when you need to visualize structure in 2D. AIENGINEERINGFROMSCRATCH.COM 155

DIMENSIONALITY REDUCTION Kernel PCA Standard PCA finds linear subspaces. It rotates your coordinate system and drops axes. But what if the data lies on a nonlinear manifold? A circle in 2D cannot be separated by any line. Standard PCA will not help. Kernel PCA applies PCA in a high-dimensional feature space induced by a kernel function, without explicitly computing the coordinates in that space. This is the kernel trick – the same idea behind SVMs. The algorithm: 1. Compute the kernel matrix K where K_ij = k(x_i, x_j) 2. Center the kernel matrix in feature space 3. Eigendecompose the centered kernel matrix 4. The top eigenvec- tors (scaled by 1/sqrt(eigenvalue)) are the projections Common kernel functions: Kernel Formula Good for RBF (Gaussian) exp(-gamma * ||x - y||^2) Most nonlinear data, smooth manifolds Polynomial (x . y + c)^d Polynomial relationships Sigmoid tanh(alpha * x . y + c) Neural network-like mappings When to use kernel PCA vs standard PCA: Criterion Standard PCA Kernel PCA Data structure Linear subspace Nonlinear manifold Speed O(min(n^2 d, d^2 n)) O(n^2 d + n^3) Interpretability Components are linear combinations of features Components lack direct feature interpretation Scalability Works on millions of samples Kernel matrix is n x n, memory-limited Reconstruction Direct inverse transform Requires pre-image approximation The classic example: concentric circles in 2D. Two rings of points, one inside the other. Stan- dard PCA projects both onto the same line – useless for classification. Kernel PCA with an RBF kernel maps the inner circle and outer circle to different regions, making them linearly separable. Reconstruction Error How good is your dimensionality reduction? You compressed 784 dimensions to 50. What did you lose? Measure reconstruction error: 1. Project data to k dimensions: X_reduced = X @ W_k 2. Reconstruct: X_hat = X_reduced @ W_k^T 3. Compute MSE: mean((X - X_hat)^2) For PCA, reconstruction error has a clean relationship to explained variance: Reconstruction error = sum of eigenvalues NOT included Total variance = sum of ALL eigenvalues Fraction lost = (sum of dropped eigenvalues) / (sum of all eigenvalues) The explained variance ratio for each component is: AIENGINEERINGFROMSCRATCH.COM 156

DIMENSIONALITY REDUCTION explained_ratio_k = eigenvalue_k / sum(all eigenvalues) Plotting cumulative explained variance against number of components gives you the “elbow” curve. The right number of components is where: - The curve flattens out (diminishing re- turns) - Cumulative variance crosses your threshold (usually 0.90 or 0.95) - Downstream task performance plateaus Reconstruction error is useful beyond choosing k. You can use it for anomaly detection: sam- ples with high reconstruction error are outliers that do not fit the learned subspace. This is the basis of PCA-based anomaly detection in production systems. Interactive figure: pca-axes. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01-math- foundations/10-dimensionality-reduction Build It Step 1: PCA from scratch import numpy as np class PCA: def __init__(self, n_components): self.n_components = n_components self.components = None self.mean = None self.eigenvalues = None self.explained_variance_ratio_ = None def fit(self, X): self.mean = np.mean(X, axis=0) X_centered = X - self.mean cov_matrix = np.cov(X_centered, rowvar=False) eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix) sorted_idx = np.argsort(eigenvalues)[::-1] eigenvalues = eigenvalues[sorted_idx] eigenvectors = eigenvectors[:, sorted_idx] self.components = eigenvectors[:, :self.n_components].T self.eigenvalues = eigenvalues[:self.n_components] total_var = np.sum(eigenvalues) self.explained_variance_ratio_ = self.eigenvalues / total_var return self def transform(self, X): X_centered = X - self.mean return X_centered @ self.components.T def fit_transform(self, X): self.fit(X) AIENGINEERINGFROMSCRATCH.COM 157

DIMENSIONALITY REDUCTION return self.transform(X) Step 2: Test on synthetic data np.random.seed(42) n_samples = 500 t = np.random.uniform(0, 2 * np.pi, n_samples) x1 = 3 * np.cos(t) + np.random.normal(0, 0.2, n_samples) x2 = 3 * np.sin(t) + np.random.normal(0, 0.2, n_samples) x3 = 0.5 * x1 + 0.3 * x2 + np.random.normal(0, 0.1, n_samples) X_synthetic = np.column_stack([x1, x2, x3]) pca = PCA(n_components=2) X_reduced = pca.fit_transform(X_synthetic) print(f"Original shape: {X_synthetic.shape}") print(f"Reduced shape: {X_reduced.shape}") print(f"Explained variance ratios: {pca.explained_variance_ratio_}") print(f"Total variance captured: {sum(pca.explained_variance_ratio_):.4f}") Step 3: MNIST digits in 2D from sklearn.datasets import fetch_openml mnist = fetch_openml("mnist_784", version=1, as_frame=False, parser="auto") X_mnist = mnist.data[:5000].astype(float) y_mnist = mnist.target[:5000].astype(int) pca_mnist = PCA(n_components=50) X_pca50 = pca_mnist.fit_transform(X_mnist) print(f"50 components capture {sum(pca_mnist.explained_variance_ratio_):.2%} of variance")↪ pca_2d = PCA(n_components=2) X_pca2d = pca_2d.fit_transform(X_mnist) print(f"2 components capture {sum(pca_2d.explained_variance_ratio_):.2%} of variance")↪ Step 4: Compare with sklearn from sklearn.decomposition import PCA as SklearnPCA from sklearn.manifold import TSNE sklearn_pca = SklearnPCA(n_components=2) X_sklearn_pca = sklearn_pca.fit_transform(X_mnist) print(f"\nOur PCA explained variance: {pca_2d.explained_variance_ratio_}") print(f"Sklearn PCA explained variance: {sklearn_pca.explained_variance_ratio_}") diff = np.abs(np.abs(X_pca2d) - np.abs(X_sklearn_pca)) print(f"Max absolute difference: {diff.max():.10f}") AIENGINEERINGFROMSCRATCH.COM 158

DIMENSIONALITY REDUCTION tsne = TSNE(n_components=2, perplexity=30, random_state=42) X_tsne = tsne.fit_transform(X_mnist) print(f"\nt-SNE output shape: {X_tsne.shape}") Step 5: UMAP comparison try: from umap import UMAP reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42) X_umap = reducer.fit_transform(X_mnist) print(f"UMAP output shape: {X_umap.shape}") except ImportError: print("Install umap-learn: pip install umap-learn") Use It PCA as preprocessing before a classifier: from sklearn.decomposition import PCA as SklearnPCA from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score X_train, X_test, y_train, y_test = train_test_split( X_mnist, y_mnist, test_size=0.2, random_state=42 ) results = {} for k in [10, 30, 50, 100, 200]: pca_k = SklearnPCA(n_components=k) X_tr = pca_k.fit_transform(X_train) X_te = pca_k.transform(X_test) clf = LogisticRegression(max_iter=1000, random_state=42) clf.fit(X_tr, y_train) acc = accuracy_score(y_test, clf.predict(X_te)) var_captured = sum(pca_k.explained_variance_ratio_) results[k] = (acc, var_captured) print(f"k={k:>3d} accuracy={acc:.4f} variance={var_captured:.4f}") Performance plateaus well before 784 dimensions. That plateau is your operating point. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering- from- scratch/tree/main/phases/01- math- foundations/10- dimensionality- reduction Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering- from- scratch/tree/main/phases/01- math- foundations/10- dimensionality- AIENGINEERINGFROMSCRATCH.COM 159

DIMENSIONALITY REDUCTION reduction/code 1. Modify the PCA class to support inverse_transform. Reconstruct MNIST digits from 10, 50, and 200 components. Print the reconstruction error (mean squared difference from the original) for each. 2. Run t-SNE on the same MNIST subset with perplexity values of 5, 30, and 100. Describe how the output changes. Why does perplexity affect cluster tightness? 3. Take a dataset with 50 features where only 5 are informative (generate one with sklearn .datasets.make_classification). Apply PCA and check whether the explained variance curve correctly identifies that the data is effectively 5-dimensional. Key Terms Term What people say What it actually means Curse of dimension- ality “Too many features” Distances, volumes, and data density all behave counterintuitively as dimensions grow. Models need exponentially more data to compensate. PCA “Reduce dimensions” Rotate your coordinate system so the axes align with the directions of maximum variance, then drop the low-variance axes. Principal component “An important direction” An eigenvector of the covariance matrix. The direction in feature space along which the data varies most. Explained variance ratio “How much info this component has” The fraction of total variance captured by one principal component. Sum the top k ratios to see how much k components preserve. Covariance matrix “How features correlate” A symmetric matrix where entry (i,j) measures how feature i and feature j move together. Diagonal entries are individual variances. t-SNE “That cluster plot” A nonlinear method that maps high-dimensional data to 2D by preserving pairwise neighborhood probabilities. Good for visualization, not for preprocessing. UMAP “Faster t-SNE” A nonlinear method based on topological data analysis. Preserves both local and some global structure. Scales better than t-SNE. Perplexity “A t-SNE knob” Controls the effective number of neighbors each point considers. Low perplexity focuses on very local structure. High perplexity captures broader patterns. Manifold “The surface the data lives on” A lower-dimensional surface embedded in a higher-dimensional space. A sheet of paper crumpled in 3D is a 2D manifold. AIENGINEERINGFROMSCRATCH.COM 160

DIMENSIONALITY REDUCTION Further Reading • A Tutorial on Principal Component Analysis (Shlens) - clear derivation of PCA from the ground up • How to Use t-SNE Effectively (Wattenberg et al.) - interactive guide to t-SNE pitfalls and parameter choices • UMAP documentation - theory and practical guidance from the UMAP authors Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/10-dimensionality-reduction • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/10-dimensionality-reduction/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/10-dimensionality-reduction The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 161

Singular Value Decomposition SVD is the Swiss Army knife of linear algebra. Every matrix has one. Every data scientist needs one. Type: Build Languages: Python, Julia Prerequisites: Phase 1, Lessons 01 (Linear Algebra Intuition), 02 (Vectors & Matrices Operations), 03 (Matrix Transformations) Time: ~120 minutes Learning Objectives • Implement SVD via power iteration and explain the geometric meaning of U, Sigma, and V^T • Apply truncated SVD for image compression and measure the compression ratio vs re- construction error • Compute the Moore-Penrose pseudoinverse via SVD to solve overdetermined least- squares systems • Connect SVD to PCA, recommendation systems (latent factors), and Latent Semantic Analysis in NLP The Problem You have a 1000x2000 matrix. Maybe it is user-movie ratings. Maybe it is a document-term frequency table. Maybe it is the pixel values of an image. You need to compress it, denoise it, find hidden structure in it, or solve a least-squares system with it. Eigendecomposition only works on square matrices. Even then, it requires the matrix to have a full set of linearly independent eigenvectors. SVD works on any matrix. Any shape. Any rank. No conditions. It decomposes the matrix into three factors that reveal the geometry of what the matrix does to space. It is the most general and most useful factorization in all of linear algebra. The Concept What SVD does geometrically Every matrix, regardless of shape, performs three operations in sequence: rotate, scale, ro- tate. SVD makes this decomposition explicit. A = U * Sigma * V^T m x n m x m m x n n x n (any) (rotate) (scale) (rotate) Given any matrix A, SVD factors it into: - V^T rotates vectors in the input space (n- dimensional) - Sigma scales along each axis (stretches or compresses) - U rotates the result into the output space (m-dimensional) AIENGINEERINGFROMSCRATCH.COM 162

SINGULAR VALUE DECOMPOSITION Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/11-singular-value-decomposition Think of it this way. You hand SVD a matrix. It tells you: “This matrix takes a sphere of inputs, first rotates it by V^T, then stretches it into an ellipsoid by Sigma, then rotates the ellipsoid by U.” The singular values are the lengths of the ellipsoid’s axes. The full decomposition For a matrix A with shape m x n: A = U * Sigma * V^T where: U is m x m, orthogonal (U^T U = I) Sigma is m x n, diagonal (singular values on the diagonal) V is n x n, orthogonal (V^T V = I) The singular values sigma_1 >= sigma_2 >= ... >= sigma_r > 0 where r = rank(A) The columns of U are called left singular vectors. The columns of V are called right singular vectors. The diagonal entries of Sigma are called singular values. They are always non- negative and conventionally sorted in decreasing order. Left singular vectors, singular values, right singular vectors Each component of the SVD has a distinct geometric meaning. Right singular vectors (columns of V): These form an orthonormal basis for the input space (R^n). They are the directions in input space that the matrix maps to orthogonal directions in output space. Think of them as the natural coordinate system for the domain. Singular values (diagonal of Sigma): These are the scaling factors. The i-th singular value tells you how much the matrix stretches vectors along the i-th right singular vector. A singular value of zero means the matrix crushes that direction entirely. Left singular vectors (columns of U): These form an orthonormal basis for the output space (R^m). The i-th left singular vector is the direction in output space where the i-th right singular vector lands (after scaling). The relationship between them: A * v_i = sigma_i * u_i The matrix A takes the i-th right singular vector v_i, scales it by sigma_i, and maps it to the i-th left singular vector u_i. This gives you a coordinate-by-coordinate picture of what any matrix does. Outer product form The SVD can be written as a sum of rank-1 matrices: A = sigma_1 * u_1 * v_1^T + sigma_2 * u_2 * v_2^T + ... + sigma_r * u_r * v_r^T Each term sigma_i * u_i * v_i^T is a rank-1 matrix (an outer product). The full matrix is the sum of r such matrices, where r is the rank. AIENGINEERINGFROMSCRATCH.COM 163

SINGULAR VALUE DECOMPOSITION This form is the foundation of low-rank approximation. Each term adds one layer of structure. The first term captures the single most important pattern. The second captures the next most important. And so on. Truncating this sum gives you the best possible approximation at any given rank. Rank-1 approx: A_1 = sigma_1 * u_1 * v_1^T (captures the dominant pattern) Rank-2 approx: A_2 = sigma_1 * u_1 * v_1^T + sigma_2 * u_2 * v_2^T (captures the two most important patterns) Rank-k approx: A_k = sum of top k terms (optimal by the Eckart-Young theorem) Relationship to eigendecomposition SVD and eigendecomposition are deeply connected. The singular values and vectors of A come directly from the eigenvalues and eigenvectors of A^T A and A A^T. A^T A = V * Sigma^T * U^T * U * Sigma * V^T = V * Sigma^T * Sigma * V^T = V * D * V^T where D = Sigma^T * Sigma is a diagonal matrix with sigma_i^2 on the diagonal. So: - The right singular vectors (V) are eigenvectors of A^T A - The singular values squared (sigma_i^2) are eigenvalues of A^T A Similarly: A A^T = U * Sigma * V^T * V * Sigma^T * U^T = U * Sigma * Sigma^T * U^T So: - The left singular vectors (U) are eigenvectors of A A^T - The eigenvalues of A A^T are also sigma_i^2 This connection tells you three things: 1. Singular values are always real and non-negative (they are square roots of eigenvalues of a positive semi-definite matrix). 2. You could compute SVD via eigendecomposition of A^T A, but this squares the condition number and loses nu- merical precision. Dedicated SVD algorithms avoid this. 3. When A is square and symmetric positive semi-definite, SVD and eigendecomposition are the same thing. Truncated SVD: low-rank approximation The Eckart-Young-Mirsky theorem states that the best rank-k approximation to A (in both Frobenius and spectral norm) is obtained by keeping only the top k singular values and their corresponding vectors: A_k = U_k * Sigma_k * V_k^T where: U_k is m x k (first k columns of U) Sigma_k is k x k (top-left k x k block of Sigma) V_k is n x k (first k columns of V) AIENGINEERINGFROMSCRATCH.COM 164

SINGULAR VALUE DECOMPOSITION Approximation error = sigma_{k+1} (in spectral norm) = sqrt(sigma_{k+1}^2 + ... + sigma_r^2) (in Frobenius norm) This is not just “a good” approximation. It is provably the best possible approximation of rank k. No other rank-k matrix is closer to A. Component Relative magnitude Kept in rank-3 approx? sigma_1 Largest Yes sigma_2 Large Yes sigma_3 Medium-large Yes sigma_4 Medium No (error) sigma_5 Medium-small No (error) sigma_6 Small No (error) sigma_7 Very small No (error) sigma_8 Tiny No (error) Keep top 3: A_3 captures the three largest singular values. Error = remaining values (sigma_4 through sigma_8). If singular values decay fast, a small k captures most of the matrix. If they decay slowly, the matrix has no low-rank structure. Image compression with SVD A grayscale image is a matrix of pixel intensities. An 800x600 image has 480,000 values. SVD lets you approximate it with far fewer. Original image: 800 x 600 = 480,000 values SVD with rank k: U_k: 800 x k values Sigma_k: k values V_k: 600 x k values Total: k * (800 + 600 + 1) = k * 1401 values k=10: 14,010 values (2.9% of original) k=50: 70,050 values (14.6% of original) k=100: 140,100 values (29.2% of original) The compression ratio improves as k gets smaller, but visual quality degrades. The key insight: natural images have rapidly decaying singular values. The first few singular values capture the broad structure (shapes, gradients). The later ones capture fine detail and noise. Truncating at rank 50 often produces an image that looks nearly identical to the original while using 85% less storage. SVD for recommendation systems The Netflix Prize made this famous. You have a user-movie ratings matrix where most entries are missing. Movie1 Movie2 Movie3 Movie4 Movie5 User1 [ 5 ? 3 ? 1 ] User2 [ ? 4 ? 2 ? ] AIENGINEERINGFROMSCRATCH.COM 165

SINGULAR VALUE DECOMPOSITION User3 [ 3 ? 5 ? ? ] User4 [ ? ? ? 4 3 ] ? = unknown rating The idea: this ratings matrix has low rank. Users do not have completely independent tastes. There are a handful of latent factors (action vs. drama, old vs. new, cerebral vs. visceral) that explain most preferences. SVD on the (filled-in) ratings matrix decomposes it into: - U: user profiles in latent factor space - Sigma: importance of each latent factor - V^T: movie profiles in latent factor space A user’s predicted rating for a movie is the dot product of their user profile with the movie’s profile (weighted by singular values). The low-rank approximation fills in the missing entries. In practice, you use variants like Simon Funk’s incremental SVD or ALS (alternating least squares) that handle missing data directly. But the core idea is the same: latent factor de- composition via SVD. SVD in NLP: Latent Semantic Analysis Latent Semantic Analysis (LSA), also called Latent Semantic Indexing (LSI), applies SVD to a term-document matrix. Doc1 Doc2 Doc3 Doc4 "cat" [ 3 0 1 0 ] "dog" [ 2 0 0 1 ] "fish" [ 0 4 1 0 ] "pet" [ 1 1 1 1 ] "ocean" [ 0 3 0 0 ] After SVD with rank k=2: Each document becomes a point in 2D "concept space." Each term becomes a point in the same 2D space. Documents about similar topics cluster together. Terms with similar meanings cluster together. "cat" and "dog" end up near each other (land pets). "fish" and "ocean" end up near each other (water concepts). Doc1 and Doc3 cluster if they share similar topics. LSA was one of the first successful methods for capturing semantic similarity from raw text. It works because synonymous terms tend to appear in similar documents, so SVD groups them into the same latent dimensions. Modern word embeddings (Word2Vec, GloVe) can be seen as descendants of this idea. SVD for noise reduction Noisy data has signal concentrated in the top singular values and noise spread across all singular values. Truncating removes the noise floor. Clean signal singular values: Component Magnitude Type sigma_1 Very large Signal AIENGINEERINGFROMSCRATCH.COM 166

SINGULAR VALUE DECOMPOSITION Component Magnitude Type sigma_2 Large Signal sigma_3 Medium Signal sigma_4 Near zero Negligible sigma_5 Near zero Negligible Noisy signal singular values (noise adds to all): Component Magnitude Type sigma_1 Very large Signal sigma_2 Large Signal sigma_3 Medium Signal sigma_4 Small Noise sigma_5 Small Noise sigma_6 Small Noise sigma_7 Small Noise Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/11-singular-value-decomposition This is used in signal processing, scientific measurement, and data cleaning. Any time you have a matrix corrupted by additive noise, truncated SVD is a principled way to separate signal from noise. Pseudoinverse via SVD The Moore-Penrose pseudoinverse A+ generalizes matrix inversion to non-square and singu- lar matrices. SVD makes computing it trivial. If A = U * Sigma * V^T, then: A+ = V * Sigma+ * U^T where Sigma+ is formed by: 1. Transpose Sigma (swap rows and columns) 2. Replace each non-zero diagonal entry sigma_i with 1/sigma_i 3. Leave zeros as zeros For A (m x n): A+ is (n x m) For Sigma (m x n): Sigma+ is (n x m) The pseudoinverse solves least-squares problems. If Ax = b has no exact solution (overdeter- mined system), then x = A+ b is the least-squares solution (minimizes ||Ax - b||). Overdetermined system (more equations than unknowns): [1 1] [3] [2 1] x = [5] No exact solution exists. [3 1] [6] x_ls = A+ b = V * Sigma+ * U^T * b AIENGINEERINGFROMSCRATCH.COM 167

SINGULAR VALUE DECOMPOSITION This gives the x that minimizes the sum of squared residuals. Same result as the normal equations (A^T A)^(-1) A^T b, but numerically more stable. Numerical stability advantages Computing eigendecomposition of A^T A squares the singular values (eigenvalues of A^T A are sigma_i^2). This squares the condition number, amplifying numerical errors. Example: A has singular values [1000, 1, 0.001] Condition number of A: 1000 / 0.001 = 10^6 A^T A has eigenvalues [10^6, 1, 10^{-6}] Condition number of A^T A: 10^6 / 10^{-6} = 10^{12} Computing SVD directly: works with condition number 10^6 Computing via A^T A: works with condition number 10^{12} (6 extra digits of precision lost) Modern SVD algorithms (Golub-Kahan bidiagonalization) work directly on A, never forming A^T A. This is why you should always prefer np.linalg.svd(A) over np.linalg.eig(A.T @ A). Connection to PCA PCA IS SVD on centered data. This is not an analogy. It is literally the same computation. Given data matrix X (n_samples x n_features), centered (mean subtracted): Covariance matrix: C = (1/(n-1)) * X^T X PCA finds eigenvectors of C. But: X = U * Sigma * V^T (SVD of X) X^T X = V * Sigma^2 * V^T C = (1/(n-1)) * V * Sigma^2 * V^T So the principal components are exactly the right singular vectors V. The explained variance for each component is sigma_i^2 / (n-1). In sklearn, PCA is implemented using SVD, not eigendecomposition. It is faster and more numerically stable. This means everything you learned about dimensionality reduction in Lesson 10 is SVD under the hood. PCA is the most common application of SVD in machine learning. Interactive figure: svd-rank-reconstruction. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/ 01-math-foundations/11-singular-value-decomposition AIENGINEERINGFROMSCRATCH.COM 168

SINGULAR VALUE DECOMPOSITION Build It Step 1: SVD from scratch using power iteration The idea: to find the largest singular value and its vectors, use power iteration on A^T A (or A A^T). Then deflate the matrix and repeat for the next singular value. import numpy as np def power_iteration(M, num_iters=100): n = M.shape[1] v = np.random.randn(n) v = v / np.linalg.norm(v) for _ in range(num_iters): Mv = M @ v v = Mv / np.linalg.norm(Mv) eigenvalue = v @ M @ v return eigenvalue, v def svd_from_scratch(A, k=None): m, n = A.shape if k is None: k = min(m, n) sigmas = [] us = [] vs = [] A_residual = A.copy().astype(float) for _ in range(k): AtA = A_residual.T @ A_residual eigenvalue, v = power_iteration(AtA, num_iters=200) if eigenvalue < 1e-10: break sigma = np.sqrt(eigenvalue) u = A_residual @ v / sigma sigmas.append(sigma) us.append(u) vs.append(v) A_residual = A_residual - sigma * np.outer(u, v) U = np.column_stack(us) if us else np.empty((m, 0)) S = np.array(sigmas) V = np.column_stack(vs) if vs else np.empty((n, 0)) return U, S, V AIENGINEERINGFROMSCRATCH.COM 169

SINGULAR VALUE DECOMPOSITION Step 2: Test and compare with NumPy np.random.seed(42) A = np.random.randn(5, 4) U_ours, S_ours, V_ours = svd_from_scratch(A) U_np, S_np, Vt_np = np.linalg.svd(A, full_matrices=False) print("Our singular values:", np.round(S_ours, 4)) print("NumPy singular values:", np.round(S_np, 4)) A_reconstructed = U_ours @ np.diag(S_ours) @ V_ours.T print(f"Reconstruction error: {np.linalg.norm(A - A_reconstructed):.8f}") Step 3: Image compression demo def compress_image_svd(image_matrix, k): U, S, Vt = np.linalg.svd(image_matrix, full_matrices=False) compressed = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :] return compressed image = np.random.seed(42) rows, cols = 200, 300 image = np.random.randn(rows, cols) for k in [1, 5, 10, 20, 50]: compressed = compress_image_svd(image, k) error = np.linalg.norm(image - compressed) / np.linalg.norm(image) original_size = rows * cols compressed_size = k * (rows + cols + 1) ratio = compressed_size / original_size print(f"k={k:>3d} error={error:.4f} storage={ratio:.1%}") Step 4: Noise reduction np.random.seed(42) clean = np.outer(np.sin(np.linspace(0, 4*np.pi, 100)), np.cos(np.linspace(0, 2*np.pi, 80))) noise = 0.3 * np.random.randn(100, 80) noisy = clean + noise U, S, Vt = np.linalg.svd(noisy, full_matrices=False) denoised = U[:, :5] @ np.diag(S[:5]) @ Vt[:5, :] print(f"Noisy error: {np.linalg.norm(noisy - clean):.4f}") print(f"Denoised error: {np.linalg.norm(denoised - clean):.4f}") print(f"Improvement: {(1 - np.linalg.norm(denoised - clean) / np.linalg.norm(noisy - clean)):.1%}")↪ Step 5: Pseudoinverse A = np.array([[1, 1], [2, 1], [3, 1]], dtype=float) b = np.array([3, 5, 6], dtype=float) AIENGINEERINGFROMSCRATCH.COM 170

SINGULAR VALUE DECOMPOSITION U, S, Vt = np.linalg.svd(A, full_matrices=False) S_inv = np.diag(1.0 / S) A_pinv = Vt.T @ S_inv @ U.T x_svd = A_pinv @ b x_lstsq = np.linalg.lstsq(A, b, rcond=None)[0] x_pinv = np.linalg.pinv(A) @ b print(f"SVD pseudoinverse solution: {x_svd}") print(f"np.linalg.lstsq solution: {x_lstsq}") print(f"np.linalg.pinv solution: {x_pinv}") Use It Full working demos are in code/svd.py. Run it to see SVD applied to image compression, recommendation systems, latent semantic analysis, and noise reduction. python svd.py The Julia version in code/svd.jl demonstrates the same concepts using Julia’s native svd() function and LinearAlgebra package. julia svd.jl This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering- from- scratch/tree/main/phases/01- math- foundations/11- singular- value- decomposition Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering- from- scratch/tree/main/phases/01- math- foundations/11- singular- value- decomposition/code 1. Implement the full SVD from scratch without using power iteration. Instead, compute the eigendecomposition of A^T A to get V and the singular values, then compute U = A V Sigma^{-1}. Compare numerical accuracy with your power iteration version and with NumPy. 2. Load a real grayscale image (or convert one to grayscale). Compress it at ranks 1, 5, 10, 25, 50, 100. For each rank, compute the compression ratio and the relative error. Find the rank where the image becomes visually acceptable. 3. Build a tiny recommendation system. Create a 10x8 user-movie ratings matrix with some known entries. Fill missing entries with row means. Compute SVD and reconstruct a rank-3 approximation. Use the reconstructed matrix to predict the missing ratings. Verify that the predictions are reasonable. 4. Create a 100x50 document-term matrix with 3 synthetic topics. Each topic has 5 asso- ciated terms. Add noise. Apply SVD and verify that the top 3 singular values are much larger than the rest. Project documents into the 3D latent space and check that docu- ments from the same topic cluster together. AIENGINEERINGFROMSCRATCH.COM 171

SINGULAR VALUE DECOMPOSITION 5. Generate a clean low-rank matrix (rank 3, size 50x40) and add Gaussian noise at different levels (sigma = 0.1, 0.5, 1.0, 2.0). For each noise level, find the optimal truncation rank by sweeping k from 1 to 40 and measuring reconstruction error against the clean matrix. Plot how the optimal k changes with noise level. Key Terms Term What people say What it actually means SVD “Factor any matrix” Decompose A into U Sigma V^T where U and V are orthogonal and Sigma is diagonal with non-negative entries. Works for any matrix of any shape. Singular value “How important this component is” The i-th diagonal entry of Sigma. Measures how much the matrix stretches along the i-th principal direction. Always non-negative, sorted in decreasing order. Left singular vector “Output direction” A column of U. The direction in output space that the i-th right singular vector maps to (after scaling by sigma_i). Right singular vector “Input direction” A column of V. The direction in input space that the matrix maps to the i-th left singular vector (after scaling by sigma_i). Truncated SVD “Low-rank approximation” Keep only the top k singular values and their vectors. Produces the provably best rank-k approximation to the original matrix (Eckart-Young theorem). Rank “True dimensionality” The number of non-zero singular values. Tells you how many independent directions the matrix actually uses. Pseudoinverse“Generalized inverse” V Sigma+ U^T. Inverts non-zero singular values, leaves zeros as zeros. Solves least-squares problems for non-square or singular matrices. Condition number “How sensitive to errors” sigma_max / sigma_min. A large condition number means small input changes cause large output changes. SVD reveals this directly. Latent factor “Hidden variable” A dimension in the low-rank space discovered by SVD. In recommendations, a latent factor might correspond to genre preference. In NLP, it might correspond to a topic. Frobenius norm “Total matrix size” Square root of the sum of squared entries. Equals the square root of the sum of squared singular values. Used to measure approximation error. Eckart- Young theorem “SVD gives the best compression” For any target rank k, the truncated SVD minimizes the approximation error over all possible rank-k matrices. AIENGINEERINGFROMSCRATCH.COM 172

SINGULAR VALUE DECOMPOSITION Term What people say What it actually means Power iteration “Find the biggest eigenvector” Repeatedly multiply a random vector by the matrix and normalize. Converges to the eigenvector with the largest eigenvalue. The building block of many SVD algorithms. Further Reading • Gilbert Strang: Linear Algebra and Its Applications, Chapter 7 - thorough treatment of SVD with applications • 3Blue1Brown: But what is the SVD? - geometric intuition for SVD • We Recommend a Singular Value Decomposition - accessible overview from the American Mathematical Society • Netflix Prize and Matrix Factorization - Simon Funk’s original blog post on SVD for rec- ommendations • Latent Semantic Analysis - the original NLP application of SVD • Numerical Linear Algebra by Trefethen and Bau - the gold standard for understanding SVD algorithms and their numerical properties Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/11-singular-value-decomposition • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/11-singular-value-decomposition/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/11-singular-value-decomposition The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 173

Tensor Operations Tensors are the common language between data and deep learning. Every image, every sentence, every gradient flows through them. Type: Build Language: Python Prerequisites: Phase 1, Lessons 01 (Linear Algebra Intu- ition), 02 (Vectors, Matrices & Operations) Time: ~90 minutes Learning Objectives • Implement a tensor class with shape, strides, reshape, transpose, and element-wise op- erations from scratch • Apply broadcasting rules to operate on tensors of different shapes without copying data • Write einsum expressions for dot products, matrix multiplications, outer products, and batched operations • Trace the exact tensor shapes through every step of multi-head attention The Problem You build a transformer. The forward pass looks clean. You run it and get: RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x768 and 512x768). You stare at the shapes. You try a transpose. Now it says Expected 4D input (got 3D input). You add an unsqueeze. Something else breaks. Shape errors are the most common bug in deep learning code. They are not hard conceptually – each operation has a shape contract – but they multiply fast. A transformer has dozens of reshapes, transposes, and broadcasts chained together. One wrong axis and the error cascades. Worse, some shape mistakes do not throw errors at all. They silently produce garbage by broadcasting along the wrong dimension or summing over the wrong axis. Matrices handle pairwise relationships between two sets of things. Real data does not fit into two dimensions. A batch of 32 RGB images at 224x224 is a 4D tensor: (32, 3, 224, 224). Self-attention with 12 heads is also 4D: (batch, heads, seq_len, head_dim). You need a data structure that generalizes to any number of dimensions, with operations that compose cleanly across all of them. That structure is the tensor. Master its operations and shape errors become trivially debuggable. The Concept What a tensor is A tensor is a multi-dimensional array of numbers with a uniform data type. The number of dimensions is the rank (or order). Each dimension is an axis. The shape is a tuple listing the size along each axis. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/12-tensor-operations AIENGINEERINGFROMSCRATCH.COM 174

TENSOR OPERATIONS Total elements = product of all sizes. A shape (2, 3, 4) holds 2 * 3 * 4 = 24 elements. Tensor shapes in deep learning Different data types map to specific tensor shapes by convention. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/12-tensor-operations PyTorch uses NCHW (channels-first). TensorFlow defaults to NHWC (channels-last). Mis- matched layouts cause silent slowdowns or errors. How memory layout works A 2D array in memory is a 1D sequence of bytes. Strides tell you how many elements to skip to move one step along each axis. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/12-tensor-operations Transpose does not move data. It swaps the strides, making the tensor non-contiguous – the elements for a row are no longer adjacent in memory. Broadcasting rules Broadcasting lets you operate on tensors of different shapes without copying data. Align shapes from the right. Two dimensions are compatible when they are equal or one is 1. Fewer dimensions get padded with 1s on the left. Tensor A: (8, 1, 6, 1) Tensor B: (7, 1, 5) Padded B: (1, 7, 1, 5) Result: (8, 7, 6, 5) Einsum: the universal tensor operation Einstein summation labels each axis with a letter. Axes in the input but not the output get summed. Axes in both are kept. Diagram. Rendered live in the web edition: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/12-tensor-operations Key patterns: i,i-> (dot product), i,j->ij (outer product), ii-> (trace), ij->ji (transpose), bij,bjk->bik (batch matmul), bhtd,bhsd->bhts (attention scores). Interactive figure: tensor-broadcast. This one moves. Watch it animate and drag its con- trols in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/12-tensor-operations Build It The code lives in code/tensors.py. Each step references the implementation there. AIENGINEERINGFROMSCRATCH.COM 175

TENSOR OPERATIONS Step 1: Tensor storage and strides A tensor stores a flat list of numbers plus shape metadata. Strides tell the indexing logic how to map multi-dimensional indices to flat positions. class Tensor: def __init__(self, data, shape=None): if isinstance(data, (list, tuple)): self._data, self._shape = self._flatten_nested(data) elif isinstance(data, np.ndarray): self._data = data.flatten().tolist() self._shape = tuple(data.shape) else: self._data = [data] self._shape = () if shape is not None: total = reduce(lambda a, b: a * b, shape, 1) if total != len(self._data): raise ValueError( f"Cannot reshape {len(self._data)} elements into shape {shape}" ) self._shape = tuple(shape) self._strides = self._compute_strides(self._shape) @staticmethod def _compute_strides(shape): if len(shape) == 0: return () strides = [1] * len(shape) for i in range(len(shape) - 2, -1, -1): strides[i] = strides[i + 1] * shape[i + 1] return tuple(strides) For shape (3, 4), strides are (4, 1) – skip 4 elements to advance one row, skip 1 element to advance one column. Step 2: Reshape, squeeze, unsqueeze Reshape changes the shape without changing element order. The total number of elements must stay the same. Use -1 for one dimension to infer its size. t = Tensor(list(range(12)), shape=(2, 6)) r = t.reshape((3, 4)) r = t.reshape((-1, 3)) Squeeze removes axes of size 1. Unsqueeze inserts one. Unsqueezing is critical for broad- casting – a bias vector (D,) added to a batch (B, T, D) needs unsqueezing to (1, 1, D). t = Tensor(list(range(6)), shape=(1, 3, 1, 2)) s = t.squeeze() v = Tensor([1, 2, 3]) u = v.unsqueeze(0) AIENGINEERINGFROMSCRATCH.COM 176

TENSOR OPERATIONS Step 3: Transpose and permute Transpose swaps two axes. Permute reorders all axes. This is how you convert between NCHW and NHWC. mat = Tensor(list(range(6)), shape=(2, 3)) tr = mat.transpose(0, 1) t4d = Tensor(list(range(24)), shape=(1, 2, 3, 4)) perm = t4d.permute((0, 2, 3, 1)) After transpose or permute, the tensor is non-contiguous in memory. In PyTorch, view fails on non-contiguous tensors – use reshape or call .contiguous() first. Step 4: Element-wise operations and reductions Element-wise ops (add, multiply, subtract) apply independently to each element and preserve shape. Reductions (sum, mean, max) collapse one or more axes. a = Tensor([[1, 2], [3, 4]]) b = Tensor([[10, 20], [30, 40]]) c = a + b d = a * 2 s = a.sum(axis=0) Global average pooling in a CNN: (B, C, H, W).mean(axis=[2, 3]) produces (B, C). Sequence mean pooling in NLP: (B, T, D).mean(axis=1) produces (B, D). Step 5: Broadcasting with NumPy The demo_broadcasting_numpy() function in tensors.py shows the core patterns. activations = np.random.randn(4, 3) bias = np.array([0.1, 0.2, 0.3]) result = activations + bias images = np.random.randn(2, 3, 4, 4) scale = np.array([0.5, 1.0, 1.5]).reshape(1, 3, 1, 1) result = images * scale a = np.array([1, 2, 3]).reshape(-1, 1) b = np.array([10, 20, 30, 40]).reshape(1, -1) outer = a * b Pairwise distance via broadcasting: reshape (M, 2) to (M, 1, 2) and (N, 2) to (1, N, 2), subtract, square, sum along last axis, take square root. Result: (M, N). Step 6: Einsum operations The demo_einsum() and demo_einsum_gallery() functions walk through every common pat- tern. a = np.array([1.0, 2.0, 3.0]) b = np.array([4.0, 5.0, 6.0]) dot = np.einsum("i,i->", a, b) A = np.array([[1, 2], [3, 4], [5, 6]], dtype=float) AIENGINEERINGFROMSCRATCH.COM 177

TENSOR OPERATIONS B = np.array([[7, 8, 9], [10, 11, 12]], dtype=float) matmul = np.einsum("ik,kj->ij", A, B) batch_A = np.random.randn(4, 3, 5) batch_B = np.random.randn(4, 5, 2) batch_mm = np.einsum("bij,bjk->bik", batch_A, batch_B) The computational cost of a contraction is the product of all index sizes (kept and summed). For bij,bjk->bik with B=32, I=128, J=64, K=128: 32 * 128 * 64 * 128 = 33,554,432 multiply-adds. Step 7: Attention mechanism via einsum The demo_attention_einsum() function implements multi-head attention end to end. B, H, T, D = 2, 4, 8, 16 E = H * D X = np.random.randn(B, T, E) W_q = np.random.randn(E, E) * 0.02 Q = np.einsum("bte,ek->btk", X, W_q) Q = Q.reshape(B, T, H, D).transpose(0, 2, 1, 3) scores = np.einsum("bhtd,bhsd->bhts", Q, K) / np.sqrt(D) weights = softmax(scores, axis=-1) attn_output = np.einsum("bhts,bhsd->bhtd", weights, V) concat = attn_output.transpose(0, 2, 1, 3).reshape(B, T, E) output = np.einsum("bte,ek->btk", concat, W_o) Every step is a tensor operation: projection (matmul via einsum), head splitting (reshape + transpose), attention scores (batch matmul via einsum), weighted sum (batch matmul via einsum), head merging (transpose + reshape), output projection (matmul via einsum). Use It Scratch vs NumPy Operation Scratch (Tensor class) NumPy Create Tensor([[1,2],[3,4]]) np.array([[1,2],[3,4]]) Reshape t.reshape((3,4)) a.reshape(3,4) Transpose t.transpose(0,1) a.T or a.transpose(0,1) Squeeze t.squeeze(0) np.squeeze(a, 0) Sum t.sum(axis=0) a.sum(axis=0) Einsum N/A np.einsum("ij,jk->ik", a, b) Scratch vs PyTorch import torch AIENGINEERINGFROMSCRATCH.COM 178

TENSOR OPERATIONS t = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float32) t.shape t.stride() t.is_contiguous() t.reshape(3, 2) t.unsqueeze(0) t.transpose(0, 1) t.transpose(0, 1).contiguous() torch.einsum("ik,kj->ij", A, B) PyTorch adds autograd, GPU support, and optimized BLAS kernels. The shape semantics are identical. If you understand the scratch version, PyTorch shape errors become readable. Every neural network layer as a tensor operation Operation Tensor Form Einsum Linear layer Y = X @ W.T + b "bd,od->bo" + bias Attention QKV Q = X @ W_q "btd,dh->bth" Attention scores Q @ K.T / sqrt(d) "bhtd,bhsd->bhts" Attention output softmax(scores) @ V "bhts,bhsd->bhtd" Batch norm (X - mu) / sigma * gamma element-wise + broadcast Softmax exp(x) / sum(exp(x)) element-wise + reduction This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/12-tensor-operations Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/12-tensor-operations/code 1. Easy – Reshape round-trip. Take a tensor of shape (2, 3, 4). Reshape it to (6, 4), then to (24,), then back to (2, 3, 4). Verify element order is preserved at each step by printing the flat data. 2. Medium – Implement broadcasting. Extend the Tensor class with a broadcast_to(s hape) method that expands dimensions of size 1 to match a target shape. Then modify _elementwise_op to automatically broadcast before operating. Test with shapes (3, 1) and (1, 4) producing (3, 4). 3. Hard – Build einsum from scratch. Implement a basic einsum(subscripts, *tens ors) function that handles at least: dot product (i,i->), matrix multiply (ij,jk->ik), outer product (i,j->ij), and transpose (ij->ji). Parse the subscript string, identify contracted indices, and loop over all index combinations. Compare your results against np.einsum. 4. Hard – Attention shape tracker. Write a function that takes batch_size, seq_len, embed_dim, and num_heads as inputs and prints the exact shape at every step of multi- head attention: input, Q/K/V projection, head split, attention scores, softmax weights, AIENGINEERINGFROMSCRATCH.COM 179

TENSOR OPERATIONS weighted sum, head merge, output projection. Verify against the demo_attention_einsu m() output. Key Terms Term What people say What it actually means Tensor “A matrix but more dimensions” A multi-dimensional array with uniform type and defined shape, strides, and operations Rank “The number of dimensions” The number of axes. A matrix has rank 2, not rank equal to its matrix rank Shape “The size of the tensor” A tuple listing the size along each axis. (2, 3) means 2 rows, 3 columns Stride “How memory is laid out” The number of elements to skip to advance one position along each axis Broadcasting “It just works when shapes differ” A strict set of rules: align from right, dimensions must be equal or one must be 1 Contiguous “The tensor is normal” Elements stored sequentially in memory with no gaps or reordering from the logical layout Einsum “A fancy way to write matmul” A general notation that expresses any tensor contraction, outer product, trace, or transpose in one line View “Same as reshape” A tensor sharing the same memory buffer but with different shape/stride metadata. Fails on non-contiguous data Contraction “Summing over an index” The general operation where a shared index between tensors is multiplied and summed, producing a lower-rank result NCHW / NHWC “PyTorch vs TensorFlow format” Memory layout conventions for image tensors. NCHW puts channels before spatial dims, NHWC puts them after Further Reading • NumPy Broadcasting – The canonical rules with visual examples • PyTorch Tensor Views – When views work and when they copy AIENGINEERINGFROMSCRATCH.COM 180

TENSOR OPERATIONS • einops – A library that makes tensor reshaping readable and safe • The Illustrated Transformer – Visualizes the tensor shapes flowing through attention • Einstein Summation in NumPy – Full einsum documentation with examples Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/12-tensor-operations • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/12-tensor-operations/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/12-tensor-operations The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 181

Numerical Stability Floating point is a leaky abstraction. It will bite you during training, and you will not see it coming. Type: Build Language: Python Prerequisites: Phase 1, Lessons 01-04 Time: ~120 minutes Learning Objectives • Implement numerically stable softmax and log-sum-exp using the max-subtraction trick • Identify overflow, underflow, and catastrophic cancellation in floating-point computa- tions • Verify analytical gradients against numerical gradients using centered finite differences • Explain why bfloat16 is preferred over float16 for training and how loss scaling prevents gradient underflow The Problem Your model trains for three hours, then the loss becomes NaN. You add a print statement. The logits are fine at step 9,000. At step 9,001 they are inf. By step 9,002 every gradient is nan and training is dead. Or: your model trains to completion but accuracy is 2% worse than the paper claims. You check everything. Architecture matches. Hyperparameters match. Data matches. The prob- lem is that the paper used float32 and you used float16 without the right scaling. Thirty-two bits of accumulated rounding error quietly ate your accuracy. Or: you implement cross-entropy loss from scratch. It works on small logits. When logits exceed 100, it returns inf. The softmax overflowed because exp(100) is larger than float32 can represent. Every ML framework handles this with a two-line trick. You did not know the trick existed. Numerical stability is not a theoretical concern. It is the difference between a training run that succeeds and one that silently fails. Every serious ML bug you will debug eventually comes down to floating point. The Concept IEEE 754: How Computers Store Real Numbers Computers store real numbers as floating point values following the IEEE 754 standard. A float has three parts: a sign bit, an exponent, and a mantissa (significand). Float32 layout (32 bits total): [1 sign] [8 exponent] [23 mantissa] Value = (-1)^sign * 2^(exponent - 127) * 1.mantissa AIENGINEERINGFROMSCRATCH.COM 182

NUMERICAL STABILITY The mantissa determines precision (how many significant digits). The exponent determines range (how large or small a number can be). Format Bits Exponent Mantissa Decimal digits Range (approx) float64 64 11 52 ~15-16 +/- 1.8e308 float32 32 8 23 ~7-8 +/- 3.4e38 float16 16 5 10 ~3-4 +/- 65,504 bfloat16 16 8 7 ~2-3 +/- 3.4e38 float32 gives you about 7 decimal digits of precision. That means it can tell apart 1.0000001 and 1.0000002, but not 1.00000001 and 1.00000002. After 7 digits, everything is rounding noise. float16 gives you about 3 digits. The largest number it can represent is 65,504. That is disturbingly small for ML where logits, gradients, and activations routinely exceed this. bfloat16 is Google’s answer to float16’s range problem. It has the same 8-bit exponent as float32 (same range, up to 3.4e38) but only 7 mantissa bits (less precision than float16). For training neural networks, range matters more than precision, so bfloat16 usually wins. Why 0.1 + 0.2 != 0.3 The number 0.1 cannot be represented exactly in binary floating point. In base 2, it is a repeating fraction: 0.1 in binary = 0.0001100110011001100110011... (repeating forever) Float32 truncates this to 23 bits of mantissa. The stored value is approximately 0.100000001490116. Similarly, 0.2 is stored as approximately 0.200000002980232. Their sum is 0.300000004470348, not 0.3. In Python: >>> 0.1 + 0.2 0.30000000000000004 >>> 0.1 + 0.2 == 0.3 False This matters for ML because: 1. Loss comparisons like if loss < threshold can give wrong answers 2. Accumulating many small values (gradient updates over thousands of steps) drifts from the true sum 3. Checksums and reproducibility tests fail if you compare floats with == The fix: never compare floats with ==. Use abs(a - b) < epsilon or math.isclose(). Catastrophic Cancellation When you subtract two nearly equal floating point numbers, the significant digits cancel and you are left with rounding noise promoted to leading digits. a = 1.0000001 (stored as 1.00000011920929 in float32) b = 1.0000000 (stored as 1.00000000000000 in float32) True difference: 0.0000001 Computed: 0.00000011920929 Relative error: 19.2% AIENGINEERINGFROMSCRATCH.COM 183

NUMERICAL STABILITY That is a 19% relative error from a single subtraction. In ML, this happens whenever you: • Compute variance of data with a large mean: E[x^2] - E[x]^2 when E[x] is large • Subtract nearly equal log-probabilities • Compute finite-difference gradients with too-small epsilon The fix: rearrange formulas to avoid subtracting large, nearly equal numbers. For variance, use the Welford algorithm or center the data first. For log-probabilities, work in log-space throughout. Overflow and Underflow Overflow happens when a result is too large to represent. Underflow happens when it is too small (closer to zero than the smallest representable positive number). Float32 boundaries: Maximum: 3.4028235e+38 Minimum positive (normal): 1.175e-38 Minimum positive (denorm): 1.401e-45 Overflow: anything > 3.4e38 becomes inf Underflow: anything < 1.4e-45 becomes 0.0 The exp() function is the primary source of overflow in ML: exp(88.7) = 3.40e+38 (barely fits in float32) exp(89.0) = inf (overflow) exp(-87.3) = 1.18e-38 (barely above underflow) exp(-104) = 0.0 (underflow to zero) The log() function hits the other direction: log(0.0) = -inf log(-1.0) = nan log(1e-45) = -103.3 (fine) log(1e-46) = -inf (input underflowed to 0, then log(0) = -inf) In ML, exp() appears in softmax, sigmoid, and probability computations. log() appears in cross-entropy, log-likelihoods, and KL divergence. The combination log(exp(x)) is a mine- field without the right tricks. The Log-Sum-Exp Trick Computing log(sum(exp(x_i))) directly is numerically dangerous. If any x_i is large, exp(x _i) overflows. If all x_i are very negative, every exp(x_i) underflows to zero and log(0) is -inf. The trick: subtract the maximum value before exponentiating. log(sum(exp(x_i))) = max(x) + log(sum(exp(x_i - max(x)))) Why this works: after subtracting max(x), the largest exponent is exp(0) = 1. No overflow is possible. At least one term in the sum is 1, so the sum is at least 1, and log(1) = 0. No underflow to -inf is possible. Proof: log(sum(exp(x_i))) = log(sum(exp(x_i - c + c))) (add and subtract c) = log(sum(exp(x_i - c) * exp(c))) (exp(a+b) = exp(a)*exp(b)) AIENGINEERINGFROMSCRATCH.COM 184

NUMERICAL STABILITY = log(exp(c) * sum(exp(x_i - c))) (factor out exp(c)) = c + log(sum(exp(x_i - c))) (log(a*b) = log(a) + log(b)) Set c = max(x) and overflow is eliminated. This trick appears everywhere in ML: - Softmax normalization - Cross-entropy loss compu- tation - Log-probability summation in sequence models - Mixture of Gaussians - Variational inference Why Softmax Needs the Max-Subtraction Trick Softmax converts logits to probabilities: softmax(x_i) = exp(x_i) / sum(exp(x_j)) Without the trick, logits of [100, 101, 102] cause overflow: exp(100) = 2.69e43 exp(101) = 7.31e43 exp(102) = 1.99e44 sum = 2.99e44 These overflow float32 (max ~3.4e38)? No, 2.69e43 < 3.4e38? Actually: exp(88.7) is already at the float32 limit. exp(100) = inf in float32. With the trick, subtract max(x) = 102: exp(100 - 102) = exp(-2) = 0.135 exp(101 - 102) = exp(-1) = 0.368 exp(102 - 102) = exp(0) = 1.000 sum = 1.503 softmax = [0.090, 0.245, 0.665] The probabilities are identical. The computation is safe. This is not an optimization. It is a requirement for correctness. NaN and Inf: Detection and Prevention nan (Not a Number) and inf (infinity) propagate virally through computation. One nan in a gradient update makes the weight nan, which makes every subsequent output nan. Training is dead within one step. How inf appears: - exp() of a large positive number - Division by zero: 1.0 / 0.0 - float32 overflow in accumulations How nan appears: - 0.0 / 0.0 - inf - inf - inf * 0 - sqrt() of a negative number - log() of a negative number - Any arithmetic involving an existing nan Detection: import math math.isnan(x) # True if x is nan math.isinf(x) # True if x is +inf or -inf math.isfinite(x) # True if x is neither nan nor inf Prevention strategies: 1. Clamp inputs to exp(): exp(clamp(x, -80, 80)) AIENGINEERINGFROMSCRATCH.COM 185

NUMERICAL STABILITY 2. Add epsilon to denominators: x / (y + 1e-8) 3. Add epsilon inside log(): log(x + 1e-8) 4. Use stable implementations (log-sum-exp, stable softmax) 5. Gradient clipping to prevent weight explosion 6. Check for nan/inf after every forward pass during debugging Numerical Gradient Checking Analytical gradients (from backpropagation) can have bugs. Numerical gradient checking verifies them by computing gradients with finite differences. The centered difference formula: df/dx ~= (f(x + h) - f(x - h)) / (2h) This is O(h^2) accurate, much better than the forward difference (f(x+h) - f(x)) / h which is only O(h). Choosing h: too large and the approximation is wrong. Too small and catastrophic cancella- tion destroys the answer. h = 1e-5 to 1e-7 is typical. The check: compute the relative difference between analytical and numerical gradients. relative_error = |grad_analytical - grad_numerical| / max(|grad_analytical|, |grad_numerical|, 1e-8)↪ Rules of thumb: - relative_error < 1e-7: perfect, gradient is correct - relative_error < 1e-5: acceptable, probably correct - relative_error > 1e-3: something is wrong - relative_error > 1: gradient is completely wrong Always check gradients when implementing a new layer or loss function. PyTorch provides torch.autograd.gradcheck() for this. Mixed Precision Training Modern GPUs have specialized hardware (Tensor Cores) that compute float16 matrix multi- plications 2-8x faster than float32. Mixed precision training exploits this: 1. Maintain float32 master copy of weights 2. Forward pass in float16 (fast) 3. Compute loss in float32 (prevents overflow) 4. Backward pass in float16 (fast) 5. Scale gradients to float32 6. Update float32 master weights The problem with pure float16 training: gradients are often very small (1e-8 or smaller). Float16 underflows anything below ~6e-8 to zero. Your model stops learning because all gradient updates are zero. The fix is loss scaling: 1. Multiply loss by a large scale factor (e.g., 1024) 2. Backward pass computes gradients of (loss * 1024) 3. All gradients are 1024x larger (pushed above float16 underflow) 4. Divide gradients by 1024 before updating weights 5. Net effect: same update, but no underflow Dynamic loss scaling adjusts the scale factor automatically. Start with a large value (65536). If gradients overflow to inf, halve it. If N steps pass without overflow, double it. AIENGINEERINGFROMSCRATCH.COM 186

NUMERICAL STABILITY bfloat16 vs float16: Why bfloat16 Wins for Training float16: [1 sign] [5 exponent] [10 mantissa] bfloat16: [1 sign] [8 exponent] [7 mantissa] float16 has more precision (10 mantissa bits vs 7) but limited range (max ~65,504). bfloat16 has less precision but the same range as float32 (max ~3.4e38). For training neural networks: • Activations and logits regularly exceed 65,504 during training spikes. float16 overflows; bfloat16 handles it. • Loss scaling is required with float16 but usually unnecessary with bfloat16 because its range covers the gradient magnitude spectrum. • bfloat16 is a simple truncation of float32: drop the bottom 16 bits of the mantissa. Con- version is trivial and lossless in the exponent. float16 is preferred for inference where values are bounded and precision matters more. bfloat16 is preferred for training where range matters more. This is why TPUs and modern NVIDIA GPUs (A100, H100) have native bfloat16 support. Gradient Clipping Exploding gradients happen when gradients grow exponentially through many layers (com- mon in RNNs, deep networks, and transformers). A single large gradient can corrupt all weights in one step. Two types of clipping: Clip by value: clamp each gradient element independently. grad = clamp(grad, -max_val, max_val) Simple but can change the direction of the gradient vector. Clip by norm: scale the entire gradient vector so its norm does not exceed a threshold. if ||grad|| > max_norm: grad = grad * (max_norm / ||grad||) Preserves the direction of the gradient. This is what torch.nn.utils.clip_grad_norm_() does. It is the standard choice. Typical values: max_norm=1.0 for transformers, max_norm=0.5 for RL, max_norm=5.0 for simpler networks. Gradient clipping is not a hack. It is a safety mechanism. Without it, a single outlier batch can produce a gradient large enough to ruin weeks of training. Normalization Layers as Numerical Stabilizers Batch normalization, layer normalization, and RMS normalization are usually presented as regularizers that help training converge. They are also numerical stabilizers. Without normalization, activations can grow or shrink exponentially through layers: Layer 1: values in [0, 1] Layer 5: values in [0, 100] Layer 10: values in [0, 10,000] Layer 50: values in [0, inf] Normalization recenters and rescales activations at every layer: AIENGINEERINGFROMSCRATCH.COM 187

NUMERICAL STABILITY LayerNorm(x) = (x - mean(x)) / (std(x) + epsilon) * gamma + beta The epsilon (typically 1e-5) prevents division by zero when all activations are identical. The learned parameters gamma and beta let the network restore any scale it needs. This keeps values in a numerically safe range throughout the network, preventing both over- flow in the forward pass and gradient explosion in the backward pass. Common ML Numerical Bugs Bug: Loss is NaN after a few epochs. Cause: logits grew too large, softmax overflowed. Or learning rate is too high and weights diverged. Fix: use stable softmax (max subtraction), reduce learning rate, add gradient clipping. Bug: Loss is stuck at log(num_classes). Cause: model outputs are near-uniform proba- bilities. Often means gradients are vanishing or the model is not learning at all. Fix: check that data labels are correct, verify the loss function, check for dead ReLUs. Bug: Validation accuracy is lower than expected by 1-3%. Cause: mixed precision without proper loss scaling. Gradient underflow silently zeroes out small updates. Fix: enable dynamic loss scaling, or switch to bfloat16. Bug: Gradient norms are 0.0 for some layers. Cause: dead ReLU neurons (all inputs negative), or float16 underflow. Fix: use LeakyReLU or GELU, use gradient scaling, check weight initialization. Bug: Model works on one GPU but gives different results on another. Cause: non- deterministic floating point accumulation order. GPU parallel reductions sum in different orders on different hardware, and floating point addition is not associative. Fix: accept small differences (1e-6), or set torch.use_deterministic_algorithms(True) and accept the speed penalty. Bug: exp() returns inf in loss computation. Cause: raw logits passed to exp() without the max-subtraction trick. Fix: use torch.nn.functional.log_softmax() which implements log-sum-exp internally. Bug: Training diverges after switching from float32 to float16. Cause: float16 can- not represent gradient magnitudes below 6e-8 or activations above 65,504. Fix: use mixed precision with loss scaling (AMP), or use bfloat16 instead. Interactive figure: logsumexp-stability. This one moves. Watch it animate and drag its controls in the web edition: https://aiengineeringfromscratch.com/lesson?path=phases/01- math-foundations/13-numerical-stability Build It Step 1: Demonstrate floating point precision limits print("=== Floating Point Precision ===") print(f"0.1 + 0.2 = {0.1 + 0.2}") print(f"0.1 + 0.2 == 0.3? {0.1 + 0.2 == 0.3}") print(f"Difference: {(0.1 + 0.2) - 0.3:.2e}") Step 2: Implement naive vs stable softmax import math AIENGINEERINGFROMSCRATCH.COM 188

NUMERICAL STABILITY def softmax_naive(logits): exps = [math.exp(z) for z in logits] total = sum(exps) return [e / total for e in exps] def softmax_stable(logits): max_logit = max(logits) exps = [math.exp(z - max_logit) for z in logits] total = sum(exps) return [e / total for e in exps] safe_logits = [2.0, 1.0, 0.1] print(f"Naive: {softmax_naive(safe_logits)}") print(f"Stable: {softmax_stable(safe_logits)}") dangerous_logits = [100.0, 101.0, 102.0] print(f"Stable: {softmax_stable(dangerous_logits)}") # softmax_naive(dangerous_logits) would return [nan, nan, nan] Step 3: Implement stable log-sum-exp def logsumexp_naive(values): return math.log(sum(math.exp(v) for v in values)) def logsumexp_stable(values): c = max(values) return c + math.log(sum(math.exp(v - c) for v in values)) safe = [1.0, 2.0, 3.0] print(f"Naive: {logsumexp_naive(safe):.6f}") print(f"Stable: {logsumexp_stable(safe):.6f}") large = [500.0, 501.0, 502.0] print(f"Stable: {logsumexp_stable(large):.6f}") # logsumexp_naive(large) returns inf Step 4: Implement stable cross-entropy def cross_entropy_naive(true_class, logits): probs = softmax_naive(logits) return -math.log(probs[true_class]) def cross_entropy_stable(true_class, logits): max_logit = max(logits) shifted = [z - max_logit for z in logits] log_sum_exp = math.log(sum(math.exp(s) for s in shifted)) log_prob = shifted[true_class] - log_sum_exp return -log_prob logits = [2.0, 5.0, 1.0] true_class = 1 print(f"Naive: {cross_entropy_naive(true_class, logits):.6f}") print(f"Stable: {cross_entropy_stable(true_class, logits):.6f}") AIENGINEERINGFROMSCRATCH.COM 189

NUMERICAL STABILITY Step 5: Gradient checking def numerical_gradient(f, x, h=1e-5): grad = [] for i in range(len(x)): x_plus = x[:] x_minus = x[:] x_plus[i] += h x_minus[i] -= h grad.append((f(x_plus) - f(x_minus)) / (2 * h)) return grad def check_gradient(analytical, numerical, tolerance=1e-5): for i, (a, n) in enumerate(zip(analytical, numerical)): denom = max(abs(a), abs(n), 1e-8) rel_error = abs(a - n) / denom status = "OK" if rel_error < tolerance else "FAIL" print(f" param {i}: analytical={a:.8f} numerical={n:.8f} " f"rel_error={rel_error:.2e} [{status}]") def f(params): x, y = params return x**2 + 3*x*y + y**3 def f_grad(params): x, y = params return [2*x + 3*y, 3*x + 3*y**2] point = [2.0, 1.0] analytical = f_grad(point) numerical = numerical_gradient(f, point) check_gradient(analytical, numerical) Use It Mixed precision simulation import struct def float32_to_float16_round(x): packed = struct.pack('f', x) f32 = struct.unpack('f', packed)[0] packed16 = struct.pack('e', f32) return struct.unpack('e', packed16)[0] def simulate_bfloat16(x): packed = struct.pack('f', x) as_int = int.from_bytes(packed, 'little') truncated = as_int & 0xFFFF0000 repacked = truncated.to_bytes(4, 'little') return struct.unpack('f', repacked)[0] AIENGINEERINGFROMSCRATCH.COM 190

NUMERICAL STABILITY Gradient clipping def clip_by_norm(gradients, max_norm): total_norm = math.sqrt(sum(g**2 for g in gradients)) if total_norm > max_norm: scale = max_norm / total_norm return [g * scale for g in gradients] return gradients grads = [10.0, 20.0, 30.0] clipped = clip_by_norm(grads, max_norm=5.0) print(f"Original norm: {math.sqrt(sum(g**2 for g in grads)):.2f}") print(f"Clipped norm: {math.sqrt(sum(g**2 for g in clipped)):.2f}") print(f"Direction preserved: {[c/clipped[0] for c in clipped]} == {[g/grads[0] for g in grads]}")↪ NaN/Inf detection def check_tensor(name, values): has_nan = any(math.isnan(v) for v in values) has_inf = any(math.isinf(v) for v in values) if has_nan or has_inf: print(f"WARNING {name}: nan={has_nan} inf={has_inf}") return False return True check_tensor("good", [1.0, 2.0, 3.0]) check_tensor("bad", [1.0, float('nan'), 3.0]) check_tensor("ugly", [1.0, float('inf'), 3.0]) See code/numerical.py for complete implementations with all edge cases demonstrated. This chapter ships an artifact. The course version of this lesson produces a reusable prompt or agent skill. It lives in the repository, ready to install: https://github.com/rohitg00/ai- engineering-from-scratch/tree/main/phases/01-math-foundations/13-numerical-stability Exercises Starter code and the lesson’s working implementation: https://github.com/rohitg00/ai-engi neering-from-scratch/tree/main/phases/01-math-foundations/13-numerical-stability/code 1. Catastrophic cancellation. Compute the variance of [1000000.0, 1000001.0, 1000002.0] using the naive formula E[x^2] - E[x]^2 in float32. Then compute it using Welford’s online algorithm. Compare the errors against the true variance (0.6667). 2. Precision hunt. Find the smallest positive float32 value x such that 1.0 + x == 1.0 in Python. This is the machine epsilon. Verify it matches numpy.finfo(numpy.float32).eps. 3. Log-sum-exp edge cases. Test your logsumexp_stable function with: (a) all values equal, (b) one value much larger than the rest, (c) all values very negative (-1000). Verify it gives correct results where the naive version fails. 4. Gradient checking a neural network layer. Implement a single linear layer y = Wx + b and its analytical backward pass. Use numerical_gradient to verify correctness for a 3x2 weight matrix. AIENGINEERINGFROMSCRATCH.COM 191

NUMERICAL STABILITY 5. Loss scaling experiment. Simulate training with float16: create random gradients in the range [1e-9, 1e-3], convert to float16, and measure what fraction become zero. Then apply loss scaling (multiply by 1024), convert to float16, scale back, and measure the zero fraction again. Key Terms Term What people say What it actually means IEEE 754 “The float standard” International standard defining binary floating point formats, rounding rules, and special values (inf, nan). Every modern CPU and GPU implements it. Machine epsilon “The precision limit” The smallest value e such that 1.0 + e != 1.0 in a given float format. For float32, it is about 1.19e-7. Catastrophic cancella- tion “Precision loss from subtraction” When subtracting nearly equal floating point numbers, significant digits cancel and rounding noise dominates the result. Overflow “Number too big” A result exceeds the maximum representable value and becomes inf. exp(89) overflows float32. Underflow “Number too small” A result is closer to zero than the smallest representable positive number and becomes 0.0. exp(-104) underflows float32. Log-sum- exp trick “Subtract the max first” Computing log(sum(exp(x))) by factoring out exp(max(x)) to prevent overflow and underflow. Used in softmax, cross-entropy, and log-probability math. Stable softmax “Softmax that does not explode” Subtracting max(logits) before exponentiating. Numerically identical result, no overflow possible. Gradient checking “Verify your backprop” Comparing analytical gradients from backpropagation against numerical gradients from finite differences to catch implementation bugs. Mixed precision “Float16 forward, float32 backward” Using lower-precision floats for speed-critical operations and higher-precision floats for numerically sensitive operations. Typical speedup is 2-3x. Loss scaling “Prevent gradient underflow” Multiplying the loss by a large constant before backprop so gradients stay in float16’s representable range, then dividing by the same constant before weight updates. bfloat16 “Brain floating point” Google’s 16-bit format with 8 exponent bits (same range as float32) and 7 mantissa bits (less precision than float16). Preferred for training. AIENGINEERINGFROMSCRATCH.COM 192

NUMERICAL STABILITY Term What people say What it actually means Gradient clipping “Cap the gradient norm” Scaling the gradient vector so its norm does not exceed a threshold. Prevents exploding gradients from ruining weights. NaN “Not a Number” Special float value from undefined operations (0/0, inf-inf, sqrt(-1)). Propagates through all subsequent arithmetic. Inf “Infinity” Special float value from overflow or division by zero. Can combine to produce NaN (inf - inf, inf * 0). Numerical gradient “Brute force derivative” Approximating a derivative by evaluating f(x+h) and f(x-h) and dividing by 2h. Slow but reliable for verification. Further Reading • What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg 1991) – the definitive reference, dense but complete • Mixed Precision Training (Micikevicius et al., 2018) – the NVIDIA paper that introduced loss scaling for float16 training • AMP: Automatic Mixed Precision (PyTorch docs) – practical guide to mixed precision in PyTorch • bfloat16 format (Google Cloud TPU docs) – why Google chose this format for TPUs • Kahan Summation (Wikipedia) – algorithm for reducing rounding error in floating point sums Continue online. The living edition of this chapter has more than the page can hold: • Animated, interactive figures and the web text: https://aiengineeringfromscratch.com/l esson?path=phases/01-math-foundations/13-numerical-stability • Runnable code for every step: https://github.com/rohitg00/ai-engineering-from-scratc h/tree/main/phases/01-math-foundations/13-numerical-stability/code • The chapter quiz, graded in the browser: https://aiengineeringfromscratch.com/lesson? path=phases/01-math-foundations/13-numerical-stability The repository moves faster than any printing. When the book and the repo disagree, trust the repo. AIENGINEERINGFROMSCRATCH.COM 193

Norms and Distances Your distance function defines what “similar” means. Choose wrong and everything downstream breaks. Type: Build Language: Python Prerequisites: Phase 1, Lessons 01 (Linear Algebra Intu- ition), 02 (Vectors, Matrices & Operations) Time: ~90 minutes Learning Objectives • Implement L1, L2, cosine, Mahalanobis, Jaccard, and edit distance functions from scratch • Select the appropriate distance metric for a given ML task and explain why alternatives fail • Connect L1 and L2 norms to LASSO and Ridge regularization and their geometric con- straint regions • Demonstrate how the same dataset produces different nearest neighbors under different metrics The Problem You have two vectors. Maybe they are word embeddings. Maybe they are user profiles. Maybe they are pixel arrays. You need to know: how close are they? The answer depends entirely on which distance function you pick. Two data points can be nearest neighbors under one metric and far apart under another. Your KNN classifier, your recommendation engine, your vector database, your clustering algorithm, your loss function – they all depend on this choice. Get it wrong and your model optimizes for the wrong thing. There is no universal best distance. L2 works for spatial data. Cosine similarity dominates NLP. Jaccard handles sets. Edit distance handles strings. Mahalanobis accounts for correla- tions. Wasserstein moves probability mass. Each one encodes a different assumption about what “similar” means. This lesson builds every major distance function from scratch, shows you when each one is the right tool, and demonstrates how the same data produces completely different nearest neighbors depending on which metric you use. The Concept Norms: measuring vector magnitude A norm measures the “size” of a vector. Every distance function between two vectors can be written as the norm of their difference: d(a, b) = ||a - b||. So understanding norms is understanding distances. AIENGINEERINGFROMSCRATCH.COM 194

NORMS AND DISTANCES L1 Norm (Manhattan distance) The L1 norm sums the absolute values of all components. ||x||_1 = |x_1| + |x_2| + ... + |x_n| It is called Manhattan distance because it measures how far you walk on a city grid where you can only move along axes. No diagonals. Point A = (1, 1) Point B = (4, 5) L1 distance = |4-1| + |5-1| = 3 + 4 = 7 On a grid, you walk 3 blocks east and 4 blocks north. When to use L1: - High-dimensional sparse data (text features, one-hot encodings) - When you want robustness to outliers (a single huge difference does not dominate) - Feature selection problems (L1 regularization promotes sparsity) Connection to L1 regularization (Lasso): adding ||w||_1 to your loss function penalizes the sum of absolute weight values. This pushes small weights to exactly zero, performing automatic feature selection. The L1 penalty creates diamond-shaped constraint regions in weight space, and the corners of diamonds lie on the axes where some weights are zero. Connection to loss functions: Mean Absolute Error (MAE) is the average L1 distance between predictions and targets. It penalizes all errors linearly, making it robust to outliers compared to MSE. L2 Norm (Euclidean distance) The L2 norm is the straight-line distance. Square root of the sum of squared components. ||x||_2 = sqrt(x_1^2 + x_2^2 + ... + x_n^2) This is the distance you learned in geometry class. Pythagoras in n dimensions. Point A = (1, 1) Point B = (4, 5) L2 distance = sqrt((4-1)^2 + (5-1)^2) = sqrt(9 + 16) = sqrt(25) = 5.0 The straight line, cutting diagonally through the grid. When to use L2: - Low-to-medium dimensional continuous data - When the feature scales are comparable - Physical distances (spatial data, sensor readings) - Image similarity at the pixel level Connection to L2 regularization (Ridge): adding ||w||_2^2 to your loss function penalizes large weights. Unlike L1, it does not push weights to zero. It shrinks all weights toward zero proportionally. The L2 penalty creates circular constraint regions, so there are no corners on axes. Weights get small but rarely exactly zero. Connection to loss functions: Mean Squared Error (MSE) is the average of L2 distances squared. Squaring penalizes large errors more heavily than small ones. MAE (L1 loss): |y - y_hat| Linear penalty. Robust to outliers. MSE (L2 loss): (y - y_hat)^2 Quadratic penalty. Sensitive to outliers. AIENGINEERINGFROMSCRATCH.COM 195

NORMS AND DISTANCES Lp Norms: the general family L1 and L2 are special cases of the Lp norm: ||x||_p = (|x_1|^p + |x_2|^p + ... + |x_n|^p)^(1/p) Different values of p produce different shaped “unit balls” (the set of all points at distance 1 from the origin): p=1: Diamond shape (corners on axes) p=2: Circle/sphere (the usual round ball) p=3: Superellipse (rounded square) p=inf: Square/hypercube (flat sides along axes) L-infinity Norm (Chebyshev distance) As p approaches infinity, the Lp norm converges to the maximum absolute component. ||x||_inf = max(|x_1|, |x_2|, ..., |x_n|) The distance between two points is determined by the single dimension where they differ the most. All other dimensions are ignored. Point A = (1, 1) Point B = (4, 5) L-inf distance = max(|4-1|, |5-1|) = max(3, 4) = 4 When to use L-infinity: - When the worst-case deviation in any single dimension matters - Game boards (a king in chess moves in L-infinity: one step in any direction costs 1) - Manu- facturing tolerances (every dimension must be within spec) Cosine Similarity and Cosine Distance Cosine similarity measures the angle between two vectors, ignoring their magnitudes. cos_sim(a, b) = (a . b) / (||a||_2 * ||b||_2) It ranges from -1 (opposite directions) to +1 (same direction). Perpendicular vectors have cosine similarity 0. Cosine distance converts it to a distance: cosine_distance = 1 - cosine_similarity. This ranges from 0 (identical direction) to 2 (opposite direction). a = (1, 0) b = (1, 1) cos_sim = (1*1 + 0*1) / (1 * sqrt(2)) = 1/sqrt(2) = 0.707 cos_dist = 1 - 0.707 = 0.293 Why cosine dominates NLP and embeddings: in text, document length should not affect sim- ilarity. A document about cats that is twice as long as another document about cats should still be “similar.” Cosine similarity ignores magnitude (length) and only cares about direc- tion. Two documents with the same word distribution but different lengths point in the same direction and get cosine similarity 1.0. When to use cosine similarity: - Text similarity (TF-IDF vectors, word embeddings, sentence embeddings) - Any domain where magnitude is noise and direction is signal - Recommendation systems (user preference vectors) - Embedding search (vector databases almost always use cosine or dot product) AIENGINEERINGFROMSCRATCH.COM 196