CPU, GPU, and Training Large Language Models
AI Tutorial: CPU/GPU and Large Model Training # This is a highly condensed reference: clearly structured, right to the point — covering CPU/GPU fundamentals, tensors and numerical precision, CUDA and PyTorch in practice, hardware selection, common interview questions, and a debugging checklist. 0. Quick Overview (30 Seconds) # CPU vs GPU: CPUs excel at general-purpose/sequential work; GPUs excel at massive parallelism (matrices/vectors). Large models need GPUs: training/inference is fundamentally matrix multiplication and parallelization — exactly what a GPU’s high concurrency + high-bandwidth memory deliver. Tensors and precision: all data becomes tensors; precision (FP16/FP8) and quantization (INT8/INT4) trade speed/VRAM against quality. The PyTorch GPU mantra: device = "cuda" if ...; model.to(device); data.to(device) Pick a GPU by VRAM first: VRAM first, then bandwidth/compute; for production, prefer full-strength high-quality models or cloud-hosted APIs. 1. CPU vs GPU: Differences, Workloads, and Analogies # 1.1 The One-Line Comparison # Dimension CPU GPU Architecture Few cores, complex control flow Massive small cores, SIMT parallelism Excels at Branching/system tasks/small-scale compute Matrix multiplication, convolution, attention, graphics rendering Task model Time-sliced, low-latency switching Batch- and throughput-oriented Typical use Business logic, scheduling, I/O Main training/inference operators (GEMM, Conv, etc.) 1.2 An Intuitive Analogy # CPU = a veteran expert: meticulous thinking, does one thing at a time with fast switching. GPU = a massive army: hordes of soldiers working simultaneously — built for parallel homogeneous small tasks. 1.3 Optional Mermaid Diagram (CPU Execution vs GPU Parallelism) # 1 2 3 4 5 6 7 8 9 10 11 flowchart LR subgraph CPU["CPU (sequential/few cores)"] A1[Task1-SliceA] --> A2[Task2-SliceB] --> A3[Task3-SliceC] end subgraph GPU["GPU (parallel/many cores)"] B1[Element1 compute]:::p B2[Element2 compute]:::p B3[Element3 compute]:::p B4[Element4 compute]:::p end classDef p fill:#e9f5ff,stroke:#3b82f6,stroke-width:1px; 2. Tensors, Precision, and Quantization (with Examples) # 2.1 Tensor Hierarchy # 0D: scalar 3.14 1D: vector [1,2,3] 2D: matrix (e.g. a 3×3 table) 3D+: still called a tensor (e.g. batch×channel×height×width) Image example: a batch of 32 224×224 RGB images → 32×3×224×224 (or N×H×W×C, depending on the framework).