https://blog.baifan.site/en/tags/attention-mechanism/Recent content in Attention Mechanism on ByF (BaiFan) · Daily Deep ThinkHugo -- gohugo.ioenblog@baifan.site (ByF (BaiFan))blog@baifan.site (ByF (BaiFan))CC BY-NC 4.0Tue, 05 Aug 2025 14:00:00 +0800https://blog.baifan.site/en/transformer-attention-mechanism-deep-dive/Tue, 05 Aug 2025 14:00:00 +0800blog@baifan.site (ByF (BaiFan))https://blog.baifan.site/en/transformer-attention-mechanism-deep-dive/<p>The physical barrier modern large language models confront when scaling context length is rarely raw floating-point throughput (FLOPs). It is the memory-access wall between high-bandwidth memory (HBM) and on-chip static random-access memory (SRAM). The standard Scaled Dot-Product Attention operator established by Vaswani et al. (2017) incurs an $O(N^2)$ memory-traffic overhead as sequence length $N$ grows, reducing self-attention primarily to a memory-bound workload.</p> <!-- more --> <p>Understanding the Transformer requires stripping away anthropomorphic cognitive metaphors and grounding the discussion in the geometry of matrix projections alongside the hardware constraints of modern GPUs.</p>