Transformer's foundation unchanged for nine years: breaking down 8 core components, distinguishing legacy from renovation
AI summary
Since "Attention Is All You Need" in 2017, models such as GPT, BERT, and LLaMA still share the same Transformer foundation; nine years of changes have concentrated on "renovation" layers such as RoPE, RMSNorm, SwiGLU, and GQA. The article breaks down the pipeline into 8 components: tokens and embeddings, positional encoding, self-attention, multi-head attention, causal masking, feed-forward networks, residual connections and normalization, and the overall assembly; among them, the self-attention formula softmax(QKᵀ/√dₖ)V remains untouched. LLaMA-7B's embedding dimension expanded from 512 in 2017 to 4096, and Llama 2 70B uses GQA to let 64 query heads share 8 groups of KV, saving about 8 times the KV cache memory.
Body
Only a summary is authorized here. Continue at the validated source link.