xarray-sql benchmark: Implementing Transformer attention in SQL with automatic gradient computation
AI summary
The benchmarks/ml/01_attention.py file in the xarray-sql repository demonstrates expressing causal attention softmax(Q·Kᵀ/√d)·V in SQL, running it on three engines—DataFusion, DataFusion with einfold, and DuckDB—with results aligned with the JAX implementation. Without writing attention-specific rules, einfold can automatically discover a FlashAttention-style tiled fusion plan from the SQL, while ddx writes the gradients of the loss with respect to Q, K, and V as a single SQL statement, which is then verified by jax.grad.
Body
Only a summary is authorized here. Continue at the validated source link.