←News detail

Hacker News TrendingUpdated Oct 31 07:56

xarray-sql benchmark: Implementing Transformer attention in SQL with automatic gradient computation

AI summary

The benchmarks/ml/01_attention.py file in the xarray-sql repository demonstrates expressing causal attention softmax(Q·Kᵀ/√d)·V in SQL, running it on three engines—DataFusion, DataFusion with einfold, and DuckDB—with results aligned with the JAX implementation. Without writing attention-specific rules, einfold can automatically discover a FlashAttention-style tiled fusion plan from the SQL, while ddx writes the gradients of the loss with respect to Q, K, and V as a single SQL statement, which is then verified by jax.grad.

Body

Only a summary is authorized here. Continue at the validated source link.

Source

Hacker News Trending