Softmax attention can be interpreted as a Nadaraya-Watson kernel regression estimator with kernel $K(q, k) = \exp(q^T k / \sqrt{d})$. What does the $\sqrt{d}$ factor in the denominator correspond to in the kernel regression interpretation?

Attention as Kernel Regression

1 selectedDifficulty 6-61 unseenView topic

IntermediateNew

0 answered

1 intermediateAdapts to your performance

Question 1 of 1

120sintermediate (6/10)conceptual

Softmax attention can be interpreted as a Nadaraya-Watson kernel regression estimator with kernel K (q, k) = exp (q^{T} k / d) . What does the d factor in the denominator correspond to in the kernel regression interpretation?