That One Memory Copy Is Killing Your Attention Speed: A PyTorch Profiler Deep Dive

Most engineers know the theory of PyTorch profiling, but watching a real trace expose a hidden inefficiency is something else entirely. In the third installment of the ‘Profiling in PyTorch’ series, the investigation moves from basic linear layers to the heart of the Transformer: the attention mechanism. We’re not here for the quadratic complexity lecture. We’re here to see what the profiler actually tells us when we run naive attention code on an NVIDIA A100.

The initial profile of a standard attention forward pass reveals exactly what you’d expect: two matmuls, a scale multiplication, causal masking, and a softmax. But the GPU trace doesn’t lie. Nestled between the expected operations is a glaring anomaly—a memory copy. As the analysis points out, this wasted operation stems from PyTorch’s default out-of-place behavior for masked_fill. The framework silently duplicates the scores tensor, applies the mask to the copy, and discards the original.

The fix is a single, almost invisible character: an underscore. Swapping scores.masked_fill(...) for the in-place scores.masked_fill_(...) eliminates the redundant memory allocation and transfer. It’s a tiny code change that the profiler instantly validates by making the offending copy vanish from the trace. This isn’t advanced kernel fusion; it’s just knowing how PyTorch manages tensor memory. The series continues to build a crucial skill—not just reading a trace, but developing a gut feeling for what operations should cost, and getting suspicious when they cost more.