GW6 Small-Head Attention Sweep: Train-MSE, LPIPS, and CLIP

Methodological Note

This page is an exploratory sweep over attention-module hyperparameters. I trained the full grid over number of heads, head size, and learning rate, using the frozen GW model. The attention modules were optimized only for normalized RepTok latent MSE; LPIPS and CLIP are measured afterward as downstream diagnostics of image quality and caption-image alignment.

There is a double-dipping risk if we simply keep the attention run with the best LPIPS or CLIP score on this page: we would be selecting a model based on the same readout used to evaluate it, and a partly random advantage could be mistaken for a stable rule. The safer interpretation is to use this grid to understand how attention capacity and optimization settings affect the downstream metrics, not to claim a final best model.

A cleaner final protocol would choose the attention configuration from a predeclared proxy, for example a latent metric or scalar-grid objective that was previously associated with better LPIPS or CLIP, or from a separate validation split. The final result should then be evaluated once on held-out tasks, ideally across the full radar plot rather than only BGE-to-RepTok. This page is therefore a preliminary tool for deciding what attention settings are worth testing next.