RepTok-v-Sonar Stage-1 BLEU
BLEU
Rows with source matched 1000-sample controller eval use the exact same ImageNet validation indices and SONAR decoder settings.
SONAR oracle: BLEU 63.58 · GW RepTok-to-SONAR: BLEU 11.65 · GW RepTok-to-SONAR + short_T5_keep MSE: BLEU 11.69 · GW RepTok-to-SONAR + short_T5_keep SONAR loss: BLEU 11.22 · GW RepTok-to-SONAR + BLEU-probe grounder: BLEU 12.34 · GW RepTok-to-SONAR + codebook short-grounder BLEU-probe: BLEU 13.84 · RepTok-v-Sonar Stage-1 adaptation: BLEU 10.24
Training Compute
FLOPs are peak-equivalent upper-bound estimates: GPU-hours × 312 TFLOP/s per A100 bf16 tensor core peak. Controller rows use elapsed time to best checkpoint from this early-stopped/manual-stop run.
GW RepTok-to-SONAR: 125.80 peak EFLOPs · GW RepTok-to-SONAR + short_T5_keep MSE: 0.19 peak EFLOPs · GW RepTok-to-SONAR + short_T5_keep SONAR loss: 0.11 peak EFLOPs · GW RepTok-to-SONAR + BLEU-probe grounder: 1.93 peak EFLOPs · GW RepTok-to-SONAR + codebook short-grounder BLEU-probe: 2.54 peak EFLOPs · RepTok-v-Sonar: 73.30 peak EFLOPs
GW RepTok-to-SONAR: 37.75M · RepTok-v-Sonar: 92.10M · GW RepTok-to-SONAR + short_T5_keep MSE: 0.53M · GW RepTok-to-SONAR + short_T5_keep SONAR loss: 0.53M · GW RepTok-to-SONAR + BLEU-probe grounder: 0.53M · GW RepTok-to-SONAR + codebook short-grounder BLEU-probe: 0.40M
Attention Trajectories
Lines are means over the matched 1K samples; shaded bands are one standard deviation over images.
Codebook Attention, Matched 1K
Whole-modality weights use a softmax across the five reconstructed modalities. Concept plots use the per-modality component weights: four dictionary codes plus the outside component, scaled so uniform selection is 1. Lines are means over 1,000 matched validation images; shaded bands are one standard deviation over images.
Decoded Codebook Concepts
dinov2 / code0
mean component weight 1.05 · prototype norm 0.71
dinov2 / code1
mean component weight 1.08 · prototype norm 0.70
dinov2 / code2
mean component weight 1.06 · prototype norm 0.52
dinov2 / code3
mean component weight 1.01 · prototype norm 0.17
dinov2 / outside
mean component weight 0.80 · prototype norm 2.62
reptok / code0
mean component weight 1.04 · prototype norm 0.07
reptok / code1
mean component weight 1.05 · prototype norm 0.66
reptok / code2
mean component weight 1.04 · prototype norm 0.13
reptok / code3
mean component weight 1.01 · prototype norm 0.75
reptok / outside
mean component weight 0.86 · prototype norm 2.95
bge / code0
mean component weight 0.83 · prototype norm 1.12
bge / code1
mean component weight 1.52 · prototype norm 1.59
bge / code2
mean component weight 1.11 · prototype norm 0.44
bge / code3
mean component weight 1.03 · prototype norm 0.11
bge / outside
mean component weight 0.51 · prototype norm 2.30
sonar / code0
mean component weight 1.37 · prototype norm 1.34
sonar / code1
mean component weight 1.28 · prototype norm 3.15
sonar / code2
mean component weight 1.00 · prototype norm 0.18
sonar / code3
mean component weight 1.33 · prototype norm 2.95
sonar / outside
mean component weight 0.03 · prototype norm 8.48
symbolic / code0
mean component weight 0.75 · prototype norm 0.60
symbolic / code1
mean component weight 1.33 · prototype norm 2.86
symbolic / code2
mean component weight 1.30 · prototype norm 2.78
symbolic / code3
mean component weight 1.57 · prototype norm 5.52
symbolic / outside
mean component weight 0.06 · prototype norm 176.54
Each image is one attention-weighted component prototype, averaged over the matched 1K validation samples and five grounding steps. The prototype is passed alone through the frozen GW fusion activation, the frozen GW RepTok decoder, and the frozen RepTok image decoder. Outside is the component left outside the four-code dictionary span for that modality.
Decoded Samples
ImageNet index 1,297
reference
A rooster with vibrant orange and red feathers on its neck and head, a prominent red comb, and a large red wattle is positioned in the foreground, occupying the left and central portion of the frame. The background consists of out-of-focus green foliage and brown earth, suggesting a natural, outdoor setting with soft, diffused lighting.
before: direct GW
A bright-faced rooster with a vivid red comb and a reddish-brown tail, a prominent left and right coat, and a dark-green wick and plumage, occupies the foreground center with a contrasting black and white tail, the background resting in the center of the stem. The background consists of a light-brown outline with dark green leaves and foliage, a background of light green leaves and ferns, featuring a clear, natural forest background, with light green lighting.
after: short_T5_keep MSE
A bright-coated rooster with a vivid red comb and tail, a centered red and yellow fleece, features a contrasting black and white stem, with a prominent brown head and tail in the center, resting on the left-hand side. The background consists of a glossy green and dark wood patches, with a light green and brown grass background, reflected by a natural wet ground, suggesting a natural background with bright light.
after: short_T5_keep SONAR loss
A bright-faced rooster with a vibrant red comb and a bright-brown coat, a red and oval tail, a prominent dark and yellow coat, and a dark vertical tail and feather, is positioned in the center of the background, with a light red-brown side. The background consists of a light-green, dusty ground with a light green foliage and foliage, with a dark green background on the outside, suggesting a natural light environment.
after: BLEU-probe grounder
A bright rooster with a vibrant red coat and crest is in the center, a distinctive red-and-yellow tail, with a bright vertical coat, red-and-yellow fleece, and a dark-brown coat and tail, depicted in the foreground with a red-and-green background. The background consists of a light-colored, dusty ground with dark green foliage, a light-green and light-green walls, underlined by a vivid, natural light, suggesting clear outdoor lighting with a dark environment.
after: codebook short-grounder BLEU-probe
A red rooster with a bright brown and black comb and a bright red stem and tail is positioned in the center of the frame, looking to the left of the frame. The background consists of a light green, dark green with a pale green texture, with a clear, green light in the background, suggesting the animal with a clear background.
ImageNet index 2,056
reference
A male house finch, characterized by its bright red head and breast, mottled brown and gray wings, and a conical beak, is perched on a thin, light-brown branch in the center of the frame. The background consists of out-of-focus green leaves and branches, with some small, unopened buds visible, suggesting a natural, leafy environment under soft, diffused lighting.
before: direct GW
A pink-headed creature with a bright brown head and chest, reddish-brown tail, brown skin, and dark-brown forehead and wing, and a narrow, grayish-brown branch, is nestled on a center-striped, dark-brown stalk in the center of the frame. The background consists of a thin, out-of-the-frame background of light-green branches and foliage, featuring a faint green and leafy outline, suggesting a natural environment, with a light, shiny outdoors, thanks to a naturalistic setting.
after: short_T5_keep MSE
A white-headed tree frog with a reddish-brown head, chest and tapered tail, dark brown and brown wings, and a thin, conical tail, rests on a dark-brown stem at the center and bottom of the frame. A dark-brown background is centered on a light, off-white frame of leaves and flowers, featuring a faint greenish-brown outline and leaves, reflecting a light greenish-green environment, with a typical dusty outline.
after: short_T5_keep SONAR loss
A white-headed tree frog with a reddish-brown head and body, a centered chestnut tail, brown and reddish feathers, and a narrow beak and tail, is depicted in a dark-brown body on the center of the stem, centered on a light-green background. The background consists of faint, out-of-the-way branches and green foliage, featuring a faint greenish-brown background and leaves, reflected by a simple, greenish-green environment, suggesting a natural outdoor light with dim shades.
after: BLEU-probe grounder
A white-headed tree frog with a brown head and chestnut forehead, reddish-brown tail and broad-brown tail, and a dark, conical tail, rests on a thin beak and stem, centered on a reddish-brown branch at the bottom of the frame. The background is a faint, light-greenish-brown background, featuring light-green and dappled leaves, with a faint green outer frame and flowers, suggesting a clear, sunny environment, with light-green leaves.
after: codebook short-grounder BLEU-probe
A brown bird's head with a bright red and brown forehead and beak, and a dark, flat head, is centered in the center of the frame, resting on a white, spotted background. The background consists of a thin, greenish-green and light-green foliage, with a clear, dusty ground, suggesting a natural environment for the plant to glow.
ImageNet index 2,708
reference
A small bird with a grey body, black head, white cheek patches, and a prominent curved crest is perched on a broken, brown, textured branch in the center-right of the frame. The background consists of a dense, out-of-focus mass of green leaves and branches, creating a natural, leafy environment with soft, diffused lighting.
before: direct GW
A small brown bird with a light brown crown, black pectoral fin, a white shell and a dark brown tail, perched on a thin, gray-brown branch branch on a dark, thick-blue foliage in the center of the frame, is centered on the outer edge. A background of light-focused branches consists of dark-green and light-brown outer frames, with a light green, branched foliage.
after: short_T5_keep MSE
A small brown bird with a black-headed crest, a white shell, and a dark brown tail on its forehead, is perched on a dark, grey leafy branch in the center with a slightly tapered back on the center. The background is centered on light-brown outlines and branching leaves, featuring a light green and brown background, suggesting a relaxed, natural vegetation with light-green shades.
after: short_T5_keep SONAR loss
A brown-headed bird with a small brown crest, black hood, white chest, and gray tail feathers on its head, is perched on a dark, light-brown branch in the center of the stem, framed by a pale-green background. The background is centered on a thin, light-green-brown branches and leaves, featuring a pale, light-green foliage from the center.
after: BLEU-probe grounder
A brown-headed bird with a dark brown crest, a white beak, a black shell head, and a gray tail on its forehead, is perched on a narrow, white-colored trunk in the center of the wing, with a dark-brown stem on the back. The background is centered around light-green branches, a pale and light-green foliage, featuring a faint green and brown leaves on the outside, with a light, naturalistic environment, reflecting off light-green foliage.
after: codebook short-grounder BLEU-probe
A brown butterfly with a black, white head and a dark brown tail, and a white center of the frame, is positioned in the center of the frame, resting on a green leaf. The background consists of a thick, green foliage with light and brown leaves, with a light, natural vegetation on the background, showing the bird with a clear setting.
ImageNet index 6,030
reference
A box turtle with a dark, glossy shell patterned with bright yellow, irregular, lightning-bolt-like markings is positioned in the center of the frame, resting on a large, gray, textured rock. The turtle's head and front legs are visible, and it is surrounded by green leaves in the upper right and lower right corners, with natural daylight illuminating the scene.
before: direct GW
A turtle shell with a dark box-shaped patch and yellow stripes are positioned in the center of the frame with green and brown stripes, reflecting a natural texture. The shell rests on a dark, green leaf-shaped texture with green, turtle-shaped leaves and grass.
after: short_T5_keep MSE
A turtle shell box with a dark, patterned turtle shell with yellow edges is positioned in the center of the box with dark and patterned shapes, its head exposed to the middle and tail. The shell rests on a dark green patch, characterized by turquoise and brown leaves, on a white ground with some darker, green leaves.
after: short_T5_keep SONAR loss
A turtle shell with a dark box-shaped turtle shell with yellow and patterned dorsal spines is positioned in the center of the box with a dark and patterned shape, the turtle rests on the middle and base of the shell. The shell is depicted with a dark green patch, surrounded by layers of turquoise and green leaves, on a clear brown ground, with some natural lighting, reflected in the middle.
after: BLEU-probe grounder
A turtle box with a dark, shaded turtle shell with yellow and patterned outlines is positioned in the center of the box, resting on the vertical side. The box is surrounded by a dark, green leaf-like texture, with some dark green and brown leaves.
after: codebook short-grounder BLEU-probe
A box turtle with a dark brown shell and yellow and spotted stripes is positioned in the center of the frame, focusing on the leaf. The background consists of a green, dark grass with brown and green leaves, with a light, natural vegetation behind the box indicating the natural environment.
ImageNet index 6,662
reference
A bright green lizard, likely an American chameleon, is in the foreground, positioned on a weathered, textured tree stump; its body is elongated and its limbs are splayed as it appears to be climbing or looking upward. The background consists of lush green foliage and small white flowers, with the scene illuminated by natural daylight, creating a vibrant, outdoor setting.
before: direct GW
A bright green leaf-green lizard with a crisp and pale skeleton and a light-skinned leg is positioned in the center of the background, resting on a narrow green stem, with the back focused on a green leaf. The background consists of obscured outgrowth foliage with green foliage and brown leaves, with a light green vegetation and foliage.
after: short_T5_keep MSE
A bright green lizard and a smooth-skinned lizard with a center leg are positioned in the background, centered on a light green leaf, with their back and head positioned in a pale green.[citation needed] The background consists of light-green plant foliage and green foliage, featuring a dark green vegetable, with a natural luminous background.
after: short_T5_keep SONAR loss
A bright green lizard with a smooth-skinned lizard and a flat, dark-skinned body and a slender tail is positioned in the center of the frame, resting on a dark, green underside of the leaf. The background consists of dim green foliage, with obscured outer leaves and green foliage.
after: BLEU-probe grounder
A bright green lizard with a textured shell and a smooth-shaped tail and a prominent crest is positioned in the center of the texture, resting on the left-hand side of the frame. The background consists of faint green foliage and leaves, with a light green background highlighting the texture.
after: codebook short-grounder BLEU-probe
A green lizard with a broad, dark brown shell and a straight-set head is located in the center of the frame, focusing on the back of the frame, wearing green cloth. The background consists of a faint, green and light grass background, with a light, natural brown space, indicating the plant with no visible scenery.
ImageNet index 6,881
reference
A vibrant agama lizard, displaying a gradient of bright purple on its head and back transitioning to deep blue on its limbs and tail, lies horizontally across the lower portion of the frame on a rough, light gray rock surface. The background consists of a darker, textured rock face above the lizard, with a distinct shadow cast to the left, indicating a light source from the right.
before: direct GW
A vibrant rose lizard with a bright blue body and a gray, light-colored tail and dark body are positioned in the center of the frame, resting on a light-colored rock surface. The background is dark, rough-textured sand, with a clear and light-colored rock on the outside.
after: short_T5_keep MSE
A vibrant lizard skull with a bright blue and red body and light brown head and tail, positioned in the center of the sculpture, is situated on a dark, textured ground. The background is light gray, with a slightly obscured rock surface on the sides.
after: short_T5_keep SONAR loss
A brilliant lizard lizard with a reddish and glossy body face, a blue skull and a light gray body, and a center and vertical tail, is positioned in the center of the sculpture, resting on a dark, brown-colored floor. The background is composed of a crisp, lighter-colored ground, with light brown and gray particles shining through the surface, a light, dark azure texture with a clear, natural background.
after: BLEU-probe grounder
A vibrant pink lizard with a glowing red body and a gray, light brown head and a textured body, and a dark brown center, is positioned in the foreground on a light brown rock, resting on the center-right side. The background is obscured by a lighter, dark brown texture, featuring light gray and brown soil underneath, with a clear natural glow, reflecting on a sunny background.
after: codebook short-grounder BLEU-probe
A red lizard with a bright brown and black spotted head and tall frame is located in the center of the frame, focusing on the back of the frame, wearing green. The background consists of a dusty, light green and brown soil, with a clear, natural vegetation underneath, indicating the body with different spots.
ImageNet index 8,699
reference
A dark, slender snake with a smooth, glossy black body and a distinct yellow ring around its neck is positioned horizontally across the center of the frame, resting on a bed of small, light-colored pebbles. The snake is nestled between large, gray, rough-textured rocks that form the background, with patches of white and gray lichen visible on the rock surfaces under diffuse, natural lighting.
before: direct GW
A dark, slender, light gray stone snake with a narrow, smooth gray arm is wrapped round the surface of the stone in a smooth, elongated frame, resting on the center edge of the rock on a smooth, dusty rock. The rock rests on a dark, flat-shaped rock surface with dark, grey stones and pebbles, with a light gray, dark rock on the sides and a light grey, sandy texture visible through the background, with dark, bright rocks.
after: short_T5_keep MSE
A dark, slender, light grey snake-like stone with a smooth edge is wrapped around a smooth body on the top of the scale, with a round, rough rock lying on the bottom of the stone. The face is composed of a narrow, rocky stone with rocks, gravel, and grey stones on the surface, with a light brown stone on the outside, and dark grey dust reflecting off the stone.
after: short_T5_keep SONAR loss
A dark, slender, light gray snake with a slender, smooth stone is wrapped around a smooth, elongated body on the top of the frame in a smooth, grey rock-like surface, rolling along the bottom of the stone in a narrow, pointed, rough texture. The rock rests on a smooth, gray stone-shaped rock with small, scratched rocks and gravel, with a dark grey surface visible on the sides, with a pale, light-colored gravel behind.
after: BLEU-probe grounder
A thin, dark grey, slender snake with a smooth, glossy long bone curls on a smooth, grey surface in the center of the frame, rolling on a smooth, gray rock on the bottom edge. The background consists of a thin, rough, scratchy stone surface with dark grey and pebbly rocks, with a pale, dark grey texture on the bottom, reflected by a light, rough stone with an outer edge.
after: codebook short-grounder BLEU-probe
A brown snake with a light brown tail and a dark, flat frame head is positioned in the center of the frame, with the head resting on a plastic bed. The background consists of a dark, green grass with light and brown surfaces, with a smooth, natural light on the side, indicating the creature's surroundings.
ImageNet index 9,684
reference
A vibrant green vine snake is positioned in the foreground, its slender, elongated body resting on a dark, textured piece of wood that occupies the lower-left portion of the frame. The background is a soft-focus blend of dark green and black, suggesting a natural, leafy environment, with the snake's head and upper body extending towards the right side of the image.
before: direct GW
A vibrant green glowing green snake with a smooth, elongated green shell with a slender, smooth-shaped body is positioned horizontally across the top edge of the tree, resting with its head in a horizontal, dark background. The background is dark-brown and light-toned, suggesting an outer, dark-brown texture, with a clear, green tree background.
after: short_T5_keep MSE
A vibrant green glowing green serpent with a slender body is situated in the upper frame, with its head slanted horizontally and slanted to the left, with its main body slanted upwards and to the top, with its mouth slanted dark-brown. The background is a clear, blurred and dark-brown texture, with a green underside suggesting a dark, textured body.
after: short_T5_keep SONAR loss
A vibrant green glowing green snake serpent with a smooth, glossy green frame is positioned in a slender, scalloped body in the upper part of the frame, with its head centered horizontally and tapered head in the upper part of the frame, with its face smooth and tapered. The background is dark-toned, with a smooth, dark-brown texture, and a light green background, suggesting a slanted, dark-brown body in the foreground, with a smooth, glossy environment that reflects a dark, snake-like base.
after: BLEU-probe grounder
A vibrant green glowing green venomous snake with a smooth, glossy stem is positioned in the center of the frame, with its head slanted back and upward, concentrating the brown body in the foreground and upper part, with its head smooth and pointed out. The background is a soft, dark-textured brush-like texture, with a light, brown and green background.
after: codebook short-grounder BLEU-probe
A green snake with a bright brown tail and a smooth, pointed body is positioned in the center of the frame, with its head positioned on the back of the frame, looking green. The background consists of a dull, green and dark leaf background, with a clear, green texture on the outside, indicating the snake's environment.
Files
metrics.csv · compute_estimates.csv · parameter_counts.csv · trajectory_stats.csv · codebook_attention_stats.csv · summary.json · decoded_examples_grounding.json
Page id: compare_reptok_vsonar_stage1_bleu