Toward Active-Vision Foundation Models

Lamarck Labs

Deep neural networks as models of vision

Let's dive into some of the background motivation behind this project.
Deep artificial neural networks have a long history
as models of the ventral visual stream.
Some of the first deep networks to predict its responses well were trained with supervised learning.
In 2014, Yamins and colleagues found that, among the classification-trained networks they used, brain predictivity tracked performance on this task.
[CLICK]
The same trend was found in later work,
[CLICK]
at least up to a point.
Above about 70% ImageNet accuracy,
the relationship was no longer significant.
[CLICK]
But such networks learn from millions of human-provided labels,
whereas humans are able to learn a lot about the world purely from visual experience.
[CLICK]
More recent work has found
that so-called unsupervised, or self-supervised networks,
which require no annotations, and thus learn purely from experience as well,
can also achieve high neural predictivity.

Siméoni et al. 2026, TMLR (DINOv3 ViT-B/16)

Siméoni et al. 2026, TMLR (DINOv3)

What's missing?

The world Sensory bandwidthComputational resources

Mnih et al. 2014, NeurIPS

End-to-endFrozen

Seeing

Observer

Choosing where to look

Policy

No policy can make up for a poor observer. A general-purpose observer lets you use any policy.

	Passive-Vision Foundation Models	Prior Active Vision Models	Active-Vision Foundation Models
Instantaneous vision	✓	✓	✓
Memory	✗	✓	✓
Visuospatial intelligence	✓	✗	✓
Task-agnostic	✓	✗	✓
Policy-agnostic	—	✗	✓

Canvas Vision Transformer

Toward Active-Vision Foundation Models

NeurIPS 2026

Let's set up the problem that CanViT is supposed to solve.
We consider a scene, which is a high-resolution image,
[CLICK]
and from that scene we take what we call a glimpse: a crop taken at a given location and scale in the image.
From that glimpse, which is the only thing the model sees, the model updates a representation that we call the canvas, which represents its current understanding of the entire scene.
On the second row, you can see a PCA visualization of the canvas after the first glimpse, and on the third row, the magnitude of the change in the canvas after that first glimpse.
[CLICK]
Then we can do the same thing with a second glimpse elsewhere in the scene.
The model keeps the understanding it acquired from its first glimpse and integrates new information into its canvas.
[CLICK]
And we can do this again,
[CLICK]
and again. The model gradually builds up an understanding of its visual surroundings, using glimpses taken at various locations and zoom levels.

Architecture

Pretraining

Experiments

Vision
Transformer

The Canvas Vision Transformer
architecture

Our architecture builds upon a standard Vision Transformer for instantaneous processing, but we augment this component with a memory module that we call the canvas.
The Vision Transformer is responsible for retinotopic, glimpse-centered visual processing,
whereas the canvas starts off without any scene-relevant information at the beginning of an episode, and then gets gradually updated through what we call Canvas Attention read and write operations.
We also carry a recurrent CLS token from one glimpse to the next, a global summary that we read out for classification.
Here we see a glimpse coming into the model. As the backbone processes it, it reads from the canvas and writes to it.
From that local evidence, the model produces a global understanding of the scene.
But of course, it hasn't seen much here: only a cat's ears. It does extrapolate that there might be something like a whole cat in the picture, but it can't know for sure.
As it gets more visual evidence, it is able to refine that understanding.
[CLICK]
Here, with a second glimpse,
[CLICK]
and now with a third, we see that the model's understanding becomes progressively sharper, with more visual evidence and more processing.

Glimpse

Glimpse patchesCanvas patches

A key component of this process is to handle space correctly.
We developed a positional encoding scheme based on a well-established technique called rotary position embeddings, or RoPE,
which allows the model to identify where each piece of evidence came from, and relate it to its spatiotopic, scene-wide canvas.
Here we can see that the coordinates of each patch in a glimpse are given to the model in scene coordinates; that's why we call this scene-relative.
The canvas patches have scene coordinates too, and those never change.
[CLICK]
If we zoom in, the glimpse patches get closer together in the scene,
and the model is able to understand the zoom level of that glimpse from the offset between neighboring glimpse patches.
[CLICK]
If we take the glimpse elsewhere, those coordinates change, while the canvas patches stay where they are.

Canvas Attention

Here's one step of Canvas Attention.
You can think of it as a cross-attention mechanism, for those who are familiar with Transformers, which uses the scene-relative rotary position embeddings that I just told you about, with a few twists.
[CLICK]
There is no self-attention and no multilayer perceptron on the canvas side, only on the glimpse side.
[CLICK]
The only fully connected projections in the architecture also lie on the glimpse side.
These are somewhat expensive operations, but they're only applied to a relatively low number of glimpse tokens.
[CLICK]
On the canvas side, we only apply basic normalization operations for stability, and position embeddings where appropriate.
[CLICK]
This means that the model is able to handle a very large canvas at low computational cost, scaling to very large canvases in a sub-quadratic manner.

Architecture

Pretraining

Experiments

Before knowing where to look,
we should know how to see.

Seeing