We define an RL problem over unlabeled images using a Markov chain perspective on image augmentations. An agent receives a view of an image $x$ and takes actions by applying an image transformation to change the view (e.g., zooming out, panning left, rotating the image, cropping to a subview), with the intention of finding views that maximize the likelihood of some specific semantic "annotation" of interest $p(\ell|x)$ (e.g., find the gentleman in the green suit, or a kite, or a boy playing).

Take two random crops of an image, as in any other SSL method, $x_1$ by cropping to bounding box $\mathbf{bb} _1$ and similarly $x_2$ to $\mathbf{bb}_2$. We interpret $(x=x_1, a=\mathbf{bb}_{1\to2}, x'=x_2)$ as a transition in our environment: applying a panning transformation from the view $x_1$ to create $x_2$. Learning a value function corresponds to, for any $\ell$, using the model's outputs at one crop $x_2$ to generate a target prediction for the other crop $x_1$: $$\min D(~Q_{AB}(\ell | x_1, a=\mathbf{bb}_{1\to2}), (1-\gamma)p(\ell | x_2) + \gamma \max_{a'} Q^{target}(\ell | x_2,a')) $$