In the mouse blastocyst, some cells of the inner cell mass (ICM) develop into primitive endoderm (PE) at the surface, while deeper cells form the epiblast. It remained unclear whether the position of cells determines their fate, such that gene expression is adjusted to cell position, or if cells are pre-specified at random positions and then sort. We have tracked and characterised dynamics of all ICM cells from the early to late blastocyst stage. Time-lapse microscopy in H2B-EGFP embryos shows that a large proportion of ICM cells change position between the surface and deeper compartments. Most of this cell movement depends on actin and is associated with cell protrusions. We also find that while most cells are precursors for only one lineage, some give rise to both, indicating that lineage segregation is not complete in the early ICM. Finally, changing the expression levels of the PE marker Gata6 reveals that it is required in surface cells but not sufficient for the re-positioning of deeper cells. We provide evidence that Wnt9A, known to be expressed in the surface ICM, facilitates re-positioning of Gata6-expressing cells. Combining these experimental results with computer modelling suggests that PE formation involves both cell sorting movements and position-dependent induction.