The Alcock-Paczynski (AP) effect uses the fact that, when analyzed with the correct geometry, we should observe structure that is statistically isotropic in the Universe. For structure undergoing cosmological expansion with the background, this constrains the product of the Hubble parameter and the angular diameter distance. However, the expansion of the Universe is inhomogeneous and local curvature depends on density. We argue that this distorts the AP effect on small scales. After analyzing the dynamics of galaxy pairs in the Millennium simulation, we find an interplay between peculiar velocities, galaxy properties and local density that affects how pairs trace cosmological expansion. We find that only low mass, isolated galaxy pairs trace the average expansion with a minimum "correction" for peculiar velocities. Other pairs require larger, more cosmology and redshift dependent peculiar velocity corrections and, in the small-separation limit of being bound in a collapsed system, do not carry cosmological information.