Even if X is not equal, if all Y are equal, we can still skip. This seems to happen more often than I expected with far away objects. Skipping here avoids filling up the queue.