Broad empirical evidence suggests that higher-level cognitive processes, such as language, categorization, and emotion, shape human visual perception. Do these higher-level processes shape human perception of all the relevant items within an immediately available scene, or do they affect only some of them? Here, we study categorical effects on visual perception by adapting a perceptual matching task so as to minimize potential non- perceptual influences. In three experiments with human adults (N = 80; N = 80, N = 82), we found that the learned higher-level categories systematically bias human perceptual matchings away from a caricature of their typical color. This effect, however, unequally biased different objects that were simultaneously present within the scene, thus demonstrating a more nuanced picture of top-down influences on perception than has been commonly assumed. In particular, perception of only the object to be matched, not the matching object, was influenced by animal category and it was gazed at less often by participants. These results suggest that category- based associations change perceptual encodings of the items at the periphery of our visual field or the items stored in concurrent memory when a person moves their eyes from one object to another. The main finding of this study calls for a revision of theories of top-down effects on perception and falsify the core assumption behind the El Greco fallacy criticism of them.