First, Freud's definition of fetishism. In typical Freudian fashion, the scene imagining the origin of fetishism is male-centric: upon seeing the female genitals for the first time, a boy supposes that there has been a castration. To shield himself from the fear of his own castration, a penis substitute for women is installed, such as the foot, and thus the fetish begins. The fetish supplements or even displaces traditional sexual pleasures. So, translating this into the traveller's scenario, the 'castration' is the fear of losing oneself in the pleasures particular to the local culture. Instead of adapting to local tastes, certain experiences (eg seeing the 'Mona Lisa') are fetishised as substitutes for an authentic experience of the destination.
In the 1950s Roland Barthes finds the tide turning against this sort of fetish (his terminology for it is 'myth') and returning to a love of the everyday: 'travel has become (or become again) a method of approach based on human realities [...] it is everyday life which is the main object of travel'. While this tide of human realities has continued to flow, the fetish of travel has yet to ebb. Marx's theory of commodity fetishism is helpful for understanding why some travellers still find fetishised objects more interesting than a destination's 'human reality'.
With the transition from feudalism to capitalism, most workers move from selling or using their products, to selling their time. In this change they become alienated from the commodities they help produce. Instead of owning the fruits of their labour, they own the money they have exchanged for their labour. As a cog in the machine of manufacturing, the workers' self-worth is diminished while products increase in worth (quite literally, as commodities are sold for more than what workers are paid to make them). Consequently, society values the relations between commodities more than between people. And persons are defined by the commodities they own.
When I recall my travels, my favourite memories are frequently the conversations I've had with locals — not the paintings I've seen or the architecture I've walked around. Not that these sights aren't also appreciated by me, but society has fetishised them above the equally interesting daily realities of the locals.