Because historically, toys in almost all cultures were something you'd engage with as a child that you'd discard as an adult. Very often as a substitute or imitation for adult activities, so adults would find it weird for you to continue to interact with them instead of doing the adult activity.
Higher castes in society do have a history of appreciating miniature models and collectibles , but that's not something most people would be familiar with