This stems from Hong Kong's time as a British colony when Western restaurants were too expensive for most people, so local cafes remade British foods like tea and toast the Cantonese way. Yum cha is all about sharing small dishes, and the desserts are light, cooling sweets for a warm climate.
