Vanilla flavors in ice cream and other foods are sometimes made from castoreum, a chemical derived from the scent gland next to a beaver's anus. It is also used in perfumes and colognes. Vanilla flavors used in ice cream and other foods are sometimes made from a substance called castoreum, which is a chemical derived from the scent gland located next to a beaver's anus. This ingredient is also used in many perfumes and colognes.