Ignorance.

Unfortunately there are some people who will always hate things they don't understand or are different from them. Many people were raised to hate. If you put toddlers of various races in a room with a bunch of toys, they will all play together regardless of what color the other child's skin is.

You are right that there are bad apples on both sides of the fence.