As a very pale person who was bullied a lot for it and probably has lasting personal doubt from that bullying, I think there is just a fundamental difference (in the US). One is “just” mean and bullying, but there is no horrific dehumanizing history behind it. Both are bad, but I don’t think they’re directly substitutable in the way you describe, as I think it cheapens the history of slavery and racism