Is the word cracker actually a racist slur and do you think I was being racist?

I just lost a friendship cause I just told them I called someone a cracker in an argument. They told me I was being racist by saying it to them. I said that they were acting like cracker is a serious slur like the n word, and he then gotten even more mad at me saying it was. He said that I couldn’t say the word cracker just like how he can’t say the n word (I’m black and he’s white). I obviously said that the n word has more depth than the word cracker and that he was just overreacting, but he ended the friendship saying he didn’t wanna argue with a dumbass like me and be friends with a racist. Was I being racist? I just think he was overreacting but was he not?