You were not being racist and he was definitely overreacting. Comparing cracker to the n word is insane.. if one of my friends made that comparison I would drop them myself. You're better off without someone that thinks a silly word like cracker is anything like the n word considering the history associated with that word.