Why does EVERYONE take me so seriously?

I've always had this problem.. I don't know if it's my tone of voice, or my facial expression, or just my overal reaction, but EVERYONE always thinks that I'm offended or that I'm freaking out when I'm not! For instance, someone will say something to me that's kind of harsh, but in a joking manner, and I'll reply (in a normal manner, i'm not offended, i'm not pissed, i'm not freaking out, I just reply) and then they'll say "i'm joking, calm down, jeez!" But I know they're joking, and I was calm! This has been happening for years, I truly don't understand why everyone takes me so seriously. People always pick on me because they think I get offended so they get enjoyment out of it and keep picking on me to see me "freak out" when I don't think I am at all. Can anyone please give me advice? I'd really appreciate it.