I'm sort of with you on this reaction. I'm allergic to molds, which is really really hard for people to understand. I'm not sure I'd be quiet as aggressive as you say, but I don't apologize for my allergies or, at this point, feel bad about having them.

I wonder if it would be worth involving the pastor in this issue, perhaps asking for a sermon that encourages people to accept each other just the way they are. Some people are missing a leg, some can't eat dairy, or gluten or whatever. Everyone has something their brain can't do, and something their brain does well: sense of direction, aptitude for math or art or... Each person is due respect for the way their body and/or brain doesn't work and how it does.