stroke (one) the wrong way
To irritate one due to someone or something's presence, nature, or habitual behavior (as opposed to directly and intentionally). The way she smiles all the time really rubs me the wrong way. It seems so fake. He's always talking about how those commercials rub him the wrong way, but I think they're cute.
stroke someone (or someone's hair) the wrong way
irritate a person.