care a whit
To care just the smallest amount. (A "whit" is an iota.) Often used in the negative. I don't care a whit what the neighbors say, I'm not taking that fence down!
didn't care a whit
and don't care a whitdidn't care at all. Sally thought Joe liked her, but he didn't care a whit about her. I don't care a whit what you do with my old clothes.