not (one's) keeper
Not responsible for one's actions, behavior, or whereabouts. Often used to express frustration that such responsibility is assumed. Of course I'm disappointed that Travis got arrested again, but I'm not his keeper. How should I know where Mary is? I'm not her keeper.
not someone's keeper
If you are asked where someone is and you answer that you are not their keeper, you are saying in quite a rude way that you do not know where they are and you cannot be expected to know. `I've no idea where he is,' Hughes replied, `I'm not his keeper.'