I'm not responsible for the actions, behavior, or whereabouts of my close relative or friend (not necessarily a brother). Often used to express frustration that such responsibility is assumed. Of course I'm disappointed that Travis got arrested again, but I'm not my brother's keeper. How should I know where Mary is? I'm not my brother's keeper.