One has to leave or depart. "Shove off" is a nautical term, referring to the act of pushing against a dock in order to move one's boat away from the shore. We both need to be up early tomorrow, so we'll have to shove off soon.