let (one) go by
To allow one to pass by or around (someone or something). I need to get into the building to retrieve my briefcase—please let me go by! I didn't want to drive any faster on the highway, so I moved closer to the curb to let the cars behind me go by.