let down (one's) guard
To become less guarded or vigilant; to stop being cautious about potential trouble or danger. We'll wait long enough for them to let down their guard before we launch our invasion. There will always be people in this business looking to exploit you, so you can never let down your guard.