try the patience of (someone)
To frustrate or annoy (someone) by continued unwanted behavior; to test the limits of someone's patience. His tangential questions are clearly trying the patience of the professor, who asked that all questions be held until the end of the lecture.