back of (one's) mind
Referring to something that one thinks of, but with little or no conscious or active consideration or reflection. Despite my success at work, there's always this worry in the back of my mind that I'm doing something wrong. You don't need to do anything about it now—just keep it in the back of your mind going forward.
back of one's mind
The remote part of one's mind or memory, as in With the idea of quitting in the back of his mind, he turned down the next assignment. [c. 1900]