How do I accept that my monogamous partner will have sexual fantasies about other people?

The general consensus is that desiring other people is natural and ok as long as they don't act upon it. I feel like if fantasizing about other people is okay because it's private business that "hurts no one", then so is cheating if they hide it completely from their partner. I am currently with someone who is very open and honest and really loves me. He says that he only wants to be with me and prefers to have me the center of his attention romantically and sexually. I feel honored, I am satisfied, and I belive him when he says that. However, it hurts to know that even someone who loves me this much will, time to time, lust and fantasize about someone else. I know that he does it for variation, not availablity because he also has wild sexual fantasies about me as well. The worst thing is that I am a hypocrite and that sometimes I will be catching myself fantasizing about other people. It feels like cheating and it bothers me so much. How do I let go of this hangup? I feel like I won't ever be happy.