Relationship Stress

I have been dating this guy for 4 months. He's so sweet and loves me so much and I don't think anybody has ever this much. He's cute but I noticed very shortly that I wasn't sexually attracted to him like I am normally to other guys but because of his love and personality I still made it work by suppressing my need. I really thought he was worth it. But as we became committed, I asked him to do an STD test so that I could be stress free about being intimate with him. The results came back for the HSV-1 and 2 and they were both positive, when HSV-2 was on a low positive. He told me he never had lesions on that area but he has had cold sores on his lip. I wasn't very afraid of cold sores but knowing he was also positive for HSV-2 has put so much stress on me. I talked to my doctor about this and she thinks it's much less likely for me getting HSV-2 when he's not breaking out and she also thinks the breakouts he has had on his lip could be the signs of HSV-2. She told me not to worry about it if he doesn't have breakouts or not feeling itchy or tingling. I'm still stressed and I don't know how I can do this. We have become very good friends and we talk every day. He's a very good guy and if I breakup with him I know I can't get over it. Please help me. I don't know what to do.