First time really talking about this

My freshman year of college, there was this boy that I liked. He was a transfer and a couple years older than me. Me being from a small town and not being allowed to do much, it was a culture shock to go to a big university much bigger than my town. He was older and had been to college for two years already. We started to hang out, as we were in the same major, and one thing kind of led to another. We started flirting and occasionally I would watch movies with him and stay in his room later than I should. He liked another girl so I didn't really think that I was putting myself in a bad position. Eventually months go by and we kiss. We make out and I'm into it. Then he starts to try to take my clothes off. I was a virgin and immediately started to get freaked out a little. I said "no this is a bad idea" "no we shouldn't do this" "stop". But he kept going. He wasn't particularly forceful with me, but I said stop. When I realized what was happening I froze. I didn't even fight. For years I was ashamed because I didn't do anything. Never even fought back. Now I have a fiancé and I am so unbelievably in love with him. He knows and is very considerate of my needs. But I have a very hard time getting physical with him. I love him. I want him. But something in my brain isn't on board. If he ever touches me and I'm not prepared or I'm taken off guard, I go into this mode where I just don't want to be touched. I want this pain to be gone. I want to have sex with my partner.