Breaking up with a friend i fell in love with but rejected him.

Hi, well this is kind of a long story, going to summarize it as best as i can.

i met my friend in 2014 well he was not my friend than. He was trying to get me to be his girlfriend. I really did like him but i didn't want to seem easy so i rejected him n told him we can only be friends...a part of me was freaking out since he would have been my first boyfriend. So we continued to be friends but deep down i was hoping he would try n ask me out again. Than i saw him with a gal n i ended up just looking for a random guy to kiss so that i can tell my friend about (to rub it in his face) but i could see the hurt in his eyes when i told him. As time went on i started to become super jealous about any girl he would hang out with but i tried to contain myself. We spent most of our time together, talking until the sun came up. 2yrs down the line i suggested we get intimate since i was curious n je told me he would have to get a girlfriend to prevent himself from falling for me n i agreed. I thought with time he would see how special he was to me n he wouldnt need to use other people to block out how he feels about me. He broke up with the girlfriend n it was just me n him for a year until there were other girls n i told him i don't like it n i want him to myself. And than a girl aproached him to ask him out, he was hesitated at first but ended up agreeing. I got furious asking him if he can't see me... and he said throughout these years he could see that i liked him but he wanted me to say it coz my rejection really broke him. I started to accept things as they were (him n his gf) coz the time we spend together never changed. We are leaving together now all was fine until i found out he is comunicating with an ex and i just lost it and trashed the house. (Even if you ask us now we will say we are friends, whereas everyone at school said we are the only people that do not know that we are in a relationship)

Are we wasting out time? (Throughout these 5years we have tried to separate n we have done so much to each other that I'm suprised we don't hate each other)

I feel like the recent fight will break us and i don't want him to leave but feel like it would be for the best to save ourselves from more heartache.

How do i let go?