Can't let go

I've been in this relationship with my fiance for a while now. We have a son and currently are expecting baby number two. Before our blessings, last year was a bit rough. On Father's day, it turned out to be one of the worst days I've went through. I know all our relationships have out ups and downs. I get it. What happened was my fiance got invited to his family BBQ. I wanted to go but his sister refused that I come. He didn't want to go but I told him he should. After he left, hours later he came back with his brother, packed his things and left. He told me 'I dont want to be with someone who's bipolar like you.' I was torn to pieces when he left. Days later i come to find out he didnt want to leave but his family kept pushing that he should. I poured my heart out because i thought he was the one but it backfired. After a few weeks he wanted to meet up and talk about it and want to work things out. I knew after that break secrets were released. As much as I wanted to be like No I think it's best we go our separate ways, I wanted him back more. So we rekindled the relationship. As months go by things between myself and the family are still tense. He still gets invited to all the family events but I still stay behind. A while back his sister went through a similar situation, took back an ex, the family said their sorrys and accepted him back. I don't understand because after I said I was sorry, nobody acknowledged it. Exactly one year after this incident, I couldnt hold myself together and lost it. I was depressed for those few days and I was an emotional wreck. Though its been a year later, i can't find myself to let this go or be happy. I've honestly from time to time find that I dont want to be married to him and into the family. Despite us having children together my happiness isn't where I thought it would be. I don't know how to get passed this or move on. Day in and day out, I find myself being hurt from the pain as well as wiping away my tears and pretending to go on.