Just lost

My boyfriend and I have been together for 20 years. I love him and he loves me….but… in the last two weeks I’ve felt like he's slipping away and our relationship has changed. So a couple months ago I dropped him off at the mall. I went to hang out with my friend. I pick him up and ask him how was his day was what happened? He said, “ It was boring walked around and got a soda and waited for me to pick him up”. Fast forward.. a couple days ago we go to his mom’s house and we are talking about random stuff. Then he brings up that he saw his brothers best friends wife at the mall recently and that they hung out and caught up. So… I feel Ike he kept that information from me. I know i personally never met the wife, but he could have told me about it. So that was one thing he did and I feel like he's keeping things from me now. Last week was another incident I am upset with him about… So our friends sister was coming in for thanksgiving and shes a cool person. My BF decides to drop everything to be at that thanksgiving party…. Back story on my bf he hates gatherings, unless its his family gatherings. When he has to be at the gatherings he can keep everyone’s attention and everyone has a good time. He is also a “work-aholic” He tells everyone work comes first and dont bother him. Also our friend has had tons of thanksgiving parties and my bf never goes to them or locks himself in his room to not be bothered. He also claims eye contact is intimate and he doesn’t do it he looks around when he talks. He also hates kids and doesn’t want any at all. Fast forward… Our friends sister comes out and my BF skips work and is looking out the window to see when she is coming over so he can talk to her. He is awkward AF waiting around and he is pacing and cleaning just to wait. Well she comes over and he is talking to her and when she leaves the room he seems to find a way to be in the same room as her. I was in the kitchen cleaning and making sure everything was in order. I do trust him not to be a complete idiot. When we were all talking I was watching him talk to her and he gave her full eye contact he didn’t look around as he talked. He is also made sure her kids are happy and taken care of… Don’t get me wrong my bf is really nice and so I know he does nice things for about anyone. He wants people to feel welcomed so the kids aren’t an issue. Now he is also making me feel crazy cause he doesn’t want me to see his phone when he’s watching YouTube or playing his games. But I have full access to his phone so its just weird that he’s being weird about it all of a sudden. After our friends sister left I confronted him about my feelings and he said, “ Nothing is happening and its my own insecurities and he feels upset that I am doubting his integrity and him after 20 years together”. He could be right and its me just putting stuff in my head. During our fight he said, “ If i accuse him of being a cheater again he will end the relationship and he doesn’t want this convo to happen again”. So now I cant tell him about my feelings or how Im feeling in the relationship or apparently it will be over. In the past I’ve been able to talk to him about anything he’s my best friend and now I cant so I'm just confused and lost. Have we taken our relationship to the end? Does he want to be with our friends sister (maybe they have a connection that him and I lost over the years?) Am I just being paranoid and dumb? Should I even be feeling like this?