Regret and Uncertainty

Hi everyone! I am new here but am going through a lot at the moment. My boyfriend and I have purchased a house together to move into with my 10 year old son. Prior to this, my boyfriend had cared for my son every Tuesday for the past year, as school as cancelled due to the pandemic. My boyfriend (Phil) does not have children of his own, and my son’s father has 50-50 shared parenting with me. We have just begun to move into the house, and honestly, it’s been awful. I should provide some background about Phil. He is in the tech industry, and we both suspect he has asperger’s tendencies. So he needs alone time, doesn’t especially enjoy meeting new people, feels clueless about dealing with people. I have no doubt that this move is causing him stress and anxiety. So our house is a bike so house, and we decided Phil would get the downstairs as his man cave. The other night, he said he did not want my son coming down there, or coming into our bedroom. The house also has an above ground pool which required a repair of a pump. Phil has said the pool is a hassle and he wants to get rid of it, and that if we keep it it will become my responsibility. My son and I are selling our house to move in with Phil into the new house (in both our names), and my son is changing schools to do this. My question is, do you think I should give Phil time and honor his wishes? Is he overwhelmed and in shut down mode? Do I argue with him about these things now because I disagree? We have been fighting almost non stop currently. I really want this to work out, but I want to make sure things are fair for everyone. Thanks in advance for any advice I might get!