My husband expects my MIL to come with us on every vacation

Hi, this is my first post, but I feel like I need to talk to someone to get some perspective.

Years ago, after my Father-in-law passed away, my Mother-in-law moved to the same city to be near us. She has never seemed to like my very much and hasn't done anthing specifically to me, but she barely talks to me and seems to avoid me when I'm around. At first it was very awkward, but every single weekend when we would go out anywhere she would come with us. Then it stopped being every weekend at least, but she would still go to a lot of outings with us, and she would often walk away from us, and my husband would have to go find her, and when I would find them to join up with them, she would walk away again. My husband has always asked her to come on every single vacation with us, and she would spend every vacation looking unhappy. It was very awkward for me because I felt like she didn't want to be around me.

My husband knows that she makes me uncomfortable, but he gets angry and says that it's "all in my head". He doesn't even ask me if she can come, it's just a "given" that she's coming with us, so secretly I cringe every time he plans a "vacation" for us because it's not an enjoyable time for me at all, but I feel that if I just refuse to go, A) it would cause a "scene"/look bad and B) I want to be there for my daughter, and I don't want to cause drama because I want it to be fun for her. It really bothers me though that we can't have one small family vation or even a weekend get-away with just the three of us (husband, daughter and I). Any time I try to discuss this with my husband he gets very defensive and angry.. and nothing changes.

Now my MIL LIVES with us in a guest house on the same property. I feel like I have even less privacy, and it's causing issues with my daughter, who now lives between the two houses. She's now a teen, so if she doesn't like something here, (like what I made for dinner, or the fact that I asked her to do something like clear her own plate from the table, or do her homework) she just goes over to my MIL's house and my MIL lets her do whatever she wants and makes food for her. My husband called tonight and he wants to go away one weekend to a nearby city. He suggested a few days, but then realized that his mom would be out of town that weekend, so he wants to postpone it until she's back! When I suggested that we could go just the three of us, and said that it would even be cheaper that way, he said no. Then I got a bit mad and said, well then maybe you could save money by taking her instead of me, and to that he said "Fine!" He said he couldn't just leave her here, and then he said he had to go.

I mean I know that he thinks he needs to take her, and since he's taken her every other time, I guess he would feel awkward just telling her that we're going away for a couple of nights and not inviting her. Personally, I think it's pretty selfish of her to expect to go on every vacation with us, and not consider that we actually might want to spend some time together just the three of us.

Am I being unreasonable wanting just one weekend away without my MIL? How do I tell him that without him blowing up and getting mad at me and in the end it won't change anything anyway?

Thanks for listening. I'm feeling helpless here :(