Anxiety with saying no

I have planned to put up my cat shelves today, and its an all day thing for me since i have to drive far to go to the store. My mom and little sister just called me and asked if I wanted to come over to have smores and play games. I told them no, that id like to finish what I was doing. I said im sorry and that id come over tomorrow instead. Why do I feel so anxious and want to cry when I tell them no, even when they tell me not to say sorry and that its fine? I feel so bad and im not sure why. I shouldnt.