He Wishes I never got pregnant

Hello,

I've been dating my bf for 3 years. We just found out a month ago we are expecting. We went back and forth and ultimately decided to keep the baby. We don't have a lot of money and we have to get married and move out fast. It's very stressful. I understand. On most days my boyfriend kisses my belly and says a lot of nice things concerning the baby. However when he is stressed or feels like he hasn't been able to "relax" he often times says "I wish I never got you pregnant" and when I get sad he gets mad. He says I should understand that now is a bad time and he is allowed to feel that way. Am I wrong for feeling heartbroken. I understand this is hard. I still work 40 hour weeks while having pregnancy insomnia. But I don't feel like it's right to wish your baby away.