What to do? Am I wrong?

When I go easy on my kids my bf would get mad/annoyed at me but when he go easy on them while I'm strict it's not a big deal and he'll shrug it off. When this happens he will ignore me for the rest of the day(or until he feel better). He is not the biological father and stated that I am too easy going on them/too loving towards them and that I needed to be more strict with them. We live together so when he is like this the mood feels tense that I would just leave him alone. I feel like I should go talk to him to make him feel better but it always end up into an argument and he brings my past into it. Am I wrong to just leave him like that until he get better? I don't know what to do. I feel like there are just some things about him that is out of my grasp and can't help him with.

Don't get me wrong, I want to put effort into talking it out but it ALWAYS backfire where everything is all my fault. I'm at the point where "If you want to ignore me, fine, I'll go do my own things."