Perhaps it is not so much about your husband leaving you, but about your own insecurities making you feel unappealing and unworthy of his love. Find confidence in yourself, do not compare yourself to others, and listen to him when he tells you he loves you but most of all believe him. :)
Past experiences can play a huge part in current relationships. Sometimes insecurities over yourself or where you are on your journey through life can impact this as well. Try to focus on the present moment and feel the love he's radiating towards you.
You may want to consider that the worry, by itself, does nothing useful for you. In fact, it may be detrimental to the relationship as the worrying will no doubt manifest itself in some negative way and therefore (paradoxically perhaps) he may well be more inclined to leave should the worry continue. Worry is an apprehension of something occurring in the future. One possible way to deal with the worry might be to say to yourself "no amount of worrying is going to make him stay if he wants to go so perhaps I should just park that thought and get on with keeping the relationship alive".
Sometimes it is hard for us to grasp that someone can love us so deeply. Even though they assure us, sometimes we need even more assurance. Trusting the fact that he loves you, and is not leaving takes time, and for you to open up a bit more to him.
Maybe it's because you have trust issues and fear of being abandoned, It's a good think he reassures you on this point, and you have to hold to that, to what he is, how you love him and how you trust him. I think with time the fear will pass, and talking about it with him is a good thing to do :)