Feeling confused

I am 31 I am married and have two kids. I was married the first time when I was 17 and married the love of my life. Because of being young and immature we’ve both made many mistakes. We have kept in contact on and off for 13 years and have even stepped out on current partners with each other. As much as I try to stay away both of us can’t. We both have families now which makes it much harder to leave. This is the only man I have been in love with and I wish it could be as easy as leaning it all behind but when there are children involved everything must be considered. All of this has made my anxiety go up because I ask myself how did I end up here? I taught my life would have been different but life isn’t always as planned. My heart wants him and I can’t fight the connection. I pray the universe starts to open a pathway for us