The only issue I see is the children. I'm Catholic myself and so is dh but if he one day decided to convert after initial shock I would let it be. With one caveat. The children will still be raised Catholic, period. It's something we agree to when we marry in the church.
H&T Mama to Faolan
6/7/12 & Malachy