If I was the other mom, I would prefer to be told these concerns rather than just avoided/dropped. Cause she might really be reaching out to you and your kid and really want you guys as friends.
I'd call, explain the 5 kids and say 'why don't you guys come here?'. Or a neutral place.
SAHM to Magnolia May (09/10) and Luke Russett (04/13) and wife and best friend to my airman.