My dad is like that. He thinks I should only have 2. I have one right now and him and anyone else who says anything (don't know if you're religious) is "it's not up to me how many kids I have. If God wants me to have these kids I will. Kids are a blessing no matter the situation". We all know that even when people have a their tubes tied, they get the snip, birth control or whatever, you still have a chance of getting pregnant. May be smaller chance but you still do. And that's also what I say.
It's not up to them how many kids you have, you're the one raising those kids not them. So I don't see why they are so concerned. I'm sorry you have to deal with people saying things. It's your time to enjoy and embrace more children and you don't need them making you feel bad about it (or your children)