If I were to have only one carrier, it would be a woven. A good woven wrap can take you from newborn to preschooler (wore my 4.75 yo in one the other day). It can do front, hip or back carries. You can wear 2 kids at once in one.
Stretchies do have some advantages, though. They are much less costly. They are snuggly soft even brand new, need no breaking in. You cant back carry with them, though, only woven or hybrid. After 15-20 lbs, baby tends to sag. I also think they are tougher to nurse in.
A hybrid, like a Je Porte Mon Bebe, might be another option.