Yes, it could be that he could handle small amounts and not larger amounts. But if you really want to be sure I would eliminate all dairy for a couple weeks and pay close attention to everything, then add back in your small amounts to be sure you don't notice any small changes or discomforts.
Single Mommy to my 3 year old DS born all natural
. We are a Cosleeping
, Babywearing Non-Vaxing