There are a variety of things that could be happening. Feedings change as the babies grow and mature, too.
Ditto the others' information on the weighing... need a really sensitive scale (reading less than ounces) and weigh before and post feeding.
Some babes are extremely efficient at nursing and it doesn't take long. Some are lazy and only nurse a short time (snacking) and then are ready/hungry again in a very short time. This can happen if they fall asleep early in the feeding. If your baby is falling asleep after 5 minutes, then wake him up and encourage him to nurse longer so he gets a complete feeding. That is how they get the "hind" milk, which is nutritionally different from the fore-milk. And doing so also will (over time) lengthen the time in between feedings. At 4 months, I would think you would be closer to a 3 hour in between feedings time frame. But, every baby is different.
Why does he stop and cry? He could be gassy. Are you burping him midway through the feedings? Or perhaps he is drinking faster than you realize and is frustrated that he isn't getting more milk at a faster rate? Or it could even be that it is his way of complaining that the "hind" milk is harder to get out (it is thicker, so requires a tad more effort on baby's part).
I remember (this has been a decade or so) the school of thought was to switch breasts halfway through the feedings. But that didn't work for us. If I tried to switch breasts, my baby was done. If I let him nurse on one side until he was done, he would take the other side if he was still hungry, AND he got the nutrition of the hind milk from the first breast. Then, at the next feeding, I just started on the "un-used" breast first. It worked for us very well. He was a pretty good nurser. He averaged 15-20 minutes per nursing session. My daughter, on the other hand, was lazy and slow. And I would sit with her for 45 as she would doze off, and I'd wake her, and she'd nurse and doze off and I'd wake her, etc.... just to empty one breast. :/ I didn't have 45 min to an hour out of every 2 hours to spend on nursing, as I had her 3 yr old brother running around to take care of as well. So we went with bottles/formula for her.
Unless he is not gaining weight, then he is probably getting adequate quantity. But if he is only nursing for 5 minutes, then he is probably missing out on the hind milk, and therefore is needing to eat more frequently. If you can get him to nurse longer and get that hind milk, I'd be willing to bet that he will go longer between feedings (both day AND night). :)
Good luck. And remember, all babies and mommies are different, and you have to do what works for you and your baby.
Congrats!