I nursed my kids for forever (well, 20 months for the first one, and almost 2 1/2 years for the second). I nursed on demand and always had enough milk. When I pumped, I also hardly got any. 4 oz was a good pumping session for me! So - do not assume that because you are only pumping a small amount that you do not have enough. I forget all the indicators you are supposed to look for - a certain number of wet diapers a day? # of nursing sessions? Babies are hungry in the beggining, and they can cluster feed, where they feed more frequently for a certain period of time, and then space them out a little more.
Often you will produce a tremendous amount of milk in the beggining, your body is trying to regulate your mild porduction with your babies appetite. Again, it does not mean that your supply is inadequate for your baby simply because it is diminishing - you may be just getting in synch with your baby. It is a delicate balance - I would absolutely hold off on formula for right now - at 3 1/2 weeks, it is so early!
If you do have a lactation consultant, use her! Ask her all these questions and see what she has to say.
Sounds like you and your baby are figuring things out - it is normal to have lots of questions about nursing! Good luck to you, sounds like it is going OK. Remember to drink lots of water, and keep up the good work!