I would review the bill and see how much was charged for the meal. depending on the type of service (buffet vs served meal) and food offered (casual vs formal) - I would just provide 15-25% of what the bill was depending on the amount of work that was done by the "server" and the cost to be safe.
My sister had a woman who catered a party at her house and did a fairly simple Mexican Themed buffet. The woman was there for about 5 hours (though prepped a lot at her own home) and my sister gave her $150 because she did an amazing job...and by herself...the day after she was in a car accident. Even a buffet can be a lot of work.
It is a hard question without more details - I was a server in the past and generally...waiting on 50 people is A LOT of work (a party at my restaurant would usually get me about $100-$300)...but depending on the age of the person helping (a 10 year old boy vs a 22 year old server is a huge difference), and what they did, you don't have to go that far.
Best bet is to look at what they did...then, I would review if you tipped enough. If he just cleaned up plates...$40 is probably adequate. If he did a great job, refilled glasses, got people items and whatnot, I would definately do more.
* If you are concerned, call the owner and sheepishly explain you are new to the catering thing, and not sure that you compensated the server enough, and ask what would be fair.
* If you want to be safe...tip 15-25% of the final bill, like you would in a restaurant.
* If you were impressed beyond belief with the service...make sure you let them know, add a little to the total (even if $40 is 15%), with a note saying "We really want to thank you for the wonderful service you provided in our time of grief, and wanted to let you know how much we appreciate how you helped us during this difficult time...could you please give this $___ to ______ and tell him how much we appreciated his hard work.
Added later ~ Catering is even harder than restaurant serving because you are always improvising. At a serving job you know where everything is (extra silverware, dishes, food), you know where everything should be placed, you know what ingredients you have in case you run out of anything, etc. There is a process. When catering, all rules are out the window. You can't send someone to the store because you are out of cream, you can use X instead of Y, you can't change up the menu because something didn't work out right. Also as a server, you are running around trying to find where the bathroom is for a guest, or digging out something from a cupboard because the guest wants this or that. It is actually a lot more stressful, too, because you are not in your comfort zone, and know that at work if you spill a glass of wine you will wipe it up and be done...at a catering job you may have just wrecked a $30,000 rug of Mrs. Jones'.
The other thing is...people go to buffets/brunches and tip less because they had to get up and get their plate. The funny thing is buffets are WAY MORE WORK for the server than a traditional meal. Generally, at a meal you will refill someone's glass once maybe twice...brunch tables sit longer and enjoy 6 or 7 "warm-ups" of coffee...and 4 glasses of champagne without blinking an eye. Instead of the server clearing a salad plate and a dinner plate, they are clearing the 7 plates your child used because each one had 4 grapes and a bagel on it. Most servers will hit your table about 3-7 times in an average meal. On a buffett with even 10 people your server is going to multiply that by 5 or 10 times just to keep dishes cleared, get grandma cream, grab another glass of water for your brother who is severely dehydrated from last night at the local watering hole, and keep on top of all the items like syrup and whatnot everyone needs.
The server gave you the look because probably they do not usually receive an outright tip...the owner probably takes it from the bill and divides it between everyone who prepped and worked on the meal (which could be 5 more people for all you know).
I would (and granted having worked in the industry for a long time) go back to the owner and fork out a bit more...IF you were pleased. Just because it was at somebody's home doesn't mean the same good service wasn't provided...but that is just my opinion...and biased because I worked in restaurants for many, many years! :)