My daughter is now 16..but when she was this little age, we ate out and were out and about all the time. She had what she called her "guys" purse. She had all sorts of small figurines...from her birthday cake she had a Pooh set (Pooh, Piglet, Rabbit, etc)..she had a few happy meal types (but those are less durable)... she had horses and animals too..and anything else I could get my hands on. They all stood up and were about 3 inches tall. There must have been 30 of them in there at any given time. She had an old "train case" style make up bag of mine with a handle on top (square type zip case, maybe 6-8 inches wide). She would carry it with her, and pull them out one by one and line them up on a table top at a restaurant, on the little table in the Dr office....on a hardcover book from the diaper bag/backpack that I would put on my lap if in a waiting room, etc. And the best part was she made up her own stories and little games with them (voices and such like scenarios)...AND they went right into the dishwasher or soapy sink so easily!
Personally I find when other children are playing noisy, light up button type toys in such public places to be extremely annoying and rude. I like the calculator suggestion someone had, tho, for the buttons. We did that too...she had her own small one in her little make up bag purse sometimes. We made DD be a part of the solution and together we packed her little bag, or just double checked it, etc before going anywhere. It worked great most of the time. We also always had coloring and sticker stuff with us (she Had a Little suitcase bag for that too..LOL). Always! Foam stickers from a craft store (in big tubs..often seasonal..you can just put some into a zipper baggie to take a few with) are great for small motor work for young children..easier for them to get the paper backs off of themselves and feel successful.
Good luck!