S.H.
If that's what you want to do, that's fine....
the invitation can state "Please join us for cake and ice-cream to celebrate Johnny's birthday..." or something like that.
That way, people will not "expect" food and to be fed.
And time the party for a non-meal time. Not lunch time or dinner time.
all the best,
Susan