J.B.
Hi!
We've had the same problem in our girls' room. It is right over the garage and with wood floors, always seems to be freezing. After two people recommended oil-filled radiators (available at Target for $20-30), we tried one and it's great!! The room is maintained at a comfortable temp. all the time now. I was really worried about fire risk, so I didn't want a space heater or anything like that. I am totally comfortable with this in their room. As long as you don't throw blankets or items directly on it, it's not a fire risk.
Another option- In the winter months when mine was little, I layered a long-sleeved onesie, then a carter's sleeper, and over that a carters fleece sleep sack (those are awesome!_ I think they may even have the sleep sacks with hand mittens on them. Good luck!
J.