yep, it happens much faster the second time. you may have just been bloated, but honestly, it does happen faster.
The first time I was pregnant, I was very very thin and in shape (Pilates daily), so I noticed some of my "skinnier" pants didn't fit even before I realized I was pg, and that was about the same time as you. I couldn't wear maternity pants until about 6 months, but yet, that trim waistline disappeared right away.
Of course, I'm on pg # 3, and I'm HUGE at 6 months this time! It all happens much quicker because your body already knows what to do, and the muscles have been stretched that way, so they stretch again much more easily.