Whether you live in an apartment or a house, chances are there's more than a bit of air coming in around your electrical outlets and switches--especially the ones in exterior walls. The technical term is air infiltration; that link from the EPA explains why it's so important to seal up your house.
A quick and easy way to cut down air infiltration (what most people simply call drafts) is to turn off the power, grab a screwdriver, and order some of these foam outlet and switch sealers.
Just pop off the plate, place the foam gasket over the switch, then replace the plate and switch. It's super cheap. Most well-stocked hardware stores have them; we paid about $4 for a pack of 10 outlet covers and 4 switches. If you have multiple outlets, you can trim and tape the gaskets together to fit.
We found packs of 10 outlets and 10 switches online for a $3.49 at AM Conservation Group. Shipping is additional.