If your outlet is loose when you go to plug something in, the problem is likely that the outlet doesn't fit snugly between the electrical box and the cover plate.
It seems harmless enough, but a loose electrical outlet could result in unsightly cracked and broken faceplates or even fire-hazard arc faults.
But this low-brow problem has an easy solution: Outlet spacers. They're little plastic pieces that wedge between the electrical box and the outlet for a better fit.
You can grab a $3 pack of outlet spacers online or at most hardware stores and they're a cinch to install: Just cut power to the outlet and wedge 'em around the screw holding your loose outlet to the electrical box until there's a tight fit. Voila!
These spacers also work well to bring an outlet flush with a thicker wall. If you're planning on installing bead board, stone or any other wall enhancement with thickness, you'll probably want to grab a set like these. They can "Lego" on to each other to bring an outlet up flush with your new wall.