When batteries are disposed of in the trash they're either incinerated, crushed or left in landfill so the metals inside are exposed to the air or ground. On a small scale that's not really a big deal, but at the 180,000 tons per year rate we were just talking about that's some serious possible damage.
Batteries are either dry or wet. Dry is the smaller stuff you use for your devices, wet is used in cars and boats, but both can be recycled. Wet cell batteries can be returned pretty much wherever you buy them. Most car dealerships, Autozone or Wal-mart will take them for free. Dry cell battery recycling depends on your area. Some municipalities have special collection days or places to drop them off, often at sites you can drop off other hazardous materials like paints, oils and chemicals. Electronics stores also sometimes have them: places like Best Buy and Office Depot.
Check with your waste disposal company and municipality to see what they recommend. Or try earth911.com, an EPA affiliated site, or calltorecycle.org, run by a non-profit that aims to promote sustainability.
Note: If you've got alkaline batteries, common in many consumer electronics, they're nonhazardous but can still be recycled so it's not dangerous to toss them, but it's not the best thing you can do either.