1. Do NOT rub it with your hands or a sponge: the wetness will basically set the stain.
2. Tape. Yep, use regular scotch tape to gently lift the pollen off of the fabric.
3. Soft Brush. You can also try a soft bristled brush to whisk it away, but this could still press it into the fabric.
4. Sunlight. I didn't need to try this (and it would be hard to get my curtains down) but many people online say that if you put the pollen in the sun for 3-4 hours that it bleaches out the color.
5. Tide Stick. I had a couple of spots left after using the tape and the tide stick did a good job of getting it out (or course the curtains are white so bleach would have done the trick but this wasn't my first battle with pollen).
How have you gotten pollen out of fabric?
Image: Laure Joliet