This tutorial gives an overview of how to make a bunch of fabric grocery bags for less than 2 dollars each. (And most of that cost is in the webbing for the handles. They'd cost less than 50 cents if you use denim from old jeans.)
Note (March 09)- I've been using these same bags for just under 2 years. We use them almost every single time we go to the grocery store. And we have 3 kids at home with us, so we shop a lot. We also use them for ALL of our errands... clothes shopping, library, even the hardware store. I also wash the bags pretty often. The point it, they have been used hard. Very hard. And even though they're made of fairly thin fabric (which is kind of the point, as I need to be able to cram all of them into their bag, week after week), none of them have ever torn or ripped.

To make these, you need access to a sewing machine, but don't even need to sew all that straight. Remember, you just need to make them prettier than the ugly plastic things you're using now. How hard can that be?