Well, it's a legitimate concern, because the information is available, and you're dependent on Apple and other application providers to handle it in a sensible manner. Meanwhile, Apple is very vague about what they'll do with the data, and that lack of specificity isn't very compatible with "we entrust you with that data". So we're left trusting and guessing.
When an iPhone app asks the phone for its location, the phone puts up a box asking the user to approve the request, so no app can get your location without your approval (assuming app store apps, on jailbroken anything goes). We "trust" that Apple honors this. But once an app has that location information, it can send that information out into the world, and provide you with some location-based services (e.g. tell your friends where you are, find the nearest Taco Bell, etc.), and it can also go dig up some ads that are specific to your location (downloading a coupon for "Taco Tico", the nearby Taco Bell competitor). At this point, your information is in the hands of that app provider and their ad network.
The reason this is coming up now, I presume, is because Apple is launching their own ad network, and providing location-aware advertising, so that a location-aware app (i.e. one that has gotten your permission to determine its location) will be able to send that location to the Apple ad network. This isn't new functionality, it could have been done on other ad networks, it's new only in that Apple now has their own ad network to compete with the others.
Apple offers a way to opt-out of this location-aware and user-aware advertising, and if you opt-out, you'll just get generic ads. Again, though, you're trusting Apple to do something sensible when you've opted out, to discard any location data and not store it, though they don't specifically say that this is what they do.
Everything I just said is possible on any smart-phone, any phone with both a GPS and internet, and certainly applies just as well to Android. And even without a data subscription, cell phone carriers have the ability to determine your approximate, or possibly precise, location as well. So this comes down to "what do they do with it?", and, again, trust.
-harry