I think my biggest gripe is people using the word so loosely that it has lost meaning. I think it's a good word, though I would never use it around little kids.
There are worse things to say, but I think most of all, we should be annoyed by the context of the words. For example, if I was to bring up the OTHER F word, I have no problem with it...BUT, I do find it wrong when it is referring to a homosexual in a pejorative way.
Aside from that, I think I get most upset over political correctness. I think it is far worse than the words themselves. It means we have lost right to say what we really want to say (despite that fact they we are thinking it).
Ultimately, I think people should be aloud to say exactly whatever they want (but keeping in mind of children who are not yet mature enough to grasp the idea of those words), but also, knowing that the context is far more important.
edit:
I think GARK is spot on with this, especially when he said that it has the biggest impact when it is used by someone who rarely swears at all. I almost feel like that was the core purposes for words like this--to express the ultimately feeling of whatever it is that you are feeling in one word.
The word has entered into our everyday vocabulary giving it very little meaning.
Consider the sexual revolution in the 1970's. Sex at one point was taboo to talk about (especially if you were a woman) but in the 70's, it became a very casual thing to do and talk about. Perhaps it was durring that same era that the word began to become more casual. the F word, but definition, means sex and it began to be used in mainstream movies in that era. I was born in the 80s, so I wouldn't know, but it's just a simple theory.