I think that in a perfect world, if you said something unintentionally derogatory, the other person would be able to ask you what you meant and not have anything to worry about. I don't think this can happen in most places right now.
This may be one of the few areas where intent itself matters a lot more than the words that are said. I've never been one to get too mad at words, or symbols, but there are those who do. This isn't anything against them, perhaps being fair-skinned just means I'm not attacked racially. I can't say what I would do if I had put up with slurs my entire life that were based on something I had no control over (I've been slurred for other reasons, but mostly for my actions, not for my genetic or cultural makeup).
I agree that "not saying the words" is not a complete solution to the problem of people not getting along. I think the idea that "watching what we say" is counter to real communication. If we're so worried about how not to offend the other person, I don't think we can ever have a real discussion (and counter to what has been occasionally suggested, I don't try to pick fights with everyone I meet, for the record).
You could follow Dumbledore, I suppose, who suggested that "fear of the word increases fear of the thing itself." If we shy away from racially- or culturally-charged words, we perhaps increase the trepidation with which we would approach someone of another race or culture. I, for one, don't try to discuss the Sioux nickname with Native students because I don't know how they are going to react, or even if they think that by bringing it up as a white person I am automatically pro-nickname (please note that I'm not suggesting that all Native students are anti-nickname, but that I simply don't know what to expect).
I am sometimes unable to communicate well with people from other cultures; I think this stems more from my natural shyness than from any racial overtones, but I really don't know. Does this make me racist, or just aware?
This topic seems to be coming up a lot recently, and I'm happy to discuss it. Whatever the case is, I think that all people (including myself of course) compartmentalize people based on past experiences. Sometimes this is based on rational thought, and other times not. I hope that someday we can learn to realize the difference and adjust ourselves accordingly (and by that, I mean the whole "do unto others" thing).