I'm going to James's party thing tomorrow. So I told him "I'm coming to your party." Now, I wouldn't tell you that I was coming to his party, and it'd be a little weird if I told James that I was "going to your party."
I mean, it's not semantically incorrect, but I guess it makes a little more sense to say you're going somewhere, and then to tell an event organizer that you're coming.
On that note, it is a lot more appropriate to tell someone "I'm coming in your butt" than it is to say "I'm going in your butt." Ewww!