Worst case, can't you just compute pairwise keys for each pair of participants? It'll add (n-2)*keylen bytes to each of your messages, but it's easy to understand and implement, and most multi-party conversations don't end up with that many participants anyway.

I think the biggest problem with trying to do anything clever here is that keeping everyone in sync while simultaneously preventing DoS by one participant would be pretty difficult. You'd need something like Paxos to ensure everybody got the messages in the same order, and you'd have to keep around old keys and messages until everyone acknowledged them, otherwise participants could easily get "lost."