A long time ago, Gábor Hojtsy came up with a simple way to allow the WYSIWYG editor to reconfigure itself like this. (I take about 2% credit for the idea since he was inspired to think of it after seeing a slide in a talk I gave, but it was pretty much all him :)
Basically, the idea is for the WYSIWYG configuration code to treat the filter system like a black box, almost as if it's performing a unit test on it. For each WYSIWYG button, we know what kind of output it produces and how that's supposed to be displayed in the end, so we can just test it and see if it works. As a simple example, a "Bold" button could call check_markup('test', $format) and see if it returns something functionally equivalant to what was put in. If it does, then the button is compatible with the text format and can be added to the WYSIWYG.
I'm not sure what happened to that idea or if it ever made it into the issue queue, but it always made sense to me and doesn't require the WYSIWYG configuration code to have deep knowledge of how the filter system works.