TinyMCE.init configuration

same problem in 7.x
i want to add some custom plugin

See wysiwyg.api.php.
With a small Drupal module you can implement hook_wysiwyg_editor_settings_alter() to override most properties on the settings object passed to the editor instance on init.
There's also hook_wysiwyg_plugin() which allows you to specify where additional plugins/extensions can be loaded from and if they have any buttons or not. Implementing this hook will let you list the plugin (if it's a button-less extensions) or its buttons under "Buttons and plugins" on the editor profile configuration page.

Writing such a module often takes no more than a few minutes, so I can put together the code if you link to the plugin you need.
If you'd like to try yourself first, I recently posted code for another TinyMCE plugin over at #1248704: Adding a simple plugin.