are there any alternatives for this like custom filter?
I'm afraid jsbeautifier and the GeSHi filter module are hard to match/combine for several reasons.
* jsbeautifier's website mentions that it can be run server side, but this requires a server side setup that is not typical for most drupal sites I guess. Apart from this, integrating it with Drupal in PHP will be non trivial and I don't have the resources to do let alone support this.
Maybe you'll have more luck at one of the syntax highlighting alternatives:http://drupal.org/project/syntaxhighlighterhttp://drupal.org/project/chili_highlighterhttp://drupal.org/project/ajax_markup
Drupal is a registered trademark of Dries Buytaert.