The code in plugins/display-edit.inc takes good care of integrating the 'settings form' callback form for custom layout plugins and handling its submission also with the 'settings submit' callback. However, these form functions are not used directly, they are used via panels_ui_layout.class.php, which does invoke the form builder from display-edit.inc to build in custom elements to the form from layout plugins but will never invoke the submission handler as part of its submission handling. It merely invokes the generic parent submission handler in ctools_export_ui.class.php and a tiny bit of modifications on its own.
The result is that the submission handler specified for the layout plugin (as well as other submission logic implemented for panels layouts) is never invoked.
Found while working on a pretty early version of http://drupal.org/project/layout. Without this patch, the layouts will not be saved properly there, because the submission process is not integrated form the layout (builder) plugin.