10 panel template in my theme , and now my site's waiting time is 18+ sec!

My site was very slow for some days so i tried evrything and finally found panle is causing the delay in waiting time. This is the steps i tried

# my site is optimized with APC and mod_deflate, so in pagespeed for chrome test, its 93/100
# real problem was waiting time . chrome network analysis shows first GET took approx 18+ sec , then loading of other elements is real quick
# after several test, i found my theme is causing delay. so i put a blank page.tpl in my theme. And you know it took 12+ sec to load that blank page!
# so problem must be with .info, disabled css, then js, still no improvement
# finally removed panel template declaration from .info , and now site took .3 sec to load!
# next i tried with only one panel template in my theme, site took 1-2 sec

so panel template affects site performance!