I wasn't able to load customer profile context from order fields in panels, found that table in foreign keys declaration for customer profile reference fields was wrong, following change fixed the issue.
--- code/sites/all/modules/contrib/commerce/modules/customer/commerce_customer.install (revision da6bc9fb81fee032d8238cda5d6227cb3ebe35cc)
+++ code/sites/all/modules/contrib/commerce/modules/customer/commerce_customer.install (revision )
@@ -166,7 +166,7 @@
'foreign keys' => array(
'profile_id' => array(
- 'table' => 'commerce_customer',
+ 'table' => 'commerce_customer_profile',
'columns' => array('profile_id' => 'profile_id'),
Hope this helps.