I have looked through the issue queue but could not find a similar issue.
Basically, when taxonomy term page's output is displayed using either Panels or Panelizer, breadcrumbs somehow pick up extra items. This happens when a term has parent terms.
For example, for Startups term which is under Small Business, instead of default:
Home » Economy » Small Business
Home » Issues » Economy » Small Business » Startups » Economy » Small Business
This happens when there are no additional breadcrumb modules/theme functions that are enabled. Adding/removing menu items for each of the levels of hierarchy does not help.
Crumbs module solves the problem though.