When I include an apache solr search results content pane into the term_view page, filter the results to the current taxonomy term, and select "Inject hierarchy of first term into breadcrumb trail" in the page settings, the breadcrumb appears as follows:

home > first term > ancestor term 3 > ancestor term 2 > ancestor term 1 > parent term

After some digging I found that apachesolr_search.module adds your search terms or filters into the breadcrumb so that what I thought was the first term was really the apache solr filter of the same name. This renders the breadcrumb out of order because of the way ctools injects the hierarchy of the term. It seems there should be a way to turn the breadcrumb functionality off in apache solr if you want to build the breadcrumb in another way. This issue may need to be switched over into apachesolr_search, but since I discovered the problem through panels I figured I would post it here.