High response time question

I think this is possibly more a drupal 7 latency issue more than xmlsitemap but I wanted to see if anyone else is getting this as well and if they have figured out a resolve. This is what google webmaster tools is telling me as a warning message for the sitemap:

Some URLs in the Sitemap have a high response time.
Some URLs listed in this Sitemap have a high response time. This may indicate a problem with your server or with the content of the page.

Specifically it seems to fail at a taxonomy term page that is only a term name (no description) with one node teaser referenced to it (no pics, just text) so there seemingly wouldn't be much load going on here but anyhow googles indexing seems to fail when trying to index taxonomy term pages. Any feedback is appreciated...