Views with fields very slow to render

I've trying to improve the performance of a Drupal 7 website and I've identified the bottleneck to be any views that contain fields. For example, one page has a large view containing 500 records per page. Here's a sample report by Devel:

Executed 52 queries in 932.49 ms. Queries exceeding 5 ms are highlighted. Page execution time was 11163.83 ms. XHProf output. Memory used at: devel_boot()=0.88 MB, devel_shutdown()=22.76 MB, PHP peak=29.25 MB.

So the queries take around 1 second and the page uses another 10 seconds to generate it. Is this normal? Here's the XHProf report:

Overall Summary
Total Incl. Wall Time (microsec): 11,098,285 microsecs
Total Incl. CPU (microsecs): 3,030,000 microsecs
Total Incl. MemUse (bytes): 22,891,752 bytes
Total Incl. PeakMemUse (bytes): 26,860,208 bytes
Number of Function Calls: 410,470

Top 10 functions sorted by exclusive CPU time:
Function Name Calls Incl. Wall Time Excl. Wall Time Excl. CPU
theme@1 3,557 4,369,824 467,463 160,000
_field_invoke_multiple 3,010 917,545 481,719 130,000
views_handler_field::theme_functions 3,500 366,360 366,360 100,000
_filter_xss_split 3,508 792,858 467,863 80,000
views_handler_field_field::set_items 1,500 3,748,347 246,105 80,000
element_children 4,687 259,358 165,944 80,000
filter_xss 3,508 1,733,676 286,365 70,000
entity_get_info 13,566 31,000 30,178 70,000
field_default_prepare_view 1,502 70,960 68,906 70,000
entity_extract_ids 9,534 249,628 229,809 70,000

-----

400,000 function calls to generate a table seems excessive.. any idea where to start to optimize this?

I know I can cache the table in different ways, but I'm not looking for those solutions. I want to improve the code that generates the table to begin with.