Views slow load time with date contextual filter

We have a Drupal site with a view of events content type (which is a list of events happening around the area). There is a display with a date contextual filter that you can filter based on a single or range of dates to return events during that time. There is another display with taxonomy term contextual filter that you can filter based on a specific category. The display with the date contextual filter takes ~32s to load compared to display with taxonomy term contextual filter which takes ~8s. Both seem very high. We can't figure out why this is the case since they are in the same view. We tried to enable view cache but I don't that would work since it's contextual filter based. Any suggestions/ideas?