Views data can contain cacheability metadata

Views data providers can detail additional cache metadata in the arrays of information they return. This is necessary because whilst cache metadata is added automatically for entity based data this is not the case for other data sources, for example the file_usage table added by \Drupal\file\FileViewsData.

To add additional cache metadata to your views data do:

$data['file_usage']['table']['base']['cache_tags'] = ['several', 'cache', 'tags'];
$data['file_usage']['table']['base']['cache_contexts'] = ['several', 'cache', 'contexts', 'that', 'exist'];
// The following would disable caching.
$data['file_usage']['table']['base']['cache_max_age'] = 0;