Drupal adds these three files the first time drupal_add_js() is called with anything in the $data argument.
In Drupal 6, we didn't call drupal_add_js() by default, at least not on visitor-facing pages.
In Drupal 7 and 8, this now happens on every request, because of the ajaxPageState setting added in drupal_theme_initialize() - note that setting is added even if ajax.js is never loaded on the request.
A lot of sites don't necessarily add any other jQuery on visitor-facing pages, so we could save a lot just by ensuring that we don't load any js by default unless absolutely necessary.
Here's an only-very-slightly tested patch that does this - when you get a setting the first time, it add the default settings, when you get something other than a setting the first time, it adds the files.