I'm working on working out the bugs for our D6 site (yeah I know, I'm a little behind) and noticed that, when visiting a user/#/edit page, it takes quite some time to finally load due to the way _uc_file_gather_files is checking every single file in the download files directory for an fid.
The problem with this, is that we have thousands of files for sale on our site. To check each one of those files, and search for its fid in the uc_files table, takes about 17 seconds on average, and this is using a pretty beefy database server.
My suggestion is to only "refresh" the uc_files table when new files are uploaded via the admin, or a "refresh files directory" button is pressed, similar to the way the theme system's "clear all cached data" is implemented. There's no reason why this directory should be refreshed every time a user needs to edit his password.
If the concern is that you want to make sure you can add files to a user, either a) just scrape the directory, and allow an fid to be assigned / corroborated after form submit, or b) add a new "Add Files" tab to the user form.
I need to get this worked on immediately, since we're hoping to migrate this week. Let me know if you'd like me to work on a patch and how you'd most like to see this fixed.