Some IDs are added multiple times to a query alter.

I didn't use array_unique() in all the places where it makes sense and added too much to the items to loop over. Let's fix that to boost performance, make tests not fail randomly and in general get the code to do what it should be doing in the first place.