Just like the "Inline image filter" can be active or not in a Input format filter, so also the img_assist should accessible or not depending on the used Input Format. Just like a change of Input filter makes the WYSIWYG Editor disappear, also the img_assist buton and access should be controllable by the active Input format filter. This can also avoid conflict between Input format filter and the use of img tag for instance (using img_assist with an inappropiate Input format filter).

May be an extra select box in the access permission to tag the valid Input formats would do?