Incompatible with Domain Taxonomy module

Domain XML Sitemap works fine. Domain Access rule can be integrated into XML Sitemap. However, XML SItemap module is not compatible with Domain Taxonomy module. After restricting access to some taxonomy terms, those terms are no longer accessible on certain domains but sitemap ignore all Domain Taxonomy settings and still generate all the taxonomy term links. This leads to 404 error to visitors after sitemap is submitted.

I think this feature is handy since the support for DA is so nice but there is this one single flaw. Many taxonomy terms cannot be shared seems they can be very irrelevant to certain websites. Any clue how to get this done?