Colorbox loads on every page

Maybe I did something wrong with the configuration, but on every page, even if there isn't a colorbox item on the page, there's a colorbox div at the end of the page. It's not visible, but it's in the code when I inspect element; it's the last div before the closing body tag.

<div style="" tabindex="-1" role="dialog" class="" id="colorbox"><div id="cboxWrapper"><div><div style="float: left;" id="cboxTopLeft"></div><div style="float: left;" id="cboxTopCenter"></div><div style="float: left;" id="cboxTopRight"></div></div><div style="clear: left;"><div style="float: left;" id="cboxMiddleLeft"></div><div style="float: left;" id="cboxContent"><div style="float: left;" id="cboxTitle"></div><div style="float: left;" id="cboxCurrent"></div><button id="cboxPrevious" type="button"></button><button id="cboxNext" type="button"></button><button id="cboxSlideshow"></button><div style="float: left;" id="cboxLoadingOverlay"></div><div style="float: left;" id="cboxLoadingGraphic"></div></div><div style="float: left;" id="cboxMiddleRight"></div></div><div style="clear: left;"><div style="float: left;" id="cboxBottomLeft"></div><div style="float: left;" id="cboxBottomCenter"></div><div style="float: left;" id="cboxBottomRight"></div></div></div><div style="position: absolute; width: 9999px; max-width: none;"></div></div>

Is there a way to remove this when colorbox isn't present on the page?

Not only does this add unnecessary DOM, but also there are 3 empty <button> tags, which goes against accessibility standards. With it being on every page, these empty buttons are throwing accessibility errors on almost every page (using this tool http://wave.webaim.org/).

Note: when a page has colorbox items with multiple images per pop-up, the buttons automatically fill in with "next" "previous".