Switch View Modes not working with views looking for NID from url

Setup:
Two Views with Block displays. Each block is now a Block Field through DS fields. Each View needs a Contextual Filter of NID provided as default from URL. When removing the Contextual Filters, the views work just fine. On the initial load, the first view works. On the switch though, the filter is lost.
I've tried the 'Use Ajax' checkbox in Views, and that didn't work. Perhaps I'm missing something? Or perhaps this just won't work with ajax loaded content? Is there a way to make the ajax loaded content url aware?