Contextual filters and OpenLayers Map

This may be by design or I may be doing it wrong but I have been trying for several days with no luck.

The quick version is that I am using the views URL, Contextual Filters and Relationships to narrow down with the view displays. If i format the view as a table or grid the contextual filter work great but as soon as I change the format to an OpenLayers Map all nodes are shown.

The long version is that I have two content types, Trips and Restaurants. Restaurants has an Entity Reference field that would allow me to reference any past trip. My view is created to display the restaurants and the URL of the view allows me to narrow down the restaurants by Trip (example.com/trips/% where % is the NID of a trip).

If I go to example.com/trips/(NID of NY trip) I expect to only see restaurants I visited while on that trip and that is the case as long as my view uses any format other than OpenLayers Map. If I am using OpenLayers Map as my format I see all restaurants from all trips. My Data Overlay is just a simple overlay that shows Restaurants.

Is this the correct behavior of OpenLayers or is a bug (or human error) causing my filters to be ignored.

Thanks.