Google Map not loading on Ajax load

TypeError: google.maps.InfoWindow is not a constructor
TypeError: google.maps.MapTypeId is undefined
I get this error when I reference a node with location field attached and "Inline entity form - Multiple Values" loads the data using AJAX.

For example:
If I create a new node and the referenced fields loads up then I don't get this problem.
but when I edited the node and edited the referenced field or click "Add new node" then I get this error.

It seems that google maps javascript is not being loaded on AJAX request.

I'm not sure yet if this is an issue with Inline Entity Form, GMAP or Location module.