I've ran into this issue and I'm posting this in case anyone else would stumble upon it.

If your responsive grid view spits out too much rows and some of the columns are just empty, specify an extra filter. The current view may collect other content that isn't of any importance.

I only had one filter enabled (Published = yes), which caused this kind of behaviour. When adding a content type filter, this was solved.