Multiple value fields causing pagination issues

I found this strange problem when testing my Views Finder earlier. My view looks like this:

Filter: Node type = 'person'
Display items: unlimited.

Fields: Node title, content: Expertise, content: Position (expertise and position are multi-value fields, and are set to 'group multiple values')

If I just run the view, I get a listing of all the persons I've added, five in total.

The finder has a textfield for searching node titles, and two select-dropdowns for Position and Expertise. Both are prepended with an empty value and I've set the matching to 'contains'. The finder pager is set to 10 items.

When I do an empty search, I'd expect to get the same five results as when running the view. Instead I get three, and a pager to the next page, which displays the last two.
Now, I did some digging in the SQL, and it seems that the first three results generate 10 SQL rows, because of the multivalue fields. Finder then seems to interpret this as 10 items, even though it just displays three.

Is there a way to configure around this problem or will some code be required?