Not blocking pages really

This sounds like a caching issue - I'm not sure if it's something that can be fixed at the module level, without disabling caching across the site. Let's do a bit of detective work on this though.
First, what caching have you got enabled on the site?

@Jaypan,

We also experienced this issue - it was caused by the dynamic page cache module (disabling the dynamic page cache module resolved the issue).

Now that the dynamic page cache module is enabled by default, we should consider using the IP cache context or a custom cache context to ensure that restricted IPs are not being served cached content.

Thanks that makes sense. I'll work on this next time I'm working on the module.

I just tried this module and it did not work as advertised. When I looked at the issues I found this one. My situation is slightly different but seems to be related. I did steps 1 (IP range rather than a list of individual IPs) & 2 as described at the beginning. However, when I tried to access the protected page from an IP not in the allowable range, I was not blocked at all. Access to the page was granted as if there was no restriction in place.

I don't use any cache module, but could the same forces be at work here? Thanks.

Cleaning up the module versions with Drupal 8 & 9 compatible 3.x release. We have to sort out outdated issues to focus on 3.x stabilization. Older Drupal 8 versions (8.x-2.x and 8.x-1.x) will be deprecated soon, so it doesn't make much sense anymore to fix them. Please instead retest if this issue is still relevant for latest 3.x version. If you should experience other issues, please create a separate issue for that, if not already existing.

If the issue from this request still exists in 3.x, please create a Merge Request (MR) against the latest 3.x-dev version to fix the issue and tell us about the actual problem and expected result.

Thank you very much! Let's get this module fixed together as community :)

Confirming the issue. I just installed the module version 3.0.0-beta4 and the dynamic cache is allowing the page to be visited if the cache has been built.

I'm wondering if you using StackMiddleware would work?

https://www.drupal.org/node/1909596#comment-11384523

Please retry with 4.x

We've looked at the code. This will likely still be an issue in 4.x

I also need to file another issue about how restrict_ip_js_alter() can break JS rendering for anonymous users. Essentially, core will cache the partial JS payload and not reset that for anon users who *are* on the proper ip.

Thanks @agentrickard! This is absolutely possible. We also found several things that could / should be improved after becoming Co-Maintainers a while ago.

Would it make sense for you to become (co-)maintainer and help to push things forward here?

I don't have a ton of time right now, though I am trying to unblock a project. It can't hurt to give me access, though I can't promise anything.

For page-level caching, we had to use a ServiceProvider override to override the core getCacheId() method to add an "ip_allowed" or "ip_deneid" string to the cache id.

@agentrickard thanks, I think anyone with a lot of experience, like you, can help to push things forward here. I'll add you as co-maintainer.
Hopefully we all can improve things here sooner or later.

PS: And we should have tests, whereever possible!

Yeah I think we should check and add tests for this with priority to see if the issue (still) exists.