Breakup tracking of content entities into smaller chunks to prevent memory limit issue

This should already work, we have batching already implemented for this case, just on a higher layer. (That's the $page being passed to that method.)
(Also, this is not indexing, but tracking.)
Could you find out the exact backtrace of the call to that method, if it really does get $page = NULL passed? If it does get a page number, the only thing that could go wrong is you having the tracking batch size (search_api.settings:tracking_page_size config value) set too high for your site.

Again, it's "tracking", not "indexing".
In any case, though, your problem description makes sense. Indeed, if running in the CLI, after saving an index the module will try to get the complete set of items already tracked right away. This was implemented in #2624424: Indexes enabled with drush aren't tracked, since it seems Drush doesn't automatically execute batches we set. (If we could manually make Drush execute the batch, that would probably be the best solution. But I don't think we can/should call drush_backend_batch_process() right in the middle of the post-save phase. Maybe with a shutdown hook – might that work?)

However, while it does currently try to load all entities in one request/process, it will already do so in batches, not load them all at once. That means that most of your patch (the whole "chunks" code) is unnecessary, we simply need to clear the static cache after loading the entities:

diff --git a/src/Plugin/search_api/datasource/ContentEntity.php b/src/Plugin/search_api/datasource/ContentEntity.php
index eb5bf56e..695c691b 100644
--- a/src/Plugin/search_api/datasource/ContentEntity.php
+++ b/src/Plugin/search_api/datasource/ContentEntity.php
@@ -747,6 +747,13 @@ public function getPartialItemIds($page = NULL, array $bundles = NULL, array $la
       }
     }
 
+    if (php_sapi_name() === 'cli') {
+      // When running in the CLI, this might be executed for all entities from
+      // within a single process. To avoid running out of memory, reset the
+      // static cache after each batch.
+      $this->getEntityStorage()->resetCache($entity_ids);
+    }
+
     return $item_ids;
   }
 

However, the attached patch is much more massive, since I also took the opportunity of refactoring the php_sapi_name() === 'cli' check (which is used quite a bit throughout our code) to a helper method on the Utility class. But the effective portion of the patch is just the above.
Please try out whether this also resolves your problem!

And, to anyone that wants to review: Do you think the new method makes sense like this? It would actually have been great to somehow make it dependency-injectable (is that a word), i.e., an instance method on some service, so it can be mocked in tests (e.g., to really test CLI compliance in the CliTest, not some state setting). However, I don't really think that would make any sense from any other point of view, so it seemed like a bit much just for the sake of slighlty better tests. This way, it's really just as simple to use as the previous code, just a bit more … "clean"?
Anyways, I'd be glad for others' opinions on this!

And, to anyone that wants to review: Do you think the new method makes sense like this? It would actually have been great to somehow make it dependency-injectable (is that a word), i.e., an instance method on some service, so it can be mocked in tests (e.g., to really test CLI compliance in the CliTest, not some state setting). However, I don't really think that would make any sense from any other point of view, so it seemed like a bit much just for the sake of slighlty better tests. This way, it's really just as simple to use as the previous code, just a bit more … "clean"?

I think this is a good idea. While it's not easy to mock this from a test it does make sense to extract this code, it makes it easier in the other code to get the actual meaning.

In any case, this patch looks solid from just a review but I haven't manually tested it yet.

Thanks a lot for testing, good to hear it works!
Also thanks for your review, Joris, of course!
Committed.