Segmentation fault during cron

I think this could be the at fault code, where its working with a large array in memory.
https://git.drupalcode.org/project/google_analytics_counter/-/blob/4.0.x...

An update, our host PHP memory limit is 128mb.

@kaszarabert Thanks for the ideas and I will use them to investigate more, It stopped working across all our Acquia environments, and they actually had different code deployed, so I think it could be an issue with the amount of data being retrieved from Google is my current theory. Since nothing on the servers changed at the point in time when things broke, at least that I have found yet.

Will research more and update tomorrow.

Still debugging, added a bunch of logging to the current cron logic, looks like the problem lies in the queryTotalPaths()

[info] Starting execution of google_analytics_counter_cron(), execution of file_cron() took 4.12ms. [19.61 sec, 64.75 MB]
 [notice] Starting cron with memory: 71827456 [19.62 sec, 64.82 MB]
 [notice] Before node count query [19.62 sec, 64.83 MB]
 [notice] Total nodes: 16481. Memory: 71827456 [19.63 sec, 64.92 MB]
 [notice] Starting main processing. Memory: 71827456 [19.63 sec, 64.92 MB]
 [notice] Before database truncate. Memory: 71827456 [19.64 sec, 64.92 MB]
 [notice] After database truncate. Memory: 71827456 [20.23 sec, 64.99 MB]
 [notice] Before queryTotalPaths. Memory: 71827456 [20.23 sec, 64.99 MB]
Segmentation fault

The problem seems to lie with the serialize() in the buildQuery method, not really sure where I go from here to debug further....

 [info] Starting execution of google_analytics_counter_cron(), execution of file_cron() took 2.59ms. [19.88 sec, 64.77 MB]
 [notice] Starting cron with memory: 71827456 [19.89 sec, 64.86 MB]
 [notice] Before node count query [19.89 sec, 64.86 MB]
 [notice] Total nodes: 16480. Memory: 71827456 [19.9 sec, 64.95 MB]
 [notice] Starting main processing. Memory: 71827456 [19.9 sec, 64.95 MB]
 [notice] Before database truncate. Memory: 71827456 [19.91 sec, 64.95 MB]
 [notice] After database truncate. Memory: 71827456 [20.52 sec, 65.02 MB]
 [notice] Before queryTotalPaths. Memory: 71827456 [20.52 sec, 65.02 MB]
 [notice] Starting queryTotalPaths [20.52 sec, 65.02 MB]
 [notice] About to buildQuery [20.53 sec, 65.02 MB]
 [notice] Starting buildQuery [20.53 sec, 65.02 MB]
 [notice] Got config [20.53 sec, 65.02 MB]
 [notice] Chunk: 1, Pointer: 0 [20.54 sec, 65.02 MB]
 [notice] Built query dates: Array
(
    [start] => 2024-10-22
    [end] => 2024-11-20
)
 [20.54 sec, 65.02 MB]
 [notice] Property ID: 371XXXX43[20.54 sec, 65.02 MB]
 [notice] Starting cache options creation [20.55 sec, 65.15 MB]
 [notice] Params structure: Array
(
    [property] => properties/371XXXX43
    [dateRanges] => Array
        (
            [0] => Google\Analytics\Data\V1beta\DateRange Object

        )

    [dimensions] => Array
        (
            [0] => Google\Analytics\Data\V1beta\Dimension Object

        )

    [metrics] => Array
        (
            [0] => Google\Analytics\Data\V1beta\Metric Object

        )

    [offset] => 0
    [limit] => 1
)
 [20.55 sec, 65.15 MB]
 [notice] Before serialize [20.55 sec, 65.15 MB]
Segmentation fault

PROBLEM:

Had a segmentation fault when trying to serialize() Google Analytics API objects (DateRange, Dimension, Metric)
These objects can't be safely serialized in PHP, causing the crash

SOLUTION:

Create two separate parameter arrays:

$parameters: Contains proper Google Analytics objects needed for the API call
$cache_params: A simple array with basic data types that can be safely serialized

WHY:

We only need serialization to create a unique cache ID (md5 hash)
The actual API call requires proper Google Analytics objects
By separating these concerns, we avoid the segfault while maintaining functionality

So going to make that proposed change, Claude.ai helped debug the issue.

UGH now getting

 [info] Starting execution of google_analytics_counter_cron(), execution of file_cron() took 3.68ms. [21.11 sec, 64.77 MB]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1732220684.750398     339 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1732220684.753480     339 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
Segmentation fault

Ok So after a lot more debugging on and off with acquia support, its something related to the Google Vendor gax, I found this comment that talks about a very hard to debug segmentation fault! And I can agree it was very hard :)

https://github.com/googleapis/gax-php/blob/140599cf5eae2432363ce6198e9fd...

And I was able to add enough debugging to pull this out of the codebase

[debug] Getting feed [37.63 sec, 116.37 MB]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1736874324.703588 708 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1736874324.703698 708 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
[error] Error: Maximum call stack size of 10436608 bytes (zend.max_allowed_stack_size - zend.reserved_stack_size) reached. Infinite recursion? in Google\ApiCore\CredentialsWrapper->Google\ApiCore\{closure}() (line 244 of /var/www/html/vendor/google/gax/src/CredentialsWrapper.php) #0 [internal function]: Google\ApiCore\CredentialsWrapper->Google\ApiCore\{closure}()
#1 /var/www/html/vendor/grpc/grpc/src/lib/UnaryCall.php(43): Grpc\Call->startBatch()
#2 /var/www/html/vendor/grpc/grpc/src/lib/BaseStub.php(295): Grpc\UnaryCall->start()

And then I found the hosting I am using has these PHP differences

grpc
grpc support => enabled
grpc module version => 1.68.0
grpc.enable_fork_support => 0 => 0
grpc.grpc_trace => no value => no value
grpc.grpc_verbosity => no value => no value
grpc.poll_strategy => no value => no value
fiber.stack_size => no value => no value
fiber.stack_size => no value => no value
zend.max_allowed_stack_size => 0 => 0
zend.reserved_stack_size => 0 => 0

Still don't have a solution, but at least the problem is narrowed down a lot more.

Thanks I tried the suggested "zend.max_allowed_stack_size: -1" but did not help :(

Were still trying to fix this, but got a new breakthrough, after many attempts to force the Google/gax to use REST instead of GRPC we still were getting a seg fault (seemed to be around accessing the credentials file), BUT we found once we removed the PHP package protobuf https://github.com/protocolbuffers/protobuf our seg faults went away, so that appears to be causing the issue.

# ddev config.yaml
webimage_extra_packages: [
  "pngcrush",
  "libjpeg-progs",
  "ffmpeg",
  "clamav",
  "php${DDEV_PHP_VERSION}-dev",
  "php${DDEV_PHP_VERSION}-grpc",
  "php${DDEV_PHP_VERSION}-oauth",
  "php${DDEV_PHP_VERSION}-http",
  "php${DDEV_PHP_VERSION}-gmp",
  "php${DDEV_PHP_VERSION}-dba",
  "php${DDEV_PHP_VERSION}-gnupg",
  "php${DDEV_PHP_VERSION}-protobuf",  <--- This is the problem, once removed and "ddev restart" and seg fault is fixed.
  "php${DDEV_PHP_VERSION}-pspell",
  "php${DDEV_PHP_VERSION}-raphf",
  "php${DDEV_PHP_VERSION}-rdkafka",
  "php${DDEV_PHP_VERSION}-ssh2",
  "php${DDEV_PHP_VERSION}-yaml"
]

Full Disclosure I would not have gotten this fixed without Calude.ai but see the new MR22 which has in it a fix that fixes the segfault issue. I will paste an AI explanation from Claude below, as this is a bit over my head.

Steps to reproduce.

1. Add GRPC and protobuf https://github.com/protocolbuffers/protobuf to your environment, I used ddev as noted above

webimage_extra_packages: [
  "php${DDEV_PHP_VERSION}-dev",
  "php${DDEV_PHP_VERSION}-grpc",
  "php${DDEV_PHP_VERSION}-http",
  "php${DDEV_PHP_VERSION}-protobuf"
]

2. Run cron see seg fault.

Claude.ai Explanation as follows:

The root cause involved three interacting issues:

Protobuf Objects and Serialization

When we tried to serialize Google Analytics' response data, we were dealing with special Protobuf objects. These objects aren't regular PHP objects - they're optimized for network communication using Google's Protocol Buffer format. Trying to serialize them directly caused conflicts between PHP's native serialization and Protobuf's internal serialization mechanisms, leading to segmentation faults.

Memory Management

The original approach tried to handle these Protobuf objects as regular PHP arrays, which created memory management problems. The segmentation faults occurred because we were trying to access memory in ways that the Protobuf extension wasn't expecting, especially when trying to serialize these objects for caching.

RepeatedField Type Safety

The final error revealed that we were treating a Google\Protobuf\Internal\RepeatedField (a special Protobuf container type) as a PHP array. This type mismatch caused PHP's array functions to fail because they expect native PHP arrays.

After testing on our Acquia hosting, this does appear to fix our issue, and I now see GRPC logging after a cron run with debug.

drush cron -vvv --debug

 [info] Starting execution of google_analytics_counter_cron(), execution of webform_scheduled_email_cron() took 5.41ms. [47.08 sec, 110.3 MB]
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1737655590.172511     303 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1737655590.176159     303 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
 [info] Execution of google_analytics_counter_cron() took 650.16ms. [47.73 sec, 111.84 MB]
I0000 00:00:1737655651.551433     245 call_credentials.c:168] GRPC_PHP: call credentials plugin function - begin
I0000 00:00:1737655651.553439     245 call_credentials.c:171] GRPC_PHP: call credentials plugin function - end
 [info] Retrieved 7350 items from Google Analytics data for paths 1 - 7350. [109.36 sec, 150.36 MB]
 [info] Merged 7349 paths from Google Analytics into the database. [138.03 sec, 150.71 MB]
 [info] Cron run completed. [138.05 sec, 150.4 MB]

Anytime, glad to have such a responsive maintainer!