If a log message which is too large is passed to this module it will emit UDP datagrams containing invalid chunks of JSON. The Lagoon logging infrastructure requires every individual UDP datagram to be a standalone valid JSON message. If it isn't the log message will be dropped. See the attached screenshot of this kind of error.
The easiest way to address this issue might be to bump the chunk size from 15000 to 65507 which is the maximum payload size of IPv4 UDP[0], and truncate the log message when it is passed to the module so that the entire JSON object is guaranteed to fit inside a single chunk. This should enough to avoid invalid JSON but it will discard part of very large log messages.
To avoid losing any log data for very large log messages, this module would have to split the incoming log data into parts, and then send the parts inside individually valid JSON-encoded UDP datagrams, with multiple UDP packets for very large log messages. I don't know if it's worth the effort, after all 65K ought to be enough for anybody :)
[0] https://en.wikipedia.org/wiki/User_Datagram_Protocol#UDP_datagram_structure
Comment | File | Size | Author |
---|---|---|---|
Screenshot_2020-07-08 Discover - Kibana.png | 22.53 KB | smlx |
Issue fork lagoon_logs-3157657
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #4
fjgarlin CreditAttribution: fjgarlin as a volunteer and commentedThere is an open issue to update the monolog library here.
So I haven't changed the chunk value yet. What I've done is when the message is to big then send multiple log entries like this `[1/3]...` `[2/3]...` and `[3/3]...`. That way we can still capture all the message.
Please test this and if you're happy mark as RTBC.