All the various update & insert methods (generally those referred to by #879858: Unify entity C(R)UD ) should be possible to be called as part of a multi-insert/update. That'll help IMMENSELY with the time spent generating all these queries.

This'll be complicated, of course, by the fact that many of these objects contain data drawn from multiple tables, so we'll still need to issue multiple queries. Which means batching each one of those (probably). Yuck.

Marking normal now, but if building the db for ALL our projects takes too long on util, we'll up this to critical.

Comments

sdboyer’s picture

sdboyer commented
Status: Active » Postponed

This is a really sticky problem, as mentioned in the unified CRUD issue. Maybe we'll work on it, maybe not...but definitely not this sprint.

marvil07’s picture

marvil07 commented

related to #484376: Use the batch API for the Fetch Now link ?

sdboyer’s picture

sdboyer commented

Sorta related, I guess. Doing batched writes would mean that db queries take less time, and therefore lessen the need, but not enough that it would ever get rid of the need to implement batch API for manual log fetching.

sdboyer’s picture

sdboyer commented
Issue tags: +git sprint 10

Part of the performance bit. Will be evaluated next week, and probably moved to phase 2 leftovers.

sdboyer’s picture

sdboyer commented
Status: Postponed » Closed (duplicate)

halstead's new approach using fast-export (#1019976-12: Improve performance of log parser) basically does this, but FAR more efficiently than anything I'd originally conceived of. Closing this as a duplicate.