So it seems that CouchDB's SocketClient can only take a certain length for a response body. So if ChangesResource supplies a response which is too big, the clients strips it, hence the replication fails without an error. An easy fix would be to add a 'content-length' header to ChangeResource's ResourceResponse. Any other ideas? I am dealing with JSON data with more than 20 000 bytes fyi.
Original report
Hi, I am having trouble syncing my pages. It seems that Relaxed\Replicator::getReplicationLog uses IDs like '_local/7c26726196f699979bc5ce33f4cbd81e' for $replicationDocId. But CouchDBClient urlencodes the ID so that it gets is encoded to '_local%2F7c26726196f699979bc5ce33f4cbd81e' which then gives me a 404 response. Removing urlencode from CouchDBClient::findDocument fixes it. Anyone getting the same error?
EDIT:
I believe this is all connected to a previous replication which set history_start_last_seq on the replication_log__history to NULL, after that no replication was possible anymore. Can anyone confirm?
Comment | File | Size | Author |
---|---|---|---|
#13 | 2658912--set-content-length-1.patch | 655 bytes | dpolant |
#9 | 2658912--set-content-length.patch | 600 bytes | dpolant |
Comments
Comment #2
alexej_d CreditAttribution: alexej_d commentedComment #3
alexej_d CreditAttribution: alexej_d commentedComment #4
jeqq@alexej_d Thank you for reporting this. In the CouchDB documentation the Changes resource doesn't return the
Content-Length
header (Relaxed Web Services implements these specifications): http://docs.couchdb.org/en/1.6.1/api/database/changes.html#db-changes. I think adding this header in the response is not the solution.I'll check why history_start_last_seq is NULL, this might be the problem.
Comment #5
alexej_d CreditAttribution: alexej_d commented@jeqq My debugging session showed that SocketClient's request method does not return the whole response body in case it is bigger than a certain amount of bytes. After that for example the getChanges method of CouchDBClient get NULL for $request->body. That's why the replication fails silently and history_start_last_seq is set to NULL. But history_start_last_seq is not the main problem here I think.
Comment #6
dpolant CreditAttribution: dpolant commentedHas anyone looked into this further? I am encountering this error on "large" replications as well. I'm seeing the symptoms posted in this ticket: https://www.drupal.org/node/2721709 but I'm posting here because it looks like this ticket will be used to track a solution.
My big concern is that this issue blocks Deploy suite from scaling. Until some kind of replications filters are working, sites are going to need to sync all of their content every time they create a new workspace, so fixing this ticket seems vital if it is indeed stopping large replications.
I'd be happy to work on a solution if someone can point me in the right direction.
Comment #7
jeqqComment #8
dpolant CreditAttribution: dpolant commentedI can confirm that the content-length header solves the problem. The couchdb changes standard specifies chunked encoding, but I couldn't get a Drupal response to implement this properly.
Setting content-length did the trick. I'll provide the patch tomorrow morning.
Comment #9
dpolant CreditAttribution: dpolant commentedHere is a patch that adds content-length header.
Comment #13
dpolant CreditAttribution: dpolant commentedLooks like HEAD doesn't play nice with content-length header. Re-rolling ..
Comment #15
jeqqThank you @dpolant!