So it seems that CouchDB's SocketClient can only take a certain length for a response body. So if ChangesResource supplies a response which is too big, the clients strips it, hence the replication fails without an error. An easy fix would be to add a 'content-length' header to ChangeResource's ResourceResponse. Any other ideas? I am dealing with JSON data with more than 20 000 bytes fyi.

Original report

Hi, I am having trouble syncing my pages. It seems that Relaxed\Replicator::getReplicationLog uses IDs like '_local/7c26726196f699979bc5ce33f4cbd81e' for $replicationDocId. But CouchDBClient urlencodes the ID so that it gets is encoded to '_local%2F7c26726196f699979bc5ce33f4cbd81e' which then gives me a 404 response. Removing urlencode from CouchDBClient::findDocument fixes it. Anyone getting the same error?

EDIT:
I believe this is all connected to a previous replication which set history_start_last_seq on the replication_log__history to NULL, after that no replication was possible anymore. Can anyone confirm?

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alexej_d created an issue. See original summary.

alexej_d’s picture

Issue summary: View changes
alexej_d’s picture

Title: getReplicationLog using slash in ID for findDocument leads to 404 errors » ChangesResource's get response is missing a "content-length" header
Issue summary: View changes
jeqq’s picture

@alexej_d Thank you for reporting this. In the CouchDB documentation the Changes resource doesn't return the Content-Length header (Relaxed Web Services implements these specifications): http://docs.couchdb.org/en/1.6.1/api/database/changes.html#db-changes. I think adding this header in the response is not the solution.

I'll check why history_start_last_seq is NULL, this might be the problem.

alexej_d’s picture

@jeqq My debugging session showed that SocketClient's request method does not return the whole response body in case it is bigger than a certain amount of bytes. After that for example the getChanges method of CouchDBClient get NULL for $request->body. That's why the replication fails silently and history_start_last_seq is set to NULL. But history_start_last_seq is not the main problem here I think.

dpolant’s picture

Has anyone looked into this further? I am encountering this error on "large" replications as well. I'm seeing the symptoms posted in this ticket: https://www.drupal.org/node/2721709 but I'm posting here because it looks like this ticket will be used to track a solution.

My big concern is that this issue blocks Deploy suite from scaling. Until some kind of replications filters are working, sites are going to need to sync all of their content every time they create a new workspace, so fixing this ticket seems vital if it is indeed stopping large replications.

I'd be happy to work on a solution if someone can point me in the right direction.

jeqq’s picture

Assigned: Unassigned » jeqq
dpolant’s picture

Assigned: jeqq » Unassigned

I can confirm that the content-length header solves the problem. The couchdb changes standard specifies chunked encoding, but I couldn't get a Drupal response to implement this properly.

Setting content-length did the trick. I'll provide the patch tomorrow morning.

dpolant’s picture

Here is a patch that adds content-length header.

Status: Needs review » Needs work

The last submitted patch, 9: 2658912--set-content-length.patch, failed testing.

The last submitted patch, 9: 2658912--set-content-length.patch, failed testing.

The last submitted patch, 9: 2658912--set-content-length.patch, failed testing.

dpolant’s picture

Looks like HEAD doesn't play nice with content-length header. Re-rolling ..

  • jeqq committed c13f3d3 on 8.x-1.x authored by dpolant
    Issue #2770483 by dpolant: ChangesResource's get response is missing a "...
jeqq’s picture

Status: Needs review » Closed (fixed)

Thank you @dpolant!