Allow for large documents to be indexed

Hello @patchak thanks for the report.

That is an internal SemanticProxy error, not sure specifically what it means (besides the text was too long) :p

We can try submitting this to the SemanticProxy team at http://www.opencalais.com/forum/213 to see what they say. Are you able to reliably reproduce the error with a specific URL? Or will sometimes the same URL submit fine and other times fail?

Thanks again.

Great news.

Cleaning up.

I'm trying to do it with a document that has almost a megabyte worth of text; at a minimum can this submit up to the max & truncate the extra, and store the truncated nid's in a table, so I know which ones it failed on.

Better yet, this splits up the doc and merges the objects... at least I think that's what it's doing. As a bonus?, my editor is removing CRLF's and making them into LF's

Above patch only gets the last one; had to go ghetto... here's a patch that works, but is much slower. Oh and it's removing extra whitespace.

nope, this doesn't work either...

Modifying the first patch so it works would be ideal, but I don't know the $keywords object structure, or if that is the right way to do this. Also one would have to get the average of the relevance to do it correctly. If Calais didn't overwrite the data, then my second way might work.

This is what I'm going to use for the time being.

@miketown2

Thanks for this. I like the idea, however your last patch does not have all of the code though, it has something for the breakup, but not the individual submits. Can you try rerolling it so that I can see if it will apply cleanly?

I only submit the first part since merging the data was a pain, due to it being stored in an object.

Semantic proxy is no longer offered and has been removed from Open Calais.