I set up a simple service to receive a binary file. If I send a file of size 10MB, memory_get_usage() reports 115601572; 20MB, 225241104; 30MB, 334880960; 40MB, 444524664, etc. (If I send a 10MB file again, the memory usage will go back down to 115601572.) Every 10MB of base64 file size increase results in ~100MB PHP memory usage increase!
Drupal's XML-RPC facility handles base64 encoding/decoding automatically, so I would expect that maybe the data gets duplicated in memory (plus 1/3x, since base64 requires 4 bytes per 3 bytes binary.) But 10X memory consumption seems like a bug.
Here's the receiver code:
function xml_file_xfer_xmlrpc() {
return array (
array (
'xml_file_xfer.upload',
'xml_file_xfer_upload',
array('string','base64','string'),
t('Upload file via XML-RPC')
)
);
}
function xml_file_xfer_upload($data, $filename) {
return file_save_data($data, $filename);
}
For a client I used a Python script (should be readable to non-Python users):
import xmlrpclib
server = xmlrpclib.Server("http://localhost/sandbox_d6/xmlrpc.php")
testFile = open("testfile", "rb")
testFileData = xmlrpclib.Binary(testFile.read())
print server.xml_file_xfer.upload(testFileData,"testfile")
I have verified that the file is received correctly and not corrupted in any way.
OS: Ubuntu Linux 8.04
PHP version: 5.2.4-2ubuntu5.2
Web server: Apache/2.2.8 (Ubuntu) PHP/5.2.4-2ubuntu5.2 with Suhosin-Patch
Drupal version: 6.3
I also tested with a CentOS Linux system as the server and got the same results.
My plan now is to set up a debugger and step through the XML-RPC base64-argument-handling code...
| Comment | File | Size | Author |
|---|---|---|---|
| #1 | mem_deltas_trace.txt | 138.21 KB | threexk |
Comments
Comment #1
threexk commentedTraced Drupal's PHP memory usage using xdebug. There are many points where the memory usage increases significantly and several are probably unnecessary. Large memory increases occur between each of these pairs of function calls:
Notes:
- This is for a 10MB file transfer (~14MB base64-encoded)
- The +number is a delta in bytes from the previous reading. It is not saying that the call listed on that line caused the increase; it is saying something before that call caused it.
- Several of these memory increases have corresponding decreases (frees). But several don't, which is the problem.
- This is running through the services module, but there is no major memory difference between just using straight XML-RPC.
I was able to cut the 28MB increase down to 14MB by using PHP string functions instead of preg_replace(). This is proof that the memory handling can be improved.
Comment #2
dpearcefl commentedDoes this issue exist in current D6?
Comment #3
dpearcefl commented