Memory leak loading genomic data (GFF, Fasta) [#2668486]

Running through an initial installation of Tripal. I have made it to loading my own feature data, however, attempting to load a GFF3 says missing scaffolding, and when I load a fasta instead first, memory appears to be leaking. Please see the error below. I upgraded my PHP.ini to allow a 4GB memory_limit, which I think is too high, and it still maxes out. Thoughts?

Calling: tripal_feature_load_fasta(/opt/bitnami/apache2/htdocs/sites/default/files/Trinity.fasta, 14, mRNA, , , , , , , , Insert only, 1, 10, Unique name, 11)

NOTE: Loading of this GFF file is performed using a database transaction. *******this should be 'fasta'
If the load fails or is terminated prematurely then the entire set of
insertions/updates is rolled back and will not be found in the database

Step 1: finding sequences
PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 72 bytes) in /usr/share/php/drush/includes/context.inc on line 80

Comments

Comment #1

14 February 2016 at 03:42

jboykin created an issue. See original summary.

Comment #2

jboykin CreditAttribution: jboykin commented 14 February 2016 at 04:29

Comment #3

jboykin CreditAttribution: jboykin commented 14 February 2016 at 04:35

Category:

Support request

» Bug report

Comment #4

laceysanderson CreditAttribution: laceysanderson commented 25 February 2016 at 03:23

Hi, I'm sorry you're running into issues! Would it be possible for you to attach the fasta file you are having difficulties with so I can test this on my system?

Comment #5

spficklin CreditAttribution: spficklin commented 25 March 2016 at 06:37

Hi Jboykin,

How big are the sequences in your FASTA file? I tested loading a FASTA file for a whole genome and I can verify that the memory does increase as the loader reads in an individual sequence, but when it moves to the next sequence in the file the memory usage drops back to a reasonable level for the next one. So, I'm not able to identify a memory leak. Below is the output from my test (note the memory usage)

Step 1: finding sequences
Parsing Line (100.00%). Memory: 34,713,896 bytes.
Step 2: Importing sequences
Importing 1 of 8. Current feature: FvbUn.
Importing 2 of 8. Current feature: Fvb1.97,640 bytes.
Importing 3 of 8. Current feature: Fvb2.34,264 bytes.
Importing 4 of 8. Current feature: Fvb3.91,744 bytes.
Importing 5 of 8. Current feature: Fvb4.99,840 bytes.
Importing 6 of 8. Current feature: Fvb5.54,336 bytes.
Importing 7 of 8. Current feature: Fvb6.96,400 bytes.
Importing 8 of 8. Current feature: Fvb7.46,144 bytes.
Sequence complete: 101.22%. Memory: 58,406,240 bytes.
Done

Your memory limit is set to 2Gb in the php.ini setting and the loader is hitting that limit and causing the loader to fail. How large are the individual sequences in your FASTA file? Can you try increasing the amount of memory you allow for scripts in your php.ini file?

It is possible there may be some obscure memory leak but we'd need to run a test on a similar dataset to what you are using to test. If you can provide more details about the number of sequences, the average size and the size of the largest.... or as Lacey suggests, provide a copy of the file your loading.