Whilst trying to get a better understanding of how some of the Migrate Plus components work, I created some basic unit tests for the File and Http data fetchers and the Json data parser, which I thought might be of use.

I've created a patch against the 8.x-3.x branch and attached it to this issue.

The patch only adds test related files (PHPUnit/test classes), it does not alter an existing code. I used the testing docs at https://www.drupal.org/docs/8/phpunit for reference.

Running the migrate_plus tests via the 'core/scripts/run-scripts.sh' results in the following output, when running the tests with this patch applied:

Drupal test run
---------------

Tests to be run:
  - Drupal\Tests\migrate_plus\Kernel\MigrationConfigEntityTest
  - Drupal\Tests\migrate_plus\Kernel\MigrationGroupTest
  - Drupal\Tests\migrate_plus\Unit\data_fetcher\FileTest
  - Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest
  - Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest

Test run started:
  Thursday, January 19, 2017 - 15:21

Test summary
------------

Drupal\Tests\migrate_plus\Kernel\MigrationConfigEntityTest     1 passes                                      
Drupal\Tests\migrate_plus\Kernel\MigrationGroupTest            2 passes                                      
Drupal\Tests\migrate_plus\Unit\data_fetcher\FileTest           3 passes                                      
Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest           4 passes                                      
Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest            9 passes                                      

Test run duration: 1 min 27 sec

Detailed test results
---------------------


---- Drupal\Tests\migrate_plus\Kernel\MigrationConfigEntityTest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      MigrationConfigEn   28 Drupal\Tests\migrate_plus\Kernel\MigrationConfigEntityTest->testCacheInvalidation()  
    


---- Drupal\Tests\migrate_plus\Kernel\MigrationGroupTest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      MigrationGroupTes   22 Drupal\Tests\migrate_plus\Kernel\MigrationGroupTest->testConfigurationMerge()        
    
Pass      Other      MigrationGroupTes   90 Drupal\Tests\migrate_plus\Kernel\MigrationGroupTest->testDelete()                    
    


---- Drupal\Tests\migrate_plus\Unit\data_fetcher\FileTest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      FileTest.php       101 Drupal\Tests\migrate_plus\Unit\data_fetcher\FileTest->testFetchFile()                
    
Pass      Other      FileTest.php       130 Drupal\Tests\migrate_plus\Unit\data_fetcher\FileTest->testFetchMultipleFiles()       
    
Pass      Other      FileTest.php       185 Drupal\Tests\migrate_plus\Unit\data_fetcher\FileTest->testFetchUnreadableFile()      
    


---- Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      HttpTest.php       102 Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest->testFetchHttpWithAuth()        
    
Pass      Other      HttpTest.php       122 Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest->testFetchHttpNoAuth()          
    
Pass      Other      HttpTest.php       142 Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest->testFetchHttpAuthFailure()     
    
Pass      Other      HttpTest.php       156 Drupal\Tests\migrate_plus\Unit\data_fetcher\HttpTest->testFetchHttp500Error()        
    


---- Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest ----


Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Pass      Other      JsonTest.php       161 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testRequiredParserConfig()      
    
Pass      Other      JsonTest.php       193 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testParseValidJson()            
    
Pass      Other      JsonTest.php       226 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testParseInvalidJson()          
    
Pass      Other      JsonTest.php       246 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testParseMultipleSources()      
    
Pass      Other      JsonTest.php       276 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testVerifyAllSourceRecordsAreRea
    
Pass      Other      JsonTest.php       309 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testIntegerItemSelector()       
    
Pass      Other      JsonTest.php       350 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testInvalidIntegerItemSelector()
    
Pass      Other      JsonTest.php       373 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testXpathLikeItemSelector()     
    
Pass      Other      JsonTest.php       411 Drupal\Tests\migrate_plus\Unit\data_parser\JsonTest->testInvalidXpathLikeItemSelector

(I had to tweak the run-scripts.sh to fully show the 'Function' column value).

CommentFileSizeAuthor
#7 interdiff_4-6.txt5.93 KBheddn
#6 2844983-6.patch5.18 KBheddn
#4 migrate_plus-initial-file-data-fetcher-tests-2844983-4.patch6.91 KBalex_sansom
#2 migrate_plus-fetcher_parser_tests-2844983-1.patch30.11 KBalex_sansom
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alex_sansom created an issue. See original summary.

alex_sansom’s picture

alex_sansom CreditAttribution: alex_sansom commented
FileSize
migrate_plus-fetcher_parser_tests-2844983-1.patch30.11 KB

Attaching .patch file.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Murphysboro, IL, USA
CreditAttribution: mikeryan as a volunteer and at Virtuoso Performance commented
Title: Tests for data fetchers (file/http) and data parser (Json) » Tests for file data fetchers
Status: Needs review » Needs work
Related issues: +#2710309: Add tests for Json parser, +#2710311: Add tests for Http fetcher

Sorry for the slow response, finally catching up on the queue now.

There were already issues for adding tests for HTTP and JSON, could you break those out of this patch and put the pieces in the appropriate places? We can do the file fetcher tests here, since there wasn't an existing issue for that.

#2710309: Add tests for Json parser
#2710311: Add tests for Http fetcher

Thanks!

alex_sansom’s picture

alex_sansom CreditAttribution: alex_sansom as a volunteer commented
Version: 8.x-3.x-dev » 8.x-4.x-dev
Status: Needs work » Needs review
FileSize
migrate_plus-initial-file-data-fetcher-tests-2844983-4.patch6.91 KB

Split the file data fetcher tests out into its own .patch file (attached).

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented
Status: Needs review » Needs work

Some small stuff. Thanks for your work on this.

  1. +++ b/tests/src/Unit/data_fetcher/FileTest.php
@@ -0,0 +1,209 @@
+   * @expectedException \PHPUnit_Framework_Error_Warning

    Let's do a try/catch and look for a specific error message. 

  2. +++ b/tests/src/Unit/data_fetcher/TestFile.php
@@ -0,0 +1,39 @@
+class TestFile extends File {

    I think this can be mocked and it should return the VFS file stream.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented
Status: Needs work » Reviewed & tested by the community
FileSize
2844983-6.patch5.18 KB

Made some (hopefully good) changes to the tests.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented
FileSize
interdiff_4-6.txt5.93 KB

And here's the missing interdiff.

  • heddn committed d40f1b0 on 8.x-4.x 
    Issue #2844983 by alex_sansom, heddn: Tests for file data fetchers
heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented
Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.