Additional repositories are not installed as expected.

I don't see the search_api dependency at http://cgit.drupalcode.org/facets/tree/facets.info.yml. Is the dependency somewhere else?

#2651854: Only look for dependencies for releases with specific project node types added

    if (preg_match('/\/tests\/|\/Tests\//', $file->uri)) {
      // Remove modules in test directories.
      continue;
    }

So modules in test directories are no longer checked for dependencies.

The way to add dependencies for tests is to use test_dependencies in info files, using the same syntax as dependencies. https://www.drupal.org/node/542202#dependencies has this documented for D7, but it doesn't look like this has made it forward to D8.

… it doesn't look like this has made it forward to D8 at https://www.drupal.org/node/2000204.

Test dependencies should still work. Attached is a screenshot of the dependencies. It looks like the test dependencies has not been updated since 8.x-1.0-alpha3. I will take a look and see why.

Drupal core 7.x and 8.x have a bunch of garbage modules and dependencies for testing the module code. I added the code to ignore tests so that these would not show up.

The DrupalCI job running probably started faster than the dependencies being updated. I kicked off a new set of tests to confirm.

#2400705-24: Outdated version of dependencies are installed on the test-infrastructure has a few more details.

Currently DrupalCI job parameters, including dependencies, are built and sent off within a minute. Dependencies are calculated after dev release packaging, which is on an every-5-minute cron. So adding new dependencies like this have a good chance of hitting this.

This uncovered a new bug. The workaround is to remove the test_dependency on search_api:search_api_test_db and run the tests again 5 or 10 minutes after commit. search_api:search_api is sufficient to get the project.

Since dependency parsing of tests was removed, search_api:search_api_test_db doesn’t have results for recent releases, so it resolves to an older search_api release. This is added in addition to the current version from the search_api:search_api dependency.

Possible improvements:

• Have project_dependency_get_external_release_dependencies() or PiftCiJob::requestTest() dedupe on project so only one version (the more recent) of each project is attempted to be checked out.
• Reparse existing dependencies so the test modules are removed completely.

Looks like Facets is testing well now. Changing the category for the potential improvements in my last comment.

If I just parse the info.yml files in Drupal 8.0.1 I get

Title	Name
<"Cat" & 'Mouse'>	block_test_specialchars_theme
Test theme	test_theme
AAA Update test	aaa_update_test
Accept header based routing test	accept_header_routing_test
Action bulk form test	action_bulk_test
Action test	action_test
Actions	action
Activity Tracker	tracker
Aggregator	aggregator
Aggregator module tests	aggregator_test
Aggregator test views	aggregator_test_views
AJAX form test mock module	ajax_forms_test
AJAX Test	ajax_test
Automated Cron	automated_cron
Ban	ban
Bartik	bartik
Batch API test	batch_test
BBB Update test	bbb_update_test
Block	block
Block Content test views	block_content_test_views
Block test	block_test
Block test theme	block_test_theme
Block test views	block_test_views
Book	book
Book module tests	book_test
Breakpoint	breakpoint
Breakpoint test module	breakpoint_module_test
Breakpoint test theme	breakpoint_theme_test
Cache test	cache_test
CCC Update test	ccc_update_test
CKEditor	ckeditor
CKEditor test	ckeditor_test
Classy	classy
Color	color
Color test	color_test
Color test theme	color_test_theme
Comment	comment
Comment empty titles test	comment_empty_title_test
Comment test	comment_test
Comment test views	comment_test_views
Common Test	common_test
Common Test Cron Helper	common_test_cron_helper
Condition Test Support	condition_test
Config collection clash test module	config_collection_clash_install_test
Config install dependency test	config_install_dependency_test
Config other module config	config_other_module_config_test
ConfigTest integration	config_integration_test
Configuration entity static cache test	config_entity_static_cache_test
Configuration events test	config_collection_install_test
Configuration events test	config_events_test
Configuration import test	config_import_test
Configuration install fail test	config_install_fail_test
Configuration Manager	config
Configuration override integration test	config_override_integration_test
Configuration override test	config_override_test
Configuration schema test	config_schema_test
Configuration test	config_test
Configuration test languages	config_test_language
Configuration Translation	config_translation
Configuration Translation Test	config_translation_test
Configuration Translation Test Theme	config_translation_test_theme
Contact	contact
Contact test module	contact_test
Contact test storage	contact_storage_test
Contact test views	contact_test_views
Content negotiation test	conneg_test
Content Translation	content_translation
Content translation test views	content_translation_test_views
Content translation tests	content_translation_test
Contextual Links	contextual
Cron Queue test	cron_queue_test
Custom Block	block_content
Custom Block module tests	block_content_test
Custom Menu Links	menu_link_content
Database Logging	dblog
Database Test	database_test
Datetime	datetime
Datetime test	datetime_test
Dblog test views	dblog_test_views
Disable user toolbar	toolbar_disable_user_toolbar
Display variant tests	display_variant_test
Drupal system listing compatible test	drupal_system_listing_compatible_test
Early rendering controller test	early_rendering_controller_test
Early translation test	early_translation_test
Entity constraints test module	entity_test_constraints
Entity CRUD Hooks Test	entity_crud_hook_test
Entity CRUD test module	entity_test
Entity Operation Test	entity_test_operation
Entity Reference	entity_reference
Entity Reference Test	entity_reference_test
Entity reference test views	entity_reference_test_views
Entity schema test module	entity_schema_test
Entity serialization test support	entity_serialization_test
Entity test extra	entity_test_extra
Error service test	error_service_test
Error test	error_test
Field	field
Field API configuration tests	field_test_config
Field API Test	field_test
Field Plugins Test	field_plugins_test
Field test views	field_test_views
Field Third Party Settings Test	field_third_party_test
Field UI	field_ui
Field UI test	field_ui_test
File	file
File test	file_test
File test	file_module_test
File test views	file_test_views
Filter	filter
Filter test module	filter_test
Filter test plugin	filter_test_plugin
FormAPI Test	form_test
Forum	forum
Forum test views	forum_test_views
HAL	hal
Help	help
help_test	help_test
History	history
Hook menu tests	menu_test
Hook path tests	path_test
HTTP Basic Authentication	basic_auth
HttpKernel test	httpkernel_test
Image	image
Image test	image_module_test
Image test	image_test
Image test views	image_test_views
Inline Form Errors	inline_form_errors
Interface Translation	locale
Internal Dynamic Page Cache	dynamic_page_cache
Internal Page Cache	page_cache
KeyValue tests	keyvalue_test
Language	language
Language config overridetest	language_config_override_test
Language form elements test	language_elements_test
Language test	language_test
Link	link
Link generation test support	link_generation_test
Locale test	locale_test
Locale Test Development Release	locale_test_development_release
Locale test translate	locale_test_translate
Menu link content dynamic route	menu_link_content_dynamic_route
Menu UI	menu_ui
Migrate	migrate
Migrate Drupal	migrate_drupal
Migrate events test	migrate_events_test
Migrate module prepareRow tests	migrate_prepare_row_test
Migrate property overwrite test	migrate_overwrite_test
Migration template test	template_test
Minimal	minimal
Module autoload test	module_autoload_test
module handler test all1	module_handler_test_all1
module handler test all2	module_handler_test_all2
module handler test module	module_handler_test
module handler test module	module_handler_test_added
module handler test multiple	module_handler_test_multiple
module handler test multiple child	module_handler_test_multiple_child
module handler test no hook	module_handler_test_no_hook
Module installer config test	module_installer_config_test
Module name length test	invalid_module_name_over_the_maximum_allowed_character_length
Module required test	module_required_test
Module test	module_test
Node	node
Node configuration tests	node_test_config
Node module access automatic cacheability bubbling tests	node_access_test_auto_bubbling
Node module access tests	node_access_test
Node module access tests language	node_access_test_language
Node module empty access tests	node_access_test_empty
Node module exception tests	node_test_exception
Node module tests	node_test
Node test views	node_test_views
Nyan cat	nyan_cat
Options	options
Options config install test	options_config_install_test
Options test	options_test
Options test views	options_test_views
Outbound route/path processing	outbound_processing_test
Page Cache Form Test	page_cache_form_test
Pager Test	pager_test
ParamConverter test	paramconverter_test
Path	path
PHPUnit Test	phpunit_test
Plugin Test Support	plugin_test
Quick Edit	quickedit
Quick Edit test	quickedit_test
RDF	rdf
RDF module conflicting namespaces test	rdf_conflicting_namespaces
RDF module namespaces test	rdf_test_namespaces
Rendering #attached test	render_attached_test
Requirements 1 Test	requirements1_test
Requirements 2 Test	requirements2_test
Responsive Image	responsive_image
Responsive image test theme	responsive_image_test_module
REST test	rest_test
REST test views	rest_test_views
RESTful Web Services	rest
Router test	router_test
Search	search
Search Embedded Form	search_embedded_form
Serialization	serialization
Serialization test module	serialization_test
Service Provider test	service_provider_test
Session test	session_test
Seven	seven
Shortcut	shortcut
Simpletest test	simpletest_test
Stable	stable
Standard	standard
Stark	stark
Statistics	statistics
Statistics test attached theme	statistics_test_attached
Statistics test views	statistics_test_views
Syslog	syslog
System	system
System dependency test	system_dependencies_test
System incompatible core version dependencies test	system_incompatible_core_version_dependencies_test
System incompatible core version test	system_incompatible_core_version_test
System incompatible module version dependencies test	system_incompatible_module_version_dependencies_test
System incompatible module version test	system_incompatible_module_version_test
System mail failure test	system_mail_failure_test
System project namespace	system_project_namespace_test
System test	system_test
System test	system_module_test
Taxonomy	taxonomy
Taxonomy CRUD tests	taxonomy_crud
Taxonomy test views	taxonomy_test_views
Telephone	telephone
Test Dynamic Page Cache	dynamic_page_cache_test
Test Module	test_module
Test page	test_page_test
Test search entity langcode	search_langcode_test
Test Search Query Alter	search_query_alter
Test Search Type	search_extra_type
Test Stable	test_stable
Test theme for configuration clash detection	config_clash_test_theme
Test theme for Nyan Cat engine	test_theme_nyan_cat_engine
Test theme libraries-extend	test_theme_libraries_extend
Test theme libraries-override	test_theme_libraries_override_with_drupal_settings
Test theme libraries-override	test_theme_libraries_override_with_invalid_asset
Test theme with a too long name	test_theme_having_veery_long_name_which_is_too_long
Test Wild West	test_wild_west
Testing	simpletest
Testing	testing
Testing config import	testing_config_import
Testing config overrides	testing_config_overrides
Testing multilingual	testing_multilingual
Testing multilingual with English	testing_multilingual_with_english
Text	text
Text Editor	editor
Text Editor test	editor_test
Theme page test	theme_page_test
Theme region test	theme_region_test
Theme suggestions test	theme_suggestions_test
Theme test	theme_test
Theme test base theme	test_basetheme
Theme test subsubtheme	test_subsubtheme
Theme test subtheme	test_subtheme
Theme test with invalid base theme	test_invalid_basetheme
Theme test with invalid core version	test_invalid_core
Theme test with invalid theme engine	test_invalid_engine
Theme test with missing content region	test_invalid_region
Token test	token_test
Toolbar	toolbar
Toolbar module API tests	toolbar_test
Tour	tour
Tour module tests	tour_test
Tracker test views	tracker_test_views
Trusted hosts test module	trusted_hosts_test
Twig	twig
Twig Extension Test	twig_extension_test
Twig Loader Test	twig_loader_test
Twig namespace test: Module A	twig_namespace_a
Twig namespace test: Module A	twig_namespace_b
Twig registry loader test	test_theme_twig_registry_loader
Twig registry loader test subtheme	test_theme_twig_registry_loader_subtheme
Twig registry loader test theme	test_theme_twig_registry_loader_theme
Twig theme test	twig_theme_test
Update Manager	update
Update script test	update_script_test
Update test	update_test
Update test	update_test_2
Update test	update_test_3
Update test	update_test_1
Update test 0	update_test_0
Update test after	update_test_postupdate
Update test base theme	update_test_basetheme
Update test failing	update_test_failing
Update test schema	update_test_schema
Update test subtheme	update_test_subtheme
Update test with 7.x updates left in the codebase.	update_test_with_7x
Update test with an invalid hook_update_8000().	update_test_invalid_hook
Url_alter tests	url_alter_test
User	user
User access tests	user_access_test
User custom phpass params test	user_custom_phpass_params_test
User module form tests	user_form_test
User module hooks tests	user_hooks_test
User Test theme	user_test_theme
User test views	user_test_views
Views	views
Views Entity Test	views_entity_test
Views Test	views_test_data
Views Test Config	views_test_config
Views Test Formatter	views_test_formatter
Views Test Language	views_test_language
Views test theme	views_test_theme
Views UI	views_ui
Views UI Test	views_ui_test

Elimination of info.yml files that are under directories tests and Tests get rid of a lot of the modules that are just used for testing. It is pretty unlikely contributed projects will have info.yml files under src/Tests and it is probably a bad idea to be requesting a dependency on a module in a test directory.

There are no components or dependencies for Drupal 7.42 and Drupal 8.0.3. See attached screenshots. Any information on why these failed?

Not really. I spot checked another recent tagged release, and dependencies did get populated. I ran the job to rebuild the drupal project's dependencies and it completed without errors. I'll see if I can remember to look for this for the next core release so I can get the packaging logs.

#2667704: Testbot is trying to checkout CTools twice and failing uncovered that the most recent ctools release wasn't parsed either, https://www.drupal.org/node/2663662. That's also too far ago, we don't have packaging console logs from that far ago. We do need to see if there is a pattern here.

The Jenkins job to rebuild dependencies does rm -rf /var/tmp/project_dependency_sourcecode_directory/avatars, which fixes the problem of the shared repo being corrupted somehow.

Packaging uses the existing repo if it is there:

15:40:40 WD project_dependency: Failed to execute git command git --work-tree    [notice]
15:40:40 '/var/tmp/project_dependency_sourcecode_directory/avatars' --git-dir
15:40:40 '/var/tmp/project_dependency_sourcecode_directory/avatars/.git' fetch
15:40:40 2>&1; output=error: Could not read
15:40:40 44b6883cab09eb4b31b5d041cf4f1330b88a86c6 fatal: unresolved deltas
15:40:40 left after unpacking fatal: unpack-objects failed

I couldn't spot any permissions issues. ‘bender’ (the robot) is the only user that interacts with these repositories on jenkins1 (the server). The whole repo is owned by that user.

Something about how that fetch runs or the previous commands used in the repo confuses Git quite a bit.

A quick, effective fix would be to rm -rf the work tree and start over if git fetch fails.

I have not dug much into this part of Project Dependency but it seems overly complicated. Code checked out from Git is persisted I suppose for efficiency reasons and then updated. But this seems to be rather prone to permission problems or just corruption. So it is a natural thing to view with suspicion when there are occasional failures.

Normal flow is process a release from one project and then a release from a different project and so on. The flow is not processing multiple releases from a single project except in special circumstances. So I do not understand what is gained by checking out the entire project. Why not check out just the code that is needed? To be concrete, use

git clone --depth 1 --branch ..

This just retrieves the requested branch instead of the entire code base.

Some time ago I coded up the following as a fallback if the fetch fails (this has not been committed):

    if ($status) {
      // Attempt using a temp directory (fallback for permission problems).
      $temp = TRUE;
      $work_tree = "/tmp/$directory" . '_' . time();
      $esc_work_tree = escapeshellarg($work_tree);
      $git_dir = "$work_tree/.git";
      $esc_git_dir = escapeshellarg($git_dir);
      $command = "git clone --depth 1 --branch $tag $url $esc_work_tree 2>&1";
      exec($command, $output, $status);
      if ($status) {
        watchdog('project_dependency', 'Failed to execute git command %cmd; output=%output', array('%cmd' => $command, '%output' => implode(' ', $output)));
        return FALSE;
      }
    }

But it would greatly simplify the code to just call this and not persist the checked-out code. Am I missing something here?

That should work. I don't think maintaining the checkouts is necessary either. Must have been a premature optimization.

I uploaded a patch to simplify the git code so it checks out the branch and then removes the branch once the files are processed.

I created a new release with the changes. If there are any problems then you should be able to roll back the release.

This has been deployed and looks like it is working well.