Distributions not spawning properly

This suggests the automatic installer failed. Greg is working on scouring the logs to see why.

I believe this is an issue with Lightning. Here is the error message I got during install. The new system should allow the end user to access the PHP error logs for their instance so that issues on install are transparent.

The website encountered an unexpected error. Please try again later.
Drupal\Core\Entity\EntityStorageException: The "oauth2_token" entity type does not exist. in Drupal\Core\Entity\Sql\SqlContentEntityStorage->save() (line 805 of core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

Drupal\Core\Entity\Sql\SqlContentEntityStorage->save(Object) (Line: 377)
Drupal\Core\Entity\Entity->save() (Line: 293)
Drupal\Core\Installer\Form\SiteConfigureForm->submitForm(Array, Object)
call_user_func_array(Array, Array) (Line: 111)
Drupal\Core\Form\FormSubmitter->executeSubmitHandlers(Array, Object) (Line: 51)
Drupal\Core\Form\FormSubmitter->doSubmitForm(Array, Object) (Line: 585)
Drupal\Core\Form\FormBuilder->processForm('install_configure_form', Array, Object) (Line: 314)
Drupal\Core\Form\FormBuilder->buildForm('install_configure_form', Object) (Line: 905)
install_get_form('Drupal\Core\Installer\Form\SiteConfigureForm', Array) (Line: 599)
install_run_task(Array, Array) (Line: 555)
install_run_tasks(Array) (Line: 117)
install_drupal(Object) (Line: 44)

Unfortunately, any time the install fails, you get the same behavior. We'll sort this one out!

Just to further confirm - This is also happening with the Sector distribution.

also affects the media_dev distribution and others I tested.

Drupal 7 distros also have problem. No DB info. They even lose their CSS formating

Yes D7 distros affected as well and I confirm css lost as well

Must be a file permissions issue with the files folder and maybe tmp as well

One of the main challenges I see here is that projects are being cloned and rsynced into the appropriate directory. This is no longer needed with Composer (at least for Drupal 8).

I'm working on a patch that should address this.

Getting there...

In a simple test to spawn password policy for D8 the following is received.

Could not parse version constraint 8.x-3.0-alpha4: Invalid version string "8.x-3.0-alpha4"

Composer only wants the "3.0-alpha4" and the major version needs stripped.

New patch

SimplyTest has build in recursive logic to look for dependencies. This is causing some issues. The following patch needs to be executed to as a test to get some details on the build.

S_CORE variable is not set so lets make one.

OK, this is getting closer.

Message:

[InvalidArgumentException]
Could not find package drupal/password_policy.

Did you mean this?
drupal/password_policy

This usually means that it's not running in core's directory.

I am out of time for tonight, so here is a patch that restores this back to the way it was for the evening. I expect to get this working tomorrow.

Just FYI, we're still actively developing the media_dev distribution 7.x branches

I don't think composer is going to help with our development.

Nothing wrong with using rsync
rsync will still be used long after I retire and my grandchildren will probably use it to combat against the robots and free themselves from the matrix.

the wetkit distribution has the same issue, no css, and no database configuration, everything is empty in the database configuration page.

@joseph.olstad - thanks for the information.

I'm not against Rsync, we just need to be using Composer for loading projects with Drupal 8 as it's better supported.

Once I get this in place, I will definitely be moving onto Drupal 7. Hang tight as we're working through this.

I misunderstood the order of operations. Projects were being added *before* core. Hence the message.

The following attempts to fix that.

The following patch has a much cleaner composer integration.

I have attempted to get Lightning working without any luck yet (but the patch shows some of the progress). I also fixed the non-interactive install issue.

I still need to test out additional projects and patches for 8 and then I will jump back to Drupal 7.

now distro profiles aren't loaded at all for 7.x, looks like the standard profile getting started by default instead of the distro profile , no way to log in either. Installation steps appear to have been skipped, goes directly to the blue login page.

Tried again after commit:
#2944232-51: Distributions not spawning properly
no change.

[D7] This worked right when Patrickd handed the keys over to nerdstein. What happened ? What changes caused the regressions? Can they be rolled back ?

Synopsis of the current issue: simplytest.me will spin up the distribution however it will not provide a login password ***EDIT*** lucky guess, admin/admin user/pass combo works ***EDIT*** so unable to log into the system once it's set up. Also, all the install forms seem to be skipped altogether, this might be intentional but just making a note of this change as it seems related.
Also, for the wetkit distribution it can be installed in either English or French, this step is skipped.
same for the media_dev distribution, the installation language option is skipped and the default is chosen.

Also, for the wetkit distro, during installation there are steps to create default content (uses the migrate module), this is skipped, no demo content is created. I imagine that the media_dev distro demo content will not be available either.

As the installation of Drupal is now automated, there's no option yet to make installation choices.

There's a feature request to add that feature back here: https://www.drupal.org/project/simplytest/issues/2948370. Contributions are, of course, encouraged!

The bug with the admin password not getting prefilled is in a couple issues, but it seems we don't yet have a dedicated issue for it.

re: #55 automated yet incorrectly automated, it skips default installation steps, things like demo content are not created that should be created when installing the wetkit distribution. This is a default option in the installation profile but is now skipped.

Are there any special arguments that need to be passed to Drush to trigger the choices?

most definately there are special arguments for drush. These drush instructions were removed long ago as the intended method of installation is with the web GUI.

drush si wetkit wetkit_theme_selection_form.theme=wetkit_bootstrap install_configure_form.demo_content=TRUE --sites-subdir=default --db-url=mysql://root:@127.0.0.1:3306/wetkit_db --account-name=admin --account-pass=WetKit@2018 --site-mail=admin@example.com --site-name='Web Experience Toolkit' --yes

would be much more convenient for testing to use the GUI don't you think? The GUI allows choosing options, and these options need to work and be tested right?

Maybe rename simplytest.me to simplyunusable.me?

Makes sense. Thanks. As I just said, we're adding the ability to do both automatic and manual installs. Feel free to contribute to the manual install issue.

The todos on that one are something like:

1. Add a checkbox to allow for manual installation.
2. Read the checkbox during build.sh.
3. Skip Drush si when the checkbox is set.
4. Prefill the credentials in the installer.

While I understand that it's frustrating that the distro is broken, we're working on it as we can.

~G

Thanks for confirming the todo list, sounds like a good plan, I approve.

Thanks Joseph,

I'll be working on this in the evenings over the course of the weekend, and it will be my first push to the website side of the project. So, I'll be sure to update this thread when it's ready for testing!

~G

Hi @Greg Boggs , how would I contribute to the manual install issue?

couldn't you just roll back to when it was working? patrickd's last commit.

https://cgit.drupalcode.org/simplytest/commit/?id=d51b15bb824b9d86019f0d...

If I was contributing to this, I'd rollback to PatrickD when everything was still working.

The University of Norway in Oslo is working on media 4.x , a lot of new code, it'd sure be nice to be able to spin up our media_dev v4.x-dev distribution on simplytest.me to test out various changes , would be so nice.

I should have done this months ago, but I've just removed all references and links from media_dev to simplytest.me , as these are just confusing people getting them into a broken environment.

Until this is fixed, Simplytest.me is limited to testing core and a few modules at a time, requiring a lot of manual setup. Or partially configured distributions.

Should someone start a 'Go fund me' campaign to get this fixed?

Joseph, sorry for the delays with all of this. Greg has been blocked and my priorities were getting out the latest release I just completed. The concept of a manual install feature is my next highest priority and I expect to work on it this week.

I just want to note that there is a substantial amount of variation that exists across distributions and between major versions. If this is a blocker for the media work, I'm happy to collaborate with you to share provisioning logs and accept patches to the d.o code base. The same offer has been extended for the OOTB initiative.

Please hang tight a bit longer while I get this moving in the coming days.

Ok, so if you can PM me with info about the provisioning logs , that might be helpful.

Reason why I followed up, two people raised issues in the media_dev queue since yesterday mentioning issues with media_dev not working in simplytest.me

how can the community help?

@nerdstein - Thank you for your work on this. I'm looking forward to getting back to testing the https://www.drupal.org/project/media_dev distro with simplytest.me. It's a good way to keep updated on all the changes happening with the D7 media module ecosystem.

The media_dev distribution is loading with a bypass install. I'm still sorting out how to get over the usability issue of populating database credentials in the installer.

The community and drupal association needs to step up here. simplytest.me is the face of Drupal , whether it is 7 or 8, we used to look awesome because of simplytest.me.

I think these changes were made for the sake of speedy installation, however now it's not installing correctly. I'd choose slow and working over fast and broken 10 times out of 10.

Hi Joseph, thanks for letting us know you're experiencing issues. The server seems to be hit with some significant load right now. We're investigating it as I'm typing this and, overall, we haven't been seeing as many issues as we had. But, things do come up with these systems.

We also just finished a development server so we could solicit more contributions and leave production for stable releases only. We also know how to set up additional servers and can look into that to boost the resourcing capabilities for the production system.

Nerdstein, do you have any documentation of the technical details of transfer of this project from PatrickD over to you?

I'd like to fork simplytest.me and get distributions working again, I've got my own servers and I own the w3bfactory.com domain so I'd just make something like simplytest.w3bfactory.com

I was thinking if I forked this project to the working state that PatrickD had it as then it'd just work. I'm not sure what happened during the takeover other than bypassing the installation profile and probably version changes of drush.

It'd be really nice to have something up and running quick.
It's been 7 months or so now and I'm starting to get impatient.

Hi Joseph, are you experiencing new issues?

The performance issues in my last comment have been addressed. There are obviously ongoing improvements we need to fix and I'm hoping to prioritize that in Q4 of this year.

If you wish to fork the project, the code is on the 7.x-1.x branch. When the project was turned over to me, there was not a clear way to provision fresh infrastructure. The cleanest way would be for me to copy several system images, but there is a lot of sensitive data that I would not be comfortable doing so.

In my previous comment, I mentioned the simplytest development server. Please let me know if you want to use this. I welcome the help and any effort you'd provide. I can refresh these systems today if you are anxious to get started -- please send me an email from the contact form so I can get the information I need to get you access.

All I ask is for you to be clear on what issues you are encountering and/or tackling and give me a clear hand-off of steps to make the production changes. I feel like this would be substantially less work for you than making a fork.

Let me know and thanks again for your interest.

I'm reviewing simplytest.me, could be that it's working better now than a while back. I was able to spin up an environment it looked like it had installed ok, I will do some more reviews, I am currently cutting some new releases , testing them.

I'm evaluating simplytest.me

I think I may have found out what is going on, the php memory limit is pretty low, 128 MB

not sure if this is explains what I am observing. Running more tests. Ran more tests.

My setup works fine with 512 MB memory limit, I tested both PHP 5.6.x and PHP 7.0.31-1

I could drop the memory limit to 128 MB , this would probably reveal the problem.

Can you please increase the php memory limit and maybe the post max and timeout limits

maybe try 192 MB as a memory limit for php
thread limit should be boosted as well.

PHP memory limit of 128 MB is too low for a distribution, maybe not for core by it'self , but a distro with a lot of modules needs more than 128MB , certainly when running i18n and other modules like entity_translation and file_entity.

Also, check your timeout limits as well, might want to increase those slightly. but I think the memory limit is the first thinig to try increasing.

basically, without more memory added, the ckeditor wyswiyg won't work properly, maybe due to the plugins being used. Works flawlessly on my server with more memory. Might be also exacerbatedby default timeout limits in php.ini

Simplytest.me status report findings on simplytest.me with media_dev-7.x-4.0-beta8

        Info
	Drupal	7.60-dev Info
	Install profile	Media 4.x Development (media_dev-7.x-4.0-beta8+0-dev) OK
	
	Database system	MySQL, MariaDB, or equivalent OK
	Database system version	5.5.46-0ubuntu0.14.04.2 OK
	Database updates	Up to date

	File system	Writable (public download method) OK
	GD library PNG support	2.2.5 OK
	GD library rotate and desaturate effects	2.2.5 OK
	Media token version	4.0 OK
	Mobile Detect	Found Loaded library from profiles/media_dev/libraries/Mobile_Detect/Mobile_Detect.php
	Navbar: Backbone library	1.1.0 (minified)
	Navbar: Underscore library	1.5.2 (minified)
	PHP	7.0.27-1+ubuntu14.04.1+deb.sury.org+1 (more information)
	PHP extensions	Enabled
	PHP memory limit	128M
	PHP register globals	Disabled
	Plupload library	1.5.8
	Unicode library	PHP Mbstring Extension
	Update notifications	Enabled
	Upload progress	Not enabled Your server is capable of displaying file upload progress, but does not have the required libraries. It is recommended to install the PECL uploadprogress library (preferred) or to install APC.
	Web server	Apache/2.4.7 (Ubuntu)

I'd suggest two things, upgrade from Ubuntu 14.04.2 to Ubuntu 18.04.1

My server is running Ubuntu 18.04.1 , the upgrade from 16.04.5 LTS went amazingly well, although I had a warm spare available with a copy of the old version just incase the upgrade failed, there were a few minor snags during the upgrade for me, but nothing major, I chose to keep most of my original conf files , the upgrade process prompts you for options.

The PHP version you're using is different from the one I am using, I am using 7.0.31, yours is older.
My memory limit is well above 128 M, yours is at 128

could be some other web server config affecting this but ya, the same distribution works flawlessly as expected on my server with more memory, but on simplytest.me, wysiwyg is not working correctly possibly due to an installation issue caused by memory constraints or a runtime issue caused by memory constraints.

RAM is cheap these days, I've got 16 GB on my servers, thinking of upgrading to 32 GB, or more, soon.
192 MB outta do it for php thread limit, change all the definitions of 128 M in your default php.ini files to 192 M.
upgrading to Ubuntu 18.04 LTS server should help you as well.

Cool thanks for the sleuthing. I'll try bumping it up and seeing if it helps and will do so this weekend. Thanks again

the wetkit distro recomments 192 M as a minimum
media_dev would be similar.

probably a pretty safe number for most distros

@nerdstein, please let me know when you increase the memory limit so that I can run a few quick tests myself.

Memory limit has been bumped. Please let me know if this helps.

ok now I'm confused, the memory change didn't seem to make a difference.

However I did find a clue.
This file moono/skin.js is not accessible
https://dmcn1.ply.st/profiles/media_dev/libraries/ckeditor/skins/moono/s...

yet the the ckeditor one is available.
https://dmcn1.ply.st/profiles/media_dev/libraries/ckeditor/ckeditor.js

However I looked at an extracted copy of the media_dev distribution version 4.0-beta8 and the skin.js IS in the filesystem
I am perplexed why it's not showing up on simplytest.me

can't explain it because I can open other js files that I expect to be there
and when I install media_dev locally, that file IS there.

temporary example:
https://media.olstad.biz/profiles/media_dev/libraries/ckeditor/skins/moo...
see, the above file is there.

Can you @nerdstein install media_dev 7.x-4.0-beta8 on simplytest.me and then go into the filesystem and check for that file?

I am guessing the file is there, but there could be some network filtering going on that is removing it

I cannot explain this.

However if I switch to the cdn can avoid this issue but that's not the point.

I also noticed that this file doesn't come through either
https://dmcn1.ply.st/profiles/media_dev/libraries/ckeditor/skins/moono/e...

In both cases the error is a 404 error
The same file exists on my installation
https://media.olstad.biz/profiles/media_dev/libraries/ckeditor/skins/moo...
but not on https://dmcn1.ply.st/profiles/media_dev/libraries/ckeditor/skins/moono/e...

Is there a site directory size limit on simplytest.me ?

maybe these files are being truncated by an artificial limit on the virtual filesystem. The media_dev distribution extracted is 111 Megabytes
there is an example mp3 file in it, some small example images, lots of modules, some of the projects are git clones so that probably takes up extra space, I might look into shrinking that by taking dev builds instead of git clones of some modules.

However, maybe just increasing the site folder limit on simplytest.me sites will help?

I am guessing this might explain things, missing / truncated files.

see my above comment, I think there's a simplytest.me site folder size limit causing the site installation to be truncated.

I will check the directory size limits.

Is this JS library in the downloaded code base or pulled from an external library? If pulled, can you point out how it's specified in your code base so I can check to see if SimplyTest is pulling it properly?

I didn't see any traces of file system limitations in an initial pass through the configs.

I've debugged this a bit further on the `dmcn1` instance. The `/home/dmcn1/www/profiles/media_dev/libraries/ckeditor/skins/moono` exists but is empty. For whatever reason, the `/home/dmcn1/www/profiles/media_dev/libraries/ckeditor/skins/moono-lisa` is not empty and seems to be loading the library and it's files properly.

Are you fully confident there is not a bug in how this library is loaded specific to the distribution? If so, please show me how this gets pulled into the distribution so I can see how it is loading this during spawning.

that library gets pulled in during build, it's included in the .tar.gz and .zip for the distribution.
It's just a file, it exists in the archive , and it exists in my copy of my installation of media_dev.
see here:
https://media.olstad.biz/profiles/media_dev/libraries/ckeditor/skins/moo...
AND
https://media.olstad.biz/profiles/media_dev/libraries/ckeditor/skins/moo...

these two files are missing on the simplytest.me instance as linked above.

this is a copy of the tar.gz from the media_dev release, there's nothing magic going on, it's just a file and doesn't move around during installation.
see the original archive here:
https://ftp.drupal.org/files/projects/media_dev-7.x-4.0-beta8-core.tar.gz

are you using the .tar.gz ? or the .zip?
I haven't tested the .zip, I use the .tar.gz

SimplyTest is leveraging Drush (D7) and Composer (D8). I do not believe either of those tools leverage tar or zip files from releases - it's my understanding they pull directly from branches/tags from the corresponding Git repo based on what the user selects.

You mentioned a "build" step. SimplyTest already performs build steps for other distributions, e.g. processing make files. Can you elaborate on this? It would be helpful to know how it is compiling these libraries into the zip/tar files.

Or, we can look to outright pull these zip/tar files if that is a better artifact to pull.

Maybe your /tmp folder doesn't have enough space

as for doing a build and a make, that is already done for you in the tar.gz , not sure why you'd want to waste all that bandwidth and cpu power to do a build, maybe that is the composer way in D8 but for D7.x you should just download the archive and use it, everything is there and cooked up correctly.

to build the media_dev distribution , clone the media_dev repo
then with Drush version 8.x you will run this command:
drush make --prepare-install build-media_dev.make SOME_OUTPUT_FOLDER_NAME

However I think it would be quicker, easier and more reliable to use the actual release build.
is it not?

Drupal.org does the make when publishing a release, so it's all packaged up for us by Drupal.org , all Drupal.org wants is the make files correctly configured and if there is a problem, Drupal.org will not publish a release of a broken make file.
If there's any errors during the build, Drupal.org aborts the release and you have to fix the issue before it will publish a release.

so if I push a change to the media_dev distribution dev branch, drupal.org does a make, but it won't upload a broken .zip or .tar.gz
if there's errors it aborts the archive and does not publish.

I think I found the issue. All CKEditor libraries seem to be failing to download.

See the following from the log, which runs the `drush make` command on the build-media_dev.make file:

bootstrap patched with php7-countable-2999011-9.patch. [ok]
Generated PATCHES.txt file for bootstrap [ok]
startupgrowth_lite-7.x-1.0 downloaded. [ok]
ckeditor downloaded from http://download.cksource.com/CKEditor/CKEditor/CKEditor%204.9.2/ckeditor.... [ok]
DOWNLOAD_ERROR: Unable to download lineutils from http://download.ckeditor.com/lineutils/releases/lineutils_4.9.2.zip. -- build forced [warning]
mobile_detect downloaded from https://github.com/serbanghita/Mobile-Detect/archive/master.zip. [ok]
DOWNLOAD_ERROR: Unable to download widget from http://download.ckeditor.com/widget/releases/widget_4.9.2.zip. -- build forced [warning]
DOWNLOAD_ERROR: Unable to download widgetselection from http://download.ckeditor.com/widgetselection/releases/widgetselection_4..... -- [warning]
build forced
DOWNLOAD_ERROR: Unable to download moono from http://download.ckeditor.com/moono/releases/moono_4.9.2.zip. -- build forced [warning]
plupload downloaded from https://github.com/moxiecode/plupload/archive/v1.5.8.zip. [ok]
plupload patched with plupload-1_5_8-rm_examples-text-1903850-26.patch. [ok]
Generated PATCHES.txt file for plupload [ok]
mediaelement downloaded from https://github.com/johndyer/mediaelement/archive/master.zip. [ok]
underscore downloaded from https://github.com/jashkenas/underscore/archive/1.5.2.zip. [ok]
modernizr downloaded from https://github.com/Modernizr/Modernizr/archive/v2.7.1.zip. [ok]
backbone downloaded from https://github.com/jashkenas/backbone/archive/1.1.0.zip. [ok]
backbone patched with backbone_source_map_distro-2315315-05.patch. [ok]
Generated PATCHES.txt file for backbone [ok]
flexslider downloaded from https://github.com/woothemes/FlexSlider/archive/2.7.1.zip.

I debugged Drush's 8.x download functionality, found that it is using wget, ran the command on the simplytest server and I can see that the ckeditor cert authority is not working:

root@worker:/home/dmcn1# wget --timeout=30 -O /tmp/test-file http://download.ckeditor.com/moono/releases/moono_4.9.2.zip
--2018-09-24 21:21:38-- http://download.ckeditor.com/moono/releases/moono_4.9.2.zip
Resolving download.ckeditor.com (download.ckeditor.com)... 173.199.183.188
Connecting to download.ckeditor.com (download.ckeditor.com)|173.199.183.188|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://ckeditor.com/cke4/sites/default/files/moono/releases/moono_4.9.2... [following]
--2018-09-24 21:21:38-- https://ckeditor.com/cke4/sites/default/files/moono/releases/moono_4.9.2...
Resolving ckeditor.com (ckeditor.com)... 54.230.95.232, 54.230.95.161, 54.230.95.79, ...
Connecting to ckeditor.com (ckeditor.com)|54.230.95.232|:443... connected.
ERROR: cannot verify ckeditor.com's certificate, issued by ‘/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Domain Validation Secure Server CA’:
Unable to locally verify the issuer's authority.
To connect to ckeditor.com insecurely, use `--no-check-certificate'.

I updated `/etc/wgetrc` to turn off certificate checks, I was able to download the library without getting an error.

Can you please retest?

Ya wget --no-check-certificate

Installed media_dev 7.x-4.something on simplytest and it successfully downloaded the ckeditor libraries and whatnot.

--> https://dmcs7.ply.st/profiles/media_dev/libraries/ckeditor/skins/moono/e...

hi @daggerhart, I tested that machine you just created, it works great.

to make a new one:
https://simplytest.me media_dev 4.0beta8

Thanks  @nerdstein, I think we finally nailed it, everything is working as expected!

@nerdstein, I think you can probably remove that message warning about an issue with certain distributions.
now that --no-check-certificate is used, should be in good shape.

Also, @Nerdstein, thanks for your work on this, great to see this issue fixed!

@nerdstein, all was working fine last time I checked about a week or two ago , now I published another release and get this:

see:
"An error occurred while downloading dependencies."

@nerdstein, can you have a look please?
even the older releases that worked last week are now failing.
https://simplytest.me/project/media_dev/7.x-4.0-beta8
or
https://simplytest.me/project/media_dev/7.x-4.0-beta9

I'll take a look, thanks for letting me know

Hi Nerdstein, it seems to be working now, I think you must have fixed it.
I just installed media_dev 7.x-2.x dev release and it worked.
Thanks!

@nerdstein: Tested Thunder 8.2.31 some minutes ago and it doesn't work yet. Do you already fixed D8 distributions?

I had same problem with D8 distributions such as Varbase, Thunder and Open Social. None of them worked :(