7.38 core update takes down entire site

This seems to be a registry issue. Core update apparently rudely disabled the domain access module. BEFORE update - disable all domain access modules - - no need to uninstall - just dis-able them => then remove the settings.php settings.inc line at the bottom - THEN update CORE - then re-enable everything. - domain access first - and put the settings.php - include line to modules/domain/settings.inc'; BACK in - then re-enable the rest of the domain modules - then to rebuild the permissions if secure pages is enabled add batch and batch/* to secure pages or the rebuild will not work. See https://www.drupal.org/node/1358736 and https://www.ostraining.com/support-forum/drupal-support/domain-access-fa... as domain access presents itself this way when the module is not enabled and registered correctly. Documented elsewhere but not centralized to find - the best answer is always no answer.

What can I say . . . David . . . it is what it is, and it did what it did - twice - until on the second try I did what I outlined above, wasting my entire day largely to figure it out and in the process

I have had quite few strange things happen with the domain module family and secure pages. Still unpredictably somethings a re-direct loop still happens - apparently due to secure pages.

One time it redirected to another of Drupal 6 site theme and all (that I manage also on the same physical server as Virtual sites under Apache) for a sub-domain completely that was NOT related to it by name at all and for a moment I thought the site had been hacked. i had to disable that sub-domain and if I re-enable it - that behavior comes back.

With all the odd things Drupal 7 does - I have been consumed in these kinds of things, where Drupal 6 NEVER did this kind of stuff, so Drupal 8 - uh really . . .not going to to even try it.

Another time data from another database showed up in this domain - and took out the entire catalog section of Ubercart, - and this is on a production server - and when it was found - and the only way it was found was by editing 125 products - to move to a new catalog section that was created and on the LAST product it turned out the SKU and prices were gone and would not edit BACK in - and the only way to fix it was to delete that product node completely and take ONLY the text body to make a new product. The node could not even be CLONED

So . . . Drupal 7 has A LOT of code that just like Windows - which I tested for pre-release when I worked at Microsoft - presents itself in odd ways only given specific scenarios - and I am quite sure NO ONE is running a vanilla fresh install of Drupal 7 as a production environment; so trying to gauge that as measure to reproduce it is meaningless . . . it is what it is, and it did what it did - twice - until on the second try I did what I outlined above.

I have noticed A LOT of deflection and cover for Drupal issues as though the issue does not exist in supporting Drupal lately - which leaves Drupal sitting as basically as an unsupported "doacracy" (do it yourself democracy) as allegedly webchick (webchick.net) calls it - but some things and today MOST things a person cannot fix on their own when the module maintainers even do not have a clue

It is nice David that you have a Drupal 7 system with secure pages and domain access enabled and working - and I will tell you - 7.38 core uploaded and upgraded fine it appeared also on my system too until something else happened like I believe a cron run overnight or maybe a robot came by - and attempted to access sub-domains that were disabled, and i awoke to the rude realization the site was down.

But . . . that which it did is a Registry issue clearly . . . but there are things that even Windows STILL does to this day that can be reproduced as a "party trick"; that no one really every got to the heart of and still exists as special scenario errors today.

My post above shows links to two other oddities that pointed me to the resolution of the problem - which two years later Domain Access STILL does - so I have seen clearly no one knows what caused it and nor have those modules been updated to prevent it (like adding " batch " and " batch /" "to the secure pages base install or as an update) - but I have been working with Drupal since 2008, and I have 16 other Drupal 6 sites that did not do this - only the Drupal 7 site did so . . . it is what it is, and it did what it did - twice - until on the second try I did what I outlined above.

It is NOT a "warm and fuzzy" feeling however

David:

As to your comment at #5 - which was posted while I drafting my reply at #6, the domain module was not touched - and CORE was simply uploaded as it comes unpacked - which has ALWAYS worked quickly and easily with no problem.

There are some modules - due their coding or what have you - that will not run from sites/all/modules. As a left-over upgrade from a Fantastico install as Drupal 6 - this is the structure that it has and works.

So as to have nothing else to blame it on - I will move the Contrib modules there and do a registry rebuild and see what happens

Gentlemen: (David and cilefen)

It simply does not matter how to reproduce it. When we (I) tested daily builds of Windows at Microsoft - the measure of how to decide which one would be the production version was the one that caused the least errors and HAD FIXES or workarounds for those errors. You two are as close as it gets to Drupal management - so read on.

You could have a host of "volunteers" building a house,but the house is no good if as soon as you move in it falls on you because there were no real and complete standards to measure the construction. No one would be interested in a free house that will harm them. That is why even habit for humanity has project managers and building codes must be followed.

I am not talking about coding style - I am talking about a testing program to intentionally stress the software to expose bugs like this and then documenting the fix and making it known by central and organized searchable publishing.

Drupal is a mission critical build of software as a CMS - that is a wonder it works as well as it does given the origins of code from all over - that is susceptible to security issues that can harm the user in the use of it - if not immediately addressed AND how the code interacts cannot always be predicted because of the complication Drupal has become, BUT certain behaviors are tell-tale by elimination. That creates a duty - if you are giving it away for free as a product in active development - you need to service it - and open source - does not mean everyone is a programmer who will use it - it could just as well mean "open sore"

This that I have was a Drupal build that was upgraded from Drupal 6 that originally was installed using "Fantastico" and the modules directory was in the Drupal root, but the issue is more likely how the Domain Access module reacted and does react. Certainly updating CORE causes other things - and in this case the tell tale is at https://www.drupal.org/node/1358736 and https://www.ostraining.com/support-forum/drupal-support/domain-access-fa... as domain access presents itself this way when the module is not enabled and registered correctly.

The way to reproduce it is to simply upload core upon the right circumstance - and here we do not know what that is and will likely NEVER know what that is.

I have since moved most all the modules to the sites/all/modules directory and that was a nightmare in and of itself - as in "theory" moving the modules and rebuilding the registry should have been all that was necessary to get them re-registered in the new location

Instead I had to move modules back one at time - as Xdebug reported what appeared to be hard-coded paths in about 15 of maybe 70 modules, Then for modules that had other dependencies - I had to disable them up their dependency tree, and then move the main module and then re-enable them all - by usually selecting the module farthest down the dependency tree - and causing the other modules to be drawn in by dependency. The registry rebuild complained of inability to re-declare the database and flushing the caches hung over and over and in fact the registry rebuild never did work - I had to truncate the entire registry file table and then i got further. In the end six modules i could not move at all - due to inability to disable them - due to other dependencies too complicated to hack apart. since i do not let drush update the core - with the --no-core flag - they are ok in the root modules directory.

These are not things end users want to be faced with - especially with no solution or known workaround and they do not know how the software really works.

In the case of the "no active batch" error that plagues at least two different kinds of updates - because of secure pages - it is documented elsewhere but there is no central way to find an answer - and the Drupal support and documentation system is not adequately searchable - nor are most maintainers FIXING these issues once discovered and solved with a module update.

So . . . when cron runs - either Drupal cron or the 'nix' cron which in my case calls Drush to do updates - any number of things might have caused this trying to update or rebuild permissions, or cron itself - but rest assured it was not the file structure of Drupal - as it was working fine for 8 years now.

What needs to happen is a better documentation system for fixes and causes - and in this case is was the registry triggered by the core update - also associated to the need to rebuild the permissions I guess when Domain Access somehow got rudely disabled.

I am not going to spend time to try to reproduce it - as this is on a production site being built even more complex each day. Everybody does not want to work for free..I am not being paid for that - and I believe Drupal is socking itself in the face to give itself its own black Eye marching on to Drupal 8 when Drupal 7 is FULL of lots of issues that need address as well as the management manner in how Drupal exists.

There is no such thing as a volunteer for Drupal - because parties who use Drupal do not want a volunteer - they want something that works or answer to the problem and when a business is affected - many offer to pay and still cannot get an answer.

No one expects a module maintainer to be able to know how the software will react in every scenario, but they should at least know the symptoms of how it occurred as to what it might be that caused it and be responsive and not just close issues as duplicates - or claim they cannot reproduce it. That is why they need to be paid - to give an incentive to do what the software they put out there needs - support and service after letting it loose in the wild.

I have given the answer to what anyone coming across this scenario needs to know - and many do not even know Drupal 7 has a registry file at all.

The sad reality is - if I had not had a full site and database backup to restore and saw the error that Xdebug showed somehow Domain Access at Drupal Root could not even PATH to the domain settings.inc file - which was still fine and actually existing - and then to hard path to it to take me to the same error that existed 2 years and 4 months ago at https://www.ostraining.com/support-forum/drupal-support/domain-access-fa... to know domain access presents itself this way when the module is not enabled and registered correctly, the site would have just been down.

I kept it running on 7.37 until I figured it out - and I am a Certified Unix Network Engineer; I know someone else not only would be panic stricken - but would have done other things to screw their own site up worse trying to fix it - because if not for XDebug reporting from my server it would have been down likely WSOD style

Drupal FREE is not a bargain when one cannot get service, and for Drupal 8 - since corporate America seems to want it now - and they are willing to pay - maybe it should be a paid solution with a module certification and real-time support process by certified developers like the program Acquia has - and Drupal 7 modules should (can) be certifiable with available paid support also with maintainer and developer parties who will answer rapidly also - because a FREE house that falls down with no rescue is no house at all. Drupal 6 can remain free and wild in open source - but heck it truly runs the best and most trouble free anyway

It is time for Drupal to grow up and to offer this kind if real support for a real product that Drupal is now directly - because even writing this response is not "free", and I have no intention and sharing code I have to repair and patch and re-write eating up MY TIME for free any longer with anyone

Damien

You did not read the post. I have been working with Drupal since 2008, and this is NOT about a module location matter - this is about sloppiness in marching forward. As a main Drupal architect - you defend and deflect but do not see the real issue of the oddities that need to be tracked down.

I am not talking about "consultants" that often largely do not know what they are doing as PHP Drupal "brand" coders - and one has no PROOF they even know the issue you need addressed of them - because usually only the module maintainer knows their own code -IF THEY WOULD even respond at all. I am talking about Drupal as a management team of "core architect developers" like you and the other powers that be actually taking the helm and realizing and acknowledging issues like this instead of deflecting as everyone does so well its seems now and organizing itself to address lower level management of the pierces of the system which are Drupal and its versions

It is completely unrealistic to test a full copy of the same up to date website on a "non-production" site as issues like the domain itself and SSL itself as well as the setup of Drupal does not allow that - and i do not happen to have a CentOS exact copy of the WHM / CPanel SSD system I use lying about - and no one does. I cannot - and I am a Network Engineer - just re-direct a Domain and the DNS and SSL to another machine and then test on it -and then just move that all back to a production site - really - did you just write that ???

It is easist just to backup the site completely before hand and restore it - if you get crappy code. This is not a "beer and pizza" coding "hobby" with time to waste for most people.

At Microsoft - daily builds went down to specific machines for testing by Brand name and BIOS and internal parts build in a lab - and guess what - in the early days - each one responded differently. That is WHY Microsoft eventually bought Compaq / Digital Equipment so as to standardize on aplatform to be their official mission critical SUPPORTED platform they developed their guaranteed 99 percent up-time applications and SDK issues for. I cannot say what you do not know about software companies and marketing a software product "FREE" as open source or not - but it appears you lack the knowledge of the right things at issue here.

At some point comments like yours are ridiculous - and this is one - it appears.

What likely caused this was core itself in a registry update or change operation - and that is just plain FACT - because the update destroyed a Drupal path or other registration issue somehow. But here is the worst part - the Drupal 6 sites behaved just fine.

I did not need anyone to know how to fix it - it was either leave it working on 7.37 - or step through by disabling what it looked like the issue was as a module and try re-installing it AFTER the update - with the back uo still sitting in the wings to get back up again if it did not work..

You as well as I know that the contributed modules directory was separated to keep Drush from backing up ALL modules when the core updates through Drush and other than that - if there is a hard coded change in pathing - it would be in the core update - and STILL it is bad practice to break working software with an update.

You do not need to preach to me about what the README.TXT says - and - in fact it actually says:

"Additionally, modules and themes may be placed inside subdirectories in a
specific installation profile such as profiles/your_site_profile/modules and
profiles/your_site_profile/themes respectively to restrict their usage to only
sites that were installed with that specific profile."

Which clearly means the modules directory is NOT A MUST BE in sites/all/modules - so perhaps YOU should read the file before relying on what you THINK it says

Moreover at another place it also says -

"The flexible hook architecture means that you should never need to directly modify
the files that come with Drupal core to achieve the functionality you want;
instead, functionality modifications take the form of modules."

Yet I found myself having to PATCH the image module in two places - a part of CORE - that it a documented set of patches from TWO YEARS AGO - for the image module which STILL IS NOT FIXED IN CORE. See https://www.drupal.org/node/1973278

So what is really happening is no one can "trust" what is being released from the Drupal core team that appears is spread way too thin anyway - developing for FREE or not (and somebody is getting paid I know that $300,000 in Drupal 8 development money is going "somewhere") - and thus you should not behave as though someone there on the core team has not made a mistake or overlooked something - because for one - a TWO YEAR OLD issue still not fixed in the image module that threw errors to fill the watchdog log - looks like the same thing I just ran into - a TWO YEAR old issue with the domain access module also that pops up where certain factors are met and are present.

This is on top of module maintainers who just do not respond to issues in their queues and I say it is time for a quality and update maintenance and certification program for Drupal management to take module code over and certify it using OTHERS if maintainers do not respond, and it is time to STOP deflecting to another untested, unknown party as a "consultant" - to fix an issue that should never have been released as a bug in the first place and which expects them to be EXPERT PHP DRUPAL "brand" coders in the first place to find the issue of someone else's code. That looks like extortion to most people.

Please . . . seriously ???

Damien

All the modules were STILL there. Nothing was "blown away" - and if you read the post - prior to 7.38 it all worked fine AND after updating the core -it STILL worked FINE - until something else happened overnight.

It is not a "rant" - it is a documentation of problems - that even Drupal core clearly has . Including issues that have not been fixed for TWO years in the image module that had to be patched for starters.

So no . . . there was no "mistake" on my part . . . I uploaded the core update it ran FINE - and overnight something happened that clearly was a registry issue and Drupal destroyed it own DRUPAL ROOT path.

This was shown by the fact hard pathing to the file found it and it ran - then showing the registry issue - where clearly after destroying the path - tit then could not locate the module is just destroyed the path to and trying to negate its installation status.

The proof is in the fact the hard path worked, and it had NOTHING to do with the module location. Are you that poor a code troubleshooter ??

STILL Drupal will not allow me to move some of the modules . . . because it will NOT re-discover them and remove the old paths to the new location - OR allow the registry to be re-built . . . and what is alarming is you seem incessant on casting blame when the README DOES NOT say the sites/all/modules directory is MANDATORY

That is also NOT my fault it is bad code from somewhere.

The README you refer to is NOT the README that came down with 7.38 - another mistake on the core team's part I guess

It has NO SUCH LANGUAGE, and even what you present says "should" - not MUST - so your argument continues it's ridiculousness

The language of easy updates has to do with Drush moving ALL modules to a back directory on a core update - NOT Drupal being able to find the module to work properly.

EVEN if this is an issue that you want to blame on me . . . then DRUPAL should NOT RUN at all with contrib modules also in the root modules directory - and the README - which is NOT the README you claim says what you imagine above changed in 2012 should been made CLEAR in a very clear manner that Druoal 7 will not longer work that way.

I have not written a "rant" - and I did not start name calling and blame pointing of any kind - but you did . . . which shows your professionalism is similar to something in the 7.38 update - it an undefined fault.

It is you that needs to be looking for what could have caused what I reported - because it is NOT the only thing that is a problem, the problem really is a system of letting buggy code get out there and then defending it - I can download modules and take a good bet with mostly anyone there will be SOME kind of problem with the code in it that is a juvenile over-looking of testing the module -and the latest TODAY is a uc_webform_panes module that has - oh joy - an error in the syntax of a query (join query) that guess what - ran fine in Drupal 6

So it is you - as a Drupal core architect that needs to realize that the entire Drupal system of producing this CMS software as a product and brand name needs a reality wake up call - starting with your attitude - because the proof is in the pudding and in writing this was NOT my fault - and nothing is documented or produced coming out of the Drupal community system that is reliable to be good functioning code without bugs because that is the nature of PHP scripting - it can fall on itself without easily traceable cause or reason..

To cilefen and Damien

This matter is closed and now you both are just embarrassing yourselves as leaders in Drupal trying to deny the obvious. Drupal has quirks - lots of them - and management issues as to how to manage support matters and make sure they get timely fixed in future releases - free or not - and this is the result of just one.

What I am telling you WOULD improve Drupal if you would just LISTEN. But Damien you did not read past counting words and calling names - as the word "rant" is a derogatory term in and of itself, -and is read that way by definition - you should see that as soon as I figured out the solution I posted it for all others to see - if they came across it. That was the end of it - until you interjected nonsense trying to deflect and defend where there was no defense.

rant, verb -1. speak or shout at length in a wild, impassioned way. BUT . . . there is is NEVER anything "wild and impassioned" about the truth as FACTS. A rant is not measured by word count - it is measured by it's "Wild and impassioned" manner. That young man IS name calling

Upon those same facts neither of you will acknowledge as "we should do better" - and now the next ridiculous position that I am supposed to accept is be one to dig for a readme - that I have NO reason to look for and that is not required - to be read as it does not not as I have said contain ANY mandatory - but all the very least rather conflicting language - about a location of modules. It simply by all accounts does not matter. It says SHOULD as in an option - and I know WHY they "should" be placed there.

The location of the modules never has been the issue that caused this.

But instead you count words, talk about capitalized keys and all other things that do not address the real issue.

To make a point has no word limit . . . and since you are immortalizing this for the world to see - I wonder how many people among all those people who have run into the same issues I have will NOT agree with me.

The facts are what they are - these things did happen and the state of Drupal is what it is that even those in your own ranks are making comments about and to deflect is just sad. See among other things https://groups.drupal.org/node/291243

You rely on a file in the sites/all/modules directory as an instruction like a person is supposed to be like a pig digging for truffles - seriously - Damien your ridiculousness just continues. Any sane Software publishing operation would ALWAYS put a readme file that is expected to be read - in the ROOT directory of the product - not buried elsewhere.

I never said Drupal "sucks" - that is immaturity re-phrasing the words of another in print making up things I never said - I have just pointed out the TRUTH that you two cannot handle. Drupal is great - it just needs to be much better - because it's ordinary everyday effort as a community - is sloppy.

Damien you wrote

"If the files were there then there would have been no reason for PHP to give that error. And it shouldn't have been a registry error - that line from the settings.php file would be executed *before* the database is ever loaded, thus no registry to be at fault."

Wow. . . now you have proven you do not know your own system. "DRUPAL ROOT" is a virtual path - and it is created by Drupal as a symbolic aliased path

DRUPAL_ROOT/modules/domain/settings.inc) is an aliased virtual path in /. . . /settings.php on line 273 while "./modules/domain/settings.inc" is hard coded actual path. Anyone who reads this is going to KNOW you do not know what you are talking about - and that seems to be your "Modus Operandii".

After Drupal destroyed the virtual path -like in a cron run or what have you - then drupal re scanned and could not find the required settings.inc file it just destroyed the path for domain access to and tried to disable the domain access module. It could not - so it took the site down with an error that it could not open the stream to the path of the file. If you read the domain access install instructions it is clear in how this file is required immediately.

A person can upload a core update - and it will not affect the other modules in the root modules directory.

The uc_webform_pane module did the same thing - and gave a White Screen Of Death - and the SQL PDO Exception could only be seen by ripping the file out of the directory from a root back door.

These are NOT graceful error handling mannerisms - and instead of marching forward to Drupal 8 - these are the kinds of things that should be addressed in Drupal 7 - first it would seem - but money is there for Drupal 8 development so a shoddy job on Drupal 7 is left to stew or at the very least people who "volunteer" are spread too thin and asked too much of "for FREE" leaving a poor open source project effort. Many "volunteers" are jumping off the infested ship - and responses like this will not encourage me to help any one any further either as in some kind of demand I stop this and help out instead . . . SERIOUSLY . . . what is wrong with you two ??

So . . . Damien you are just WRONG again. Looking at your Drupal resume - on the profile page - I think I see why you have worked for so many companies at such a young age.

This kind of response to ANYONE at any company would have you in the "boss's" office and it seems you cannot help yourself - with a fight you are loosing or I should say already lost on the facts alone before you even got involved.

I am not going to re-write code for someone else - to improve the situation especially given it is not mine - and as core code - that should not be done anyway - but if the TWO of you would just read and listen to what is being said - this WOULD improve Drupal with standards for support and testing of anomalies - instead of trying to see who can spray a stream of nonsense the farthest.

You do not put out buggy code that you expect someone to then pay "consultants" - by holding them hostage tothose who are yourselves in a family way and community way to then fix - which really looks like bait and switch or extortion. That is how a person with a Law Degree as I have would always see it - and free or not - there is liability for such actions even if the software is free if the intentions are negligent and /or fraudulent. or both to begin with/ You cannot have both - offer working software to some -and refuse to do the same to others as in even a best effort. To argue with me is not a best effort

I am working on proprietary code - and I have had to fix a lot of Drupal 7 modules - but I have no intention on sharing it with anyone for free - as I am not being paid for even this reply either. - but lots of people will see EXACTLY what i am talking about because it makes SENSE and especially if they will have been in a similar scenario. I see lots of issue queues where people have actually written they just gave up - with no answer forthcoming.

What caused this was the core update dropping that DRUPAL ROOT path at some time later on some other event in th middle of the night - not domain access - as it was working FINE before the core update and worked again FINE in the root modules directory after even being re-installed - before i moved it and most others - as I outlined after the core update - so please -just stop - do not point fingers again.

This matter is closed - and solved as much as it can be - but not really ever fixed as to cause - so - just STOP - you both have already made enough of a fool of yourselves for the world to see.

The printout of this support post would make great reading on any blogpost for the kind of people who behind drupal - just stop.

Oh I enjoy using Drupal, and I did not attack anyone - the converse is true

Because someone is wrong and attacks me when I am right - is sad

The personal attack was from the other two - and anyone reading this will see that - if they can actually read a flow of English

Hard heads however, is not productive -coming to their defense when the post reads otherwise - does not look good at all.

You should direct your comment to Damien and cilefen - I did my contribution - I came across a problem, documented it and when no one responded with a known fix for the issue I found a way to fix it and posted it and then closed the issue - and that was that until the other two started in.

Again, this post of yours does not address any fix for the issue - but to shun me - when the issue in the software and the issues of proper support and code that is not as good as it code be remains. The truth is what it is.

People do not respond to issues in their projects, and modules can fill the log with errors - but this means fix it or remove it and use another module that often does the same thing. An example is accountmenu - which fills the log with $name class errors - but user menu does the same thing and has no errors. there is often a lot of duplication of effort - which in this case finds often way better code - and the truth is what is is - management is lacking addressing this - it does not mean I don't like Drupal. it means management of the Drupal "brand name" is not being done well - and he we are. The truth is what it is.

You should re-direct your comments to who was actually the ones with "poor manners" and also WRONG about the issue - calling derogatory remarks out towards me and making statements directed at me I did not say...just like you have done and "assumed" here also.

Good reading on what is going on here . . . http://press.princeton.edu/titles/8468.html

Jaypan:

As always from you - a great constructive thought as a solution !! However - and assuming your are referring to "Acquia Cloud" => https://www.acquia.com/blog/drupal-dev-workflow-everyone-git-flow-or-jus... and https://www.acquia.com/products-services/acquia-cloud - ; I am not allowed to use cloud services to put the client's code on any server which is not controlled and secure as a root device.

This is not to mention the extra hard drive/SSD space to have a duplicate site existing in the hosting environment

Also there are sync issues in the DNS - switching it back and forth to on the box directories of the different installations - which are not worth the issues for what is normally a "once in a blue-moon" issue like I ran into.

Simply backing up the site - and then restoring it if there is a problem with new software - like a core update I do not control - as software is the more realistic way to go about it

Drupal 7 -and especially now with Drupal 7 - it can consume lots of time and at some point the testing and use of redundancy tools becomes un-needed and un-wanted redundant work.

I use PHP Storm and a staging server adds complexity all around - and I am fairly sure of how this issue presented itself - I would have NEVER seen it coming in a test environment - because it is NOT my code anyway to mod and change as Drupal core.

The truth is I would never build a test environment with the root modules directory having the contributed modules anyway as this was an upgraded Drupal 6 install from 2008 as a Fantastico install which put the contributed modules in the root modules directory - and to build a test environment as the site it has now become and is morphing into would also make this unrealistic to do also.

I use the old fashioned push a module (or file) up and stress test for as many scenarios as I can for the options it is supposed to have as its features; and if one is broken - or not working - I take it back down off the production machine and put the backup copy back up and then debug the file and try again.

I change things one at a time - so as to track issues back down easier

This way the client's code never leaves my control as the client requires - so it remains unrealistic to create a staging and production server in my case as i said.