Hi,

I tried a D6->D7 migration with a small number of very simple content types (diabled all contributed modules, upgraded core, then enabled contributed modules one by one). The database updates went smoothly, and I ran ./admin/structure/content_migrate, which claimed to have successfully converted all CCK2 fields. The used field types mostly come from CCK2 core (like integer, and text fields; various optionwidgets from CCK2 core like checkboxes).

However, the content types can no longer be edited as on D6/CCK2; some field values, e.g. from textfield/checkboxes, are not being saved, and I'm getting lots of error messages when saving nodes:

Printed in red:

Notice: Undefined index: max_length in text_field_widget_form() (line 513 of /var/www/drupal/modules/field/modules/text/text.module).
PDOException: SQLSTATE[HY000]: General error: 1366 Incorrect integer value: '' for column 'field_capacity_value' at row 1: INSERT INTO {field_data_field_capacity} (entity_type, entity_id, revision_id, bundle, delta, language, field_capacity_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => 66 [:db_insert_placeholder_2] => 66 [:db_insert_placeholder_3] => cli [:db_insert_placeholder_4] => 0 [:db_insert_placeholder_5] => und [:db_insert_placeholder_6] => ) in field_sql_storage_field_storage_write() (line 425 of /var/www/drupal/modules/field/modules/field_sql_storage/field_sql_storage.module).

On the bottom: "The website encountered an unexpected error. Please try again later."

Additonally, I'm getting this red notice when viewing a node:

Notice: Undefined index: max_length in text_field_widget_form() (line 513 of /var/www/drupal/modules/field/modules/text/text.module).

And yes, newly created nodes are being saved, but usually do not contain the values from checkboxes, or texts entered in textfields. Also I noticed that i can not change exisitng checkbox values when editing an existing node; I can click on (another) checkbox, but this value isn't saved, either.

The field configuration at ./admin/structure/types was - at least in some cases - changed significantly; e.g., I wasn't using key|value pairs, just values; those became key|value pairs like

1|value1
2|value2
...

Also, defined minimum/maximum values for integer number disappeared during the CCK field migration.

Please advise if I'm posting in the wrong issue queue, but I really do not know where issues like this belong (fields in core? CCK migration?).

Thanks & greetings, -asb

Comments

Agileware’s picture

I have had the same problem upgrading a site from drupal 6 to drupal 7.

My preexisting textarea fields started giving the error (not sure if it happens for other field types as :

Notice: Undefined index: max_length in text_field_widget_form() (line 513 of /modules/field/modules/text/text.module).

The error was occurring when editing nodes with the fields or editing the field settings or on the managing fields page of the content type edit area.

Agileware’s picture

Title:SQLSTATE[HY000]: General error: 1366 Incorrect integer value» Text fields (textfield widget) are not being properly migrated to drupal 7 fields
Version:6.x-3.x-dev» 7.x-2.x-dev
StatusFileSize
new1.68 KB
new1.8 KB
new2.6 KB
new1.52 KB
new49.08 KB
new47.94 KB

Here is my explanation:

The problem is that fields of type text with the textfield widget are being migrated into longtext fields and the widget is not coming across properly (for longtext it should be textarea, not textfield).

If you go to the widget type settings page (admin/structure/types/manage/CONTENTTYPE/fields/FIELDNAME/widget-type) and save the settings the error messages go away because the textarea widget is properly assigned.
But then the field has changed from being a text field in D6 to a text area in D7, which isn't ideal.

For extra details here is some data for a field (field_interviewee) I migrated.

The screenshots show the manage fields screen for my content type before and after the widget-type settings workaround mentioned above.

The text files have the field information from before and after the D6 to D7 upgrade (using content_fields() in D6 and field_info_field() in D7)
and also the field instance info from before and after the widget-type settings workaround mentioned above (using field_info_instance()).

Agileware’s picture

Priority:Normal» Major

This is pretty major actually, as it could mess up a lot of data for people that can't easily be fixed.

I was lucky to only have two text fields and knowledge enough to manually fix it.

asb’s picture

Thanks, Agileware, for looking a little bit deeper into this. This issue is annoying, but I'm not sure if it's a "major" issue - compared to all the other data that is being lost during the D6->D7 migration. The sites I've seen in the past couple of weeks rely heavily on customized content types with numerous (more complex and non CCK core) CCK fields, and currently none of those could be migrated without losing actually "major" parts of the content. I really hope that the migration tools will evolve greatly, or all those sites will be stuck on D6 for a very long time. Well, let's hope for the best...

KarenS’s picture

Status:Active» Postponed (maintainer needs more info)

These messages are related to the specific way the fields were configured in D6. There isn't enough information here to tell how they were configured.

A field with a blank value for 'max_length' would have been created as 'big' text and should be 'big' text after the migration. A field with a value for 'max_length would have been created as 'varchar' and should be 'varchar' after the migration.

A field that had an input filter in D6 should have a 'format' field in the D6 data and in D7 after the migration.

The issues about widgets are separate from the issues about the data type. If the data is right and the widget is wrong, it should be pretty easy to change the widget.

To do anything with this I need to know what the D6 data looked like, what the D7 data looks like (*without* making manual changes to it), and see an export of the D6 field and what the D7 field array ended up looking like.

The best I can do from the above information is make some guesses about what you started with, which is not much help.

asb’s picture

Is there a way to locate the information you need after the update (in D7), or is this only available in D6 and gets lost during the update/upgrade?

Agileware’s picture

Priority:Major» Normal
Status:Postponed (maintainer needs more info)» Active

I still have an old version of my site and I can confirm that on the old site the fields did not have a maximum length set.

Do you know if there is any documentation on this somewhere just so people know to make sure they have maximum length set?

Putting back to normal seeing as it is a site config issue.

KarenS’s picture

The new field should just be a copy of the old field. If you look in the old database, what you see there is what you will get in D7. There is not any real documentation for this whole process, but it would be good to recommend that you look at what you have in D6 and make changes before you upgrade if you don't like it. There's another issue about documenting this process, are you volunteering to write some documentation?

Agileware’s picture

I don't think my amount of spare time at the moment extends to that unfortunately, it was more of a question out of curiosity.

KarenS’s picture

Status:Active» Fixed

So I think there is no 'bug' here, right?

asb’s picture

Status:Fixed» Active

Sorry, but I have to disagree. If the upgraded website continues to output messages like "The website encountered an unexpected error", gives red warnings like "Undefined index: max_length in text_field_widget_form() (line 513 of /var/www/drupal/modules/field/modules/text/text.module)" when editing a node, and simply does not store content on the migrated nodes anymore, how can this be not a bug?

Whatever the migration process treid to accomplish technically, it needs to conserve the ingress data, and after finishing the upgrade, the data has to be editable. Everything else is simply not working as it should since keeping the exisiting data is the whole point of a "migration path", right?

KarenS’s picture

Status:Active» Postponed (maintainer needs more info)

I can't reproduce any error. The comment above indicated that the 'error' was that the field was not configured as he expected, but it was configured to match the way it was configured in D6, which is not a bug.

If you are seeing an error, I need more information about how the fields you are trying to migrate were originally configured -- what widget? what field? what settings?

KarenS’s picture

Note that there is a separate issue about bugs in the migration of number fields, so stick to errors on text fields in this issue.

asb’s picture

The errors I got are given in the inital posting; I'm not familiar with PDOExceptions, so I can not interpret them properly. Regarding the exact configuration of the fields in the D6 site: If these settings are being lost during the upgrade, I'll have to restore a server that has been taken offline in the meantime, and restore a database backup. I'll probably get to it not before the next weekend (May 14./15.).

Agileware’s picture

Status:Postponed (maintainer needs more info)» Active

Looking back through the issue I would say there is still one part that seems like a bug that can be fixed.

From my post in #2:

If you go to the widget type settings page (admin/structure/types/manage/CONTENTTYPE/fields/FIELDNAME/widget-type) and save the settings the error messages go away because the textarea widget is properly assigned.

So in the upgrade process if a text field with no max length is migrated into a text area, the "Widget type" needs to be updated to text area.
It seems this is not being done and is what is causing the error in this case.

asb’s picture

OK, here are two sample D6 field configs from the custom content type 'person':

  • Field name: 'field_name'
  • Field type: 'Text'
  • Widget: 'Text field'
  • Size of text field: '60'
  • Global settings: 'Required', Number of values: '1'
  • Text processing: 'Simple text'
  • Available values: N/A
  • Field name: 'field_geschlecht'
  • Field type: 'Text'
  • Widget: 'Checkbox/Select buttons'
  • Size of text field: N/A
  • Global settings: 'Not required', Number of values: '1'
  • Text processing: 'Simple text'
  • Available values: 'männlich', 'weiblich'

The content type has also a number of other fields and field types (date, calculated, location, etc.), but let's stick to the simplest cases for now. On D7, those fields become:

  • Field name: 'field_name'
  • Field type: 'Long text'
  • Widget: 'Text field (multiple lines)' <- changed
  • Size of text field: '60'
  • Global settings: 'Not required' <- changed, Number of values: '1'
  • Text processing: 'Simple text'
  • Available values: N/A

When editing the cont type definition (./admin/structure/types/manage/person/fields/field_name), I'm getting this red message:

Notice: Undefined index: max_length in text_field_widget_form() (line 513 of /var/www/intranet/modules/field/modules/text/text.module).

  • Field name: 'field_geschlecht'
  • Field type: 'List (text)' <- changed
  • Widget: 'Checkbox/Select buttons'
  • Size of text field: N/A
  • Global settings: 'Not required', Number of values: '1'
  • Text processing: (?) <- N/A, so also changed
  • Available values: '0|männlich', '1|weiblich' <- changed to key/value pairs (unwanted!)

When editing a node of the content type 'person', I'm getting the error message mentioned above again, multiple times:

Notice: Undefined index: max_length in text_field_widget_form() (line 513 of /var/www/intranet/modules/field/modules/text/text.module).

The content from 'field_name' has vanished; also, 'field_geschlecht' has select buttons, but none is selected. So the content from both fields was either not properly migrated, or not migrated at all.

Since I don't know exactly where CCK and D7 field API/whatever is supposed to store it's data, the following is more or less guessing.

The database table in D6:

mysql> describe content_type_person;
+--------------------------+------------------+------+-----+---------+-------+
| Field                    | Type             | Null | Key | Default | Extra |
+--------------------------+------------------+------+-----+---------+-------+
| vid                      | int(10) unsigned | NO   | PRI | 0       |       |
| nid                      | int(10) unsigned | NO   | MUL | 0       |       |
| field_geburtsdatum_value | varchar(20)      | YES  |     | NULL    |       |
| field_sterbedatum_value  | varchar(20)      | YES  |     | NULL    |       |
| field_geburtsort_value   | longtext         | YES  |     | NULL    |       |
| field_sterbeort_value    | longtext         | YES  |     | NULL    |       |
| field_name_value         | longtext         | YES  |     | NULL    |       |
| field_geschlecht_value   | longtext         | YES  |     | NULL    |       |
| field_name_vater_value   | longtext         | YES  |     | NULL    |       |
| field_name_mutter_value  | longtext         | YES  |     | NULL    |       |
| field_anmerkungen_value  | longtext         | YES  |     | NULL    |       |
| field_anmerkungen_format | int(10) unsigned | YES  |     | NULL    |       |
| field_alter_value        | varchar(3)       | YES  |     | NULL    |       |
| field_ort_location_lid   | int(10) unsigned | YES  |     | NULL    |       |
+--------------------------+------------------+------+-----+---------+-------+
14 rows in set (0.00 sec)

The database table in D6:

mysql> describe content_type_person;
+--------------------------+------------------+------+-----+---------+-------+
| Field                    | Type             | Null | Key | Default | Extra |
+--------------------------+------------------+------+-----+---------+-------+
| vid                      | int(10) unsigned | NO   | PRI | 0       |       |
| nid                      | int(10) unsigned | NO   | MUL | 0       |       |
| field_geburtsdatum_value | varchar(20)      | YES  |     | NULL    |       |
| field_sterbedatum_value  | varchar(20)      | YES  |     | NULL    |       |
| field_geburtsort_value   | longtext         | YES  |     | NULL    |       |
| field_sterbeort_value    | longtext         | YES  |     | NULL    |       |
| field_name_value         | longtext         | YES  |     | NULL    |       |
| field_geschlecht_value   | longtext         | YES  |     | NULL    |       |
| field_name_vater_value   | longtext         | YES  |     | NULL    |       |
| field_name_mutter_value  | longtext         | YES  |     | NULL    |       |
| field_anmerkungen_value  | longtext         | YES  |     | NULL    |       |
| field_anmerkungen_format | int(10) unsigned | YES  |     | NULL    |       |
| field_alter_value        | varchar(3)       | YES  |     | NULL    |       |
| field_ort_location_lid   | int(10) unsigned | YES  |     | NULL    |       |
+--------------------------+------------------+------+-----+---------+-------+
14 rows in set (0.00 sec)

At least this table looks absolutlely identical to me. But D7 has other tables, that do not exist in D6:

mysql> describe field_data_field_name;
+-------------------+------------------+------+-----+---------+-------+
| Field             | Type             | Null | Key | Default | Extra |
+-------------------+------------------+------+-----+---------+-------+
| entity_type       | varchar(128)     | NO   | PRI |         |       |
| bundle            | varchar(128)     | NO   | MUL |         |       |
| deleted           | tinyint(4)       | NO   | PRI | 0       |       |
| entity_id         | int(10) unsigned | NO   | PRI | NULL    |       |
| revision_id       | int(10) unsigned | YES  | MUL | NULL    |       |
| language          | varchar(32)      | NO   | PRI |         |       |
| delta             | int(10) unsigned | NO   | PRI | NULL    |       |
| field_name_value  | longtext         | YES  |     | NULL    |       |
| field_name_format | varchar(255)     | YES  | MUL | NULL    |       |
+-------------------+------------------+------+-----+---------+-------+
9 rows in set (0.00 sec)
mysql> describe field_data_field_geschlecht;
+------------------------+------------------+------+-----+---------+-------+
| Field                  | Type             | Null | Key | Default | Extra |
+------------------------+------------------+------+-----+---------+-------+
| entity_type            | varchar(128)     | NO   | PRI |         |       |
| bundle                 | varchar(128)     | NO   | MUL |         |       |
| deleted                | tinyint(4)       | NO   | PRI | 0       |       |
| entity_id              | int(10) unsigned | NO   | PRI | NULL    |       |
| revision_id            | int(10) unsigned | YES  | MUL | NULL    |       |
| language               | varchar(32)      | NO   | PRI |         |       |
| delta                  | int(10) unsigned | NO   | PRI | NULL    |       |
| field_geschlecht_value | varchar(255)     | YES  | MUL | NULL    |       |
+------------------------+------------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

Let's look at some sample data:

mysql> select * from field_data_field_name;
+-------------+--------+---------+-----------+-------------+----------+-------+-----------------------+-------------------+
| entity_type | bundle | deleted | entity_id | revision_id | language | delta | field_name_value      | field_name_format |
+-------------+--------+---------+-----------+-------------+----------+-------+-----------------------+-------------------+
| node        | person |       0 |         4 |           4 | und      |     0 | B*******, A*** S***** | NULL              |
| node        | person |       0 |         5 |           5 | und      |     0 | L***, K********       | NULL              |
...

(Data in 'field_name_value' anonymized since this is real data). However, the database seems to still have the data in 'field_name_value', for some unknown reason it appears to be not displayed after the migration.

mysql> select * from field_data_field_geschlecht;
+-------------+--------+---------+-----------+-------------+----------+-------+------------------------+
| entity_type | bundle | deleted | entity_id | revision_id | language | delta | field_geschlecht_value |
+-------------+--------+---------+-----------+-------------+----------+-------+------------------------+
| node        | person |       0 |         4 |           4 | und      |     0 | m�nnlich               |
| node        | person |       0 |         5 |           5 | und      |     0 | weiblich               |
...

It's similar with 'field_geschlecht_value': The data is still there, but it does not match the (unwanted) key/value definition of the D7 fields. Seems that not much migration happened here at all.

I hope this rough description helps to get the picture. If I can provide more information, e.g. data from other db tables, please feel free to ask. I'm attaching an export of the sample content type; in D7 I havn't yet found a way to export content type definitions.

asb’s picture

StatusFileSize
new38.82 KB
Agileware’s picture

Title:Text fields (textfield widget) are not being properly migrated to drupal 7 fields» Text fields (textfield widget) with no max length are not being properly migrated to drupal 7 fields

Making title more descriptive.

@asb:
For the cause of this problem see posts #5 and #15. The problem is not having the max length for the fields set in drupal 6 before you upgrade to drupal 7.

So you either have to redo your upgrade from drupal 6 to 7 (making sure to set the max length on your text fields first) or manually fix your drupal 7 database. You could do something like this (for each field that is messed up):
1. create a new field field_oldfield (this field will be temporary so call it anything)
2. copy data from your original text area into field_oldfield (a quick sql query will be easiest)
3. delete the original text area (all the data will be gone so don't miss step 2)
4. recreate your original text area as a text field
5. copy the data from your field_oldfield into the new text field (a quick sql query will be easiest)
6. delete field_oldfield

If you just want to keep your d7 text areas as is but want the error to go away then do:

If you go to the widget type settings page (admin/structure/types/manage/CONTENTTYPE/fields/FIELDNAME/widget-type) and save the settings the error messages go away because the textarea widget is properly assigned.

There is however still one bug here, which i outlined in post #15.

asb’s picture

Sorry, I do not follow. "Size of textfield" is a required setting and set to '60' in 'field_name', so how could not having the max length for the fields be the problem?

And even if the value wouldn't be required, how would I know how the actual max lengh would have to be (to avoid cutting of text), and why would the user be forced to figure out such things for hundreds of textfields? Isn't this exactly the point of what is called a migration path?

Seems like it'll be a loooong time before I'll be touching D7 a upgrade attempt again. Hopefully it'll be many years before D6 reaches it's EOL.

KarenS’s picture

'Size of textfield' is not the size of the value stored in the database, it is the size of the widget presented to the user. The size of the database field is set further down, in the global settings.

I realize this can be confusing, but it has been this way for several years now. If you look at your database you will see what you have. This migration code is not going to alter the database storage or fix D6 UX considerations, just get the data migrated.

asb’s picture

You're referring to the global setting, called in a localized Drupal "Maximallänge", OK. The help text for this setting explains: "Leave empty for an unlimited length", so the D6 UI explicitly allows for an unlimited length. Because this setting is valid, a migration path needs to take care that the field's data is still available - meaning: viewable, editable, and unaltered - after the migration.

It's absurd if the user is being forced to guess what actual lenght his/her unlimited length fields might need (scroll through hundeds of thousands of records on the mysql prompt and count characters?), and that he/she is supposed to touch every existing text field before starting the migration (I don't think I've very often limited the size of text fields at all). And honestly I not even want to limit the length of these text fields because I actually care if the name is Karen or Karen S.

Geez, these are text fields, the simplest data types of all. What are we supposed to do when trying to migrate more complex data types like imagesfields or geo point data - hack the database files with a hex editor?

Agileware’s picture

I guess the idea is that usually a text field is for relatively short amounts of text and if you want a lot of text then use a text area.
So 255 characters should be enough for a text field.
It isn't very user friendly to have a single line text field that allows so many characters and the user only be able to see a small amount of the text in the field because their screen isn't 60 inches wide.
As you have said your field size is set to 60, that size text field isn't going to show anywhere near 255 characters without the user scrolling through the text a lot.

Finding out what the length of the longest value in a database column is not as hard as you make out.
You can just run one of these SQL queries depending on where your cck field is stored in the database:
1. SELECT MAX(LENGTH(field_MYFIELD_value)) FROM content_field_MYFIELD;
2. SELECT MAX(LENGTH(field_MYFIELD_value)) FROM content_type_MYCONTENTTYPE;

asb’s picture

Finding out what the length of the longest value in a database column is not as hard as you make out. You can just run one of these SQL queries depending on where your cck field is stored in the database (...)

According to the project page, CCK ist currently installed and active on 248,029 Drupal sites. Honestly, how many of these sites are properly maintained and administered by someone who knows enough about MySQL or PostgreSQL, how many have even access to Phpmyadmin or a database shell? The last time I checked, major parts of the Drupal users were discussing how many clicks the installer should require, or which WYSIWYG editor looks prettier. Lots of users are aware of security issues, but don't update their modules because they consider it too complicated. I faintly remember also that D7 had usability initiatives to make using it simpler for non techy users, not harder even for people with some kind of technical background.

However, even if we don't care about our users and their real wishes end needs, I still have (would have, if I'd still consider the migration) the personal problem of hundreds of text fields on my Drupal sites. Roughly estimated, I have about 10 content types per site, and an average of 5 text fields per content type; it'd take me days to just touch every text field in this way, and on top I'd lose lots of flexibilty due to the forced limited field lenghts, plus there are still the unwanted data modifications (key/value pairs insted of plain text storage). Not much motivation left for more upgrade attempts, I think.

KarenS’s picture

This module is designed to migrate the existing data from D6 to D7. If it is a text field in D6, it will end up as a text field in D7. IF it was a varchar field in D6, it will be a varchar field in D7.

The migration process is enormously complicated, and we haven't even begun to address all the edge cases for odd modules. I can't even fathom the complexity of not only migrating the existing data to the same field type but also trying to guess what the owner might have meant it to be and altering it (and how in the world could we possibly be sure that it was *meant* to be a varchar or text field just from the length of text in the database). I do NOT want to be responsible for altering data for a couple hundred thousand sites.

We are talking about making the D7 field EXACTLY THE SAME as whatever it was in D6. That ought to be sufficient. You have been using that data for months or years just as it is, apparently without problems. If you feel a need to change it, change it. If you haven't even noticed it it probably doesn't need to be changed.

KarenS’s picture

Sorry for the outburst, I was getting frustrated about being chewed out over the D6 CCK UI when I am spending all this time trying to figure out how to get the data migrated.

The fundamental problem is that D6 CCK was structured in a way that it was possible to create a long text field that used a textfield widget. Yes, we probably should have reworked the code to make that impossible, but it is what it is and there are hundreds of thousands of sites out there now. And I have no time to re-architect D6 CCK and be sure that it won't break anything (which would also involve creating an update hook to alter data on all those hundreds of thousands of sites). So I am going to just focus on how to make the migration work well.

Originally I thought I would just leave the data alone, migrate like to like, but since it is no longer possible to have the same widget, it is already not like to like. So maybe the migration should try to be a little smarter and switch the field type in that case. But I'm really torn about adding that complexity into this process. Still mulling this over.

asb’s picture

@KarenS: Thanks for your clarifications. I'm glad to hear that the basic intention of the migration procedure is to maintain the data as is and not to make guesses about what the user might have wanted and make estimates based on these guesses. If I understand you correctly, things like the alteration from plain text storage to key/value pairs are not intended ("FieldUpdateForbiddenException: Cannot update a list field to not include keys with existing data [...]").

Btw, I don't think that the CCK design decision to allow a long text field to use a textfield widget was or is wrong. It just accomodates properly for real-world data which is not always the way an engineer wants it to be. Example: If you design a field to store a places or a citys name, the engineer might decide that a max langth of 50 character will suffice. But in the real world, we might want to store the name of a hill in New Zealand, or a certain town in Wales:

Taumatawhakatangihangakoauauotamateapokaiwhenuakitanatahu (85 characters)
Llanfairpwllgwyngyllgogerychwyrndrobwllllantysiliogogogoch (58 characters)

There are numerous cases like this, not only in geography, but also in chemistry, biology, and many other disciplines, so from a user's perspective flexibility is always appreciated. If D7 doesn't allow this real-world requirements anymore, it's imho D7 that is poorly designed, not D6's CCK. We all know that D7 has also other terrible design decisions, so this wouldn't come a surprise.

Last but not least I have to apologize as well since the migration module is still under development and obviously not yet intended to deliver results for "production". I think my frustration comes from my prior experiences with Drupal upgrades from the past years and the insidious data degradation that seems to be so inevitably connected with it. It's simply so frustratrating to spend months and years to build a site and lose more or less significant parts of it on a regular basis (like book menu structures, the Views1->Views2 migration, or data from unported modules), on every major Drupal upgrade, and in many cases it felt more like a data grave than like a reliable data storage mechanism.

I am fully aware that you are doing an incredible job to avoid such data degradation for the CCK migration, and that you should get much more help and appreciation for that. So thank you very much for your hard work, and please don't make me figure out the max length for all text fields manually before being allowed to attempt another D7 migration ;)

KarenS’s picture

The question about key/value pairs is separate from the max length issue. That is the one change we do have to make because in D6 values used in a select list are stored as key/value strings in a textarea but in D7 they must be in an array. If we did not make that change no one with select lists in D6 would have anything functional in D7. But please don't pollute this issue with a discussion about that, there is no way for me to manage issues where people keep dumping new topics into the mix. I need one issue for each topic so other people can identify it and I can clearly see what needs to be fixed.

I guess you are saying you are not in favor of altering the unlimited text values on migration. If the text values are not altered, the remaining fix is the one discussed earlier, to make sure the textarea widget gets swapped in because the textfield widget will not work for a long text field in D7. So the result would be that your data would be a long text field in both D6 and D7 but you would end up with a textarea widget in D7, regardless of whether you started out with a textfield widget or a textarea widget.

This is probably the safest course (not to change anything we don't absolutely have to), so I think that is what I will commit. Working up a commit soon.

KarenS’s picture

And to clarify further, the title of this issue is the *only* thing I am trying to fix in this issue -- a D6 text field that had an unlimited length and used a textfield widget.

KarenS’s picture

Status:Active» Fixed

I ended up going the other way with this after trying to migrate a bunch of test fields. It is so easy to create a long text field with a textfield widget that most likely almost everyone who has done it has done it accidentally. I am going to make an assumption that if they are viewing the field through a textfield widget they really don't have more than 255 characters of data and ended up with that setting accidentally. If I did the opposite, left it as a long text field and gave it a textarea widget, there is no way, other than custom code or an additional migration to get that field converted to varchar with a textfield widget.

I did not test to see if any of the values were more than 255 characters long. If I did that I would need to add a really complex additional if/then switch to handle the field differently depending on the results of that search. I am hoping that the warning that the field length will be set to 255 characters will be sufficient.

Since that is probably true for most but may not be true for everyone, we need a way to notify the user what is going on. We would need that no matter which way we fix this problem. So I just posted #1164530: Add message system to flag that I have added a messaging system to the migration. Now when you view the fields you can see, *before* you migrate them, what kinds of changes are going to be made. If you don't like the changes you can not migrate that field and handle it in whatever other way you like. For instance, if you really *do* want that field to be a long text field, you can change the widget in D6 to be a textfield widget and the migration will then happily migrate it as long text.

asb’s picture

Re. #27: "The question about key/value pairs is separate from the max length issue. ... But please don't pollute this issue ... keep dumping new topics into the mix..."

Sorry, but the key/value pair issue was part of the initial issue I posted. Sad if we lose the plaintext fields as well.

If I understand the messaging system correctly, the workflow is now:

  1. Migrate Drupal core from D6 to D7
  2. then start ./admin/content/field_convert in D7 and check the changes field convert mentions
  3. if unwanted changed show up, cancel field convert, drop the D7 site, and go back to D6
  4. repeat the D6 -> D7 migration, and start field convert again

Interesting approach (dropping the migrated D7 site).

KarenS’s picture

I give up. I have spent hours and hours and hours on this and no one is happy.

asb’s picture

If I understood your reply correctly, the migration workflow actually involves a number of "loops" between D6, the core migration, and the CCK migration on D7. This would be perfectly fine if one such "loop" would catch all CCK fields, but I'm feeling uneasy since probably other fields will require similar postprocessing steps, leading to other "loops" back to D6, and requiring to restart the D6-D7 migration again.

According to what I've seen in #16, may I suggest to distinguish between (a) preprocessing on D6, and (b) postprocessing on D7? Since the CCK field data seems to remain untampered in the database after the D6->D7 migration, wouldn't it be possible to move the CCK migration completely either into a preprocessing step on D6, or a postprocessing on D7?

steinmb’s picture

Status:Fixed» Active

Let us leave this open then it is not fixed, and @asb pls. stop giving Karen a hard time. She do a fantastic job on the CCK module, and have put in so many hours working on the migration path for us, that I personally can't thank here enough.

Back to the issue: Got the exact same problem @abs have migration a D6 site from a client. The CCK migration script correctly detect that this might get tricky, and warns me that it will try to fit the field into a 255 large field. This fail, not 100% sure why, but from the error message and checking the length of my longest row (600char) I'll guess it just didn't not fit.
An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=400&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'field_contributors_value' at row 1: INSERT INTO {field_revision_field_contributors} (entity_type, bundle, entity_id, revision_id, language, delta, field_contributors_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => work [:db_insert_placeholder_2] => 714 [:db_insert_placeholder_3] => 1555 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => Doménico Chiappe, novel, songs, and multimedia screenplay; Andreas Meier, programmer, artworks and graphic and multimedia design; Ramón Leún (artworks), Manuel Gallardo (artworks), Humberto Mayol (photography), Edgar Galíndez (photography), Pedro Ruiz (photography), Raúl Alemán (music). ) in drupal_write_record() (line 6851 of /drupal-7.2/includes/common.inc).

Have been reading up on this issue, and find your comments in #5 very interesting. Any suggestions on how this db. could have ended up with a incorrect formatted field in CCK 6.x?

steinmb’s picture

StatusFileSize
new92.32 KB

And I confirm that simply changing the text field widget from "text field" to "text area", is enough to make the migration work again.

Cheers
Stein

steinmb’s picture

This is rather difficult for users with no deep knowledge of Drupal or/and db to figure out. Any idea how we could move this alonge?

asb’s picture

#32. However, this issue is closed.

jan.s’s picture

I found a solution, but it is a little hacky. Use this solution if you already upgraded to Drupal 7 and cannot change the widget in Drupal 6 anymore.

In the Drupal 7 database go to the old Drupal 6 table 'drupal_content_node_field_instance' and replace "textfield" with "textarea" for the needed fields.

Now you can use the migrate tool and it converts the content to longtext.

yurtboy’s picture

Thanks! Worked for me.

clemens.tolboom’s picture

@jan.s thanks for the workaround.

I used this shotgun

update content_node_field_instance set widget_type = 'text_textarea' where widget_type='text_textfield'

as a quick fix.

seaneffel’s picture

Priority:Normal» Major

Regarding #37 and #39: Nope, changing the field widget value in the table does not solve the problem. Sure, you can force the field to migrate, but you'll get the wrong widget in the end.

This bug effectively kills your migration path if the values in your D6 text fields used a certain widget - and I don't think we should break our users data, hence the upgrade in priority.

seaneffel’s picture

If you need a workaround while this is being resolved, this is the way I handled it based on comments from here: http://drupal.org/node/1105554

I opened up the backup of my original D6 site and set the maximum length of each text field 255. From the D6 database I SQL dumped only the content_node_field table and used it to replace the table of the same name in my D7 upgrade-in-progress. For the text fields definitions that were edited to 255 or less I was able to successfully clear the warning on the Structure > Migrate page and migrate them with no max_length error messages.

If you understand your database better than I do, which is likely, it might be more elegant to manually add the "max_length" value within the "global_settings" column of that same table. It would keep the collateral damage of replacing whole database tables to a minimum.

For the whole solution it would be nice just import text fields and automatically set the new migrated field to a max length of 255. You have a good warning system in place that should set users up to review their migrated fields for accuracy. You could update the message to remind them specifically to look for fields with truncated values in case some original data had 255 or more characters. If they discovered those issues then they can roll back the field migration and look for a solution. I bet that very few users if any have field values longer than 255 if they were using text_textfield as their widget.

Just saying.

clemens.tolboom’s picture

Just for the record my fix from #39 is broken due to a new D7 core version. So I need to check this again.

My textfields have way more characters than 254 ... the largest has ~4000 chars so reducing this field length to 254 is not an option for me :(

seaneffel’s picture

So the fix has to detect the length of the values in the field and advise the user how to treat their specific field migration.

clemens.tolboom’s picture

@seaneffel It looks like you have a plan for a patch?

I had this shell command for my known fields on my Drupal 5 upgrade path as that version allowed for fields too long. So maybe my upgrade procedure is wrong after all. Fixing something in a D5 context to fix a D7 problem. The FIELDS list was produced on a D7 CCK field migration so I knew which to fix on D5 :(

function trouble_fields_by_length {
  local FIELDS
  local FIELD
  local SQL
  FIELDS="field_x_value field_y_value ..."

  for FIELD in $FIELDS
  do
    SQL="SELECT '$FIELD', nid, length($FIELD) FROM content_type_content WHERE length($FIELD) > 240"
    echo $SQL
    drush @${SA_DEST} sql-query "$SQL"
  done
}

Did this help? (I'm puzzled again)

giorgio79’s picture

I just faced with the content_migrate module, since during migration it always halted at too long fields.

I had text fields that had text longer than 255 characters.

The solution is simple.

If you fail, roll back with content migrate module, and in field settings you can increase the max length in field settings since the field will now be empty and you can change the field settings. Now you can start the migration again.

PS: If content migrate would delete the field during a rollback, just create the field manually with custom length (the same name content migrate would create)

rbuckbailey’s picture

StatusFileSize
new1.23 KB

I put together this quick hack to allow me to translate a D6 "unlimited" text field to a 255-character D7 text area. Now my "long title" field doesn't break when I migrate!

I did not patch the error output, so this will still complain that you're doing something stupid or impossible, but it worked in my case. Hope this helps somebody else with the same upgrade troubles I had.

Summit’s picture

Hi, would be great if you could make a patch for it!
Greetings, Martijn

markj’s picture

I finally finished a successful D6->D7 upgrade after giving all my CCK fields of "Text field" widget type a maximum length value. I've posted a comment on https://drupal.org/node/1144136 cross referencing this issue.

Others may find this script useful - it will tell you which fields have values longer than 255 characters.

https://gist.github.com/mjordan/6175870

markj’s picture

Issue summary:View changes

This isn't about D6 to D6.

dmundra’s picture

Thanks markj for the script. I adjusted it to work with drush.

https://gist.github.com/dmundra/15db7a8f46005db6d38b

GopherWild’s picture

I was having the same problem. I tried changing datatypes and setting max values in the database (both before and after migration), and I tried picking a different widget, but none of these worked. The datatypes in the database were correct and so was the widget (text area).

The really stupid solution that *did* work happened when I went into my Content Type > manage display. My fields were listed as:

Date ... Display dates using...settings
Directions ... Display dates using...settings
Body

But both Date and Directions had Date settings information (for some odd reason). So... I dragged Directions below Body, saved the settings, and it worked.

Date ... Display dates using...settings
Body
Directions

When I dragged it back so Directions was under Date and saved again, it took. No issues now.

I have ZERO idea why Drupal thought Directions needed date settings, nor do I have any idea why dragging it away worked to fix the issue.