Text fields (textfield widget) with no max length are not being properly migrated to drupal 7 fields

I have had the same problem upgrading a site from drupal 6 to drupal 7.

My preexisting textarea fields started giving the error (not sure if it happens for other field types as :

Notice: Undefined index: max_length in text_field_widget_form() (line 513 of /modules/field/modules/text/text.module).

The error was occurring when editing nodes with the fields or editing the field settings or on the managing fields page of the content type edit area.

Here is my explanation:

The problem is that fields of type text with the textfield widget are being migrated into longtext fields and the widget is not coming across properly (for longtext it should be textarea, not textfield).

If you go to the widget type settings page (admin/structure/types/manage/CONTENTTYPE/fields/FIELDNAME/widget-type) and save the settings the error messages go away because the textarea widget is properly assigned.
But then the field has changed from being a text field in D6 to a text area in D7, which isn't ideal.

For extra details here is some data for a field (field_interviewee) I migrated.

The screenshots show the manage fields screen for my content type before and after the widget-type settings workaround mentioned above.

The text files have the field information from before and after the D6 to D7 upgrade (using content_fields() in D6 and field_info_field() in D7)
and also the field instance info from before and after the widget-type settings workaround mentioned above (using field_info_instance()).

This is pretty major actually, as it could mess up a lot of data for people that can't easily be fixed.

I was lucky to only have two text fields and knowledge enough to manually fix it.

These messages are related to the specific way the fields were configured in D6. There isn't enough information here to tell how they were configured.

A field with a blank value for 'max_length' would have been created as 'big' text and should be 'big' text after the migration. A field with a value for 'max_length would have been created as 'varchar' and should be 'varchar' after the migration.

A field that had an input filter in D6 should have a 'format' field in the D6 data and in D7 after the migration.

The issues about widgets are separate from the issues about the data type. If the data is right and the widget is wrong, it should be pretty easy to change the widget.

To do anything with this I need to know what the D6 data looked like, what the D7 data looks like (*without* making manual changes to it), and see an export of the D6 field and what the D7 field array ended up looking like.

The best I can do from the above information is make some guesses about what you started with, which is not much help.

I still have an old version of my site and I can confirm that on the old site the fields did not have a maximum length set.

Do you know if there is any documentation on this somewhere just so people know to make sure they have maximum length set?

Putting back to normal seeing as it is a site config issue.

The new field should just be a copy of the old field. If you look in the old database, what you see there is what you will get in D7. There is not any real documentation for this whole process, but it would be good to recommend that you look at what you have in D6 and make changes before you upgrade if you don't like it. There's another issue about documenting this process, are you volunteering to write some documentation?

I don't think my amount of spare time at the moment extends to that unfortunately, it was more of a question out of curiosity.

So I think there is no 'bug' here, right?

I can't reproduce any error. The comment above indicated that the 'error' was that the field was not configured as he expected, but it was configured to match the way it was configured in D6, which is not a bug.

If you are seeing an error, I need more information about how the fields you are trying to migrate were originally configured -- what widget? what field? what settings?

Note that there is a separate issue about bugs in the migration of number fields, so stick to errors on text fields in this issue.

Looking back through the issue I would say there is still one part that seems like a bug that can be fixed.

From my post in #2:

If you go to the widget type settings page (admin/structure/types/manage/CONTENTTYPE/fields/FIELDNAME/widget-type) and save the settings the error messages go away because the textarea widget is properly assigned.

So in the upgrade process if a text field with no max length is migrated into a text area, the "Widget type" needs to be updated to text area.
It seems this is not being done and is what is causing the error in this case.

Making title more descriptive.

@asb:
For the cause of this problem see posts #5 and #15. The problem is not having the max length for the fields set in drupal 6 before you upgrade to drupal 7.

So you either have to redo your upgrade from drupal 6 to 7 (making sure to set the max length on your text fields first) or manually fix your drupal 7 database. You could do something like this (for each field that is messed up):
1. create a new field field_oldfield (this field will be temporary so call it anything)
2. copy data from your original text area into field_oldfield (a quick sql query will be easiest)
3. delete the original text area (all the data will be gone so don't miss step 2)
4. recreate your original text area as a text field
5. copy the data from your field_oldfield into the new text field (a quick sql query will be easiest)
6. delete field_oldfield

If you just want to keep your d7 text areas as is but want the error to go away then do:

If you go to the widget type settings page (admin/structure/types/manage/CONTENTTYPE/fields/FIELDNAME/widget-type) and save the settings the error messages go away because the textarea widget is properly assigned.

There is however still one bug here, which i outlined in post #15.

'Size of textfield' is not the size of the value stored in the database, it is the size of the widget presented to the user. The size of the database field is set further down, in the global settings.

I realize this can be confusing, but it has been this way for several years now. If you look at your database you will see what you have. This migration code is not going to alter the database storage or fix D6 UX considerations, just get the data migrated.

I guess the idea is that usually a text field is for relatively short amounts of text and if you want a lot of text then use a text area.
So 255 characters should be enough for a text field.
It isn't very user friendly to have a single line text field that allows so many characters and the user only be able to see a small amount of the text in the field because their screen isn't 60 inches wide.
As you have said your field size is set to 60, that size text field isn't going to show anywhere near 255 characters without the user scrolling through the text a lot.

Finding out what the length of the longest value in a database column is not as hard as you make out.
You can just run one of these SQL queries depending on where your cck field is stored in the database:
1. SELECT MAX(LENGTH(field_MYFIELD_value)) FROM content_field_MYFIELD;
2. SELECT MAX(LENGTH(field_MYFIELD_value)) FROM content_type_MYCONTENTTYPE;

This module is designed to migrate the existing data from D6 to D7. If it is a text field in D6, it will end up as a text field in D7. IF it was a varchar field in D6, it will be a varchar field in D7.

The migration process is enormously complicated, and we haven't even begun to address all the edge cases for odd modules. I can't even fathom the complexity of not only migrating the existing data to the same field type but also trying to guess what the owner might have meant it to be and altering it (and how in the world could we possibly be sure that it was *meant* to be a varchar or text field just from the length of text in the database). I do NOT want to be responsible for altering data for a couple hundred thousand sites.

We are talking about making the D7 field EXACTLY THE SAME as whatever it was in D6. That ought to be sufficient. You have been using that data for months or years just as it is, apparently without problems. If you feel a need to change it, change it. If you haven't even noticed it it probably doesn't need to be changed.

Sorry for the outburst, I was getting frustrated about being chewed out over the D6 CCK UI when I am spending all this time trying to figure out how to get the data migrated.

The fundamental problem is that D6 CCK was structured in a way that it was possible to create a long text field that used a textfield widget. Yes, we probably should have reworked the code to make that impossible, but it is what it is and there are hundreds of thousands of sites out there now. And I have no time to re-architect D6 CCK and be sure that it won't break anything (which would also involve creating an update hook to alter data on all those hundreds of thousands of sites). So I am going to just focus on how to make the migration work well.

Originally I thought I would just leave the data alone, migrate like to like, but since it is no longer possible to have the same widget, it is already not like to like. So maybe the migration should try to be a little smarter and switch the field type in that case. But I'm really torn about adding that complexity into this process. Still mulling this over.

The question about key/value pairs is separate from the max length issue. That is the one change we do have to make because in D6 values used in a select list are stored as key/value strings in a textarea but in D7 they must be in an array. If we did not make that change no one with select lists in D6 would have anything functional in D7. But please don't pollute this issue with a discussion about that, there is no way for me to manage issues where people keep dumping new topics into the mix. I need one issue for each topic so other people can identify it and I can clearly see what needs to be fixed.

I guess you are saying you are not in favor of altering the unlimited text values on migration. If the text values are not altered, the remaining fix is the one discussed earlier, to make sure the textarea widget gets swapped in because the textfield widget will not work for a long text field in D7. So the result would be that your data would be a long text field in both D6 and D7 but you would end up with a textarea widget in D7, regardless of whether you started out with a textfield widget or a textarea widget.

This is probably the safest course (not to change anything we don't absolutely have to), so I think that is what I will commit. Working up a commit soon.

And to clarify further, the title of this issue is the *only* thing I am trying to fix in this issue -- a D6 text field that had an unlimited length and used a textfield widget.

I ended up going the other way with this after trying to migrate a bunch of test fields. It is so easy to create a long text field with a textfield widget that most likely almost everyone who has done it has done it accidentally. I am going to make an assumption that if they are viewing the field through a textfield widget they really don't have more than 255 characters of data and ended up with that setting accidentally. If I did the opposite, left it as a long text field and gave it a textarea widget, there is no way, other than custom code or an additional migration to get that field converted to varchar with a textfield widget.

I did not test to see if any of the values were more than 255 characters long. If I did that I would need to add a really complex additional if/then switch to handle the field differently depending on the results of that search. I am hoping that the warning that the field length will be set to 255 characters will be sufficient.

Since that is probably true for most but may not be true for everyone, we need a way to notify the user what is going on. We would need that no matter which way we fix this problem. So I just posted #1164530: Add message system to flag that I have added a messaging system to the migration. Now when you view the fields you can see, *before* you migrate them, what kinds of changes are going to be made. If you don't like the changes you can not migrate that field and handle it in whatever other way you like. For instance, if you really *do* want that field to be a long text field, you can change the widget in D6 to be a textfield widget and the migration will then happily migrate it as long text.

I give up. I have spent hours and hours and hours on this and no one is happy.

Let us leave this open then it is not fixed, and @asb pls. stop giving Karen a hard time. She do a fantastic job on the CCK module, and have put in so many hours working on the migration path for us, that I personally can't thank here enough.

Back to the issue: Got the exact same problem @abs have migration a D6 site from a client. The CCK migration script correctly detect that this might get tricky, and warns me that it will try to fit the field into a 255 large field. This fail, not 100% sure why, but from the error message and checking the length of my longest row (600char) I'll guess it just didn't not fit.
An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /batch?id=400&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'field_contributors_value' at row 1: INSERT INTO {field_revision_field_contributors} (entity_type, bundle, entity_id, revision_id, language, delta, field_contributors_value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => work [:db_insert_placeholder_2] => 714 [:db_insert_placeholder_3] => 1555 [:db_insert_placeholder_4] => und [:db_insert_placeholder_5] => 0 [:db_insert_placeholder_6] => Doménico Chiappe, novel, songs, and multimedia screenplay; Andreas Meier, programmer, artworks and graphic and multimedia design; Ramón Leún (artworks), Manuel Gallardo (artworks), Humberto Mayol (photography), Edgar Galíndez (photography), Pedro Ruiz (photography), Raúl Alemán (music). ) in drupal_write_record() (line 6851 of /drupal-7.2/includes/common.inc).

Have been reading up on this issue, and find your comments in #5 very interesting. Any suggestions on how this db. could have ended up with a incorrect formatted field in CCK 6.x?

And I confirm that simply changing the text field widget from "text field" to "text area", is enough to make the migration work again.

Cheers
Stein

This is rather difficult for users with no deep knowledge of Drupal or/and db to figure out. Any idea how we could move this alonge?

I found a solution, but it is a little hacky. Use this solution if you already upgraded to Drupal 7 and cannot change the widget in Drupal 6 anymore.

In the Drupal 7 database go to the old Drupal 6 table 'drupal_content_node_field_instance' and replace "textfield" with "textarea" for the needed fields.

Now you can use the migrate tool and it converts the content to longtext.

Thanks! Worked for me.

@jan.s thanks for the workaround.

I used this shotgun

update content_node_field_instance set widget_type = 'text_textarea' where widget_type='text_textfield'

as a quick fix.

Regarding #37 and #39: Nope, changing the field widget value in the table does not solve the problem. Sure, you can force the field to migrate, but you'll get the wrong widget in the end.

This bug effectively kills your migration path if the values in your D6 text fields used a certain widget - and I don't think we should break our users data, hence the upgrade in priority.

If you need a workaround while this is being resolved, this is the way I handled it based on comments from here: http://drupal.org/node/1105554

I opened up the backup of my original D6 site and set the maximum length of each text field 255. From the D6 database I SQL dumped only the content_node_field table and used it to replace the table of the same name in my D7 upgrade-in-progress. For the text fields definitions that were edited to 255 or less I was able to successfully clear the warning on the Structure > Migrate page and migrate them with no max_length error messages.

If you understand your database better than I do, which is likely, it might be more elegant to manually add the "max_length" value within the "global_settings" column of that same table. It would keep the collateral damage of replacing whole database tables to a minimum.

For the whole solution it would be nice just import text fields and automatically set the new migrated field to a max length of 255. You have a good warning system in place that should set users up to review their migrated fields for accuracy. You could update the message to remind them specifically to look for fields with truncated values in case some original data had 255 or more characters. If they discovered those issues then they can roll back the field migration and look for a solution. I bet that very few users if any have field values longer than 255 if they were using text_textfield as their widget.

Just saying.

Just for the record my fix from #39 is broken due to a new D7 core version. So I need to check this again.

My textfields have way more characters than 254 ... the largest has ~4000 chars so reducing this field length to 254 is not an option for me :(

So the fix has to detect the length of the values in the field and advise the user how to treat their specific field migration.

@seaneffel It looks like you have a plan for a patch?

I had this shell command for my known fields on my Drupal 5 upgrade path as that version allowed for fields too long. So maybe my upgrade procedure is wrong after all. Fixing something in a D5 context to fix a D7 problem. The FIELDS list was produced on a D7 CCK field migration so I knew which to fix on D5 :(

function trouble_fields_by_length {
  local FIELDS
  local FIELD
  local SQL
  FIELDS="field_x_value field_y_value ..."

  for FIELD in $FIELDS
  do
    SQL="SELECT '$FIELD', nid, length($FIELD) FROM content_type_content WHERE length($FIELD) > 240"
    echo $SQL
    drush @${SA_DEST} sql-query "$SQL"
  done
}

Did this help? (I'm puzzled again)

I just faced with the content_migrate module, since during migration it always halted at too long fields.

I had text fields that had text longer than 255 characters.

The solution is simple.

If you fail, roll back with content migrate module, and in field settings you can increase the max length in field settings since the field will now be empty and you can change the field settings. Now you can start the migration again.

PS: If content migrate would delete the field during a rollback, just create the field manually with custom length (the same name content migrate would create)

I put together this quick hack to allow me to translate a D6 "unlimited" text field to a 255-character D7 text area. Now my "long title" field doesn't break when I migrate!

I did not patch the error output, so this will still complain that you're doing something stupid or impossible, but it worked in my case. Hope this helps somebody else with the same upgrade troubles I had.

Hi, would be great if you could make a patch for it!
Greetings, Martijn

I finally finished a successful D6->D7 upgrade after giving all my CCK fields of "Text field" widget type a maximum length value. I've posted a comment on https://drupal.org/node/1144136 cross referencing this issue.

Others may find this script useful - it will tell you which fields have values longer than 255 characters.

https://gist.github.com/mjordan/6175870

This isn't about D6 to D6.

Thanks markj for the script. I adjusted it to work with drush.

https://gist.github.com/dmundra/15db7a8f46005db6d38b

I was having the same problem. I tried changing datatypes and setting max values in the database (both before and after migration), and I tried picking a different widget, but none of these worked. The datatypes in the database were correct and so was the widget (text area).

The really stupid solution that *did* work happened when I went into my Content Type > manage display. My fields were listed as:

Date ... Display dates using...settings
Directions ... Display dates using...settings
Body

But both Date and Directions had Date settings information (for some odd reason). So... I dragged Directions below Body, saved the settings, and it worked.

Date ... Display dates using...settings
Body
Directions

When I dragged it back so Directions was under Date and saved again, it took. No issues now.

I have ZERO idea why Drupal thought Directions needed date settings, nor do I have any idea why dragging it away worked to fix the issue.

I enhanced markj's code at #48 and made it into a module. Hopefully people will find that useful: https://www.drupal.org/project/report_oversized_text_fields