Attachment Imports

Is it just the attachments that are not showing?

#670424: Missing posts after migration?

To see what the problem is, compare the data in the files table with what is actually on the filesystem - pick any attachment and have a look at what path is being given for it, and then check if it is the correct one.

I can see that the "files/1_501c1ae9df445fe2fb84483312f7619d" bit is wrong.

You should be able to fix this in phpmyadmin with an sql script to replace the filepath with the real file path plus whatever is in the filename column.

(if you have uploaded ohter files using other means, you may want to then limit the query for the relevant fid's)

thanks for that screenshot - it shows that I was wrong with my last instructions. If you have the attachments in the files folder, then it should work as expected.

If you have them in some other folder, then you need to replace the "files/" bit with whatever the correct path is.

EDIT - just to check if it is not the filename, edit one file to give it an extension, and then edit the files data to point to the correct place and see if that gets it shown.

Chances are that you have uploaded the files to the wrong place. Have you also checked the permissions to see if people have the permission to see the uploaded files?

in phpmyadmin, try something like:

UPDATE files SET filepath = REPLACE(filepath, "files/", "sites/default/files/") WHERE fid < X

(where X is teh maximum fid for the import... you can remove the WHERE statement if there are no other files... and change it if there were other files before the import.)

Also, i had to hack your phpbb2druapl.module and i'll tell you why. I was in the situation where i am migrating on behalf of a third party. This means I am not THE admin in the sense of the board founder/amdin. What was happening was my drupal admin account was overwriting the true admin/founder lol ( i dont think hed be too happy!! :) ) So i removed some code and lowered the user id grab to 1 from 2, removed any reassginment of posts etc from user 2 (true admin of phpboard in this case was user 2).

here you could have created a new user and then in the phpbb2drupal settings page chosen to map the details to the new user without hacking the module.

do you have the phpass module installed? (there is a "fixed" working version downloadable from my site).

What encryption scheme does smf use? phpbb3 uses phpass, same with the upcoming drupal 7. but standard drupal 6 only uses md5, so the extra module is needed to allow the phpass passwords to be used.

try copying the password into the phpass table to see if it makes any difference.

you can replace it if you wish - I have not checked to see if there is an updated version available, but I had made no changes to that file, so it does not hurt to replace it - it was just a matter of convenience.

ok, that is probably not something that can be avoided - you may need to force all the users to reset their passwords.

Either that, or you can create a new module that overrides the authentication proces a'la phpass and then use that to authenticate the users, but this will be more effort and you will need to find out exactly how smf encoded its users.

Or you can ask all the users to request a new password.

Hi,

I also have a attachment import problem. I've made sure that the files are at the correct location. But all the "filepath"s are like "files/1_501c1ae9df445fe2fb84483312f7619d". When I click the link, my browser asks for downloading the file, but it sees the file as "1_501c1ae9df445fe2fb84483312f7619d" without any attachements (and the file name doesn't make any sense, although on the filename column I see that the file name is "something.pdf").

So there are two problems here:
1. The files does not have attachments (and since there are different types of files -jpg, gif, pdf, etc.) I'm not sure how I can give those files correct extensions (maybe using the data from the filename column, but how?).
2. The file names, while downloading, does not make any sense. Although this is not a major issue, I feel these two issues can be solved at once. I only lack the knowledge to do so.

Some more info:
When I rename "1_501c1ae9df445fe2fb84483312f7619d" to "1_501c1ae9df445fe2fb84483312f7619d.pdf" on the filepath, it downloads as a PDF file, just normal (only the file name is gibberish). But when I rename it to "something.pdf", the download link takes me to the File Not Found page.

Thanks in advance

Is there any way you can find out what all the file types are? are they stored anywhere? You will need to compare in the database.

As for the filename, that is a different issue and since the files are named in such ways before, that bit cannot be fixed by this module, though it may be possible to create a mass renaming script?

Hi,

Thanks for the response. For the file types there are two sources as far as I can see:
1. The filename column has the filenames as "something.pdf", so there is the information about the file extension
2. There is a filemime column in the table which includes info like "image/jpeg", "application/octet-stream", "application/pdf", "application/zip" etc about the file type. This second one maybe more difficult to use.

If you think the first info can be used, could you also help me creating the phpmyadmin SQl query?

Cheers

are the actual filenames in the director ending in eg ".pdf" or do they have no file endings?

If they do have the filetype endings, then you can redo the import and in phpbb2drupal.module, on lines 662 and 822 (in both cases jsut before the db_query line and after the db_set_active('default');) add

$file->physical_filename = $file->physical_filename . '.' . end(explode(‘.’,$file->real_filename));

amd then re do the import.

If however the files do not have the filetype in their physical file names, that will not work.

(there may also be a phpmyadmin query to update the filepath using the filename data, but I am not sure how you would get a substring there...)

Thanks for the info. I'll try this approach.

(Yes, the filename column includes the extensions.)

(I meant the actual filenames in the filesystem as opposed to the filenames as stored in the database)

No the actual files don't have the file extensions either.

Hmm, this is getting trickier that I thought it would be.

That complicates things as then you need something to change the actual filenames too. This would require a more complicated script to be written, one that manipulates the files too.

I don't have access to phpmyadmin right now, so I don't know if this is possible or not.

Maybe I can try to rename the files before starting the migration. If such info exists in the phpbb tables I can do the rename first and then use the un-hacked version (excluding the hack you shared a few comments ago) of php2drupal migrate to do the migration.

I'm just noting the idea here. I'll have a look in a few hours.

Hi again,

Sorry for the late response, but I couldn't find any time for this.

I've just checked the PhpBB tables and in table phpbb_attachments I found two coulmns, physical_filename and real_filename.

phyical_filename is something like this: 2_a8b580ad650e236892ba294fd13f565b
corresponding real_filename is: play.jpg

So, the PhpBB database contains the file extensions as part of real_filename. Do you think this will help?

We are already importing the physical filename as the filepath and the real filename as the filename.

Now the issue is how to display the file with the correct filename - drupal shows the filepath filename as the real filename... and that is not an issue I know how to fix.

The only way I can think of is to rename all the files using a bulk file renaming utility to their "real_filename"s. And then replacing the filepath in Drupal with the filename. In that case, filepath, filename and the actual file name of the file would be all the same.

I only need to export the PhpBB table in a file (if I can) and run the renaming utility using that file to do the renaming. This may be tricky...

I just wonder why I don't have the attachments of the files...

I am unsure why it does not work, but changingthe filepath to inclide the .pdf etc to add an attachment does seem to indicate it is more an issue of what types of atttachment are allowed and not allowed attachments will probably not show. But this is a total guess.

After a year there is something new about this problem with attachments "real names" vs "physical name"?

I imported from phpbb 3.08 yesterday and have this problem with attachments.

By reading around the issues (and applying a patch) I was able to import attachments into posts (I can see their real name in drupal posts), but they get not rendered because when I click on them they goes to the physical name.

example:
in a post I see attachment "img_1234.jpg", but when I click on it goes to "/sites/default/files/files/53_ddc2953156972e826ff9c6160757f53c" and the browser can't render that file without extension.

I copied all the PHPBB3 attachment folder to where it points, so it finds them. But the browser can't render a non extension file (under windows)

It's a problem in the module...
How this can be solved? without mass file renaming & relative sql editing?

I need attachments working, otherwise phpbb2drupal it's not useful in my case.

thanks,
Roberto.

I renamed all attachments files with a script, giving them the real name extracted from phpbb3 table.

Now I can see attachments and they all open fine in the browser.

I have some issues with post count, need to double check if it imported everything.

to help other users, do you mind posting your script that you used?

I renamed all the attachments using their physical_name and migrate is fetching the images and creating files in drupal. But they are not being attached to the posts or comments. When I check the migrate logs I get this error message:

Informational Undefined property: stdClass::$field_attach File /var/www/drupal/phpbb/sites/all/modules/contrib/phpbb2drupal/phpbb2drupal_core.inc, line 788(file: /var/www/drupal/phpbb/sites/all/modules/contrib/phpbb2drupal/phpbb2drupal_core.inc, line 788)

I created a file field for the Forum Topic content type and also added this existing field to the Comment fields. I'm so close, please anyone give me some ideas how to resolve this, thanks.

Hi. I am working on this now. You shouldn't need to rename the files. It should now copy using the physical file name and save using the real file name.

So I am roughly where you are. FWIW I am getting.

Informational Trying to get property of non-object File /Users/jfrench/Documents/workspace/drupal-7.15/modules/file/file.field.inc, line 220(file: /Users/jfrench/Documents/workspace/drupal-7.15/modules/file/file.field.inc, line 220)
48 Informational Undefined property: stdClass::$uri File /Users/jfrench/Documents/workspace/drupal-7.15/includes/file.inc, line 570(file: /Users/jfrench/Documents/workspace/drupal-7.15/includes/file.inc, line 570)

Changing issue to be D7 version of the module, I think that is what most active people are using now.

I have added a field_attach to forum comments and to forum posts and it seems to attach the attachments.

So I will change the install hook to add that to the content types.

I then need to work out how to get them to display inline. bbcode doesn't seem to recognize the [attachment] tag.

I tried latest dev release (14 november). I have some issues, I'll open new issue reports soon, one of them being WEIRD posts /post count/deleted user issues I never had before.

------------
About attachments (obliviously without media):
It copies the files and names them with the attach id to prevent duplicates, all automatically, that's good!

But I don't see them attached as you said.

I think I'm missing some files too, about 5 of 1800, but I'll double check before give false warning, could be orphans.

I see file_managed table is populated, but file_usage table is empty. IIRC (can't verify now) drupal 7 populates this table to attach images to posts/content.

Make a new clean drupal installation, write a forum post, attach an image from disk, then check those tables and you'll understand how it works. I'd like to help more but I'm not at my pc right now, surely I'll test this next days.

I GOT IT

(how to attach imported files to content and show them, without media module)
This done with a clean d7.17, migrate 2.5, and your phpbb2drupal 7.3.dev 14nov2012

THIS IS AFTER RUNNING THE MIGRATION, so these are the missing pieces on your code:

1) You MUST have a field in "forum topic" content type:

give it a name, I'd say call it attachments, type: file, widget: file.

Then give it:
Allowed file extensions, all those you can remember, we'll change this after the migration to limit if needed.
tick "enable description field" (I'll explain at #4 lower how to import descriptions from phpbb3),
number of values: unlimited
save settings

2) populate the "file_usage" table

next you have it to write some code to populate file_usage table. That table needs:

fid: File ID. (You already have this in the file_managed table, already populate from your module)
module The name of the module that is using the file. (file)
type The name of the object type in which the file is used. (node)
id The primary key of the object using the file. (node id, of the forum post where the attachment is)
count The number of times this file is used by this object.

3) populate the "field_data_field_NAMEOFTHEFIELDCHOOSEDATPOINT1 table"

when you get it to populate that, you have to populate too the table "field_data_field_NAMEOFTHEFIELDCHOOSEDATPOINT1" (I said call it attachments, so it would be field_data_field_attachments)

This needs:
entity_type The entity type this data is attached to (node)
bundle The field instance bundle to which this row belongs, used when deleting a field instance (forum)
deleted A boolean indicating whether this data item has been deleted (0)
entity_id The entity id this data is attached to revision_id The entity revision id this data is attached to, or NULL if the entity type is not versioned (node ID where attachment is)
language The language for this data item. (und)
delta The sequence number for this data item, used for multi-value fields, THIS IS IMPORTANT, SHOULD INCREASE IF MORE THAN ONE ATTACHMENTS FOR SAME NODE ID, so delta0 for first attachment, delta1 for the second and so on...
field_file_fid The file_managed.fid being referenced in this field. (file ID same as file_managed table)
field_file_display Flag to control whether this file should be displayed when viewing content.
field_file_description A description of the file. (THIS SHOULD IMPORTED FROM PHPBB3 TABLE, READ BELOW)

4) importing file description/captions from phpbb3 to drupal

about file descriptions/captions, call them how you like (those words under a picture)
You know that on phpbb3 and drupal when you attach a file to content you can add a description/caption to it...

Here some info to help to add this function:

- in phpbb3 attachment description can be found in table "phpbb_attachments", look at the "attach_comment" row

- in drupal7 that must be copied to "field_data_field_NAMEOFTHEFIELDCHOOSEDATPOINT1" table, "file_NAMEOFTHEFIELDCHOOSEDATPOINT1_description" row.

=======================================================
I tried manually to write some rows to attach some images and it works. But I can't write a line of code (sadly, must learn).

The only thing still missing is the inline thing. Didn't check that.

Thanks for the update. I'm looking at creating the file field as part of the install. You also need to add it to comments.

Once I have done that I'll work out populating the other tables.

Right, a few changes committed which should help this work. If you can start from a clean DB that would be great....

If not I have just committed an update function as well. so you can run update.php.

So it should import, copy and attach the files. It won't display them as yet. But you can change this by editing the field settings. Check "Enable Display field" and "Files displayed by default"

This is working for me (without media module) but it would be good to get some confirmation of this.

OK, I will try this in a few hours and will report here.
Thanks for the update.php thing, but I always prefer to start from clean db and new module release, to avoid mods I could have done in between, and so new potential issues. But I will test both ways if you want.

EDIT> sorry my mistake

Drat. Thanks for testing.

Does the file at http://localhost/sitename/files/2_13c48fe3ccfd04b1d091048c2fb215e2 resolve if you try and open in a browser?

Is your files directory writable?

Tables won't be populated if this error occurs.

sorry my fault, had renamed phpbb files directory and made a typo giving back its name :D
i have lot of folders of drupal with similar names and was watching the wrong instance.
give me 5 minutes and I'll try it

EDIT:

at a quick glance everything seems working fine!

even pics captions are there. I have a post count issue Ill'try to diagnose this night, must go out now.

thanks

As I said above, attachments are now working fine, fields are created, tables are populated.

First bug I found, I'll post here since is related to attachment migration:

- topic/post date is set to "date of migration" only on posts with attachments.
The topics/posts without attachments show the correct time.

In those with attachments BOTH "created" and "changed" rows on node table are set to migration date/time, so all posts/threads in forum are not in the correct order.

Just committed a fix which looks to resolve this issue. Had to manually tweak the DB after saving nodes and comments to keep the modified date correct.

Have made some improvements to this now. It should now import if using the media module too. It also is inserting a [media] tag, however this isn't quite working as expected. Will look into that some more and get a resolution before closing this as fixed.

This should now be working even with media module turned on. To see the attachments inline you will need to change your input filter and add the media filter.

hey nbz,
could you help me to run an update query to change [attachment] to
Basically I converted all crazy looking paths images from phpbb to regular image files.
So now in posts I have:

ex.[attachment=0]forum.jpg[/attachment]

Now if I run a query to replace
[attachment=0]  to <img src="/files/forum/
and
[/attachment] to "</img>

I want them to show in posts rather than being attachments.
I could dump database and use simple find and replace but that way I won't learn anything about SQL queries so please help

just brainstorming ideas, but maybe you want to create a filter instead of hardlinking the files?

Something like converting them to a format for inline module: https://drupal.org/project/inline

(I am not sure what the definitive future proof solution is, I think there were some plans to get something that allows custom image placement into Drupal 8 - I would guess a backport of that solution into drupal 7 would be the best way.)

Filter idea works great. customfilter module rocks
changed [attachment] to [img]
Now the only issue I have is I need to insert some php code to give full path to the image because bbcode won't show as images if the full path is not provided so I would need to add

[img]PhotoAug1703822.jpg[/img] - doesn't show image
[img]http://example.com/files/images/PhotoAug1703822.jpg[/img] - does show image

Any ideas? thanks again