Problem/Motivation

Back on May 24, 2016, Twitter made a couple of announcements about upcoming changes to tweet formats:

Then on September 26, 2017, they made another one:

The first two were about doing more with the same 140 characters. The third one is about having up to 280 characters. In making those changes, Twitter updated their tweet object and added tweet "modes." The two modes are "Compatibility" and "Extended." The details of modes and the related changes to the objects can be found on the Tweet updates page.

Due to those changes, the creation of a number of seemingly unrelated issues in this queue about missing or incomplete tweet data have been created. Also, there have been situations where fixing the problem for one tweet mode breaks it or something else for the other.

The upshot of those changes for this module is that the code now needs to take into account the possibility that a tweet may be a "Classic" tweet in compatibility mode or an "Extended" tweet in extended mode, and the appropriate mode needs to be used to retrieve all the relevant data.

At some point in the future, Twitter plans on making these changes the de facto standard:
In the "Rendering modes" section of the Tweet updates page:

In the future, an additional announcement will be made when the time is right to make a change for the rendering mode to default to extended mode.

In the "Standard streams" section of the Tweet updates page:

In the future, a date for a switchover to extended mode will be announced, after which time apps should be able to process the newer Tweet payloads.

Until then, however, I do not think we can completely ignore the compatibility mode tweets.

Proposed resolution

Implement extended_mode tweets in this module.

Things to consider when implementing this:

  • Creating the relevant URL in create_url() in twitter.lib.php.
  • Updating the calls a Twitter API endpoint in call() in twitter.lib.php.
  • Accounting for tweet data in different places in the TwitterStatus constructor in twitter.lib.php (for 7.x-5.x). (This is in a differnt place in the 7.x-6.x branch.)
  • Possibly also accounting for tweet data in different places twitter_status_save() in twitter.inc.

I believe that if the data retrieved from Twitter can be placed in this module's twitter objects early enough on in the API call process, very little of the rest of the module will need to be touched, but that yet remains to be seen.

This will need to be done for both 7.x-5.x and 7.x-6.x branches.

Remaining tasks

  1. Fix the issue
  2. Create a patch
  3. Review the patch
  4. Commit the patch

User interface changes

To be determined.

API changes

To be determined.

Data model changes

To be determined.

Original report by iancawthorne

(Previous title: Sometimes Tweet images show, sometimes they don't)

I've tried with various accounts. With images attached to a Tweet in the format https://pbs.twimg.com/media/image-name.jpg, sometimes the image will be there, sometimes it won't. This seems to apply inconsistently for both Tweets and Re-tweets.

Checking the entities column of the twitter table, it looks like the image doesn't get added to here when it doesn't work. But it's there in [media] when it does.

As of yet, I've been unable to work out "why" it sometimes does not pull the image through.

CommentFileSizeAuthor
#25 twitter_implement-extended-mode-tweets_7x5x_2907103_25.patch5.1 KBoadaeh
#19 twitter_implement-extended-mode-tweets_2907103_19.patch4.73 KBoadaeh
#18 twitter_fatal-error-during-cron_2803293_18.patch779 bytesoadaeh
#17 twitter_fatal-error-during-cron_2803293_17.patch4.73 KBoadaeh
#8 show_tweet_images-2907103-8.patch880 bytesalex_daniels
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

iancawthorne created an issue. See original summary.

iancawthorne’s picture

iancawthorne CreditAttribution: iancawthorne commented
Issue summary: View changes
iancawthorne’s picture

iancawthorne CreditAttribution: iancawthorne commented

I have worked out what the problem is here. The api call needs to have "?tweet_mode=extended" added on the end.

This is found in:

function create_url()

This line:
$url = variable_get('twitter_api', TWITTER_API) .'/1.1/'. $path . $format;

Becomes:
$url = variable_get('twitter_api', TWITTER_API) .'/1.1/'. $path . $format.'?tweet_mode=extended';

Would the preferred outcome to this be to provide a patch as per above, or should there be a way to add parameters in a less hard coded way?

iancawthorne’s picture

iancawthorne CreditAttribution: iancawthorne commented

However, when "tweet_mode=extended" is specified, the tweet message itself becomes blank.

Melju’s picture

Melju CreditAttribution: Melju commented

When you change your URL to be "extended".
You have to change this line :

$this->text = $values['text'];

to this :

$this->text = $values['full_text'];

The text will show up again with the pictures this time ;)

patgibbs’s picture

patgibbs CreditAttribution: patgibbs commented

Has anyone managed to find a workable solution to this issue?

I'm dealing with a couple of organisations whose Twitter feeds contain a lot of visual material - it is critical that the organisation's website reflects this in its Twitter feed. However, the Twitter module in its current state only displays a fraction of the feed's images in what appears to be a random fashion.

iancawthorne mentioned above that the 'tweet_mode=extended' parameter was missing in the twitter.lib.php file (although this proved to be unreliable for other reasons as he mentioned). This thread states that the images are broken when links are included in the message body (although this has not proved to be the case in my experience, since several of our tweets that have included links also display their images).

I'm afraid I'm not adept at PHP enough to look into this myself, but I feel that this is a significant bug that needs addressing - images are a vital part of Twitter!!

Thanks in advance.

patgibbs’s picture

patgibbs CreditAttribution: patgibbs commented
Priority: Normal » Major
alex_daniels’s picture

alex_daniels CreditAttribution: alex_daniels commented
Status: Active » Needs review
FileSize
show_tweet_images-2907103-8.patch880 bytes

I've found a workable fix (although I should say I'm on 7.x-6.x-dev and not 5.x-dev) and created a patch for the 5.x-dev version using the same fix that worked on my end.

Hope this works for anyone who is still looking for a fix.

alex_daniels’s picture

alex_daniels CreditAttribution: alex_daniels commented

I've found a workable fix (although I should say I was working on 6.x-dev not 5.x-dev).

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Status: Needs review » Postponed (maintainer needs more info)

@iancawthorne and @patgibbs are there any error notices in watchdog (/admin/reports/dblog) when those tweets without images get pulled in?

iancawthorne’s picture

iancawthorne CreditAttribution: iancawthorne commented

No error notices in the logs no. I see three success notices in the logs, which are:

Finished downloading tweets.
Downloaded 20 tweets for the xxxxxxxx account.
Starting to download tweets.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Assigned: Unassigned » oadaeh
Status: Postponed (maintainer needs more info) » Active

I did some research into the extended tweet mode when I posted my last comment and I did quite a bit more tonight, and I agree with @iancawthorne that the problem is due to the change in tweet structure as outlined in this post: https://developer.twitter.com/en/docs/tweets/tweet-updates

The changes for supporting extended tweet modes will be more extensive than just those two lines, however, as several things in the structure of the tweet object got shifted a bit and multiple places in the code will need to be checked and verified they still work correctly.

I have begun work on it, and I would have gotten farther, but I wanted to see what the future for compatible tweets were, and I finally found it in the "Rendering modes" section of the document I linked to:

In the future, an additional announcement will be made when the time is right to make a change for the rendering mode to default to extended mode.

EDIT: I will be updating this ticket with more information about the actual scope of the issue later on.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented

I marked #2869088: Tweets doesn't display images when include links as a duplicate of this ticket.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Title: Sometimes Tweet images show, sometimes they don't » Implement extended_mode tweets
Version: 7.x-5.x-dev » 7.x-6.x-dev
Issue summary: View changes
oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Issue summary: View changes
oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented

I have already begun working on this, and I have updated both the create_url() and call() functions in twitter.lib.php for both branches. I have also updated the TwitterStatus constructor in twitter.lib.php for the 7.x-5.x branch, but I am still looking for where or how that is done in the 7.x-6.x branch, since the code is not the same. I have yet to look at anything beyond that.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Assigned: oadaeh » Unassigned
Status: Active » Needs review
FileSize
twitter_fatal-error-during-cron_2803293_17.patch4.73 KB

Here is a patch to implement extended_mode tweets on the 7.x-6.x branch. It also includes something similar to what I posted in #2803293-15: Fatal Error During Cron because I think it might be related.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
FileSize
twitter_fatal-error-during-cron_2803293_18.patch779 bytes

Here is a patch to implement extended_mode tweets on the 7.x-5.x branch. It also includes something similar to what I posted in #2803293-15: Fatal Error During Cron because I think it might be related.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
FileSize
twitter_implement-extended-mode-tweets_2907103_19.patch4.73 KB

Here's the real patch for 7.x-6.x.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
hosef’s picture

hosef CreditAttribution: hosef at Hook 42 commented
Status: Needs review » Reviewed & tested by the community

I tested the patch for the 7.x-6.x branch on my local environment.

The patch applied cleanly. The code changes looked good and the documentation changes were easy to understand. I was able to successfully download tweets from my twitter account.

Marking as RTBC.

andrey.troeglazov’s picture

andrey.troeglazov CreditAttribution: andrey.troeglazov at DrupalJedi commented
Assigned: Unassigned » andrey.troeglazov

andrey.troeglazov’s picture

andrey.troeglazov CreditAttribution: andrey.troeglazov at DrupalJedi commented
Assigned: andrey.troeglazov » Unassigned
Status: Reviewed & tested by the community » Fixed

Thanks, @oadaeh and @hosef for patch and testing, I have reviewed the patch it seems to be ok for me. I have commited it to 7.6 branch.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Version: 7.x-6.x-dev » 7.x-5.x-dev
Status: Fixed » Needs review
FileSize
twitter_implement-extended-mode-tweets_7x5x_2907103_25.patch5.1 KB

I also intended to attach a patch for the 7.x-5.x branch, but instead I attached one for the other issue. (Sorry for the confusion.)
The patch I am attaching to this comment is the correct patch to the 7.x-5.x branch to apply the same fix as has now been applied to the 7.x-6.x branch.

Status: Needs review » Needs work

The last submitted patch, 25: twitter_implement-extended-mode-tweets_7x5x_2907103_25.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented

The tests are failing because CTools is broken.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented
Status: Needs work » Needs review

andrey.troeglazov’s picture

andrey.troeglazov CreditAttribution: andrey.troeglazov at DrupalJedi commented
Status: Needs review » Fixed

Looks good for me, commited. Thank you.

oadaeh’s picture

oadaeh CreditAttribution: oadaeh at Hook 42 commented

Excellent. Thank you. Now we need to see how many open issues can be closed because of this. :)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Katy Jockelson’s picture

Katy Jockelson CreditAttribution: Katy Jockelson commented

Can you please confirm which branches (for D7) this has been committed to?
I'm not getting images or retweets with either the 7.x-6.2 or the 7.x-6.x-dev version.
I see the note at the top of the module page 'View embedded images media from tweets (not in 7.x-6.x branch yet).' - does that mean we should use 5x if we want to see images and retweets?

Thanks a lot

pixel6’s picture

pixel6 CreditAttribution: pixel6 commented

It works! Thanks for all.