There is a use case where instead of "downloading" photos some associations would prefer that the images are "hotlinkekd" when displayed. Need to add this option in however there are a few considerations:

  • Need to determine if the getObject on the server supports returning "urls" instead of photo data
  • Need to determine a table schema to hold the urls for the hotlink data vs. storing a file object in the file schema won't be possible
  • Need to workout how images would be handled in views (do we need a new field_handler?)
  • I'm sure there are more considerations ...

Thoughts / Discussion?

CommentFileSizeAuthor
#60 ImageHotlink-1296680-60.patch10.53 KBbloomt
#57 Screen Shot 2015-09-02 at 8.18.37 AM.png84.54 KBbrighteridea
#53 image-url.patch19.09 KBjbrown

Comments

camidoo’s picture

camidoo commented

Any of you guys/gals out there in RETS land know of or have a server that allows or even prefers hot linking, i'm thinking that there may be some sort of url "pattern" or tokenized url string that we can come up with to solve this.

Just need to find someone or somehow to test out my theory.

webavant’s picture

webavant commented

I just emailed FBS (FlexMLS) to ask them for their policies and tips on hotlinking. My MLS has tutorials for their RETS implementation and they only mention getObject, not mentioning hotlinking at all. As far as I can tell, there is no field with the hotlink URL, and it's not one of the options for getObject types. I seem to remember having difficulty trying to construct a proper hotlink URL in the past.

I would love to hotlink, as it eliminates the possibility for a lot of errors, although I would insist on keeping the getObject option as well.

webavant’s picture

webavant commented

I got a response from a RETS support rep at FBS Data. Of course this only applies to Flex MLS.

We would indeed prefer that images be hotlinked instead of pulling the image file. Generally it would be possible to pull an image file URL by appending the "Location=1" parameter to your RETS photo query. This will instruct the RETS server to return the URL instead of the photo file itself.

webavant’s picture

webavant commented

Actually, this really eliminates any need for getObject in my case, as long as hotlinks aren't loading too slow.

camidoo’s picture

camidoo commented

but it would really through a wrench in the ability to use the different things you've been talking about in #1397392: Theming Entitities Without Writing PHP. I'll do some work on this soon.

webavant’s picture

webavant commented

Absolutely true. Definitely field the whole userbase. Even I'm not sure what I would prefer.

ralva83638’s picture

ralva83638 commented

Here's what my provider said, too bad, I like the idea of links much better:

"
My partner informed me today that we are getting hit with dozens of blank
queries from your userid. He sees regular queries requesting Commercial data
and then dozens of Rets requests with no query information. Please check
your program and see if you can stop this.
You had also asked about using us as a photo server and that is not the way
IDX is supposed to work. You will need to download all the images that you
are going to provide to customer sites.
Thanks,
"

camidoo’s picture

camidoo commented

yea, some mls's have a cdn that they put the images on, some don't support it at all, is pretty frustrating that there is no "standard" way of doing anything.

i'm pretty close to having a way to choose either linking to images or downloading them, and a way to check if linking is supported.

jday’s picture

jday commented

When the import is processing the images, does it always fetch all images from scratch or only fetch the new, update revised and delete old? The image processing stage of the import pretty much runs a good 20 hours or more for me...

camidoo’s picture

camidoo commented

wow, that's a long time to fetch the images.

What actually happens is this, on the 'first' run it will download all the images. Then, on subsequent runs it will delete all images for for any listing that has a change in it, so if the 'public remarks' field changes or any field changes, all images for that listing_key will be removed and that listing is marked that it needs it's images downloaded again.

What i'm finding is that after the initial import, if you setup a cron entry to run the drush import command like once an hour, or every two hours, there are typically only a handful of (2 or 3) listings that actually change, which means that there are only a (relatively speaking) few listings that are updated / imported on each run.

jday’s picture

jday commented

What exactly triggers the image retrieval? If the listing price is changed will the drush command re-fetch all the photos even though they have not changed? What if an agent does update the photos but does not make a change to the rest of the listing, will the new images be pulled?

camidoo’s picture

camidoo commented

if the hash changes the photo's will be re-downloaded. The hash is made up of all the fields combined into one long string and an md5 performed on the string:


$string = '';
foreach ($rets->field as $field) {
  $string .= $field->value;
}

$hash = md5($string);

so typically if a photo only is changed the photo modification timestamp will change, thus causing the hash to change and the images to be re-downloaded.

for now it works, however there is the side effect that if any field changes all the images are downloaded. Granted it's not the most elegant solution, however it is on the list of things that do need to be tweaked.

kevinquillen’s picture

kevinquillen commented

Some servers provide a way to hotlink to an image, some do not. For another property service we integrated with, I was able to use Media + Remote Stream Wrapper to facilitate this. There are probably a handful of ways to approach it, though.

kevinquillen’s picture

kevinquillen commented
Version: 7.x-2.x-dev » 7.x-3.0-alpha1
camidoo’s picture

camidoo commented
Status: Active » Postponed

Marking this as postponed for now, need to get to a stable release and i don't see this as a show stopper, however i think that it is a feature that we should implement eventually.

kevinquillen’s picture

kevinquillen commented

For what its worth, I recently gutted remote image support in another property integration module in Drupal 7 because it was killing performance with Views and Imagecache and Field Limiter.

kevinquillen’s picture

kevinquillen commented
Version: 7.x-3.0-alpha1 » 7.x-3.x-dev
Assigned: camidoo » kevinquillen
Status: Postponed » Closed (won't fix)

After working with this for quite a while, remote image support in Drupal is hard on site performance. For listings that have up to 25 images, it can take up to 30-60 seconds to load the page (image style generation) - longer if the file(s) aren't found or the server is not responsive enough. Remote stream wrappers are kind of a pain to work with as well.

bloomt’s picture

bloomt commented

I am using flexMLS and am not allowed to download images larger than 300 by 250px. In order to get larger images I must hotlink to them. According to their HOW TO I must use a query with a location=1 which will retrieve the photo url. I cannot seem to figure this out, anyone have any ideas?

bloomt’s picture

bloomt commented
Issue summary: View changes
Status: Closed (won't fix) » Active
rexington’s picture

rexington commented

I'd be very interested in being able to hot-link the images. My provider requires it -- I can't get any actual images returned as objects. Any chance this feature will be made possible?

kevinquillen’s picture

kevinquillen commented

Who are you using? I've noticed recently that new things have been coming through to my Innovia feed, like high res image fields and other things that can be used. Hot linking probably isn't that far fetched now.

rexington’s picture

rexington commented

This is MLSListings.com -- they're a SF Bay Area provider. There's three image objects I can grab, and they require hot-linking for all three: Thumbnail, Photo and HRPhoto. I sure hope their CDN is fast!

rexington’s picture

rexington commented

An additional note -- I think it's been covered earlier, but just to reiterate: there apparently is a simple way to get the getObject() to return a URL instead of an image. It's covered in Rets Technical Specs 1.7.2 section 5.4.1

webavant’s picture

webavant commented

I haven't looked at the getObject() code in Drealty, but I think it's already doing the necessary stuff to get the MLS URLs for hotlinking, it just needs to provide an option to store the image differently, whether by creating a new themable field widget, or perhaps by storing the URL in a field using the widgets provided by FileField Sources + Remote File Source. getting the image URL from the HTTP response headers before downloading them, because my MLS requires doing so in order to get the hi-res images that Drealty is downloading, so in that case, I believe that Drealty must also already be requesting the getObject URL with the Location=1 parameter, as described in the RETS spec.

rexington’s picture

rexington commented

It seems to me that it's actually not getting the URLs. I'm having trouble getting debugging output from PHRETS (won't write to the log file for some reason), but when I try to process images, it's just telling me that all of the listings "had no images" even though my query requires that they have at least two images in the image count field. And from my brief glance at the code, it doesn't *seem* that it's passing the "location=1" parameter.

x_v’s picture

x_v commented

I am using MLSlistings, and even though the import is working very well, none of the images are ever imported. This is the output I receive:

Listing MLxxxxxxxx had no images. Marking as processed.

My property image field is correctly mapped as well.

MLSlisting sent me this as a reminder:

Thank you for being a valued data customer of MLSListings. You are receiving this email because we have made a change to our download policy that requires a modification to your system access by January 31, 2014. Specifically, the change is that we now require data accounts to use image URLs, rather than downloaded images that are stored on your local servers.

We did a review in November and found that your RETS account is downloading image objects from the MLSListings RETS server rather than using the image URL. We are requesting that you make the switch by revisiting your “Get Object” procedure and updating it to use the image URL by using the setting “Location=1” in the “GetObject” request.

Will this require a change in dRealty deamon code?

Thanks for your help.

bloomt’s picture

bloomt commented

It does seem at though the module will need to process images in a different manner in order to use a location=1 request. Has anyone figured how to do this?

bloomt’s picture

bloomt commented
Priority: Normal » Major
webavant’s picture

webavant commented

Changing the issue priority seems appropriate here, since it seems to be affecting at least three people, although I'm not sure hotlinking is the exact issue here, or if this is a compound and related issue, or a separate issue altogether. Since it already appears to be working with FlexMLS, and although Drealty is storing the images and not hotlinking them in my case, I believe that Drealty must be using be using the phRets getObject() method to retrieve the images' URLs, which are the same URLs that should be used for hotlinking, according to the FlexMLS docs. I believe this would be the same for other MLSes, but I am not sure. I'd like to think I could test this and submit a patch for this, however I somehow doubt I'll get around to it as fast as some of the other members here.

osman’s picture

osman
Location Copenhagen
commented

Apparently there is a hidden sub-module in dRealty, "drealty_image"

The intention seems obvious, storing image URLs, but I don't think it is currently functioning.

Does anyone have any idea about the near future plans of this module?

In 11 days image downloads will not be available anymore. Thoughts?

bloomt’s picture

bloomt commented

webavant are you able to get images that are larger than 300 x 225 px from flexMLS? I have not figured this out yet and it is really slowing my development down.

bloomt’s picture

bloomt commented

I have noticed that line 141 of phrets.php specifies location = 0, I changed this to location = 1 which should make the query pull a url to the image instead of downloading it. The problem is that the drealty.daemon.php is still expecting to download the image. The code that gets the images from the rets server is at line 436 in the drealty.daemon.php file.

Any thoughts?

bloomt’s picture

bloomt commented
Title: Image retrieval Hot Linking vs. Downloading » Retrieve URL to photo instead of Downloading Image
camidoo’s picture

camidoo commented
Title: Retrieve URL to photo instead of Downloading Image » Image retrieval Hot Linking vs. Downloading
camidoo’s picture

camidoo commented

the problem isn't getting the url from the RETS server, if the RETS server supports it, it is fairly easy to grab the URL. The problem is, what to do with the URL after it's retrieved. The url is basically just a string, so, where does that string get stored, is it another 'field'? How is the field displayed? Since it is a "remote" image, you can't use any of the features in image ui / functionality built into core as it is a remote image.

So, as was decided long ago, it's a bigger issue than just "can we", and more of a question of "how complicated does it need to be to please the masses". Which then equates to 'who' is going to take on the lion's share of the work?

If any of you guys would like to submit a patch, or offer up some solid suggestions on how this should work outside of, modifying files that don't need to be modified ( phrets.php ), I'm sure that Kevin would be more than happy to take this into consideration.

@osman as far as the drealty_image sub-modulel found in the drealty module source, it's not really 'hidden', however it's there, to be honest i don't recall where i left off on that and no idea if kevin or his team has even touched it. However if i'm not mistaken, i created that module as an attempt to solve this problem, but i'm not sure what the reasoning for not finishing it was.

The problem that many don't understand is that there are a TON of moving parts when dealing with RETS and the many many many implementations of it. While there is a standard, it's not implemented the same from provider to provider, and from mls to mls, so when trying to wrangle all this into a 'project' that attempts to cater to the 'masses' there are inevitably areas where functionality is going to fall short for some.

My best suggestion is if this is functionality that you really need, or is a must have, either become an expert on the module and see if you can roll a patch and contribute that way, or contact kevin (at) inclindinc and see if he and his team are available to take on this functionality and fund their development, which will then be contributed back to the community. I'm sure they'd appreciate the gesture, and appreciation for all the work they have put into this project in the last several years and selflessly contributed back to the community.

kevinquillen’s picture

kevinquillen commented

Yes. I know things are quiet in here, but we've got a few real estate projects going on, in which I am putting time in solving some issues in the queue.

In our case, our RETS vendor doesn't allow hot linking. Secondary to that, I tried this with remote image stream wrappers for a rental service. When you get above 10 images, page load kind of takes a crap waiting for them to come down the pipe.

So we decided it was easier to just fetch and store them.

One thing that makes development of a universal RETS handler lethargic is that as camidoo said, different parts of the country leverage different RETS vendors, who use different versions of RETS from 1.5 and up. This creates inconsistencies and differences, like field names, field values, data types and so forth.

Anyway, if you could figure out how to hotlink images in a meaningful way, feel free to go for it and patch it. As stated, the only way I could think of is storing the URL as a link field type and theming it as an image, which could be a pain in the ass. How would you, for example, tell a link field to theme itself as a slideshow as easily as you can an image field? Or can Image now store URLs? I'm not really sure.

rexington’s picture

rexington commented

@kevinquillen and @camidoo, I see what you mean by this not being as simple as it may seem. Especially since it involves so many potential implementations of the standard.

Here's my general thinking on the solution -- I'm going to see if we can provide a patch that does this:

In the admin interface (where "process images" is), we include a toggle for "Grab Image URL Only" or some such. Then, for now, we just map this to a link or text field. We can deal with the problem of theming/rendering separately.

Thoughts/feedback?

osman’s picture

osman
Location Copenhagen
commented

@rexington, I like the idea of having a toggle option for using Image URLs vs. downloads.

Also perhaps Link Image Formatter's module or something similar could be used to format image urls to display as images.

osman’s picture

osman
Location Copenhagen
commented

FYI. I just received an email from MLSListings Inc.

all data feed customers (Vendors and Brokers) are going to be required to start requesting the photo URL’s only (Location=1 in your GetObject request) and no longer download photos directly.

kevinquillen’s picture

kevinquillen commented

Really - does that mean photos can no longer be downloaded and stored for anyone? That's kind of a pain in the ass to change on people like that.

rexington’s picture

rexington commented

Yeah, this vendor is making it a requirement. Can't even retrieve the images at all right now. We're working on making some (hopefully patchable) modifications to the module. I'll keep you guys posted.

osman’s picture

osman
Location Copenhagen
commented

@kevinquillen, here is the email I received. We really don't have much time left.

Is there any plan to enable image hot-linking? Or any previous discussion? I'd like to help where I can. But if there already is a plan, that would be helpful of course.

Thank you for being a valued data customer of MLSListings. You are receiving this email because we have made a change to our download policy that requires a modification to your system access by January 31, 2014. Specifically, the change is that we now require data accounts to use image URLs, rather than downloaded images that are stored on your local servers.

We did a review in November and found that your RETS account is downloading image objects from the MLSListings RETS server rather than using the image URL. We are requesting that you make the switch by revisiting your “Get Object” procedure and updating it to use the image URL by using the setting “Location=1” in the “GetObject” request.

We would like to make this transition as easy as possible for you, so we are here to help. Feel free to contact us at data@mlslistings.com with any questions or concerns you might have.

Regards,

MLSListings Data Support

bloomt’s picture

bloomt commented

Anyone have any luck with this?

rexington’s picture

rexington commented

We're working on it. We've got URLs being retrieved and we've added admin interface components for it. But we want to clean things up and make it something that can be cleanly patched in.

bloomt’s picture

bloomt commented

Any updates on the status of this patch?

bloomt’s picture

bloomt commented

Sorry to press the issue but I am curious if there has been any progress on this lately, or if anyone can give me any pointers to help me accomplish this.

rexington’s picture

rexington commented

We've had it working for a while on our server, but we don't really have a clean patch to submit. I'll see what we can do, but no guarantees of time or workability.

osman’s picture

osman
Location Copenhagen
commented

Are we going to see a DEV release at least sometime soon?

bloomt’s picture

bloomt commented

I would also love to see a patch for this, it would be nice to have a base point to start with instead of trying to make my own patch.

bloomt’s picture

bloomt commented

I guess I will make my own solution here, I am looking into the unfinished drealty image module that is hidden and unfinished. I will let you guys know where I get with this.

lauggh’s picture

lauggh commented

Hi guys,
Have you figured out a solution yet? I am stuck here as well. :)

Thanks!

jrcholtz’s picture

jrcholtz commented

Need help asap, in the above mentioned comment " it is easy to get the url of the images" could you please explain how to modify the code or whatever to how to do this? My MLS really screwed me over on this..

jbrown’s picture

jbrown commented
StatusFileSize
new image-url.patch19.09 KB

I was hired to replace the downloading functionality with hotlinking by Project6 Design for DeLeon Realty.

Here's the patch.

I'm available for further consultancy for this module.

x3appdev’s picture

x3appdev commented

CRMLS now requires using media urls too. They have it set up as a separate resource, class.(Media, Media) I am able to pull all of the urls in with drealty using a manually supplied rets query. Each media listing is stored as a separate drealty listing. My problem is how to combine/join each media listing with the same key to each other, implement the media order, and then how to reference them to the other property bundles with the same key value, but different system name for the field that stores them. If anybody can point me in the right direction I would really appreciate it. I've been bouncing around from tokens to relation to entity reference to computed fields and my brain feels like it is tied in a knot right now. If anyone has the same setup with image urls stored in a different resource, let me know if you need help querying to pull in the media listings. I did not have any luck with the patch in #53.

duntuk’s picture

duntuk commented

@jbrown Can you please explain how to setup drealty after applying your patch. I've tried it, and can't get it to work with drealty-dev-2013 (the latest last version; which I'm assuming you made the patch against) and drealty-beta-6 (current version).

Any input would help. Thanks.

brighteridea’s picture

brighteridea commented

I'm receiving an error when trying to save this. Has anyone gotten the patch to work

It is a standard Error Page

"Error - The website encountered an unexpected error. Please try again later."

brighteridea’s picture

brighteridea commented
StatusFileSize
new Screen Shot 2015-09-02 at 8.18.37 AM.png84.54 KB

So I figured it out and I hope this helps people.

I was getting the error in my database, according to my logs from the field "image_link_fields" in "drealty_classes" it wasn't long enough at 255 characters, I upped it to 2048. If you don't see this field it because it is created during the install process, so you will need to completely uninstall the module and reinstall it or create it by hand in the database structure.

After that I created specific link fields for every entry created.

shauntyndall’s picture

shauntyndall commented
Component: Code » Drealty Image
Assigned: kevinquillen » Unassigned
Status: Active » Needs work

Doing an issue queue audit. Seems like patch #53 needs a bit of work but could possibly resolve #2384463: GetObject method not supported. Can't pull images from CRMLS.org.

bloomt’s picture

bloomt commented

@jbrown your patch does not apply cleanly. I am using drealty-7.x-3.x-dev.

bloomt’s picture

bloomt commented
StatusFileSize
new ImageHotlink-1296680-60.patch10.53 KB

Here is an updated version of @jbrown patch from comment 53 that will apply to the latest dev version

This is my first patch so don't be surprised if its not totally conventional

bloomt’s picture

bloomt commented

Also I should mention that you will need to install the link module

shauntyndall’s picture

shauntyndall commented
Status: Needs work » Fixed

See: https://www.drupal.org/project/drealty/releases/7.x-3.0-rc1 & https://www.drupal.org/node/1951580#comment-11360001

The implementation of media objects changed completely with 7.x-3.0-rc1. Our tests encompassed all the use cases described (as best we could given access). I think the biggest challenge is the lack of documentation on the mapping of media objects. We are working on that.

If you are interested in trying the new approach on a project please open an issue and we'll try to address your specific RETS instance to ensure the module facilitates a reasonable solution for handling media.

bloomt’s picture

bloomt commented

Documentation would be very helpful, I cannot get RC1 to process media in any way.

shauntyndall’s picture

shauntyndall commented

See #2759473-4: Mark all Media Objects for update? - Broken for some guidance. We are trying to update documentation for the next release.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.