A very common configuration is to have a rule that sets the order to complete when "the order is first paid in full".

However, normal checkout completion stuff normally happens "on completing the checkout process".

However, per this helpful drupalcommerce.org post, when using Paypal WPS, the "order first paid in full" event (on IPN) may happen before the user redirects back to Commerce (and , which means that the "Completing the checkout process" is skipped and never fires.

This causes all sorts of pain, because we have generic things we want to happen on completing the checkout process (like creating a user for anon users, etc.)

Thanks to spoonbow for the detailed debugging over on drupalcommerce.org.

Workaround, if you run into this issue (from #62):

Test carefully, this may work or not in your individual environment based on your setup and rules!
1. DISABLE "Permit checkout completion logic to be invoked multiple times per order. " at admin/commerce/config/order
2. Import the following rule:

{ "rules_fix_commerce_checkout_complete_racecondition_ipn_callback" : {
    "LABEL" : "Execute commerce_checkout_complete() if IPN came back before the order was completed (#1460964)",
    "PLUGIN" : "reaction rule",
    "WEIGHT" : "-10",
    "OWNER" : "rules",
    "TAGS" : [ "bugfix", "commerce_checkout_complete" ],
    "REQUIRES" : [ "php", "rules", "commerce_payment" ],
    "ON" : { "commerce_payment_order_paid_in_full" : [] },
    "DO" : [
      { "php_eval" : { "code" : "watchdog(\u0027rules\u0027, \u0027Triggering commerce_checkout_complete() from rule \u0022rules_checkout_complete_if_ipn_came_back_first\u0022 for order ID: \u0022\u0027 . $commerce_order-\u003Eorder_id . \u0027\u0022 as IPN came back first.\u0027, array(), WATCHDOG_NOTICE, url(\u0027admin\/commerce\/orders\/\u0027 . $commerce_order-\u003Eorder_id));\r\ncommerce_checkout_complete($commerce_order);\r\n" } }
    ]
  }
}
CommentFileSizeAuthor
#54 simulate_checkout.txt734 bytesMikechr
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

rfay’s picture

Project: Commerce PayPal » Commerce Core
Component: PayPal WPS » Payment

Studying this, I believe this is a commerce issue.

In Commerce_payment.module, we have this code. The code inside the "if" doesn't fire if the order status is 'completed'. So the correct rule never fires. Would it be possible to just change it to say if ($order_status['state'] == 'checkout' || $order_status['state'] == 'completed') ?

This seems rather fragile for the case that a user doesn't return from paypal even though they completed the purchase, too...

function commerce_payment_redirect_pane_next_page($order) {
  // Load the order status object for the current order.
  $order_status = commerce_order_status_load($order->status);

  if ($order_status['state'] == 'checkout' && $order_status['checkout_page'] == 'payment') {
    $payment_page = commerce_checkout_page_load($order_status['checkout_page']);
    $next_page = $payment_page['next_page'];

    $order = commerce_order_status_update($order, 'checkout_' . $next_page);

    // Inform modules of checkout completion if the next page is Completed.
    if ($next_page == 'complete') {
      commerce_checkout_complete($order);
    }
  }
}
bago’s picture

Wouldn't it be better to move the call to "commerce_checkout_complete" into the "commerce_order_status_update" so that it will be fired even if the status is changed from the "when the order is first payed in full" rule?

DC cannot expect people to return to the main website after the payment, because this often doesn't happen, so.. once we have the payment we should move on to the next step without waiting for user interaction. (UPDATED: I just tested this again after removing the rule that changed the order status on IPN and it seems other checkout completion rules are now fired)

This way the same events would fire also when an administrator administratively change an order status. Currently I can mark an order as "completed", but all of my rules won't trigger...

rszrama’s picture

Priority: Critical » Normal

It seems to me this is a problem that can be solved through an alternate configuration: namely, you know the IPN is coming in, and we now see that changing the order status on "When an order is first paid in full" will prevent the IPN activated "When completing the checkout process" event from firing - thus opening the door for that event to never fire in the likely event that the customer doesn't return to the site. In this case, the order status update can just be moved from the "When an order is first paid in full" action to the "When completing the checkout process" event, and a condition can be used to check the balance of the order and ensure that payment was actually received.

As to the actual solution, I'm not sure. It could be we just need to change the checkout page before we save the transaction in the PayPal module and advise other modules to do the same. However, that would be contrary to the way on-site payment works with respect to those events, and I think it's better to keep them uniform. The other option is to trigger the "When completing the checkout process" event based on some alternate criteria. The conditions in that function that Randy pointed out are there to ensure this event doesn't get invoked twice on any given order; if we see the order is still in checkout, then we know it needs to be "completed." It could be we just depend on some alternate bit of data tucked away in the order object like we do for ensuring "When an order is first paid in full" only executes once.

However it works out, this isn't really a critical issue, so I've demoted it to normal. Until we solve the core issue, please use the alternate configuration I recommended above.

rfay’s picture

Unfortunately, the alternate config that you suggest never fires for *anything* if the customer doesn't return to the site to complete the checkout process. So then they've *paid* and done everything we've asked of them, but because they didn't follow funky paypal links back to our site, they're left with a not-completed order. However, your proposed workflow is better than what I'm doing now. The condition would be "Order balance Comparison".

Edit: Ryan's alternate workflow then *only* works when the IPN comes in before "when completing the checkout process". If the race condition has the user completing before the IPN, again, we lose the ability to respond to the fact that the wonderful event has happened.

rszrama’s picture

If the user beats the IPN back to your site, presumably "When completing the checkout process" has already fired before the IPN came. That event gets invoked when the user returns to the site and is forwarded on to the checkout completion page.

To summarize: whether the user returns or not, you will always have the "When completing the checkout process" event. It will either be triggered by the IPN - assuming you haven't changed the order status on "When an order is first paid in full" - or by the user advancing to the completion page. If this is your only payment method and the only way for people to access this event, then there's no harm in updating the order status on that event, assuming the order was paid in full, instead of "When an order is first paid in full."

rfay’s picture

My actual experience here now that I've changed it as suggested is that *some* orders (presumably the ones where the user does not return to the site) are being left in the Checkout:Payment state after successfully paying and after the IPN coming through. So I suspect that #5 may not be the whole picture.

So for users of Paypal WPS, I maintain this is a critical #fail.

rszrama’s picture

Hmm, ok. It'd be good to figure out why these orders are not progressing. Perhaps you can setup another administrative e-mail or something that also sends an e-mail to you on the checkout completion event. This will let us know if the event itself isn't being triggered or if something is wrong with the Rule. For a counter-example, on RealMilkCheese.com we use PayPal WPS for payment and have never had an order get stuck in Checkout: Payment whether the customer came back to our site or not.

rfay’s picture

I set up a notification email for "When completing the checkout process". We'll see if we can track this down.

rszrama’s picture

Ok, great. I got to thinking to that we can probably confirm in the watchdog that the IPN was even received / verified and processed. It could be something actually prevented the processing or introduced an error in that process somewhere such that the loop was never completed. I'm not aware of any problems with the IPN handling code, but it's never something I feel I've "nailed."

rfay’s picture

It would be great to have that in the watchdog... But in this case I'm getting the payment completely updated by the IPN, so I'd say that probably the IPN does not necessarily cause "When completing the checkout process" to fire.

rszrama’s picture

Hmm, I'm not sure what you're saying - you're not seeing IPN debug logs in the watchdog even with logging turned on but somehow they're still being processed? If you look at the top of the function commerce_paypal_wps_paypal_ipn_process() where PayPal WPS IPNs are actually processed, you can see that it's bailing out for unknown payment statuses. Perhaps your successful IPNs are submitting a status other than "Completed"? Is that something you can investigate in the debug logs?

rfay’s picture

So I've had one right and one not today;

The failing order (left in Checkout:Payment) had two IPN watchdogs, "IPN Validated" and then "IPN Processed". But the "When completing the checkout process" event did not fire. There were no errors or time-related entries in the watchdog around the time of the processed payment. I have to assume that these people just didn't come back from paypal.

rszrama’s picture

I'm 99.99% sure that's a red herring, because most of my customers don't come back and the code simply doesn't leave that option open (a processed IPN for an order on the Checkout:Payment status without a checkout completion event being fired). If you have a message in the watchdog that the IPN processed, the line right before that message in commerce_paypal_wps_paypal_ipn_process() calls commerce_payment_redirect_pane_next_page(). This is the function that moves an order forward if the order is still in the payment page status, the exact same function that will be called if the customer happens to return to the site before the IPN.

Can you confirm this code in your version of commerce_paypal_wps.module? The lines in question are 236-237. If they're different, feel free to post the function and we can take it from there. Otherwise you might want to put an additional watchdog() or two in those functions to track it down further. If you need a temporary fix, too, you can implement hook_commerce_paypal_ipn_process() in a custom module to update orders that got skipped for whatever reason.

rfay’s picture

Ouch. I had code from October 6, not sure how. Did drush dl fail me?

Anyway, that was before

commit cb907d270df27674ae00596c1ec4237be8a21b6f
Author: rszrama <ryan.szrama@gmail.com>
Date:   Fri Nov 11 06:17:51 2011 -0500

    Update orders on incoming IPNs.

So I'm betting that's going to be the thing here.

However, the OP here remains a key issue regardless of my experience from here, since we've been telling people from the beginning of time "Just update the order status when first paid in full"

rfay’s picture

With the (ahem) correct current code, just did a formal test with real payment and did not return to the originating site, and all worked out fine.

I really appreciate the help, and sorry for the facepalm.

IMO, the OP is a real and significant issue.

rfay’s picture

Just a thought as I'm sheepishly trying to figure out how I had code that old. On this site I tried to use all stable releases, but apparently failed. I note that Commerce Paypal doesn't have one though. Couldn't it have an alpha release at least? That way update module would have informed me of the error of my ways.

VladimirAus’s picture

Might help some people.
I tried to recreate "Completing the checkout process" rule from scratch to add extra functionality and it didn't work.
When I cloned the original rule "Send an order notification e-mail" and modified it, it worked no problems.
Sounds like a bug as two rules were exactly the same.

BD3’s picture

This is happening for me as well using Commerce 7.x-1.4 and Commerce Paypal 7.x-1.0-rc1 and not going back to the website. I tried patching with http://drupal.org/node/1130166#comment-4813116 and it still did not work.

More details: on the Paypal side of things, I did not pay with an account, only with a credit card as a Guest account. The order user is never created and the order stays in Checkout:Review.

Is this bug still out there? From the comments above it sounded like rfay was just using old code, but perhaps it wasn't committed?

BD3’s picture

My issue ended up being related with #1552478: After payment accepted, User redirected to payment review page. My mistake!

acb’s picture

Confirming this is happening still. Isn't there a simple way around this issue? Perhaps add another rule event?

rszrama’s picture

Nope, you'd just need to change the way you're using checkout completion rules so the order status isn't being updated on "When an order is first paid in full" if the order status is still in checkout. Conversely, make it so the order status is being updated on "When the customer completes checkout" if the order balance is <= 0.

acb’s picture

Thanks, as ever, Ryan. Is there any way we could get a rule export of what you describe in this thread? It might be a great help to many.

a.ross’s picture

So if I understand this comment right:

[...] the line right before that message in commerce_paypal_wps_paypal_ipn_process() calls commerce_payment_redirect_pane_next_page(). This is the function that moves an order forward if the order is still in the payment page status, the exact same function that will be called if the customer happens to return to the site before the IPN.

...then making sure the order progresses to order complete (in the case the user doesn't trigger it) is the responsibility of the Payment module? I think this is an important issue, as other Payment modules clearly have a bug in this regard.

chefnelone’s picture

Priority: Normal » Major

Same problem here, the rule "Commerce order message: order notification e-mail" is not fired whatever I try.

micnap’s picture

+1 for #22. Could we get an export of the rule described in #21?

Als’s picture

I confirm that Ryan's suggestion at #3 re-explained at #21 works.
Using Commerce Kickstart (commerce_kickstart-7.x-2.6) with Paypal module 7.x-2.0-beta1

Edit: for some strange reason I initially referred to the wrong comments. Now corrected.

drugan’s picture

Hope it will be useful for you to review this post:
Add "Checkout complete page is viewed" event

xrampage16’s picture

Issue summary: View changes

Same issue. I have multiple orders that have come on, and IPN did not report the data correctly. Now there are numerous orders which do not have payment posted, without a method to post payment after the fact.

A method to post payment should exist manually for such situations.

Currently am having to look up other methods of posting payments (COD)?

discipolo’s picture

Looks like it works for me if I enable the paypal express checkout module, even if i disable the rule and stick to wps

see https://drupal.org/node/2264291

or is this not the same issue?

sundersingh’s picture

I had the same issue.
Disabling the rule to complete the order when first paid in full resolved the PayPal IPN problem.
Paypal Express Checkout was not needed for me.

vaccinemedia’s picture

I'm also having the same issues and the emails are not being sent to either the customer or the store owner when adding them into the mix / emails which are sent. Ryan: can you post an example rule which you descried earlier for us please?

cornelyus’s picture

Ok,

So.. just to be sure.

Example 1. I have a rule that acts When an order is first paid in full. I change the order status of it.
IPN reaches first than user returns to site, so all rules from Completing the checkout process.
Hence, I don't send no e-mail to user, etc etc.

Example 2. I remove rule that acts When an order is first paid in full. So IPN reaches the website, but nothing happens.
User doesn't return to website with the link it should. My order will stay in checkout: payment status forever.

What's not the worst situation then?
Maybe i'm missing something, but it seems I'm always dependent of a user returning from the paypal with the link that's provided there, right? That's the only way Completing the checkout process will react.

I'll just ask, even if it sounds "dumb". What if I use the hook hook_commerce_payment_order_paid_in_full and inside the hook i'll just invoke the rules with event Completing the checkout process ?

cornelyus’s picture

So,

I went inspecting this, and one thing I wasn't truly aware. My dev environment can't receive IPN's, so I didn't know that Completing the checkout process event can be triggered by 2 things. The user returning to the checkout process, or the IPN is received. So the example 2 I referenced previously never happens.

Ok, reaching a better scenario. But, on that event the status order is only changed to "Pending", I didn't have yet a way of changing the status of the order, reflecting that the payment is accepted. I can't use When an order is first paid in full because it is fired first that event before.

A bit more of searching and this thread shed some more light on how everything works, which led into and consequently to

Bottom line, now I think I am more aware of the explanations given on #3 and #21. The trick, for now is really to combine two more rules for Completing the checkout process and When an order is first paid in full.
To refer that I added another condition on the When an order is first paid in full event, to see if order is not on cart, and if not is on checkout also.

vaccinemedia’s picture

So am I right in thinking that the default "out of the box" checkout rules which come with commerce kickstart never work unless the customer clicks on the link after paying with paypal to get back to the site? Also is the solution to this to edit all checkout rules to have the event "when first paid in full" rather than "on completing the checkout process"?

I'm only confused as the defaults don't appear to work and I just need to know what to edit in order for the notification emails to be sent out and anonymous account creations to happen etc...

funkeyrandy’s picture

RE #34....so what is the answer here?

edit all checkout rules to have the event "when first paid in full" rather than "on completing the checkout process"?

will that do it?

vaccinemedia’s picture

@funkeyrandy I believe so. That's what I did and the client hasn't reported the issue anymore so I'm assuming everything is working ok now.

funkeyrandy’s picture

hmmm i dont think so...my status is stuck in checkout:payment, so the order balance would never show paid in full...any ideas?

funkeyrandy’s picture

im still having this issue....in my case the ipn status in paypal is stuck at "retrying" event thought the callback url is correct...pasting the address in the browser hits the ipn succesfully...any ideas??

vaccinemedia’s picture

Is this a local dev version of the site? Paypal has to be able to "see" the website publicly in order to ping back the IPN data

funkeyrandy’s picture

no its not...although telnetting to port 443 is taking like 10 sec....paypal says their threshold is 3....ALSO
Im seeing in the logs that the ipn address is getting hit, but there is no post data...any ideas?

quercus020’s picture

I'm having the same issues.
Everything works perfectly when using the Example Payment method but when I enable PayPal WPS the orders get stuck in the shopping cart page with status Checkout: Confirm order. Emails are not sent out to the order email and accounts for anonymous users are not created.
I have a fairly vanilla setup with commerce discounts and coupons.

quercus020’s picture

...in fact none of my checkout rules were firing as the event 'completing the checkout process' was never reached.
My only workaround was by adding 'When an order is first paid in full' as the event trigger instead of 'Completing the checkout process' to every rule in the checkout rules set.

discipolo’s picture

changing the rules event or adding a new rule on "order paid in full" didnt help me.
i also tried #1460964: "When completing the checkout process" never fires if a rule sets order complete on IPN
i cant help but think all this needs is a session set somewhere #1579948: Checkout page not found when order created through API for not authenticated (anonymous) user

Marko B’s picture

Quercus020 I have the same issue. Some orders over WPS just get stuck in "Checkout: Confirm order" Will need to debug what happens. IPN is on. I am assuming that maybe user never gets back to Drupal page and that is why order is not completed.

mtrax’s picture

So it looks like this issue is stuck, so while we are waiting for a fix is there a method or script we can use to push these orders stuck in status "Checkout: Confirm order".
Other than editing each order one by one, is there a view or rule we can execute which pushes these out of the Shopping cart ?

Marko B’s picture

iwant2fly’s picture

I am also having this issue with orders that have a 100% off coupon so the order completes but there is no real payment information being processed.

Anybody’s picture

This issue is still a big problem... will someone have a look at it again? How can we proceed?

lnittoli’s picture

there's no solution for me if user not go back to the site after paypal payment...

Anybody’s picture

I think we're talking about a general race condition problem in Drupal Commerce here for payment (provider) modules and their call to commerce_checkout_complete. additionally to the call in http://www.drupalcontrib.org/api/drupal/contributions!commerce!modules!p... which is not always guaranteed if the user does not switch back to the order complete step.

We're having a similar discussion in https://www.drupal.org/node/2424203#comment-10518584

@srzrama, could you perhaps have a general look if you had a minute? The problem pops up again and again in many DC projects...

Anybody’s picture

This is still a major problem and I think if we'll ever be able to fix this, we'd also have to look at this as a result: #2609176: Add an order setting to disable multiple checkout completion invocations per order

hkovacs’s picture

I wonder if this module will help with this issue. https://www.drupal.org/project/commerce_paypal_ipn_rules_integration

I have not used it, but seems like it might do the trick.

rp7’s picture

Not tested on a live site yet, but I tested the https://www.drupal.org/project/commerce_paypal_ipn_rules_integration module (as mentioned in #52) and the first signs look good. Installed that module & added an extra event ("Process Paypal IPN") to the "Send an order notification e-mail" rule.

Mikechr’s picture

FileSize
734 bytes

I've managed to fix this issue by creating a rule that checks the order status if the paypal payment ipn comes back first and simulates the checkout process before firing the rest of the "paid in full" rules (change weight of rule accordingly currently it's set at "-1")

{ "rules_simulate_checkout_complete_if_ipn_came_back_first" : {
    "LABEL" : "Simulate checkout complete if ipn came back first",
    "PLUGIN" : "reaction rule",
    "WEIGHT" : "-1",
    "OWNER" : "rules",
    "TAGS" : [ "simulate checkout complete" ],
    "REQUIRES" : [ "rules", "php", "commerce_payment" ],
    "ON" : { "commerce_payment_order_paid_in_full" : [] },
    "IF" : [
      { "data_is" : { "data" : [ "commerce-order:status" ], "value" : "checkout_payment" } }
    ],
    "DO" : [
      { "php_eval" : { "code" : "commerce_checkout_complete($commerce_order);\r\nwatchdog(\u0027simulate_checkout_complete\u0027, $commerce_order-\u003Eorder_id, array(), WATCHDOG_NOTICE, \u0027link\u0027);" } }
    ]
  }
}
buerorezo’s picture

I had the same problem when using the Mollie payment integration module. A similar approach as Mikechr's one is to listen to the update of the payment entity status in a custom module and set the order status to checkout_complete to fire the corresponding rules.

/**
 * Implements hook_entity_update().
 */
function HOOK_entity_update($entity, $type) {
  if ($type == 'payment') {
    if (isset($entity->statuses) && is_array($entity->statuses)) {

      foreach ($entity->statuses as $key => $status) {
        if ($status->status == 'payment_status_success') {
          $order = commerce_order_load($entity->context_data['order_id']);
          $order = commerce_order_status_update($order, 'checkout_complete');
        }
      }
    }
  }
}
xaris.tsimpouris’s picture

For a project of mine, I cannot accept for the checkout event to be triggered from PayPal as cookies must be utilized (affiliate network). To be honest, the asynchronous behavior of IPN makes everything messy, will it happen before checkout? after checkout? Maybe checkout complete should never be triggered from "When an order is first paid in full", but make (some) rules to be triggered by both "Completing the checkout process" and "When an order is first paid in full" in a way, only the first time will do the trick. For example "Assign an anonymous order to a pre-existing user" can be triggered by any of the above, but only the first time. Rule condition cuts off the second time.

For me it gets down to commerce_payment_redirect_pane_next_page($order); in file commerce_paypal_wps.module (commerce_paypal module). This triggers "Completing the checkout process" AFTER "When an order is first paid in full" is triggered some lines above at commerce_payment_transaction_save. So I commented and changed rules to be triggered in a way that makes sense.

sjmobley’s picture

Issue tags: +hold
Anybody’s picture

Status: Active » Closed (duplicate)

Closing this as duplicate of #2036149: PayPal WPS not updating completing checkout to join forces, because I guess all these issues (see #2036149-33) are based on the same problem.

I you guess I'm wrong and this issue is not a duplicate, please feel free to reopen. Otherwise please help to finally fix that issue! :)

ckng’s picture

This is a DC issue, not Paypal specific, other payments are affected as well. Do not think it is a good idea to group them under a Paypal issue queue, this will just lead to more issues being created, IMO.

Anybody’s picture

a.ross’s picture

I don't really have an opinion on where this should be fixed, but maybe my old comment will give an extra insight: #1460964-23: "When completing the checkout process" never fires if a rule sets order complete on IPN

I should also mention that, iirc, I have patched payment modules used in a commerce site I built to include that extra line of code to advance checkout one step. I've never had checkout flow issues since I did that.

Anybody’s picture

Issue summary: View changes
Status: Closed (duplicate) » Active

As I was the one who closed this as duplicate (of the PayPal issue), but now also experienced the issue with other payment modules, not only PayPal, I agree this seems to be a design issue in Drupal Commerce.

For that reason, I'm reopening this issue at Drupal Commerce. The race condition should definitely be handled by Drupal Commerce Checkout, whatever race condition happens with the IPN and if the user sees the checkout complete page or not (see comments above).

Now with #2609176: Add an order setting to disable multiple checkout completion invocations per order fixed and committed we're lucky as we have a workaround for this using custom code or rules. Still I hope this will be fixed in Drupal Commerce Core. Based on #54 you can do the following (I added this to the issue summary):

Workaround, if you run into this issue (from #62):

Test carefully, this may work or not in your individual environment based on your setup and rules!
1. DISABLE "Permit checkout completion logic to be invoked multiple times per order. " at admin/commerce/config/order
2. Import the following rule:

{ "rules_fix_commerce_checkout_complete_racecondition_ipn_callback" : {
    "LABEL" : "Execute commerce_checkout_complete() if IPN came back before the order was completed (#1460964 workaround)",
    "PLUGIN" : "reaction rule",
    "WEIGHT" : "-10",
    "OWNER" : "rules",
    "TAGS" : [ "bugfix", "commerce_checkout_complete" ],
    "REQUIRES" : [ "php", "rules", "commerce_payment" ],
    "ON" : { "commerce_payment_order_paid_in_full" : [] },
    "DO" : [
      { "php_eval" : { "code" : "watchdog(\u0027rules\u0027, \u0027Triggering commerce_checkout_complete() from rule \u0022rules_checkout_complete_if_ipn_came_back_first\u0022 for order ID: \u0022\u0027 . $commerce_order-\u003Eorder_id . \u0027\u0022 as IPN came back first.\u0027, array(), WATCHDOG_NOTICE, url(\u0027admin\/commerce\/orders\/\u0027 . $commerce_order-\u003Eorder_id));\r\ncommerce_checkout_complete($commerce_order);\r\n" } }
    ]
  }
}