A very common configuration is to have a rule that sets the order to complete when "the order is first paid in full".
However, normal checkout completion stuff normally happens "on completing the checkout process".
However, per this helpful drupalcommerce.org post, when using Paypal WPS, the "order first paid in full" event (on IPN) may happen before the user redirects back to Commerce (and , which means that the "Completing the checkout process" is skipped and never fires.
This causes all sorts of pain, because we have generic things we want to happen on completing the checkout process (like creating a user for anon users, etc.)
Thanks to spoonbow for the detailed debugging over on drupalcommerce.org.
Workaround, if you run into this issue (from #62):
Test carefully, this may work or not in your individual environment based on your setup and rules!
1. DISABLE "Permit checkout completion logic to be invoked multiple times per order. " at admin/commerce/config/order
2. Import the following rule:
{ "rules_fix_commerce_checkout_complete_racecondition_ipn_callback" : {
"LABEL" : "Execute commerce_checkout_complete() if IPN came back before the order was completed (#1460964)",
"PLUGIN" : "reaction rule",
"WEIGHT" : "-10",
"OWNER" : "rules",
"TAGS" : [ "bugfix", "commerce_checkout_complete" ],
"REQUIRES" : [ "php", "rules", "commerce_payment" ],
"ON" : { "commerce_payment_order_paid_in_full" : [] },
"DO" : [
{ "php_eval" : { "code" : "watchdog(\u0027rules\u0027, \u0027Triggering commerce_checkout_complete() from rule \u0022rules_checkout_complete_if_ipn_came_back_first\u0022 for order ID: \u0022\u0027 . $commerce_order-\u003Eorder_id . \u0027\u0022 as IPN came back first.\u0027, array(), WATCHDOG_NOTICE, url(\u0027admin\/commerce\/orders\/\u0027 . $commerce_order-\u003Eorder_id));\r\ncommerce_checkout_complete($commerce_order);\r\n" } }
]
}
}
Comment | File | Size | Author |
---|---|---|---|
#54 | simulate_checkout.txt | 734 bytes | Mikechr |
Comments
Comment #1
rfayStudying this, I believe this is a commerce issue.
In Commerce_payment.module, we have this code. The code inside the "if" doesn't fire if the order status is 'completed'. So the correct rule never fires. Would it be possible to just change it to say
if ($order_status['state'] == 'checkout' || $order_status['state'] == 'completed')
?This seems rather fragile for the case that a user doesn't return from paypal even though they completed the purchase, too...
Comment #2
bago CreditAttribution: bago commentedWouldn't it be better to move the call to "commerce_checkout_complete" into the "commerce_order_status_update" so that it will be fired even if the status is changed from the "when the order is first payed in full" rule?
DC cannot expect people to return to the main website after the payment, because this often doesn't happen, so.. once we have the payment we should move on to the next step without waiting for user interaction. (UPDATED: I just tested this again after removing the rule that changed the order status on IPN and it seems other checkout completion rules are now fired)
This way the same events would fire also when an administrator administratively change an order status. Currently I can mark an order as "completed", but all of my rules won't trigger...
Comment #3
rszrama CreditAttribution: rszrama commentedIt seems to me this is a problem that can be solved through an alternate configuration: namely, you know the IPN is coming in, and we now see that changing the order status on "When an order is first paid in full" will prevent the IPN activated "When completing the checkout process" event from firing - thus opening the door for that event to never fire in the likely event that the customer doesn't return to the site. In this case, the order status update can just be moved from the "When an order is first paid in full" action to the "When completing the checkout process" event, and a condition can be used to check the balance of the order and ensure that payment was actually received.
As to the actual solution, I'm not sure. It could be we just need to change the checkout page before we save the transaction in the PayPal module and advise other modules to do the same. However, that would be contrary to the way on-site payment works with respect to those events, and I think it's better to keep them uniform. The other option is to trigger the "When completing the checkout process" event based on some alternate criteria. The conditions in that function that Randy pointed out are there to ensure this event doesn't get invoked twice on any given order; if we see the order is still in checkout, then we know it needs to be "completed." It could be we just depend on some alternate bit of data tucked away in the order object like we do for ensuring "When an order is first paid in full" only executes once.
However it works out, this isn't really a critical issue, so I've demoted it to normal. Until we solve the core issue, please use the alternate configuration I recommended above.
Comment #4
rfayUnfortunately, the alternate config that you suggest never fires for *anything* if the customer doesn't return to the site to complete the checkout process. So then they've *paid* and done everything we've asked of them, but because they didn't follow funky paypal links back to our site, they're left with a not-completed order. However, your proposed workflow is better than what I'm doing now. The condition would be "Order balance Comparison".
Edit: Ryan's alternate workflow then *only* works when the IPN comes in before "when completing the checkout process". If the race condition has the user completing before the IPN, again, we lose the ability to respond to the fact that the wonderful event has happened.
Comment #5
rszrama CreditAttribution: rszrama commentedIf the user beats the IPN back to your site, presumably "When completing the checkout process" has already fired before the IPN came. That event gets invoked when the user returns to the site and is forwarded on to the checkout completion page.
To summarize: whether the user returns or not, you will always have the "When completing the checkout process" event. It will either be triggered by the IPN - assuming you haven't changed the order status on "When an order is first paid in full" - or by the user advancing to the completion page. If this is your only payment method and the only way for people to access this event, then there's no harm in updating the order status on that event, assuming the order was paid in full, instead of "When an order is first paid in full."
Comment #6
rfayMy actual experience here now that I've changed it as suggested is that *some* orders (presumably the ones where the user does not return to the site) are being left in the Checkout:Payment state after successfully paying and after the IPN coming through. So I suspect that #5 may not be the whole picture.
So for users of Paypal WPS, I maintain this is a critical #fail.
Comment #7
rszrama CreditAttribution: rszrama commentedHmm, ok. It'd be good to figure out why these orders are not progressing. Perhaps you can setup another administrative e-mail or something that also sends an e-mail to you on the checkout completion event. This will let us know if the event itself isn't being triggered or if something is wrong with the Rule. For a counter-example, on RealMilkCheese.com we use PayPal WPS for payment and have never had an order get stuck in Checkout: Payment whether the customer came back to our site or not.
Comment #8
rfayI set up a notification email for "When completing the checkout process". We'll see if we can track this down.
Comment #9
rszrama CreditAttribution: rszrama commentedOk, great. I got to thinking to that we can probably confirm in the watchdog that the IPN was even received / verified and processed. It could be something actually prevented the processing or introduced an error in that process somewhere such that the loop was never completed. I'm not aware of any problems with the IPN handling code, but it's never something I feel I've "nailed."
Comment #10
rfayIt would be great to have that in the watchdog... But in this case I'm getting the payment completely updated by the IPN, so I'd say that probably the IPN does not necessarily cause "When completing the checkout process" to fire.
Comment #11
rszrama CreditAttribution: rszrama commentedHmm, I'm not sure what you're saying - you're not seeing IPN debug logs in the watchdog even with logging turned on but somehow they're still being processed? If you look at the top of the function commerce_paypal_wps_paypal_ipn_process() where PayPal WPS IPNs are actually processed, you can see that it's bailing out for unknown payment statuses. Perhaps your successful IPNs are submitting a status other than "Completed"? Is that something you can investigate in the debug logs?
Comment #12
rfaySo I've had one right and one not today;
The failing order (left in Checkout:Payment) had two IPN watchdogs, "IPN Validated" and then "IPN Processed". But the "When completing the checkout process" event did not fire. There were no errors or time-related entries in the watchdog around the time of the processed payment. I have to assume that these people just didn't come back from paypal.
Comment #13
rszrama CreditAttribution: rszrama commentedI'm 99.99% sure that's a red herring, because most of my customers don't come back and the code simply doesn't leave that option open (a processed IPN for an order on the Checkout:Payment status without a checkout completion event being fired). If you have a message in the watchdog that the IPN processed, the line right before that message in commerce_paypal_wps_paypal_ipn_process() calls commerce_payment_redirect_pane_next_page(). This is the function that moves an order forward if the order is still in the payment page status, the exact same function that will be called if the customer happens to return to the site before the IPN.
Can you confirm this code in your version of commerce_paypal_wps.module? The lines in question are 236-237. If they're different, feel free to post the function and we can take it from there. Otherwise you might want to put an additional watchdog() or two in those functions to track it down further. If you need a temporary fix, too, you can implement hook_commerce_paypal_ipn_process() in a custom module to update orders that got skipped for whatever reason.
Comment #14
rfayOuch. I had code from October 6, not sure how. Did drush dl fail me?
Anyway, that was before
So I'm betting that's going to be the thing here.
However, the OP here remains a key issue regardless of my experience from here, since we've been telling people from the beginning of time "Just update the order status when first paid in full"
Comment #15
rfayWith the (ahem) correct current code, just did a formal test with real payment and did not return to the originating site, and all worked out fine.
I really appreciate the help, and sorry for the facepalm.
IMO, the OP is a real and significant issue.
Comment #16
rfayJust a thought as I'm sheepishly trying to figure out how I had code that old. On this site I tried to use all stable releases, but apparently failed. I note that Commerce Paypal doesn't have one though. Couldn't it have an alpha release at least? That way update module would have informed me of the error of my ways.
Comment #17
VladimirAusMight help some people.
I tried to recreate "Completing the checkout process" rule from scratch to add extra functionality and it didn't work.
When I cloned the original rule "Send an order notification e-mail" and modified it, it worked no problems.
Sounds like a bug as two rules were exactly the same.
Comment #18
BD3 CreditAttribution: BD3 commentedThis is happening for me as well using Commerce 7.x-1.4 and Commerce Paypal 7.x-1.0-rc1 and not going back to the website. I tried patching with http://drupal.org/node/1130166#comment-4813116 and it still did not work.
More details: on the Paypal side of things, I did not pay with an account, only with a credit card as a Guest account. The order user is never created and the order stays in Checkout:Review.
Is this bug still out there? From the comments above it sounded like rfay was just using old code, but perhaps it wasn't committed?
Comment #19
BD3 CreditAttribution: BD3 commentedMy issue ended up being related with #1552478: After payment accepted, User redirected to payment review page. My mistake!
Comment #20
acb CreditAttribution: acb commentedConfirming this is happening still. Isn't there a simple way around this issue? Perhaps add another rule event?
Comment #21
rszrama CreditAttribution: rszrama commentedNope, you'd just need to change the way you're using checkout completion rules so the order status isn't being updated on "When an order is first paid in full" if the order status is still in checkout. Conversely, make it so the order status is being updated on "When the customer completes checkout" if the order balance is <= 0.
Comment #22
acb CreditAttribution: acb commentedThanks, as ever, Ryan. Is there any way we could get a rule export of what you describe in this thread? It might be a great help to many.
Comment #23
a.ross CreditAttribution: a.ross commentedSo if I understand this comment right:
...then making sure the order progresses to order complete (in the case the user doesn't trigger it) is the responsibility of the Payment module? I think this is an important issue, as other Payment modules clearly have a bug in this regard.
Comment #24
chefnelone CreditAttribution: chefnelone commentedSame problem here, the rule "Commerce order message: order notification e-mail" is not fired whatever I try.
Comment #25
micnap CreditAttribution: micnap commented+1 for #22. Could we get an export of the rule described in #21?
Comment #26
AlsI confirm that Ryan's suggestion at #3 re-explained at #21 works.
Using Commerce Kickstart (commerce_kickstart-7.x-2.6) with Paypal module 7.x-2.0-beta1
Edit: for some strange reason I initially referred to the wrong comments. Now corrected.
Comment #27
drugan CreditAttribution: drugan commentedHope it will be useful for you to review this post:
Add "Checkout complete page is viewed" event
Comment #28
xrampage16 CreditAttribution: xrampage16 commentedSame issue. I have multiple orders that have come on, and IPN did not report the data correctly. Now there are numerous orders which do not have payment posted, without a method to post payment after the fact.
A method to post payment should exist manually for such situations.
Currently am having to look up other methods of posting payments (COD)?
Comment #29
discipolo CreditAttribution: discipolo commentedLooks like it works for me if I enable the paypal express checkout module, even if i disable the rule and stick to wps
see https://drupal.org/node/2264291
or is this not the same issue?
Comment #30
sundersingh CreditAttribution: sundersingh commentedI had the same issue.
Disabling the rule to complete the order when first paid in full resolved the PayPal IPN problem.
Paypal Express Checkout was not needed for me.
Comment #31
vaccinemedia CreditAttribution: vaccinemedia commentedI'm also having the same issues and the emails are not being sent to either the customer or the store owner when adding them into the mix / emails which are sent. Ryan: can you post an example rule which you descried earlier for us please?
Comment #32
cornelyus CreditAttribution: cornelyus commentedOk,
So.. just to be sure.
Example 1. I have a rule that acts When an order is first paid in full. I change the order status of it.
IPN reaches first than user returns to site, so all rules from Completing the checkout process.
Hence, I don't send no e-mail to user, etc etc.
Example 2. I remove rule that acts When an order is first paid in full. So IPN reaches the website, but nothing happens.
User doesn't return to website with the link it should. My order will stay in checkout: payment status forever.
What's not the worst situation then?
Maybe i'm missing something, but it seems I'm always dependent of a user returning from the paypal with the link that's provided there, right? That's the only way Completing the checkout process will react.
I'll just ask, even if it sounds "dumb". What if I use the hook hook_commerce_payment_order_paid_in_full and inside the hook i'll just invoke the rules with event Completing the checkout process ?
Comment #33
cornelyus CreditAttribution: cornelyus commentedSo,
I went inspecting this, and one thing I wasn't truly aware. My dev environment can't receive IPN's, so I didn't know that Completing the checkout process event can be triggered by 2 things. The user returning to the checkout process, or the IPN is received. So the example 2 I referenced previously never happens.
Ok, reaching a better scenario. But, on that event the status order is only changed to "Pending", I didn't have yet a way of changing the status of the order, reflecting that the payment is accepted. I can't use When an order is first paid in full because it is fired first that event before.
A bit more of searching and this thread shed some more light on how everything works, which led into and consequently to
Bottom line, now I think I am more aware of the explanations given on #3 and #21. The trick, for now is really to combine two more rules for Completing the checkout process and When an order is first paid in full.
To refer that I added another condition on the When an order is first paid in full event, to see if order is not on cart, and if not is on checkout also.
Comment #34
vaccinemedia CreditAttribution: vaccinemedia commentedSo am I right in thinking that the default "out of the box" checkout rules which come with commerce kickstart never work unless the customer clicks on the link after paying with paypal to get back to the site? Also is the solution to this to edit all checkout rules to have the event "when first paid in full" rather than "on completing the checkout process"?
I'm only confused as the defaults don't appear to work and I just need to know what to edit in order for the notification emails to be sent out and anonymous account creations to happen etc...
Comment #35
funkeyrandy CreditAttribution: funkeyrandy commentedRE #34....so what is the answer here?
edit all checkout rules to have the event "when first paid in full" rather than "on completing the checkout process"?
will that do it?
Comment #36
vaccinemedia CreditAttribution: vaccinemedia commented@funkeyrandy I believe so. That's what I did and the client hasn't reported the issue anymore so I'm assuming everything is working ok now.
Comment #37
funkeyrandy CreditAttribution: funkeyrandy commentedhmmm i dont think so...my status is stuck in checkout:payment, so the order balance would never show paid in full...any ideas?
Comment #38
funkeyrandy CreditAttribution: funkeyrandy commentedim still having this issue....in my case the ipn status in paypal is stuck at "retrying" event thought the callback url is correct...pasting the address in the browser hits the ipn succesfully...any ideas??
Comment #39
vaccinemedia CreditAttribution: vaccinemedia commentedIs this a local dev version of the site? Paypal has to be able to "see" the website publicly in order to ping back the IPN data
Comment #40
funkeyrandy CreditAttribution: funkeyrandy commentedno its not...although telnetting to port 443 is taking like 10 sec....paypal says their threshold is 3....ALSO
Im seeing in the logs that the ipn address is getting hit, but there is no post data...any ideas?
Comment #41
quercus020 CreditAttribution: quercus020 commentedI'm having the same issues.
Everything works perfectly when using the Example Payment method but when I enable PayPal WPS the orders get stuck in the shopping cart page with status Checkout: Confirm order. Emails are not sent out to the order email and accounts for anonymous users are not created.
I have a fairly vanilla setup with commerce discounts and coupons.
Comment #42
quercus020 CreditAttribution: quercus020 commented...in fact none of my checkout rules were firing as the event 'completing the checkout process' was never reached.
My only workaround was by adding 'When an order is first paid in full' as the event trigger instead of 'Completing the checkout process' to every rule in the checkout rules set.
Comment #43
discipolo CreditAttribution: discipolo commentedchanging the rules event or adding a new rule on "order paid in full" didnt help me.
i also tried #1460964: "When completing the checkout process" never fires if a rule sets order complete on IPN
i cant help but think all this needs is a session set somewhere #1579948: Checkout page not found when order created through API for not authenticated (anonymous) user
Comment #44
Marko B CreditAttribution: Marko B commentedQuercus020 I have the same issue. Some orders over WPS just get stuck in "Checkout: Confirm order" Will need to debug what happens. IPN is on. I am assuming that maybe user never gets back to Drupal page and that is why order is not completed.
Comment #45
mtrax CreditAttribution: mtrax commentedSo it looks like this issue is stuck, so while we are waiting for a fix is there a method or script we can use to push these orders stuck in status "Checkout: Confirm order".
Other than editing each order one by one, is there a view or rule we can execute which pushes these out of the Shopping cart ?
Comment #46
Marko B CreditAttribution: Marko B commentedLook at mine posts here.
https://www.drupal.org/node/2036149#comment-9876161
https://www.drupal.org/node/2036149#comment-9887487
Comment #47
iwant2fly CreditAttribution: iwant2fly commentedI am also having this issue with orders that have a 100% off coupon so the order completes but there is no real payment information being processed.
Comment #48
AnybodyThis issue is still a big problem... will someone have a look at it again? How can we proceed?
Comment #49
lnittoli CreditAttribution: lnittoli commentedthere's no solution for me if user not go back to the site after paypal payment...
Comment #50
AnybodyI think we're talking about a general race condition problem in Drupal Commerce here for payment (provider) modules and their call to commerce_checkout_complete. additionally to the call in http://www.drupalcontrib.org/api/drupal/contributions!commerce!modules!p... which is not always guaranteed if the user does not switch back to the order complete step.
We're having a similar discussion in https://www.drupal.org/node/2424203#comment-10518584
@srzrama, could you perhaps have a general look if you had a minute? The problem pops up again and again in many DC projects...
Comment #51
AnybodyThis is still a major problem and I think if we'll ever be able to fix this, we'd also have to look at this as a result: #2609176: Add an order setting to disable multiple checkout completion invocations per order
Comment #52
hkovacs CreditAttribution: hkovacs as a volunteer commentedI wonder if this module will help with this issue. https://www.drupal.org/project/commerce_paypal_ipn_rules_integration
I have not used it, but seems like it might do the trick.
Comment #53
rp7 CreditAttribution: rp7 commentedNot tested on a live site yet, but I tested the https://www.drupal.org/project/commerce_paypal_ipn_rules_integration module (as mentioned in #52) and the first signs look good. Installed that module & added an extra event ("Process Paypal IPN") to the "Send an order notification e-mail" rule.
Comment #54
Mikechr CreditAttribution: Mikechr at Tabs & Spaces commentedI've managed to fix this issue by creating a rule that checks the order status if the paypal payment ipn comes back first and simulates the checkout process before firing the rest of the "paid in full" rules (change weight of rule accordingly currently it's set at "-1")
Comment #55
buerorezo CreditAttribution: buerorezo commentedI had the same problem when using the Mollie payment integration module. A similar approach as Mikechr's one is to listen to the update of the payment entity status in a custom module and set the order status to checkout_complete to fire the corresponding rules.
Comment #56
xaris.tsimpouris CreditAttribution: xaris.tsimpouris commentedFor a project of mine, I cannot accept for the checkout event to be triggered from PayPal as cookies must be utilized (affiliate network). To be honest, the asynchronous behavior of IPN makes everything messy, will it happen before checkout? after checkout? Maybe checkout complete should never be triggered from "When an order is first paid in full", but make (some) rules to be triggered by both "Completing the checkout process" and "When an order is first paid in full" in a way, only the first time will do the trick. For example "Assign an anonymous order to a pre-existing user" can be triggered by any of the above, but only the first time. Rule condition cuts off the second time.
For me it gets down to
commerce_payment_redirect_pane_next_page($order);
in filecommerce_paypal_wps.module
(commerce_paypal
module). This triggers "Completing the checkout process" AFTER "When an order is first paid in full" is triggered some lines above atcommerce_payment_transaction_save
. So I commented and changed rules to be triggered in a way that makes sense.Comment #57
sjmobley CreditAttribution: sjmobley at Centarro commentedComment #58
AnybodyClosing this as duplicate of #2036149: PayPal WPS not updating completing checkout to join forces, because I guess all these issues (see #2036149-33) are based on the same problem.
I you guess I'm wrong and this issue is not a duplicate, please feel free to reopen. Otherwise please help to finally fix that issue! :)
Comment #59
ckngThis is a DC issue, not Paypal specific, other payments are affected as well. Do not think it is a good idea to group them under a Paypal issue queue, this will just lead to more issues being created, IMO.
Comment #60
AnybodyI agree with ckng, see #2609176: Add an order setting to disable multiple checkout completion invocations per order and help fixing this.
Comment #61
a.ross CreditAttribution: a.ross commentedI don't really have an opinion on where this should be fixed, but maybe my old comment will give an extra insight: #1460964-23: "When completing the checkout process" never fires if a rule sets order complete on IPN
I should also mention that, iirc, I have patched payment modules used in a commerce site I built to include that extra line of code to advance checkout one step. I've never had checkout flow issues since I did that.
Comment #62
AnybodyAs I was the one who closed this as duplicate (of the PayPal issue), but now also experienced the issue with other payment modules, not only PayPal, I agree this seems to be a design issue in Drupal Commerce.
For that reason, I'm reopening this issue at Drupal Commerce. The race condition should definitely be handled by Drupal Commerce Checkout, whatever race condition happens with the IPN and if the user sees the checkout complete page or not (see comments above).
Now with #2609176: Add an order setting to disable multiple checkout completion invocations per order fixed and committed we're lucky as we have a workaround for this using custom code or rules. Still I hope this will be fixed in Drupal Commerce Core. Based on #54 you can do the following (I added this to the issue summary):
Workaround, if you run into this issue (from #62):
Test carefully, this may work or not in your individual environment based on your setup and rules!
1. DISABLE "Permit checkout completion logic to be invoked multiple times per order. " at
admin/commerce/config/order
2. Import the following rule: