[Parent] Store more transaction specific information on Payment entities

IMO we can have a single field - additional_data where payment gateway plugins can store required information for later investigations.

I'm in general agreement here but going to convert this issue to serve as the parent for another approach. Basically, we saw the similar need to store payment transaction related results in a more structured format - e.g. AVS response code - but I didn't identify the need to store the current payment method state with the transaction - e.g. a stored credit card's current expiration date. There's a sense in which updates to the payment method are irrelevant, since the reason we maintain the association is to process subsequent transactions - i.e. issue a credit against a card, and if it happened to expire but was updated, we don't really care if the expiration date differs on the credit than the initial charge.

In any event, to get this started off, we'll focus on adding an AVS response code field to the Payment entity that can be used by those payment gateways that need it. I don't think we'll add a generic "additional data" array as proposed in the prior patch - if anything we'd create a general storage mechanism for API responses. For now, I'd like to focus on structured data that is meaningful to present in the UI for merchant decision makers, which is why we'll focus on AVS and not CVV to start, because a CVV failure will decline the transaction while an AVS response may trigger a manual merchant review process.

Hi Ryan,

Thanks for explaining the goal and it makes sense to have structured fields for some of the fields which we want to show to merchants.

Additional data here is mainly to store the information we had sent on a specific day and received back for the order/payment to allow investigations later. I've been working closely since last few years with Magento as backend and Drupal as frotend and I've observed that we become really helpless if we don't have such information and that was the reason I wanted to add it. Do you think it makes sense and/or it deserves separate ticket?

@nikunjkotecha Yeah, I've noticed the same thing. In Commerce 1.x we did have a base field for storing API responses, and it just got missed or descoped in 2.x for some reason. A lot of payment gateway integration modules include some option to log API requests / responses to the watchdog, but that's only good for moderate, timely debugging. I think we can do better than 1.x with a little more structure, though.

The latest idea we discussed was to make use of the core commerce_log module to create logs against the Payment. We don't typically log API requests (harder to keep secrets out of the database), but I'd envision us logging the fact that a request was made. That would just be a simple template identifying the type of API request and the URL (assuming it doesn't contain secrets). We could then have another template for logging the response and likely yet another for asynchronous messages (as in the case of PayPal Instant Payment Notifications).

This will of course be dependent on us getting a full page view for Payment entities, which we don't currently provide today. I thought we had an issue for that but am not currently finding it. We can make it if need be.

(Note: this is different from logging payment operations to the order itself as in #2845321: Add payment logging to orders. Logs against the Payment would be technical and governed by their own view permission, while the logs on the order are more about giving merchants the information they need in customer service, not necessarily debugging.)

That sounds interesting, I'll check and also try to contribute

Forgive me at the following stupid questions. "Dumb questions" hour?

I was looking a bit at various issues regarding payments and credit card transactions. One of the things that has appeared relatively clear (to me) is that there is a desire to not only store more transaction specific information on the payments, but also catalog things for investigatory purposes. That said, I have a somewhat different approach to the idea of the issue...

I have been dragged, kicking and screaming no less, into the idea that the receipt is the actual thing the payment is hinged upon. The payment is one side of the transaction, but the receipt is the other. In its purest form, the term "receipt" is the acceptance/receipt of a thing, an object, a widget, whatever. So, let me ask this:

Does there _actually_ need to be more information stored in the payment table, or payment method tables?

It appears (to me) the storage mechanism for the results of a processed credit card/crypto/whatever is _actually_ the receipt not the payment. The result of the transaction processed by the payment gateway is the receipt of funds. The receipt of funds occurs through a merchant id (gateway?), a transaction number (provided by the gateway, but specific to the receipt of that particular transaction), and potentially batch number or response codes... Those are all in connection with a receipt/response from the gateway.

If the gateway fails to process the transaction (response code indicating failure/no receipt?), then there probably is no reason to generate a receipt record. Am I way off base in this?

I mentioned on slack a few days ago there may be some merit to querying the commerce_payment table in connection with the receipt table, that was to suggest the legitimate payments are only those that are receipted. However, the receipt only occurs when something is received.

If the gateway fails for one reason or another, perhaps that is a separate type of record? (Not bundle, because the receipt _is_ _the receipt_.) But something different (entity) ... something to catalog the request as a failure, thereby not an actual receipt, but a forensic tool of some sort?

Is it possible there are valid receipts and invalid receipts? If that is the case, then bundles need to be classified as valid versus invalid, or... JOIN()s need to be created from payment against two separate entities (receipt and this other thing), with which ever entity holding the corresponding record is the result of the receipt, valid or invalid...

Hmm...

Does any of this make sense, or am I just wildly off in my procedural thinking? It is a complex problem, but there has to be a way to solve it.

Re-reading your question you keep referring to a receipt table. What is this? Is this some contrib module?

Yes. Commerce_Receipt. Nevermind, I thought there was a contrib module..

Receipts are a general contract/business process that acknowledges the parties, each, received ("receipt of") their ends of the agreed bargain (order). Commerce received its payment (as opposed to a promise to pay), and the customer received their products.

The abstraction of receipts allow them to be associated with Payments as well as a Shipment or other type of fulfillment.

The payment record is a promise to pay. It is usually then sent to a payment processor, (sometimes a physical cash register, other times an omline payment processor) which actually conducts (read: verifies) the financial transaction, and then a receipt is generated. (Authorization code, typically in combination with a merchant id and/or batch process number.)

The receipt for the transaction comes from the gateway saying "Yup, we're good" or "Nosirree, Bob. We have to try again for X reason."

The receipt of funds, unless it is cash, needs to be verified by a third party. That is largely to ensure protection on both sides of the transaction. The payor has the ability to stop payment, and the receiver has the ability to prove the payor got their end of the agreement, and thereby are required to release payment... That is the reason for third parties...receipt of the agreed upon terms, for both sides.

Organizations, in general, do not provide things up front on a promise to pay. There is a receipt that says the organization got its money, (the transaction receipt from payment processor) and the customer got their purchased products or services (receipt from shipper or other provider). It is a transaction: this for that.

All of that said, payments themselves are not technically receipts, hence the reasoning for looking at commerce_receipt as the appropriate place to store receipt information. (Batch codes, auth codes, etc.)

Since the payment method is not static and can change at any moment after the payment is recorded the information about the payment method can be lost.

That is the exact reason to store the info on the receipt. The payment method information can change, but the receipt of funds, and the delivery process (auth codes, response codes, etc) should not.

Please do not take any of this as combative, as it is not intended to be. It is just how I've come to understand the processes at play.

Please, argue with me! I say that, not in jest or sarcasm, rather as a way to truly help determine the best possible solution. Whenever I say "argue with me," it is out of respect for you and your knowledge, expertise, and experience. It is also a way for you to tell me, here's why you're wrong, or otherwise put me in my place. I can say, without hesitation, sometimes that's what I need.

The payment being the receipt does offer some beauty in its simplicity. At the same time, it does remove some possibilities.

The ability for organizations to do projected cash flow or forecasted liabilities is hinged upon the receipt record. Consider projected revenues based on...what? Projections cannot exist without some basis in reality. The reality is some payments are promised, but are not real until they're received. The receipt is what allows projections versus realized revenues or liabilities.

If the promise to pay is not a thing, then it simultaneously removes the ability to do things like purchase orders. It constricts an organization to always requiring the immediate receipt of funds to recognize a legitimate contract, even when the payment of the order is been contractually required, or previously agreed.

There are modules, like Invoice and Invoice Payment. It appears those exist to handle the aforementioned issues. The promise to pay versus the receipt of funds.

I need some more clarification here. Are you arguing that details about the payment method should not be stored on the payment? Or that a more robust receipt storage is missing in commerce?

TL;DR;TL;DR; In my view, the storage of payment process/receipt information should be separate from the payment itself. The payment signals intent, while the receipt indicates action.

TL;DR: I, personally, believe the receipt should be separate from the payment. I also believe there should be a more robust storage of receipts in Commerce. The reason, in my view, for the separation is that the receipt is both the acceptance and the offset record. It turns an asset into a liability and vice versa, depending on the View and Context. (View and Context used in Drupal's terminology.)

Let's say you're a small e-commerce site, and your niche is specialty products. Customer John Doe logs in and creates an order with a promise to pay because your specialized products are costly to make. You don't have a lot of inventory, and you don't know when you'll have access to more to fulfill Doe's order. The promise to pay is an asset in a forecast/projection model, but becomes a liability once it's accepted/receipted. Simultaneously, the specialty product sitting on the shelf is a liability until it's received by John Doe, then it becomes an asset. Those are the payment vs receipts in the context of a business.

It all depends on context and views. (No pun intended)

Many will say, "You've got that backwards, the things sitting on your shelf are assets!" My response is, "Are they?"

It seems to me, if the context is business then the product sitting on the shelf is a liability ... because it's been paid for/manufactured and it's still sitting on the shelf and not out being enjoyed by a buyer. If the context of that exact same situation is in appraisals (instead of business), then the products on the shelf are assets (because they've been paid for/manufactured), until they (heaven forbid) become liabilities upon receipt by an insurance company/adjuster covering a claim.

Now, before ripping me apart...

The goal of this issue is to store more information about the payment transaction on the payment itself (easy) instead of having to retrieve the information from the 3rd party payment processor (complicated).

There is beauty in simplicity. I mean that. No sarcasm. Simplicity makes it easier for small businesses to run, because there is less complexity. There is a reason Quicken and Quickbooks are a thing. It does not (necessarily) get into the nitty, gritty, nuance; it caters to the vast majority of use cases. That is 1000% understandable, laudable, achievable, and a directed goal.

I am not arguing Commerce should go one way or the other, as it is not my choice. I am simply trying to provide a little more information as to why complexity in the backend is sometimes necessary to garner the most market share, and provide the most business insight. That, should not translate to complexity in the front end.

If it is easy enough to use for mom-and-pop shops, but powerful enough for fintech and Fortune 50s... At the same time, if there are only 50 applications of the feature, it may not be worth it to incorporate.

In either case, _I_ would be hesitant to mess with the existing payment entity. (A happy medium may be another entity which handles the "Supplemental Payment Info," which holds the credit card processing info noted in the issue summary. It could be classified as a receipt, or an attempted receipt.) The presumption that the payment is the receipt removes the ability to promise to pay and forecasting/projections (both assets and liabilities). Projections and forecasting are often necessary to secure outside financing/investment, or simply budgeting based on knowing what contracts (read: commerce_order) are ready to go, but have not been accepted/receipted yet. The presumption of promise to pay not being a thing may not work for organizations that work in arrears. I don't know what percentage of organizations operate that way, but an educated guess would put it fairly low.

Edit: For those organizations that operate in arrears, their outbound payments may come prior to their incoming offset payments. In other words, those organizations may operate at a loss until there is receipt of counterbalancing funds to get back to baseline. The payment being the receipt makes it more difficult to track what has actually been receipted.

"The check's in the mail!" (That used to be a thing, and for some organizations still is.) Imagine an organization issued a check for something. It is upon the recipient's deposit of the check that a receipt record would be added to the payment sender's system saying "those funds are gone." (This is the process of reconciling a check register.) The receipt record would change the meaning of the payment (on the sender's side) from an outgoing asset into a paid-off liability.

When all is said and done, I may just be full of wind and so far off base that there is a necessary disregard for my commentary/noise.

I’ve applied the existing patch for Commerce 2.x and upgraded the site to Commerce 3.x. Is there a Commerce 3.x–compatible patch available, or has anyone started work on one?