We're having a lot of trouble getting the normalization of data and execution of queries right.
First we fixed bugs in #3224085: XeroListNormalizer does not use XeroNormalizer, #3156893: XeroQuery with 'get' method is fatally broken unless you explicitly call setFormat('xml'), and #3102407: Support partial updates to Xero items.
But we missed, or introduced new ones: #3227619: Clarify the uuid requirement for POST queries, #3231251: When requesting Invoices or BankTransactions, Contacts data is cut. and #3227991: Incorrect XML for PUT/POST queries containing multiple data items.
The codebase in XeroQuery and the normalizer would be easier to understand if we only had to support json and ditched support for xml. Xero doesn't even provide documentation or substantial examples that I can find on the XML api, so it's extremely hard to figure out.
Users of this module should be using our API, not providing us with raw xml or json to throw at Xero. Let's assume we own the data format that is being used and drop support for xml.
Issue fork xero-3233375
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
mradcliffeAgreed. It looks like that's the direction Xero has been moving in.
Initially everything was in XML for Xero on the documentation. So it was easy to see and use.
Then the docs swapped over to JSON, but you could still switch it to XML output.
Now it seems XML is only mentioned a few times in the docs and it's not possible to do it without actually using the Api.
The main difference between the two is that the extra wrapping layer in XML.
which becomes
JSON doesn't have this:
which becomes
Additionally the end result from XML also changed form based on whether there was 1 or more element so
becomes
Comment #4
jonathanshawI've stripped out the XML handling everywhere except XeroQueryCacheTest and XeroQueryExecuteTest. To fix those, we need an example of a full json response.
Comment #5
jonathanshawXeroNormalizer::deNormalize() was heavily assuming xml. I suspect this might be why we have struggled with using json recently. I've attempted a fix.
Comment #6
jonathanshawThe XeroQueryCacheTest unit test was failing for reasons I wasn't clear about, so I replaced it with a much simpler kernel test in XeroQueryExecuteTest. At the same time, I noticed that XeroQueryExecuteTest was getting a xero_user response to a xero_contact query, so fixed that.
Comment #7
jonathanshawI suspect the test fail in XeroDeserializationTest is a genuine fail indicating a bug.
Comment #8
jonathanshawOK, the test fail turned up an unrelated bug I've opened a new issue for #3242097: Allow deserializing readonly properties.
I think this is now ready.
We will probably need a follow-up to address deserialization of json date values, as these are untested and may not be working.
I'm not sure how Drupal will handle:
"UpdatedDateUTC": "\/Date(1615463133377+0000)\/",To compound the problem, our data types sometimes specify datetime_iso8601 and sometimes string. I seem to remember we've also hit problems with the Drupal datetime_iso8601 expecting an explicit timezone but Xero implying UTC.
We may need to develop our data type e.g.
xero_iso8601that can accomodate our needs, use it consistently on all date properties, and then custom deserialize these properties to digest Xero's numeric json date values. Or maybe we just switch all our typed data to use timestamps.Comment #9
jonathanshawOn reflection, I suggest we fix the dates here. Otherwise HEAD might be rather broken.
Let's fix it by deserializing any date values into DrupalDateTime objects and calling
$item->setDateTime($date). This will work with any typed data property that implements DateTimeInterface.Here's how we parse dates:
per https://stackoverflow.com/a/56380900
Comment #10
mradcliffeYeah, fixing dates is probably important.
I committed some test-only fixes to the merge request, but the changes look fairly straightforward so far.
I found that XML is still supported by Xero officially and XML is the default response type still - https://developer.xero.com/documentation/api/accounting/requests-and-res.... But they've literally removed every other option for testing XML.
Comment #11
mradcliffeChanging status to Needs work for the date stuff.
Comment #13
jonathanshawI've added a sane & straightforward way of handling Xero dates.
(This added even more complexity to the tangle that is XeroNormalizer::denormalize() so I opened #3245232: Refactor XeroNormalizer::denormalize into multiple services.)
However, what I've done may not work because our typed data's date handling is not sane and straightforward. For example, InvoiceDefinition is doing this:
We're not consistent in using date definitions on date properties.
Getting an invoice according to Xero docs returns things like this:
Although in some cases in the invoice getting docs I also see
Posting invoices looks like this:
(DueDateString is a weird poorly documented convenience alternative I think, there is no comparable ExpectedPaymentDateString).
It looks like our current code could be use either string or datetime_iso8601 as the definition for date properties, because Xero's xml would return a string time format that was straightforwardly understood by both string and datetime_iso8601. And we could post xml "DueDateString" with value "2009-06-06T00:00:00" and Xero would be OK.
I need to check whether Xero will magically understand
"DueDate": "2009-06-06T00:00:00". If it doesn't, then I see no alternative but to get more consistent about how we define date properties so we know when Xero is going to expect us something formatted in a special way. This could be a BC break.Comment #14
jonathanshawOK, this is going to be easier than I feared. Xero allows us to post in either date format.
Therefore we don't need to worry about normalization of dates at all, we can continue to use the existing output of our string/datetime_iso8601 properties.
And therefore for denormalization all we have to do at this point is detect examples of Xero's weird json Date format and convert it into an iso8601-like string of the kind the XML api has been giving us.
We can leave our data types untouched for now.
Comment #15
jonathanshawThe last commit significantly refactors XeroNormalizer::denormalize().
This has become necessary because the remaining test fail indicates ou denormalization of nested json items is failing. I believe this i because the denormalizer was refusing to denormalize anything that wasn't wrapped in an array keyed by plural name, such as the invoice contact in a json response.
Comment #16
jonathanshawThis is ready now I think.
Comment #17
mradcliffeOkay, let's do it. Publishing the change record after this comment.