In #2552735: Add endpoint validation we added endpoint validation, but not surprisingly there are some edge cases that our approach there doesn't cover.

At the moment I have an endpoint for which the metadata API returns a valid response, but the actual SODA2 API returns a 500 error:

So we need to tune our approach to validation.

CommentFileSizeAuthor
#6 socrata-validate-2689575-6.patch2.05 KBarh1
#4 socrata-validate-2689575-4.patch802 bytesarh1

Comments

arh1 created an issue. See original summary.

arh1’s picture

arh1 commented
Issue summary: View changes
papagrande’s picture

papagrande
Primary language English
commented

Might be worth pinging Socrata on that one if their APIs return different data.

arh1’s picture

arh1 commented
Status: Active » Needs review
Related issues: +#2552735: Add endpoint validation
StatusFileSize
new socrata-validate-2689575-4.patch802 bytes

Let's just validate against a (limit 1) SODA2 API request. If there was a reason we didn't do that the first time we looked at validation, I'm not sure what it was :)

Patch attached.

papagrande’s picture

papagrande
Primary language English
commented
Status: Needs review » Needs work

I was able to put in https://data.seattle.gov/api/views/rhf2-u5fm.json as an endpoint and it validated, but it shouldn't. The endpoint then doesn't show up on the view content list.

arh1’s picture

arh1 commented
Status: Needs work » Needs review
StatusFileSize
new socrata-validate-2689575-6.patch2.05 KB

I was able to put in https://data.seattle.gov/api/views/rhf2-u5fm.json as an endpoint and it validated, but it shouldn't.

Ah, good point. That's a separate case, but also important to catch. We should ensure we only allow SODA2 API URLs when we validate.

The attached patch just does some string-matching for that. I'm not sure if there's a better way to check...

I also tweaked wording of the validation error messages, as well as the URL form field help text.

  • PapaGrande committed f2525ee on 7.x-1.x authored by arh1 
    Issue #2689575 by arh1: Fix endpoint validation to cover edge cases
papagrande’s picture

papagrande
Primary language English
commented
Status: Needs review » Closed (fixed)

Looks good. Committed.

I took out your comment

// @todo: Is there a better way to check this?

because according to https://dev.socrata.com/docs/endpoints.html, that is the correct way to format the URL.

All resources are accessed through a common base path of /resource/ along with their dataset identifier. This paradigm holds true for every dataset in every SODA API. All datasets have a unique identifier - eight alphanumeric characters split into two four-character phrases by a dash. For example, ydr8-5enu is the identifier for the Building Permits. This identifier can then be appended to the /resource/ endpoint to construct the API endpoint.

papagrande’s picture

papagrande
Primary language English
commented
Status: Closed (fixed) » Needs work

I'm reopening this because I'm able to put in https://data.seattle.gov/resource/55n4-ddnu.json, which comes from https://data.seattle.gov/Transportation/Downtown-Seattle-Bike-Racks/55n4... API, as a valid endpoint. However, that is a map and not a dataset so a query to https://data.seattle.gov/resource/55n4-ddnu.json returns an empty array. We could test for that, but it would be nice if the Socrata metadata API offered an easier way to validate that an endpoint is indeed a dataset with valid data.

papagrande’s picture

papagrande
Primary language English
commented
Status: Needs work » Closed (fixed)

Moved the latest issue to #2700071: Endpoint Validation Fails on Map Endpoints since the earlier patches are already committed.