A very common mysql technique for joining against a table in an external DB is to prefix the table with the name of the DB. This technique does not work in DBTNG because it strips period from table names. An example of such a query is SELECT * FROM users u JOIN external.sales e ON u.uid=e.uid

This patch allows period for mysql tables. Other engines are unaffected.

CommentFileSizeAuthor
#8 escapeTable.diff461 bytesmoshe weitzman
escapeTable.diff713 bytesmoshe weitzman
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

chx’s picture

Status: Needs review » Reviewed & tested by the community

Looks good to me. Can't imagine why a dot would hurt other DBs but this way we know what we are doing.

Damien Tournoud’s picture

Status: Reviewed & tested by the community » Needs work

That will fail horribly if there is a default prefix. Not sure how to workaround that.

Crell’s picture

moshe weitzman’s picture

Title: Dynamic queries cannot join against external tables in Mysql » Over aggressive escaping of table names
Status: Needs work » Reviewed & tested by the community

Retitle to reflect the intent here. I am not trying to add a feature for prefix sites or anyone else. The driver is simple too aggressive without any benefit.

Crell’s picture

Status: Reviewed & tested by the community » Needs work

What about Damien's comment in #2? Just overriding the method without thinking of the impact on other weird things we're doing to the table name is a hack.

moshe weitzman’s picture

I did read #2 and discussed it with DamZ in IRC.

So, let me ask. Why does the mysql driver need to strip periods from table names? The only answer I am hearing is "We need to impede developers who don't use table prefixes from doing stuff that is impossible on sites that do use table prefixes." You can probably tell that I think this is going much too far.

Crell’s picture

If we're going to allow periods, shouldn't that at least be allowed for all drivers, not just the MySQL one? We want to make certain that we can get the behavior of all supported databases as close as possible.

moshe weitzman’s picture

Status: Needs work » Needs review
FileSize
461 bytes

Now allowing period for all drivers.

Crell’s picture

Er, wait. I just realized something I should have realized days ago.

The period needs to be OUTSIDE the table name. Prefixing happens on the table name, not the DB name. So if you're joining across databases, you should be doing this anyway:

db_query("SELECT * FROM {table1} INNER JOIN otherdb.{table2}");

Shouldn't you? Is there a detail here I'm missing?

moshe weitzman’s picture

The problem comes in dynamic queries where table names get escaped. I don't see how your comment is applicable there. Consider:

db_select('external.users)->execute();. That period gets stripped today.

Crell’s picture

Ah! Hm. Well, that's going to be a problem anyway if there's a prefix defined, because it will prefix the DB name, not the tablename, wouldn't it?

DBTNG was not designed for cross-DB joins in part because that's a MySQLism AFAIK, and very much an edge-case.

If we're going to allow periods in table names, then the patch in #8 looks correct with the caveat that it will still break with prefixing. I'm torn on whether or not that's a deal-breaking caveat.

I'd prefer to get input from the rest of the DB team here.

moshe weitzman’s picture

I completely agree that DBTNG was not designed to support periods in table names. Even after this is committed, it won't be a supported feature. We don't need to actively prevent it either.

aaron’s picture

Considering: db_select('external.users)->execute(); I really don't see, considering {prefixing} how it would be fixed -- if you had, say a prefix of mydb_, that would be turned into "SELECT ... FROM mydb_external.users ..." I think that it would be better to somehow allow external db's; perhaps like db_select('users')->external('external')?

However, in that case, an external db might not want to have prefixes anyway. I think the whole thing is opening a big can of worms, and needs to be thought out in more detail. Committing this patch would achieve absolutely nothing, IMHO.

moshe weitzman’s picture

@aaron. your reply suggests that you think we are trying to add a feature here and that feature is poorly implemented. Thats not I intend.

Lets turn this around. Code should exist for a reason. Why does the code for stripping periods need to exist?

Crell’s picture

Well just to confuse the issue, I checked through the CVS records. And I found no period. I just did a cvs up, and the escapeTable() in database.inc has no period. In fact I found no record of a period in the CVS logs anywhere.

Someone please explain to me why I am going rapidly insane, because I don't know how that's possible if #8 above is passing the bot.

Frando’s picture

-    return preg_replace('/[^A-Za-z0-9_]+/', '', $table);
+    return preg_replace('/[^A-Za-z0-9_.]+/', '', $tabl

The patch in #8 is adding the dot to the characters that are not stripped.

Crell’s picture

OK, so I am just going crazy. That's what I get for trying to do 5 things at once this morning. *sigh*

In that case, the current whitelist has been the same going back to 4.7. Before that it doesn't seem to exist. I couldn't tell you why that was selected originally. For DBTNG I just carried that escaping logic forward as-is.

I suspect it was to avoid "whoops, wrong database!" problems, since a period is NOT a valid character in a table name.

moshe weitzman’s picture

I have never seen any sanitizing of table names before DBTNG landed. We only used placeholders in WHERE. So this particular escaping does not go back to 4.7 AFAICT ... Now I feel like the one going crazy.

catch’s picture

Crell is referring to db_escape_table() no?

http://api.drupal.org/api/function/db_escape_table

If so, that was only applied to four functions in D6, not to db_query() or update_sql().

Dries’s picture

Looks like this might be a small design flaw in the API?

I'll leave it up to Crell to sign off on the final fix.

catch’s picture

Status: Needs review » Reviewed & tested by the community

Maybe Crell will take another look if I mark this RTBC. We've confirmed that this is a new restriction in D7, given the old one didn't apply to select queries at all. I'm not sure what else there is to do here.

Crell’s picture

Since I don't know what the original logic was for db_escape_table() in the first place, I'm OK with this unless someone from the security team sees a reason not to.

(Sorry, this patch fell off my radar for a bit.)

Dries’s picture

Status: Reviewed & tested by the community » Fixed

Committed to CVS HEAD. Thanks.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.