Make our own token_scan() variant

This issue was split off from #1547160: Support variable substitution in direct input mode, but it was never updated/closed when further progress was made on #1547160: Support variable substitution in direct input mode

Specifically, in #1547160-33: Support variable substitution in direct input mode it was decided to handle the issue of the token_scan() variant in that issue, rather than in this one:

let's do it properly *now*

Likewise, in #1547160-36: Support variable substitution in direct input mode it was concluded:

No need for custom token_scan()

And further, the issue that motivated splitting this off was solved in #812058: add token support for non-entities without the need for a custom token_scan().

Those issues were solved long ago, so there's nothing that needs to be done here.