Allow variable and tag end characters to be quoted.

[dylanahsmith]

@pushrax & @trishume for review

For Shopify/liquid#623

Problem

Shopify/liquid#170 tried to allow curly braces in variables, but was rejected for performance reasons. However, parsing is a lot faster now that we are using liquid-c, so I think I don't think performance would be as much of a concern now.

Solution

When encountering a quote character inside a variable or tag, the tokenizer will include everything up to the end quote character (if one is present) even if it includes the tag/variable end characters.

[pushrax]

I don't really like the idea of coupling the definition of a quoted string literal to both the tokenizer and lexer, but if we have to then let's use the same function in both places (i.e. simple logic around scan_past).

[dylanahsmith]

I don't really like the idea of coupling the definition of a quoted string literal to both the tokenizer and lexer

Well, a tokenizer is a lexer. The problem is that the tokenization was split into multiple passes originally. With strict parsing we could actually parse variables in the same pass, but lax parsing makes that more complicated.

if we have to then let's use the same function in both places (i.e. scan_past).

scan_past is a very thin wrapper around memchr which doesn't provide any benefit here. It is also awkward, since it makes the assumption that the cursor is on the quote character, and I found naming the return value more difficult (e.g. after_end_quote vs end_quote). I think using memchr is easier to read, since it is a standard API.

[pushrax]

{% " %} asdf {% " %} will change behaviour right?

[dylanahsmith]

Yes, unterminated quotes would be a behaviour change. Should we check existing templates to see how often this happens, and how much of an effect it would have on those templates?

[trishume]

👍

But you should definitely test it for breakage of existing templates