Skip to content

Fix lexing of string literals ending in an escaped backslash#132

Merged
jg-rp merged 1 commit into
jg-rp:mainfrom
gaoflow:fix-lexer-escaped-backslash-string-literals
Jul 5, 2026
Merged

Fix lexing of string literals ending in an escaped backslash#132
jg-rp merged 1 commit into
jg-rp:mainfrom
gaoflow:fix-lexer-escaped-backslash-string-literals

Conversation

@gaoflow

@gaoflow gaoflow commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

The lexer's quoted-string patterns locate the closing quote with a single-character look-behind:

self.double_quote_pattern = r'"(?P<G_DQUOTE>(?:(?!(?<!\\)").)*)"'
self.single_quote_pattern = r"'(?P<G_SQUOTE>(?:(?!(?<!\\)').)*)'"

(?<!\\) cannot tell \\" (an escaped backslash followed by the real closing quote) from \" (an escaped quote), so for a name selector whose key ends in an escaped backslash the pattern runs past the real closing quote and a following selector then mis-pairs with a later quote.

The visible effect is that JSONPath produces a normalized path it cannot itself read back, which violates RFC 9535 §2.7 (a normalized path must select exactly its node):

>>> import jsonpath
>>> jsonpath.findall(r"$['a\\']['b']", {"a\\": {"b": 1}})
jsonpath.exceptions.JSONPathSyntaxError: unexpected token "'"

That $['a\\']['b'] is python-jsonpath's own generated normalized path for the node, so findall(match.path, doc) round-trips break.

The fix replaces the look-behind with the standard disjoint "unescaped char or backslash-escape" form, which is unambiguous and backtracking-free:

self.double_quote_pattern = r'"(?P<G_DQUOTE>(?:[^"\\]|\\.)*)"'
self.single_quote_pattern = r"'(?P<G_SQUOTE>(?:[^'\\]|\\.)*)'"

This is the same look-behind bug class fixed in #124 for the regex-literal pattern; that change left the two string-literal patterns untouched. The re_pattern regex-literal pattern is out of scope here.

tests/test_escaped_backslash_string_literals.py covers lexing/parsing of keys ending in escaped backslashes plus normalized-path round-trip and idempotence over several documents. Full suite passes (2793), ruff and mypy --strict clean.

@jg-rp jg-rp merged commit b01c7d1 into jg-rp:main Jul 5, 2026
14 checks passed
@jg-rp

jg-rp commented Jul 5, 2026

Copy link
Copy Markdown
Owner

Thanks @gaoflow 👍

@jg-rp jg-rp added the bug Something isn't working label Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants