perf(es/lexer): Use logos lexer as a sub-lexer

[kdy1]

Description:

• Define RawToken.

  • It should be declared in a standalone crate because it requires heavy code generation. logos generates a deterministic finite state machine that consists of lookup tables and jump tables.
  • In the future, RawToken will be renamed to Token and used directly by the parser. I need to investigate much more about API.
  • For performance, RawToken should be single-byte-sized.

• Adjust lexer to work on RawToken instead of char.

  • logos takes &str and generates Iterator<Item = RawToken>.
  • Some ECMAScript code cannot be lexed using FSM. logos provides callback API, and that's how we should handle ambiguous tokens.
  • The regex syntax of logos is a bit inferior to that of regex. In other words, even the regex is valid, the logos lexer may not generate matching RawToken. This is for performance, and it's documented here.

• Wrapper: logos::Lexer => RawLexer => Lexer => Parser

  • The callback API of logos is not enough for lexing ECMAScript. We have to wrap the logos lexer with our lexer and call logos lexer only if it's valid to do. Well, it's valid most of the time, but currently, we handle regex and template literals using a separate method.
  • I introduced RawLexer as a sort of buffer (based on peek_nth from itertools), but I found that it's a good place to have various lexing methods, so I added read_regexp. read_regexp uses another logos token definition, so it should be in the swc_ecma_raw_lexer crate.

• Fix tests

  • Currently, some of spans are wrong.
  • Currently, processed values of AST are simply filled with the raw value from the lexer. These should have the correct values. For example, Str has the value of \\\\ and the raw value of \\\\ for input string \\\\. But the value field should be \\ instead.

Related issue (if exists):

[changeset-bot]

⚠️ No Changeset found

Latest commit: c862038

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[GiveMe-A-Name]

We do not need maintain the cur_pos message in parser.lexer. How about get span from logos.lexer ? It's a better way to reduce ecma.parser.lexer complexity.

[kdy1]

I agree that it can be a better design. You would only need to add start_pos to those.