plrs

#

The multi-tool of lexical analysis and tokenization.

Install #

pip install plrs

Build #

maturin build

Examples #

• zipfs-law

API #

Global variables #

EOF_TOKEN

Classes #

Tokens
Settings
Token
  - part
  - token
  - set_part
  - set_token
  - __str__
  - __repr__

Lexer
  - new
  - char_forward
  - skip_over_char_set
  - next

Functions #

is_char_symbol
is_char_operator
is_char_whitespace
is_char_numeric
is_single_quote
is_double_quote
ends_token
is_part_numeric
tokenize