`tokens`¶

Utilities for working with tokens.

class macro_polo.tokens.Delimiter(open_type, open_string, close_type, close_string)[source]¶

Represents a delimiter which must be kept balanced.

Parameters:

close_string: str | None¶

String representation of the closing token.

If None, only close_type is checked.

static from_token(token)[source]¶

Attempt to create a Delimiter from a Token.

matches_close(token)[source]¶

Check if the given token matches the delimiter’s close token.

matches_open(token)[source]¶

Check if the given token matches the delimiter’s open token.

open_string: str | None¶

String representation of the opening token.

If None, only open_type is checked.

class macro_polo.tokens.Token(type, string)[source]¶

Minimal representation of a token.

Parameters:

class macro_polo.tokens.TokenTree(*args)[source]¶

A delimited sequence of tokens.

macro_polo.tokens.desimplify(tokens, *, indent=' ')[source]¶

Revert simplifications made by lex() and fix indentation.

Only reverts simplifications that change semantics.

Parameters:

Return type:

Iterator[Token]

macro_polo.tokens.lex(source)[source]¶

Create a simplified token stream from source code.

Some simplifications are applied to make matching easier:

Semantically innert tokens, such as NL and COMMENT, are stripped.
NEWLINE-INDENT and NEWLINE-DEDENT pairs are reduced to INDENT and DEDENT, respectively. (This is reversed by desimplify().)
INDENT and NEWLINE tokens’ strings are normalized.
The trailing NEWLINE and ENDMARKER are stripped.

macro_polo.tokens.stringify(tokens)[source]¶

Construct source code from a token stream.

tokens¶