Procedural Macros¶

For more complex macros, you can define a macro as a Python function that takes a sequence of tokens as input and returns a new sequence of tokens as output. These are referred to as “procedural macros” or “proc macros”.

Types of Proc Macros¶

There are three types of procedural macros:

function-style macro¶

Implements the Macro protocol.

Invoked as:

macro_name!(input tokens)

macro_name![input tokens]

macro_name!{input tokens}

macro_name!:
    input
    tokens

Important

Due to the way Python’s tokenizer works, indentation and newlines are only preserved by the last (block) style.

When invoked, the registered macro is called with a single argument, the token sequence passed as input.

Note

macro_rules are function-style macros.

module-level macro¶

Implements the ParameterizedMacro protocol.

Invoked with ![name(parameters)] or ![name] (equivalent to ![name()]).

Module-level macro invocations must come before all other code (with the exception of a docstring), and must each appear on their own line.

When invoked, the registered macro is called with two arguments:

parameters (as a token sequence)
the remainder of the module starting from the line immediately following the invocation (as a token sequence).

decorator-style macro¶

Implements the ParameterizedMacro protocol.

Invoked with @![name(parameters)] or @![name] (equivalent to @![name()]).

Decorator-style macro invocations must immediately precede a “block”, defined as either a single newline-terminated line, or a line followed by an indented block.

When invoked, the registered macro is called with two arguments:

parameters (as a token sequence)
the block immediately following the invocation (as a token sequence).

When multiple decorator-style macros are stacked, they are invoked from bottom to top.

Exporting Proc Macros¶

Important

Proc macros cannot be invoked in the same module in which they are defined.

To make a function usable as a macro, you use one of the three predefined decorator macros function_macro, module_macro, and decorator_macro to mark a macro for export. You can then import it using the predefined import module macro.

All three export macros take an optional name parameter as an alternative name to use when exporting the macro. By default the name of the function is used.

Example:

# coding: macro-polo
"""An example of a module proc macro that adds braces-support to Python."""

import token

from macro_polo import Token


@![module_macro]
def braces(parameters, tokens):
    """Add braces support to a Python module.

    The following sequences are replaced:
    - `{:` becomes `:` followed by INDENT
    - `:}` becomes DEDENT
    - `;` becomes NEWLINE
    """
    output = []
    i = 0
    while i < len(tokens):
        match tokens[i : i + 2]:
            case Token(token.OP, '{'), Token(token.OP, ':'):
                output.append(Token(token.OP, ':'))
                output.append(Token(token.INDENT, ''))
                i += 2
            case Token(token.OP, ':'), Token(token.OP, '}'):
                output.append(Token(token.DEDENT, ''))
                i += 2
            case Token(token.OP, ';'), _:
                output.append(Token(token.NEWLINE, '\n'))
                i += 1
            case _:
                output.append(tokens[i])
                i += 1

    return output

We can then import and invoke our braces macro:

Source

# coding: macro-polo
"""An example of using the `import` macro and invoking a module macro."""
![import(braces)]
![braces]


for i in range(5) {:
    print('i =', i);
    if i % 2 == 0 {:
        print(i, 'is divisible by 2')
    :}
:}

Expanded

"""An example of using the `import` macro and invoking a module macro."""

for i in range(5):
    print('i =', i)
    if i % 2 == 0:
        print(i, 'is divisible by 2')

Output

$ python3 examples/proc_macros/uses_braces.py 
i = 0
0 is divisible by 2
i = 1
i = 2
2 is divisible by 2
i = 3
i = 4
4 is divisible by 2

Practically, you’ll probably want to use macro_polo’s lower-level machinary, instead of re-implementing things like matching, transcribing, and scanning.