Macros!

CIS 198 Lecture 13

What Are Macros?

• In C, a macro looks like this:

`` 1. #define FOO 10 // untyped integral constant 2. #define SUB(x, y) ((x) - (y)) // parentheses are important! 3. #define BAZ a // relies on there being ana` in context!

1. int a = FOO;
2. short b = FOO;
3. int c = -SUB(2, 3 + 4);
4. int d = BAZ; ```
5. The preprocessor runs before the compiler, producing:

1. int a = 10; // = 10 2. short b = 10; // = 10 3. int c = -((2) - (3 + 4)); // = -(2 - 7) = 5 4. int d = a; // = 10

Why C Macros Suck¹

• C does a direct token-level substitution.

• The preprocessor has no idea what variables, types, operators, numbers, or anything else actually mean.

• Say we had defined SUB like this:

``` 1. #define SUB(x, y) x - y

1. int c = -SUB(2, 3 + 4); ```
2. This would break terribly! After preprocessing:

1. int c = -2 - 3 + 4; // = -1, not 5.

¹ GCC Docs: Macro Pitfalls

Why C Macros Suck

• Further suppose we decided to rename a:

`` 1. #define FOO 10 2. #define BAZ a // relies on there being ana` in context!

1. int a_smith = FOO;
2. int d = BAZ; ```
3. Now, the preprocessor produces garbage!

1. int a_smith = 10; // = 10 2. int d = a; // error: `a` is undeclared

Why C Macros Suck

• Since tokens are substituted directly, results can be surprising:

``` 1. #define SWAP(x, y) do { \ 2. (x) = (x) ^ (y); \ 3. (y) = (x) ^ (y); \ 4. (x) = (x) ^ (y); \ 5. } while (0) // Also, creating multiple statements is weird.

1. int x = 10;
2. SWAP(x, x); // x is now 0 instead of 10 ```
3. And arguments can be executed multiple times:

``` 1. #define DOUBLE(x) ((x) + (x))

1. int x = DOUBLE(foo()); // foo gets called twice ```

Why C Macros Suck

• C macros also can’t be recursive:

``` 1. #define foo (4 + foo)

1. int x = foo; ```
2. This expands to

1. int x = 4 + foo;

• (This particular example is silly, but recursion is useful.)

Why C Macros Suck

• In C, macros are also used to include headers (to use code from other files):

1. #include <stdio.h>

• Since this just dumps stdio.h into this file, each file now gets bigger and bigger with additional includes.
• This is a major contributor to long build times in C/C++ (especially in older compilers).

Rust Macros from the Bottom Up

• Almost all material stolen from Daniel Keep’s excellent book:
• The Little Book of Rust Macros (TLBORM).
• This section from Chapter 2.

Rust Syntax Extensions

• Rust has a generalized system called syntax extensions. Anytime you see one of these, it means a syntax extension is in use:

• #[foo] and #![foo]

  • These are used for attributes.
• foo! arg
  • Always foo!(...), foo![...], or foo!{...}
  • Sometimes means foo is a macro.
• foo! arg arg
  • Used only by macro_rules! name { definition }
• The third form is the one used by macros, which are a special type of syntax extension - defined within a Rust program.
• These can also be implemented by compiler plugins, which have much more power than macros.

Rust Macros

• A Rust macro looks like this:

``` 1. macro_rules! incr { // define the macro 2. // syntax pattern => replacement code 3. ( $x:ident ) => { $x += 1; }; 4. }

1. let mut x = 0;
2. incr!(x); // invoke a syntax extension (or macro) ```
3. So… this is totally foreign. The heck’s going on?

Rust Syntax - Token Streams

• Before we dive in, we need to know a little about Rust’s lexer and parser.

• A lexer is a compiler stage which turns the original source (a string) into a stream of tokens.

• A string of code like ((2 + 3) + 10) will turn into a stream like
  • '(' '(' '2' '+' '3' ')' '+' '10' ')'.

• Tokens can be:

• Identifiers: foo, Bambous, self, we_can_dance, …

• Integers: 42, 72u32, 0_______0, …
• Keywords: _, fn, self, match, yield, macro, …
• Lifetimes: 'a, 'b, …
• Strings: "", "Leicester", r##"venezuelan beaver"##, …
• Symbols: [, :, ::, ->, @, <-, …
• In C, macros see the token stream as input.

Rust Syntax - Token Trees

• After lexing, a small amount of parsing is done to turn it into a token tree.

• This isn’t a full-fledged AST (abstract syntax tree).

• For the token stream '(' '(' '2' '+' '3' ')' '+' '10' ')', the token tree looks like this:

  1. Parens 2. ├─ Parens 3. │ ├─ '2' 4. │ ├─ '+' 5. │ └─ '3' 6. ├─ '+' 7. └─ 10 - In Rust, macros see one token tree as input.

• When you do println!("{}", (5+2)), the "{}", (5+2) will get parsed into a token tree, but not fully parsed into an AST.

Rust Syntax - AST

• The AST (abstract syntax tree) is the fully-parsed tree.

• All syntax extension (and macro) invocations are expanded, then parsed into sub-ASTs after the initial AST construction.

  • Syntax extensions must output valid, contextually-correct Rust.

• Syntax extension calls can appear in place of the following syntax kinds, by outputting a valid AST of that kind:

• Patterns (e.g. in a match or if let).

• Statements (e.g. let x = 4;).
• Expressions (e.g. x * (y + z)).
• Items (e.g. fn, struct, impl, use).

• They cannot appear in place of:

• Identifiers, match arms, struct fields, or types.

Macro Expansion

• Let’s parse this Rust code into an AST:

1. let eight = 2 * four!();

1. Let { name: eight 2. init: BinOp { op: Mul 3. lhs: LitInt { val: 2 } 4. /* macro */ rhs: Macro { name: four 5. /* input */ body: () } } } - If four!() is defined to expand to 1 + 3, this expands to:

1. Let { name: eight 2. init: BinOp { op: Mul 3. lhs: LitInt { val: 2 } 4. /* macro */ rhs: BinOp { op: Add 5. /* output */ lhs: LitInt { val: 1 } 6. /* */ rhs: LitInt { val: 3 } } } }

1. let eight = 2 * (1 + 3);

Macro Rules

• Put simply, a macro is just a compile-time pattern match:

1. macro_rules! mymacro { 2. ($pattern1) => {$expansion1}; 3. ($pattern2) => {$expansion2}; 4. // ... 5. } - The four! macro is simple:

1. macro_rules! four { 2. // For empty input, produce `1 + 3` as output. 3. () => {1 + 3}; 4. }

Macro Rules

• Any valid Rust tokens can appear in the match:

``` 1. macro_rules! imaginary { 2. (twentington) => {"20ton"}; 3. (F00 & nee) => {"f0e"}; 4. }

1. imaginary!(twentington);
2. imaginary!(F00&nee);
3. imaginary!(schinty six); // won't compile; is a real number ```

Macro Rules - Captures

• Portions of the input token tree can be captured:

1. macro_rules! sub { 2. ($e1:expr, $e2:expr) => { ... }; 3. } - Captures are always written as $name:kind.

• Possible kinds are:
  • item: an item, like a function, struct, module, etc.
  • block: a block (i.e. { some; stuff; here })
  • stmt: a statement
  • pat: a pattern
  • expr: an expression
  • ty: a type
  • ident: an identifier
  • path: a path (e.g. foo, ::std::mem::replace, …)
  • meta: a meta item; the things that go inside #[...]
  • tt: a single token tree

Macro Rules - Captures

• Captures can be substituted back into the expanded tree

1. macro_rules! sub { 2. ( $e1:expr , $e2:expr ) => { $e1 - $e2 }; 3. } - A capture will always be inserted as a single AST node.

• For example, expr will always mean a valid Rust expression.
• This means we’re no longer vulnerable to C’s substitution problem (the invalid order of operations).

• Multiple expansions will still cause multiple evaluations:

  ``` 1. macro_rules! twice { 2. ( $e:expr ) => { { $e; $e } } 3. }

  1. fn foo() { println!("foo"); }

  2. twice!(foo()); // expands to { foo(); foo() }: prints twice ```

Macro Rules - Repetitions

• If we want to match a list, a variable number of arguments, etc., we can’t do this with the rules we’ve seen so far.
• Repetitions allow us to define repeating subpatterns.
• These have the form $ ( ... ) sep rep.
  • $ is a literal dollar token.
  • ( ... ) is the paren-grouped pattern being repeated.
  • sep is an optional separator token.
  • Usually, this will be , or ;.
  • rep is the required repeat control. This can be either:
  • * zero or more repeats.
  • + one or more repeats.
• The same pattern is used in the output arm.
  • The separator doesn’t have to be the same.

Macro Rules - Repetitions

• We can use these to reimplement our own vec! macro:

1. macro_rules! myvec {
2. (   $(              // Start a repetition
3. $elem:expr  // Each repetition matches one expr
4. ) ,            // Separated by commas
5. *            // Zero or more repetitions
6. ) => {
7. { // Braces so we output only one AST (block kind)
8. let mut v = Vec::new();

10. $(                 // Expand a repetition
11. v.push($elem); // Expands once for each input rep
12. ) *               // No sep; zero or more reps

14. v                  // Return v from the block.
15. }
16. }
17. }
18. println!("{:?}", myvec![3, 4]);

Macro Rules - Repetitions

• Condensed:

1. macro_rules! myvec {
2. ( $( $elem:expr ),* ) => {
3. {
4. let mut v = Vec::new();
5. $( v.push($elem); )*
6. v
7. }
8. }
9. }
10. println!("{:?}", myvec![3, 4]);

Macro Rules - Matching

• Macro rules are matched in order.
• The parser can never backtrack. Say we have:

1. macro_rules! dead_rule { 2. ($e:expr) => { ... }; 3. ($i:ident +) => { ... }; 4. } - If we call it as dead_rule(x +);, it will actually fail.

• x + isn’t a valid expression, so we might think it would fail on the first match and then try again on the second.
• This doesn’t happen!
• Instead, since it starts out looking like an expression, it commits to that match case.
  • When it turns out not to work, it can’t backtrack on what it’s parsed already, to try again. Instead it just fails.

Macro Rules - Matching

• To solve this, we need to put more specific rules first:

1. macro_rules! dead_rule { 2. ($i:ident +) => { ... }; 3. ($e:expr) => { ... }; 4. } - Now, when we call dead_rule!(x +);, the first case will match. - If we called dead_rule!(x + 2);, we can now fall through to the second case. - Why does this work? - Because if we’ve seen $i:ident +, the parser already knows that this looks like the beginning of an expression, so it can fall through to the second case.

Macro Expansion - Hygiene

• In C, we talked about how a macro can implicitly use (or conflict) with an identifier name in the calling context (#define BAZ a).

• Rust macros are partially hygenic.

• Hygenic with regard to most identifiers.

  • These identifiers get a special context internal to the macro expansion.

• NOT hygenic: generic types (<T>), lifetime parameters (<'a>).

  ``` 1. macro_rules! using_a { 2. ($e:expr) => { { let a = 42; $e } } 3. } // Note extra braces ^ ^

  1. let four = using_a!(a / 10); // this won't compile - nice! ```
  2. We can imagine that this expands to something like:

  1. let four = { let using_a_1232424_a = 42; a / 10 };

Macro Expansion - Hygiene

• But if we want to bind a new variable, it’s possible.

• If a token comes in as an input to the function, then it is part of the caller’s context, not the macro’s context.

  ``` 1. macro_rules! using_a { 2. ($a:ident, $e:expr) => { { let $a = 42; $e } } 3. } // Note extra braces ^ ^

  1. let four = using_a!(a, a / 10); // compiles! ```
  2. This expands to:

  1. let four = { let a = 42; a / 10 };

Macro Expansion - Hygiene

• It’s also possible to create identifiers that will be visible outside of the macro call.

• This won’t work due to hygiene:

  ``` 1. macro_rules! let_four { 2. () => { let four = 4; } 3. } // ^ No extra braces

  1. let_four!();
  2. println!("{}", four); // four not declared ```
  3. But this will:

  ``` 1. macro_rules! let_four { 2. ($i:ident) => { let $i = 4; } 3. } // ^ No extra braces

  1. let_four!(myfour);
  2. println!("{}", myfour); // works! ```

Nested and Recursive Macros

• If a macro calls another macro (or itself), this is fine:

1. macro_rules! each_tt { 2. () => {}; 3. ( $_tt:tt $($rest:tt)* ) => { each_tt!( $($rest)* ); }; 4. }

• The compiler will keep expanding macros until there are none left in the AST (or the recursion limit is hit).

• The compiler’s recursion limit can be changed with #![recursion_limit="64"] at the crate root.

  • 64 is the default.
  • This applies to all recursive compiler operations, including auto-dereferencing and macro expansion.

Macro Debugging

• Rust has an unstable feature for debugging macro expansion.

• Especially recursive macro expansions.

  ``` 1. #![feature(trace_macros)] 2. macro_rules! each_tt { 3. () => {}; 4. ( $_tt:tt $($rest:tt) ) => { each_tt!( $($rest) ); }; 5. }

  1. trace_macros!(true);
  2. each_tt!(spim wak plee whum);
  3. trace_macros!(false); ```
  4. This will cause the compiler to print:

  1. each_tt! { spim wak plee whum } 2. each_tt! { wak plee whum } 3. each_tt! { plee whum } 4. each_tt! { whum } 5. each_tt! { } - More tips on macro debugging in TLBORM 2.3.4

Macro Scoping

• Macro scoping is unlike everything else in Rust.

• Macros are immediately visible in submodules:

  ``` 1. macro_rules! X { () => {}; }

  1. mod a { // Or mod a could be in a.rs.
  2. X!(); // valid
  3. } ```
  4. Macros are only defined after they appear in a module:

  ``` 1. mod a { / X! undefined here / }

  1. mod b {
  2. / X! undefined here /
  3. macro_rules! X { () => {}; }
  4. X!(); // valid

  5. }

  6. mod c { / X! undefined / } // They don't leak between mods. ```

Macro Scoping

• Macros can be exported from modules:

``` 1. #[macro_use] // outside of the module definition 2. mod b { 3. macro_rules! X { () => {}; } 4. }

1. mod c {
2. X!(); // valid

3. } ```

4. Or from crates, using #[macro_export] in the crate.

5. There are a few other weirdnesses of macro scoping.

6. See TLBORM 2.3.5 for more.

7. In general, to avoid too much scope weirdness:

8. Put your crate-wide macros at the top of your root module (lib.rs or main.rs).

Rust Macros Design Patterns

• This section from TLBORM Chapter 4.
• I won’t cover most of the chapter.

Macro Callbacks

• Because of the way macros are expanded, “obviously correct” macro invocations like this won’t actually work:

``` 1. macro_rules! expand_to_larch { 2. () => { larch }; 3. }

1. macro_rules! recognise_tree {
2. (larch) => { println!("larch") };
3. (redwood) => { println!("redwood") };
4. ($($other:tt)*) => { println!("dunno??") };

5. }

6. recognise_tree!(expand_to_larch!()); ```

7. This will be expanded like so:

   1. -> recognize_tree!{ expand_to_larch ! ( ) }; 2. -> println!("dunno??"); - Which will match the third pattern, not the first.

Macro Callbacks

• This can make it hard to split a macro into several parts.

• This isn’t always a problem - expand_to_larch ! ( ) won’t match an ident, but it will match an expr.

• The problem can be worked around by using a callback pattern:

``` 1. macro_rules! call_with_larch { 2. ($callback:ident) => { $callback!(larch) }; 3. }

1. call_with_larch!(recognize_tree); ```

2. This expands like this:

   1. -> call_with_larch! { recognise_tree } 2. -> recognise_tree! { larch } 3. -> println!("larch");

Macro TT Munchers

• This is one of the most powerful and useful macro design patterns. It allows for parsing fairly complex grammars.

• A tt muncher is a macro which matches a bit at the beginning of its input, then recurses on the remainder of the input.

• ( $some_stuff:expr $( $tail:tt )* ) =>

• Usually needed for any kind of actual language grammar.
• Can only match against literals and grammar constructs which can be captured by macro_rules!.
• Cannot match unbalanced groups.

Macro TT Munchers

1. macro_rules! mixed_rules {
2. () => {}; // Base case
3. (trace $name:ident ; $( $tail:tt )*) => {
4. {
5. println!(concat!(stringify!($name), " = {:?}"), $name);
6. mixed_rules!($($tail)*);  // Recurse on the tail of the input
7. }
8. };
9. (trace $name:ident = $init:expr ; $( $tail:tt )*) => {
10. {
11. let $name = $init;
12. println!(concat!(stringify!($name), " = {:?}"), $name);
13. mixed_rules!($($tail)*);  // Recurse on the tail of the input
14. }
15. };
16. }

Macros Rule! Mostly!

• Macros are pretty great - but not perfect.
• Macro hygiene isn’t perfect.
• The scope of where you can use a macro is weird.
• Handling crates inside of exported macros is weird.
• It’s impossible to construct entirely new identifiers (e.g. by concatenating two other identifiers).
• …
• A new, incompatible macro system may appear in future Rust.
• This would be a new syntax for writing syntax extensions.