Lexer

Lexica is a whitespace delimited language and so indentation is used for lexical scoping. This means that indentation tokens must be parsed into logical block open and close tokens.

The lexer consists of a pipeline from a string to a stream of tokens. This stream differs from a traditional iterator in that it always produces a token. End of input is denoted with a special token variant. This simplifies parsing the language.

Each stage in the pipeline transforms the provided input. An intermediary LexerToken type is used to thread Indent tokens through to further stages.

Stage

Description

Source Split

Splits a string into string slices with span information

Lexer Tokenize

Maps each string slice into a LexerToken

Indent Lexer

Maintains the indentation level and produces logical blocks

Space Lexer

Modifies logical tokens based on surrounding context