`logos.std.compiler.tokens`

Module std · package logos-std

Types

enum Delim

Delimiter kind enclosing a TokenTree::Group.

Variants

Brace

Bracket

Paren

struct Ident

Identifier token text as a non-owning (ptr, len) byte view.

Implements: Copy

Fields

len: u64

Length of the identifier text in bytes.

ptr: *const u8

Pointer to the first byte of the identifier text (UTF-8, not owned).

enum LitKind

Lexical category of a TokenTree::Literal token.

Variants

Bool

Char

Float

Int

Str

enum Spacing

Whether a Punct is glued to the following punct as part of a multi-char operator.

Variants

Alone

Joint

struct Span

Source location of a token.

Fields

col: i32

Column number within the line.

file_id: i32

Id of the source file; -1 marks an unknown location.

line: i32

Line number within the file.

struct TokenStream

Flat sequence of TokenTrees; nesting lives inside Group trees.

Fields

span: Span

Outer span of the whole stream, set by the producer.

trees: Vec<TokenTree>

Trees in source order.

Methods

fn is_empty(self: &TokenStream) -> bool

Returns true if the stream contains no trees.

fn length(self: &TokenStream) -> i64

Returns the number of top-level trees in the stream.

fn push(self: &mut TokenStream, t: TokenTree) -> void

Appends a tree to the end of the stream.

enum TokenTree

One node of a token stream: a leaf token or a balanced delimited subgroup.

Variants

Group(Delim, TokenStream, Span)

Ident(Ident, Span)

Literal(LitKind, String, Span)

Punct(char, Spacing, Span)

fn span_unknown() -> Span

Returns the sentinel Span for tokens with no source location (file_id == -1).

fn token_stream_new() -> TokenStream

Creates an empty TokenStream with an unknown span.