Class Tokenizer

Namespace

Farkle.Parser.Tokenizers

Assembly

Farkle.dll

Provides factory methods to create Tokenizer<TChar>s.

public static class Tokenizer

Inheritance

object

Tokenizer

Methods

CreateChain<TChar>(ReadOnlySpan<ChainedTokenizerComponent<TChar>>, IGrammarProvider?, Tokenizer<TChar>?)

Creates a Tokenizer<TChar> from a tokenizer chain.

public static Tokenizer<TChar> CreateChain<TChar>(ReadOnlySpan<ChainedTokenizerComponent<TChar>> components, IGrammarProvider? grammar = null, Tokenizer<TChar>? defaultTokenizer = null)

Parameters

components ReadOnlySpan<ChainedTokenizerComponent<TChar>>

grammar IGrammarProvider

The IGrammarProvider to pass to the delegates given in Create(Func<IGrammarProvider, Tokenizer<TChar>>). This parameter is optional if no such delegates have been added.

defaultTokenizer Tokenizer<TChar>

The tokenizer to use in place of Default.

Returns

Tokenizer<TChar>

Type Parameters

TChar

The type of characters the tokenizer accepts.

Exceptions

InvalidOperationException

grammar is null and components contains a grammar-dependent tokenizer, or defaultTokenizer is and components contains the default tokenizer.

Create<TChar>(Grammar)

Creates a Tokenizer<TChar>.

public static Tokenizer<TChar> Create<TChar>(Grammar grammar) where TChar : unmanaged, IComparable<TChar>

Parameters

grammar Grammar

The Grammar the tokenizer will use.

Returns

Tokenizer<TChar>

Type Parameters

TChar

The type of characters the tokenizer accepts.

Exceptions

ArgumentNullException

grammar is null.

NotSupportedException

TChar is not char.

InvalidOperationException

The grammar cannot be used for tokenizing.