Class Tokenizer<TChar>

Namespace

Farkle.Parser.Tokenizers

Assembly

Farkle.dll

Provides an interface to break a sequence of characters into tokens.

public abstract class Tokenizer<TChar>

Type Parameters

TChar

The type of characters the tokens are made of.

Inheritance

object

Tokenizer<TChar>

Constructors

Tokenizer()

Creates a Tokenizer<TChar>.

protected Tokenizer()

Methods

TryGetNextToken(ref ParserInputReader<TChar>, ITokenSemanticProvider<TChar>, out TokenizerResult)

Tries to get the next token from the input.

public abstract bool TryGetNextToken(ref ParserInputReader<TChar> input, ITokenSemanticProvider<TChar> semanticProvider, out TokenizerResult result)

Parameters

input ParserInputReader<TChar>

A ParserInputReader<TChar> with the input and the ParserState.

semanticProvider ITokenSemanticProvider<TChar>

An ITokenSemanticProvider<TChar> to create the semantic values for the tokens.

result TokenizerResult

Will hold the TokenizerResult if the method returns true.

Returns

bool

true if the tokenizer has found a token or failed with an error. In this case the result will be written to result.

false in one of the following cases:

• The tokenizer needs more input to make a decision.
• Input has ended and the IsFinalBlock property of input is true.
• Not applicable for tokenizers that are wrapped in a chain: The tokenizer has encountered a noise symbol.

A tokenizer object is considered to be wrapped in a chain if it was returned by CreateChain<TChar>(ReadOnlySpan<ChainedTokenizerComponent<TChar>>, IGrammarProvider?, Tokenizer<TChar>?) or by any other public API of Farkle.