Class ParserInputReaderTokenizerExtensions

Namespace

Farkle.Parser.Tokenizers

Assembly

Farkle.dll

Provides extension methods on ParserInputReader<TChar> specific to tokenizers.

public static class ParserInputReaderTokenizerExtensions

Inheritance

object

ParserInputReaderTokenizerExtensions

Remarks

Calling these methods outside of a tokenizer is undefined behavior.

Methods

IsSingleTokenizerInChain<TChar>(in ParserInputReader<TChar>)

Returns whether the tokenizer chain in a parsing operation consists of only one tokenizer.

public static bool IsSingleTokenizerInChain<TChar>(this in ParserInputReader<TChar> input)

Parameters

input ParserInputReader<TChar>

The state of the parsing operation.

Returns

bool

Type Parameters

TChar

The type of characters the tokenizer processes.

Remarks

If this property is true, tokenizers can avoid exiting when they encounter a noise symbol, and instead continue tokenizing. Checking for this property is optional and provides a performance improvement.

SuspendTokenizer<TChar>(ref ParserInputReader<TChar>, Tokenizer<TChar>)

Suspends the tokenization process and sets it to resume at the specified Tokenizer<TChar>.

public static void SuspendTokenizer<TChar>(this ref ParserInputReader<TChar> input, Tokenizer<TChar> tokenizer)

Parameters

input ParserInputReader<TChar>

The state of the parsing operation.

tokenizer Tokenizer<TChar>

The tokenizer to resume at.

Type Parameters

TChar

The type of characters the tokenizer processes.

Remarks

This method is intended to be called by tokenizers that want to keep running when they return. The reasons they want that can be to ask for more characters or to produce many tokens consecutively.

• If the tokenizer is part of a chain and does not return a result, subsequent components of the chain will not be invoked. Instead the parent tokenizer will return without a result, causing the parser to return and request more input.
• When the parent tokenizer gets invoked again, it will first invoke tokenizer and if needed, continue the chain at the tokenizer after the tokenizer that initially suspended.

Because suspending tokenizers more than once is not allowed, tokenizers should return right after they call this method.

Exceptions

ArgumentNullException

tokenizer is null.

InvalidOperationException

The tokenizer is already suspended.

SuspendTokenizer<TChar, TArg>(ref ParserInputReader<TChar>, ITokenizerResumptionPoint<TChar, TArg>, TArg)

Suspends the tokenization process and sets it to resume at the specified ITokenizerResumptionPoint<TChar, TArg>.

public static void SuspendTokenizer<TChar, TArg>(this ref ParserInputReader<TChar> input, ITokenizerResumptionPoint<TChar, TArg> resumptionPoint, TArg arg)

Parameters

input ParserInputReader<TChar>

The state of the parsing operation.

resumptionPoint ITokenizerResumptionPoint<TChar, TArg>

The resumption point to resume at.

arg TArg

An argument to pass to resumptionPoint when it gets invoked.

Type Parameters

TChar

The type of characters the tokenizer processes.

TArg

The type of arg.

Remarks

This method is intended to be called by tokenizers that want to keep running when they return. The reasons they want that can be to ask for more characters or to produce many tokens consecutively.

• If the tokenizer is part of a chain and does not return a result, subsequent components of the chain will not be invoked. Instead the parent tokenizer will return without a result, causing the parser to return and request more input.
• When the parent tokenizer gets invoked again, it will first invoke resumptionPoint and if needed, continue the chain at the tokenizer after the tokenizer that initially suspended.

Because suspending tokenizers more than once is not allowed, tokenizers should return right after they call this method.

Exceptions

ArgumentNullException

resumptionPoint is null.

InvalidOperationException

The tokenizer is already suspended.