Interface ITokenizerResumptionPoint<TChar, TArg>

Provides a way for a Tokenizer<TChar> to resume at multiple points after suspending.

public interface ITokenizerResumptionPoint<TChar, in TArg>

TChar: The type of characters the tokenizer reads.
TArg: A type a value of which will be passed to the resumption point. Implementing this interface with multiple TArg types allows defining many suspension points.

Methods

Tries to get the next token from the input. This method is identical to TryGetNextToken(ref ParserInputReader<TChar>, ITokenSemanticProvider<TChar>, out TokenizerResult) with the only addition of the arg parameter.

bool TryGetNextToken(ref ParserInputReader<TChar> input, ITokenSemanticProvider<TChar> semanticProvider, TArg arg, out TokenizerResult result)

input ParserInputReader<TChar>: A ParserInputReader<TChar> with the input and the ParserState.
semanticProvider ITokenSemanticProvider<TChar>: An ITokenSemanticProvider<TChar> to create the semantic values for the tokens.
arg TArg: The value that had been passed in SuspendTokenizer<TChar, TArg>(ref ParserInputReader<TChar>, ITokenizerResumptionPoint<TChar, TArg>, TArg).
result TokenizerResult: Will hold the TokenizerResult if the method returns true.

bool: Whether the tokenizer was given enough characters to either find a token or conclusively fail. See TryGetNextToken(ref ParserInputReader<TChar>, ITokenSemanticProvider<TChar>, out TokenizerResult) for more details about the semantics of the method's return type.