java.lang.Object
net.luis.utils.io.token.TokenReader
A class that reads tokens from a string input based on defined token definitions.
It processes the input string, identifies tokens, and returns a list of tokens.
It handles escape sequences and allows for custom token definitions and separators.
It processes the input string, identifies tokens, and returns a list of tokens.
It handles escape sequences and allows for custom token definitions and separators.
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static final class
A class that tracks the position of the current character in the input string.
It keeps track of the line number, character position, and character index in the line. -
Field Summary
FieldsModifier and TypeFieldDescriptionThe set of allowed characters that can be part of tokens.private final Set
<TokenDefinition> The set of token definitions used to identify tokens in the input string.The set of characters that are considered as separators in the input string. -
Constructor Summary
ConstructorsConstructorDescriptionTokenReader
(@NotNull Set<TokenDefinition> definitions, @NotNull Set<Character> allowedChars, @NotNull Set<Character> separators) Constructs a new token reader with the specified token definitions, allowed characters, and separators.
The characters '\\' and '\n' are added to the set of separators.
The list of separators is added to the list of allowed characters to ensure they are valid. -
Method Summary
Modifier and TypeMethodDescriptionprivate void
addToken
(@NotNull List<Token> tokens, @NotNull String word, @NotNull TokenPosition startPosition, @NotNull TokenPosition endPosition) Adds a token to the list of tokens.
It uses the token definitions to determine the type of token and creates a new token object.
If no matching definition is found, a word token is created.readTokens
(@NotNull String input) Reads tokens from the input string and returns a list of tokens.
It processes the input string, identifies tokens based on the defined token definitions, and handles escape sequences.
-
Field Details
-
definitions
The set of token definitions used to identify tokens in the input string. -
allowedChars
The set of allowed characters that can be part of tokens. -
separators
The set of characters that are considered as separators in the input string.
-
-
Constructor Details
-
TokenReader
public TokenReader(@NotNull @NotNull Set<TokenDefinition> definitions, @NotNull @NotNull Set<Character> allowedChars, @NotNull @NotNull Set<Character> separators) Constructs a new token reader with the specified token definitions, allowed characters, and separators.
The characters '\\' and '\n' are added to the set of separators.
The list of separators is added to the list of allowed characters to ensure they are valid.- Parameters:
definitions
- The set of token definitionsallowedChars
- The set of allowed charactersseparators
- The set of separators- Throws:
NullPointerException
- If any of the parameters are nullIllegalArgumentException
- If the definitions contain the word token definition, or if the allowed characters or separators are empty
-
-
Method Details
-
readTokens
Reads tokens from the input string and returns a list of tokens.
It processes the input string, identifies tokens based on the defined token definitions, and handles escape sequences.- Parameters:
input
- The input string to read tokens from- Returns:
- An unmodifiable list of tokens read from the input string
- Throws:
NullPointerException
- If the input string is nullIllegalStateException
- If an undefined character is encountered in the input string- See Also:
-
addToken
private void addToken(@NotNull @NotNull List<Token> tokens, @NotNull @NotNull String word, @NotNull @NotNull TokenPosition startPosition, @NotNull @NotNull TokenPosition endPosition) Adds a token to the list of tokens.
It uses the token definitions to determine the type of token and creates a new token object.
If no matching definition is found, a word token is created.- Parameters:
tokens
- The list of tokens to add the new token toword
- The word to be added as a tokenstartPosition
- The start position of the token in the input stringendPosition
- The end position of the token in the input string- Throws:
NullPointerException
- If any of the parameters are null
-