Class TokenReader

java.lang.Object
net.luis.utils.io.token.TokenReader

public class TokenReader extends Object
A class that reads tokens from a string input based on defined token definitions.
It processes the input string, identifies tokens, and returns a list of tokens.
It handles escape sequences and allows for custom token definitions and separators.
  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    private static final class 
    A class that tracks the position of the current character in the input string.
    It keeps track of the line number, character position, and character index in the line.
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    private final Set<Character>
    The set of allowed characters that can be part of tokens.
    private final Set<TokenDefinition>
    The set of token definitions used to identify tokens in the input string.
    private final Set<Character>
    The set of characters that are considered as separators in the input string.
  • Constructor Summary

    Constructors
    Constructor
    Description
    TokenReader(@NotNull Set<TokenDefinition> definitions, @NotNull Set<Character> allowedChars, @NotNull Set<Character> separators)
    Constructs a new token reader with the specified token definitions, allowed characters, and separators.
    The characters '\\' and '\n' are added to the set of separators.
    The list of separators is added to the list of allowed characters to ensure they are valid.
  • Method Summary

    Modifier and Type
    Method
    Description
    private void
    addToken(@NotNull List<Token> tokens, @NotNull String word, @NotNull TokenPosition startPosition, @NotNull TokenPosition endPosition)
    Adds a token to the list of tokens.
    It uses the token definitions to determine the type of token and creates a new token object.
    If no matching definition is found, a word token is created.
    @NotNull @Unmodifiable List<Token>
    readTokens(@NotNull String input)
    Reads tokens from the input string and returns a list of tokens.
    It processes the input string, identifies tokens based on the defined token definitions, and handles escape sequences.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

    • definitions

      private final Set<TokenDefinition> definitions
      The set of token definitions used to identify tokens in the input string.
    • allowedChars

      private final Set<Character> allowedChars
      The set of allowed characters that can be part of tokens.
    • separators

      private final Set<Character> separators
      The set of characters that are considered as separators in the input string.
  • Constructor Details

    • TokenReader

      public TokenReader(@NotNull @NotNull Set<TokenDefinition> definitions, @NotNull @NotNull Set<Character> allowedChars, @NotNull @NotNull Set<Character> separators)
      Constructs a new token reader with the specified token definitions, allowed characters, and separators.
      The characters '\\' and '\n' are added to the set of separators.
      The list of separators is added to the list of allowed characters to ensure they are valid.
      Parameters:
      definitions - The set of token definitions
      allowedChars - The set of allowed characters
      separators - The set of separators
      Throws:
      NullPointerException - If any of the parameters are null
      IllegalArgumentException - If the definitions contain the word token definition, or if the allowed characters or separators are empty
  • Method Details

    • readTokens

      @NotNull public @NotNull @Unmodifiable List<Token> readTokens(@NotNull @NotNull String input)
      Reads tokens from the input string and returns a list of tokens.
      It processes the input string, identifies tokens based on the defined token definitions, and handles escape sequences.
      Parameters:
      input - The input string to read tokens from
      Returns:
      An unmodifiable list of tokens read from the input string
      Throws:
      NullPointerException - If the input string is null
      IllegalStateException - If an undefined character is encountered in the input string
      See Also:
    • addToken

      private void addToken(@NotNull @NotNull List<Token> tokens, @NotNull @NotNull String word, @NotNull @NotNull TokenPosition startPosition, @NotNull @NotNull TokenPosition endPosition)
      Adds a token to the list of tokens.
      It uses the token definitions to determine the type of token and creates a new token object.
      If no matching definition is found, a word token is created.
      Parameters:
      tokens - The list of tokens to add the new token to
      word - The word to be added as a token
      startPosition - The start position of the token in the input string
      endPosition - The end position of the token in the input string
      Throws:
      NullPointerException - If any of the parameters are null