Package ai.djl.huggingface.tokenizers

package ai.djl.huggingface.tokenizers
Contains classes to interface with the underlying Huggingface tokenizers.
  • Classes
    Class
    Description
    Encoding
    A class holds token encoding information.
    HuggingFaceTokenizer
    HuggingFaceTokenizer is a Huggingface tokenizer implementation of the Tokenizer interface that converts sentences into token.
    HuggingFaceTokenizer.Builder
    The builder for creating huggingface tokenizer.