Package ai.djl.huggingface.tokenizers.jni

package ai.djl.huggingface.tokenizers.jni
Contains classes to interface with the underlying Huggingface tokenizers.
  • Classes
    Class
    Description
    CharSpan
    A class holds character span information.
    LibUtils
    Utilities for finding the Huggingface tokenizer native binary on the System.
    TokenizersLibrary
    A class containing utilities to interact with the Tokenizer JNI layer.