Tokenization¤

Module: generative_models.extensions.nlp.tokenization

Source: generative_models/extensions/nlp/tokenization.py

Overview¤

Advanced tokenization for text generation tasks.

This module provides JAX-compatible tokenization utilities for text processing and generation tasks.

class AdvancedTokenization

def __call__()

def __init__()

def add_special_tokens()

def apply_masking()

def compute_token_frequencies()

def create_attention_mask()

def create_position_ids()

def decode_batch()

def detokenize()

def encode_batch()

def get_vocabulary_info()

def tokenize()

def truncate_sequences()