Inference Pipeline

This section contains documentation for inference pipeline.

Modules

Adaptation

• adapter
• lora
• prefix_tuning
• prompt_tuning

Batching

• dynamic
• padding
• streaming

Conversion

• onnx
• tensorrt
• tfjs

Generators

• autoregressive_generator
• diffusion_generator
• energy_generator
• flow_generator
• gan_generator
• vae_generator

Inference

• base
• pipeline

Metrics

• latency
• memory
• throughput

Optimization

• caching
• compilation
• distillation
• production
• pruning
• quantization

Sampling

• ancestral
• beam_search
• classifier_free
• nucleus
• temperature
• top_k

Serving

• grpc
• middleware
• rest
• stateless

Statistics

• Total modules: 37
• Total classes: 7
• Total functions: 17