Transformer Text Generation

Coming Soon

This example is planned for a future release. Check back for updates on transformer-based text generation.

Overview

This example will demonstrate:

• Transformer architecture for text
• Autoregressive text generation
• Tokenization and vocabulary handling
• Nucleus and top-k sampling

Planned Features

• GPT-style decoder-only transformer
• Positional encodings
• Multi-head self-attention
• Layer normalization and residual connections

Related Documentation

• Text Compression
• Seq2Seq Models
• Autoregressive Models

References

• Vaswani et al., "Attention Is All You Need" (2017)
• Radford et al., "Language Models are Unsupervised Multitask Learners" (2019)