🎤 Presenters: Dan from Together AI and Eugene, CEO and co-founder of Fedas.

📈 Overview: Discussion on advancements in non-post Transformer architectures, focusing on scaling models and improving efficiency.

🔍 Key Points:

• Scaling: Recent years have seen significant increases in model parameter sizes and context lengths.
• Compute Efficiency: Exploring alternatives to traditional attention mechanisms to reduce computational costs.
• Quadratic Scaling: Attention mechanisms scale quadratically with context length, prompting the search for more efficient models.

🚀 Advancements Since 2020:

• State Space Models: Introduced in 2022, combining principles from signal processing to improve quality and efficiency.
• Specialized Kernels: Development of efficient kernels like Flash FFT to enhance performance.
• Selection Mechanisms: Improved methods for selecting relevant information from hidden states to boost model quality.

📊 Current State: As of yesterday, notable models include:

• Jamba: A hybrid model from AI2, currently leading in non-Transformer architectures.
• Sauna: A diffusion model from Nvidia and MIT, utilizing linear attention for larger sequences.
• Gated State Space Models: Achieving significant results in various applications, including DNA modeling.

💡 Future Directions: Focus on hardware-efficient designs and exploring new paradigms for long context processing.

❓ Q&A Highlights:

• Discussion on the relevance of long context lengths and the potential for models to handle infinite context.
• Exploration of how models can learn and remember information over extended periods.

🌟 Conclusion: Exciting developments in non-Transformer architectures are paving the way for more efficient AI models.