In the ever-evolving landscape of artificial intelligence, groundbreaking advancements are constantly reshaping the boundaries of what’s possible. NVIDIA, a pioneer in the field, has recently unveiled its latest triumph – the Nemotron-4 340B family of large language models. This remarkable achievement not only pushes the limits of AI capabilities but also ushers in a new era of open access and collaborative innovation.
The Nemotron-4 340B family comprises three main versions: Nemotron-4-340B-Base, Nemotron-4-340B-Instruct, and Nemotron-4-340B-Reward. What sets these models apart is their open access under the NVIDIA Open Model License Agreement, empowering developers, researchers, and enthusiasts worldwide to freely use, modify, and distribute them, even for commercial applications.
Diving into the technical intricacies of the Nemotron-4-340B-Base, we uncover a standard decoder-only Transformer architecture with causal attention masks and a SentencePiece tokenizer for efficient vocabulary management. The model employs squared ReLU activation in the MLP layers, grouped query attention (GQA) for optimized query handling, and boasts an impressive array of hyper-parameters, including 96 transformer layers, a hidden dimension of 18432, 96 attention heads with 8 KV heads, a sequence length of 4096, and a staggering vocabulary size of 256,000.
Video About NVIDIA’s Nemotron-4 340B : https://youtu.be/QanTJ6raKI8
Training such a behemoth was no simple task. NVIDIA harnessed the power of 768 DGX H100 nodes, each equipped with 8 H100 GPUs, combining tensor parallelism, pipeline parallelism, and data parallelism to tackle the challenge. The model was trained on a staggering 9 trillion tokens, encompassing 70% English data, 15% multilingual data covering 53 languages, and 15% source code data from 43 programming languages, ensuring a diverse and comprehensive knowledge base.
But the true testament to the Nemotron-4 340B’s prowess lies in its benchmark performance. In commonsense reasoning tasks like ARC-Challenge, Winogrande, and Hellaswag, it scored impressively high, with results ranging from 85.30% to 94.28%. On popular aggregated benchmarks like MMLU (5-shot) and BBH (3-shot), it achieved scores of 81.10% and 82.40%, respectively. In coding tasks, the Nemotron-4-340B-Base excelled with a 57.32% score on HumanEval (0-shot).
The Nemotron-4-340B-Instruct variant outperformed other instruct models in instruction following and chat capabilities, while the Nemotron-4-340B-Reward topped the charts on RewardBench, surpassing even proprietary models like GPT-4o-0513 and Gemini 1.5 Pro-0514.
One of the most remarkable aspects of these models is their reliance on synthetic data. Over 98% of the data used in model alignment was synthetically generated, a feat made possible by NVIDIA’s open-source synthetic data generation pipeline. This pipeline includes synthetic prompt generation, response and dialogue generation, quality filtering, and preference ranking, enabling the creation of high-quality data for a wide range of domains.
As we explore the capabilities of the Nemotron-4 340B family, we are reminded of the incredible potential that lies within the realm of AI. With open access and collaborative efforts, these models have the power to propel innovation across industries, revolutionizing fields such as natural language processing, content creation, and beyond.
The future of AI is bright, and NVIDIA’s Nemotron-4 340B is a shining example of what can be achieved when cutting-edge technology meets the spirit of open collaboration. As we continue to push the boundaries of what’s possible, we can look forward to a world where AI serves as a catalyst for progress, unlocking new realms of human potential and shaping the course of our collective future.
Resources:
https://huggingface.co/nvidia/Nemotron-4-340B-Instruct
https://research.nvidia.com/publication/2024-06_nemotron-4-340b