7B parameters on-premise large language model

Our Approach:

Our dedicated team quickly embarked on the task, parsing through terabytes of country-specific data, and scraping billions of web pages. This meticulous attention to detail was crucial in ensuring the model’s training material included up-to-date, language-specific information reflecting the cultural context accurately.

We curated over 60,000 high-quality bilingual training instructions, providing an all-encompassing coverage of key training topics. Understanding the importance of legal compliance and cultural appropriateness, we prepared several auxiliary datasets. These datasets were primarily focused on fine-tuning the mode to maximize the necessary alignment. The model was trained on 1 trillion tokens, using 440 GPUs.

The team deployed a proprietary state-of-art retriever model to ensure that a real-time search to any connected data source is possible and that the data is seamlessly counted in as a part of the prompt before the generator creates a reply.

Results Delivered:

Despite a tight deadline, the team completed the tasks on time. After several stages of training and testing, we reached the planned results. Within a single month, we created an on-premise large language model tailored to meet the customer’s specific needs. The model stands out for its superior quality, up-to-date information, cultural appropriateness, and legal compliance. Our solution has surpassed GPT 3.5 on six benchmarks in the country language and has been successfully deployed, and is now in active use.