New Open-Source Framework to Improve AI for Telecom-Specific Applications

The GSMA Foundry, the innovation hub of the GSMA, has announced the launch of GSMA Open-Telco LLM Benchmarks, an open-source community designed to enhance the performance, efficiency, and safety of large language models (LLMs) for telecom applications.

Supported by Hugging Face, Khalifa University, The Linux Foundation, and leading mobile network operators (MNOs) like Deutsche Telekom, LG Uplus, SK Telecom, and Turkcell, this initiative provides a standardized framework to evaluate AI models in real-world telecommunications use cases.

The Need for AI-Specific Benchmarks in Telecom Networks

As AI adoption in telecommunications accelerates, current LLMs still struggle to handle technical knowledge, network troubleshooting, and regulatory compliance. Recent benchmark results highlight these gaps:

• GPT-4 scored less than 75% on TeleQnA, a dataset assessing telecom knowledge.
• GPT-4 performed below 40% on 3GPP Docs Classification, a test based on 3GPP standards.
• Microsoft’s Phi-2 scored only 10% on MATH500, a benchmark of 500 general math questions.

These results underscore the need for AI models tailored to telecom—capable of handling network management, diagnostics, and compliance. The GSMA Open-Telco LLM Benchmarks aims to bridge this gap by providing a transparent and industry-wide evaluation framework.

GSMA Open-Telco LLM: Key Features & Industry Benefits

1. Open-Source AI Benchmarks: Improving Telecom LLM Performance

The GSMA Open-Telco LLM Benchmarks will provide a structured framework for assessing LLMs across three critical areas:

• Telecom domain expertise – Can AI models accurately interpret and respond to industry-specific queries?
• Energy efficiency – How much computing power does each AI model require?
• Safety and compliance – Does AI-generated output meet telecom industry security and regulatory standards?

To ensure transparency, these benchmarks will be hosted on Hugging Face, where the community can contribute and collaborate.

2. Telecom AI Collaboration: How Industry Leaders Are Shaping AI

This initiative unites network operators, AI researchers, and developers to submit datasets, AI models, and telecom use cases for evaluation. By fostering an open AI ecosystem, the telecom industry can accelerate innovation and reliability in AI applications.

3. AI Testing in Telecom: Real-World Use Cases & Applications

The GSMA Open-Telco LLM Benchmarks will evaluate AI models in real-world telecom scenarios, including:

• Network optimization – AI-driven solutions for improving mobile network performance.
• Fault detection & troubleshooting – Using AI for predictive maintenance and diagnostics.
• Customer service automation – Enhancing AI-powered chatbots and virtual assistants.
• Energy efficiency – Measuring AI’s role in reducing network power consumption.

Industry Leaders on the Importance of AI Benchmarking

GSMA – “Today’s AI models struggle with telecom-specific queries, often producing inaccurate, misleading, or impractical recommendations,” said Louis Powell, Head of AI Initiatives at GSMA. “By creating an industry-wide set of benchmarks, we’re ensuring AI in telecoms is safe, reliable, and aligned with real-world operational needs.”

Hugging Face – “Hugging Face is the leading open platform for AI builders, and we’re thrilled to support and host the GSMA Open-Telco LLM Benchmarks to advance telecoms AI adoption and innovation,” said Jeff Boudier, Head of Product and Growth, Hugging Face.

Khalifa University – “Academia plays a crucial role in advancing AI for telecommunications by ensuring rigorous benchmarking and scientific integrity,” said Prof. Merouane Debbah, Director of the 6G Research Center at Khalifa University. “This initiative will drive innovation and enhance the reliability of AI models in telecom applications.”

LG Uplus – “We stand at a turning point in AI, and telcos will play a vital role in enabling autonomous networks,” said Sangyeob Lee, CTO at LG Uplus. “We welcome GSMA Open-Telco LLM Benchmarks as a guiding light toward assured AI-driven intelligence services.”

The Linux Foundation – “By establishing open, standardized benchmarks, this initiative brings much-needed transparency, enabling operators to deploy domain-specific AI with confidence,” said Arpit Joshipura, General Manager for Networking, Edge, and IoT at The Linux Foundation.

SK Telecom – “This initiative is a pivotal step in ensuring AI solutions for telecoms are robust, reliable, and precise,” said Eric Davis, Head of AI Tech Collaboration at SK Telecom.

Responsible AI in Telecom: Ensuring Ethical AI Deployment

The launch of GSMA Open-Telco LLM Benchmarks follows the industry’s broader commitment to ethical and sustainable AI adoption. Central to this effort is the GSMA Responsible AI Maturity Roadmap, which provides best-practice guidelines for AI deployment in telecommunications.

AI at MWC25: Exploring Generative AI in Telecom Innovation

The ‘Gen AI Summit: Experimentation to Transformation’ at MWC25 Barcelona will explore the practical applications of generative AI in telecoms. Key topics include:

• AI-driven network optimization
• Personalized customer experiences
• Generative AI for 5G and beyond

Featured speakers include:

• Harry Singh, Chief Digital Officer at BT
• Harrison Lung, Group Chief Strategy Officer at e&
• Kaniz Mahdi, Director of Technology at AWS Industries
• Laurent Leboucher, CTO at Orange

Additionally, AI visionary Arthur Mensch, CEO of Mistral AI, will speak in Keynote 7: Tech Game Changers, discussing the latest AI developments in telecom.

In Keynote 10: Why AI Agents Will Change Everything, Bret Taylor, CEO of Sierra and Board Member at OpenAI, will explore how AI agents are set to revolutionize business and enterprise connectivity.

Join the GSMA Open-Telco AI Benchmarking Initiative

The GSMA Open-Telco LLM Benchmarks community is now inviting contributions from:

• Mobile network operators
• AI developers and researchers
• Technology vendors and startups

Organizations can submit AI models, datasets, and telecom use cases for evaluation by contacting [email protected].

For more details, visit: www.gsma.com/get-involved/gsma-foundry/gsma-open-telco-llm-benchmarks.

The Future of AI in Telecom: Smarter, Safer, and More Efficient

With the launch of GSMA Open-Telco LLM Benchmarks, the telecom industry is taking a critical step toward AI-driven innovation. By establishing transparent, standardized AI evaluations, this initiative will:

• Enhance telecom AI performance
• Ensure compliance with industry standards
• Drive efficiency and sustainability in telecom networks

As AI continues to shape the future of connectivity, GSMA’s initiative ensures that telecom AI remains reliable, scalable, and ready for the challenges of 5G, 6G, and beyond.