The Rise of Language Models in Mining Software Repositories: A Survey

Miguel Romero-Arjona, Saman Barakat, Ana B. Sánchez, Sergio Segura

SCORE Lab, I3US Institute, Universidad de Sevilla, Spain

Mining software repositories (MSR) means analyzing the data that software projects leave behind, such as code, bug reports and developer discussions. Language models have gone from a niche technique to one of its main tools. We reviewed 177 studies published up to 2026 to map what the models are used for, which data are mined, which models are chosen, and how reproducible the research is.

  • 177 analyzed studies
  • 8 application types
  • 163 distinct language models
  • 173 data sources mined

A field that took off after 2022

The first studies appeared in 2020, but the topic remained almost unexplored until 2022. Since then, at least 17 studies have been published every year, with a peak of 79 in 2025. The total number of studies grows faster each year, closely following a quadratic curve (R² = 0.97).

  1. 1 2020
  2. 4 2021
  3. 18 2022
  4. 17 2023
  5. 38 2024
  6. 79 2025
  7. 20 2026*

* 2026 covers only the first months of the year.

RQ1

What are language models used for in MSR?

We found 280 applications of language models (LMs), since a single study can use them for several tasks, and grouped them into eight types. Classification dominates, followed by generation, detection and extraction. In each application, the LM plays one of two roles:

Main contribution
The LM output is the result of the study or tool. For example, a model that automatically sorts issue reports (bug reports, feature requests, questions) into categories.
Enabling contribution
The LM is a means to an end: it prepares or filters data so that researchers can then analyze it at scale. For example, a classifier that picks out gender-related app reviews, which researchers then analyze by hand.
  • 82 studies use LMs only as the main contribution, 71 only as an enabling contribution, and 24 in both roles.
Share of applications by type (n = 280)
  • Classificatione.g., labeling issue reports 38.2%
  • Generatione.g., review comments, pull request titles 21.4%
  • Detectione.g., uncivil or confusing comments 15%
  • Extractione.g., details from AI model documentation 14.3%
  • Enhancemente.g., fixing Jupyter notebooks 3.9%
  • Assessmente.g., scoring review comments 3.9%
  • Retrievale.g., recommending microservices 2.2%
  • Predictione.g., whether a code change needs review 1.1%

RQ2

Which data do these studies mine?

We identified 173 data sources, almost all of them public. GitHub is by far the most mined repository, and a small set of curated datasets (developer–ChatGPT conversations, code reviews and issue reports) is reused again and again. Source code, issue reports and discussion threads are the most commonly mined types of data.

  • The studies released 102 new datasets, 87 of them publicly available.
  • The new datasets mostly contain the same types of data that were mined: studies tend to enrich existing data rather than create new kinds.
Most used data sources (number of studies)
  • GitHubcode hosting platform 55
  • DevGPTdeveloper–ChatGPT conversations 10
  • Google Play Storeapp reviews 8
  • Li et al. (2022)642k code reviews 8
  • Hugging Facehub of AI models 7
  • Stack OverflowQ&A posts 5
  • Kallis et al. (2022)803k labeled issues 5
Share of each data type in the mined data (n = 360)
  • Source code 17.5%
  • Issue reports 15%
  • Discussion threads 14.5%
  • Repository metadata 8.3%
  • Requirements 7.8%
  • User feedback 7.2%
  • Code reviews 7.2%
  • Commits 6.1%
  • Pull requests 5.8%
  • AI model metadata 3.6%
  • Conversations with LMs 3.6%
  • Text 2.5%
  • Multimedia 0.8%

RQ3

Which language models are used, and how has that changed?

Small models such as BERT and RoBERTa are still the workhorse, especially for classification. Researchers usually fine-tune them, that is, retrain them on labeled examples from their own data. Since 2023, the field has shifted towards large general-purpose models, mostly OpenAI's GPT models accessed online, which are simply given instructions in plain language (prompting) instead of being retrained.

  • Models from the BERT family account for 29.3% of all uses and GPT models for 21.9%. Newer families such as DeepSeek, Mistral, Qwen, Claude and Gemini are starting to appear.
  • Where the size is known, 65% of uses involve small models, under 0.5 billion parameters (the usual measure of a model's size), and most of them are fine-tuned on repository data.
Share of LM uses by family in 2020–22 (n = 38)
  • BERT 63%
  • GPT 11%
  • T5 16%
  • Llama 0%
  • Mistral 0%
  • DeepSeek 0%
  • Qwen 0%
  • Claude 0%
  • Gemini 0%
  • Other 11%

* 2026 covers only the first months of the year. 2020–2022 are grouped because there were few uses.

Most used language models (number of uses)
  • BERT-baseopen, 0.11B 44
  • RoBERTa-baseopen, 0.125B 35
  • GPT-4oclosed 27
  • GPT-4closed 25
  • GPT-3.5-turboclosed 20
  • DistilBERT-baseopen, 0.066B 16
  • CodeBERTopen, 0.125B 15

Open models can be downloaded and run by anyone; closed models are only available through the provider's service. B = billions of parameters.

RQ4

Can these studies be reproduced and reused?

Most studies share their materials (data, code and scripts), but reproducing them is harder in practice: many rely on proprietary models, available only through a company's service and liable to change without notice; costs are rarely reported; and few studies release tools that are maintained after publication.

  • Materials are hosted mainly on GitHub (81), Zenodo (44) and figshare (13). Unlike GitHub, Zenodo and figshare are designed for long-term preservation.
  • Only 7 of the 30 proposed tools were updated after their release year.
Each square is one of the 177 studies
  • 148 Shared and accessible
  • 8 Mentioned but unavailable
  • 21 Not provided

Open challenges

From the analysis we derive seven open challenges, each with concrete actions for the research community.

  1. Beyond code

    Most studies mine code or data closely tied to it, such as issues and commits. Requirements, design documents, images and videos, or specifications of AI agents remain largely unexplored.

  2. Generate, not just classify

    Generation accounts for only 21.4% of applications. LMs could also enrich repositories by writing requirements, software models, records of design decisions or user tutorials.

  3. Industrial evidence

    166 of the 177 studies are led by academic researchers, usually working with open-source data. Approaches still need to be validated with real company data and constraints, such as cost.

  4. Reusable tools

    Only 30 tools were proposed, and just seven were updated after their release year. Tools should be recognized as research contributions in their own right.

  5. Report the cost

    Only 18 of the 93 studies that use proprietary models report how much they spent. Cost should be evaluated alongside performance.

  6. Benchmarks and datasets

    The field lacks shared benchmarks (standard tests for each task) to help choose a model, its size and how to use it, and datasets are often poorly documented.

  7. AI agents for MSR

    We found no study that uses AI agents, systems that combine LMs with tools to pursue a goal on their own, for MSR. Their effectiveness, cost and reliability remain to be explored.

Cite this work

@article{romero2026rise,
  author    = {Romero-Arjona, Miguel and Barakat, Saman and S{\'a}nchez, Ana B. and Segura, Sergio},
  title     = {The Rise of Language Models in Mining Software Repositories: A Survey},
  year      = {2026},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  issn      = {1049-331X},
  doi       = {10.1145/3861019},
  journal   = {ACM Trans. Softw. Eng. Methodol.},
  month     = oct
}

This work is part of the project PID2024-156482NB-I00, funded by MICIU/AEI/10.13039/501100011033 and by ERDF/EU, and by grant DGP_PRED_2024_00262, funded by the Junta de Andalucía/CIIU and the FSE+.