It is possible to shake up traditional research methods with the use of AI, but there are still challenges left to overcome.
Artificial Intelligence. It is not a new term to researchers, but in the past year it has become a widely discussed topic in the media. A big part of this is due to OpenAI’s big language-model based chatbot ChatGPT that grew into a worldwide phenomenon. There are both opportunities and threats to its use. In research for example, it is possible to automate certain tasks by using ChatGPT. At the same time, however, there is a fear of AI taking over jobs and threatening traditional methods of studying.
But how can a big language-model be used as a tool in research?
We are studying how well a big language-model based chatbot can replace a human in doing qualitative research. In this case we use the language-model to sort research data into different categories. Traditionally, the categorizing is done by at least two people who can compare and discuss the results, and, by doing so, make sure that the categorization is done carefully.
Preliminary results show that it is possible to replace real people with ChatGPT when categorizing research data. This is an important observation, as categorizing data manually is extremely time consuming. If it is possible to replace one or multiple people with a big language-model, it will result in considerable time savings. However, there are still challenges associated with the use of ChatGPT, because it is not perfect and it makes mistakes too.
What are the best ways to utilize ChatGPT then? Our research found that the language-model is very sensitive. Therefore it is important to pay attention to the word choices and repetition of different prompts. You should also not assume everything ChatGPT says is true, and instead have multiple rounds of conversations with it and analyze its answers carefully in order to reach better results. How can this be done in practice? More detailed instructions can be found in our upcoming conference paper.
Leevi Rantala is a doctoral researcher at the Empirical Software Engineering in Software, Systems and Services (M3S) research group at the University of Oulu. It is one of Europe’s largest research units in the software sector. Within the software engineering research field, Leevi’s interests include topics such as technical debt, natural language processing tools, sentiment analysis, and adaptive and learning programs. Read more about Leevi’s work at the University of Oulu repository.
In Researcher Blogs KAUTE’s grant recipients tell about their research in their own words.