study: charlie brown
A comparative analysis of LLM and human ability to illicit unexpressed constraints about software requirements.
HUMANS VS LLM
How they compare and compete when it comes to creating software requirements.
WHO CREATES BETTER REQUIREMENTS?
The majority of research (and funding) today in the AI space is focused on code generation. This is interesting as there’s overwhelming evidence showing that poor requirements lead to project failures. We set out to test how well an LLM performed against a human expert is creating first draft requirements for a software product.
WHAT DOES THIS MEAN?
Based on our findings in this study LLMs will play in increasing larger roll as human accelerators when creating software requirements. Acting as first draft engineers, LLMs will be able to speed up initial phases of the SDLC helping create more aligned, complete, and accurate requirements.
ALIGNMENT
+1.12
LLM improvement over human experts in creating aligned requirements
COMPLETENESS
+10.2%
Requirements generated by theLLM were considered more complete
SPEED
720x
(LLMs)Times faster in creating initial draft requirements.
COST
0.06%
LLMs were less than .1% the cost of human experts.
study implications
Our findings in this study highlight several key insights into the potential role of LLMs in requirements engineering (RE). At the initial stages of RE, particularly in requirements elicitation, LLMs demonstrated a significant ability to create aligned and comprehensive requirements quickly.
We expect that humans will still play a critical role in enabling a successful RE process — as LLMs and human experts possess complementary strengths in requirements generation: LLMs excel at producing comprehensive and syntactically correct requirements with human experts bringing domain knowledge, contextual understanding, and the ability to identify nuanced stakeholder needs. As LLMs (and AI) continue to evolve, they will likely take on a more integrated role in the software development lifecycle, with human experts transitioning to roles that focus on orchestration and oversight.
