Mid Sweden University

Testing is a critical part of software development, used to ensure that the created software system meets expected functional requirements and to identify potential errors.The testing process is also one of the most time-consuming and expensive tasks during development. A central part of contemporary software testing is referred to as unit testing, where test cases are created to ensure the validity of individual components of a software system. While there already exist tools that can automatically generate tests, Large Language Models present new possibilities. Even if the tests are generated it is important that they are maintainable as tests often require modification to align with evolving software. Comprehensibility is a main component of maintainability. In this research, we have conducted a comparative analysis of the comprehensibility of tests generated by different LLM-based tools and compared them to manually written tests. Our research indicates that the general comprehensibility of LLM-generated test cases can surpass those of manually written test cases. Accordingly, we further found that LLM-generated test suites tend to be less complex and include less smelly code while being comparable in general readability. Lastly, the underlying functionality of LLM-based test generators, including design choices and model selection, significantly impacts how well LLM-based unit test generators create comprehensive tests. These same design and model choices might further affect the line and branch coverage of the resulting test suites. Lastly, we found that LLM-generated tests tend to have worse line and branch coverage compared to manually written tests.