A metamorphic testing approach for supporting program repair without the need for a test oracle

Jiang, Mingyue; Chen, Tsong Yueh; Kuo, Fei-Ching; Towey, Dave; Ding, Zuohua

Abstract

Test suite based automated program repair (APR) relies on a test oracle to determine the execution result of individual test cases. The applicability of APR techniques, therefore, is limited by the fact that test oracles may not exist. Metamorphic Testing (MT) is a testing approach that, rather than checking the correctness of individual test outputs, checks testing results through verification of relations among multiple test cases and their outputs: MT can therefore be applied without test oracles. This paper presents an integration of MT with APR that enables application of APR without the need for a test oracle. Two important issues for this integration which have been thoroughly investigated and addressed are: (1) feasibility — which is addressed by proposing a framework to support the integration, and then presenting MT-GenProg, a tool incorporating MT with the popular APR technique GenProg; and (2) effectiveness — which is confirmed through an empirical study of GenProg and MT-GenProg on 1,143 program versions from the IntroClass benchmark suite, demonstrating MT-GenProg’s comparable performance to GenProg, in terms of repair effectiveness. We conclude that the proposed integration is both practically feasible and effective, and thus successfully extends APR techniques to a broader application domain.

Citation

Jiang, M., Chen, T. Y., Kuo, F.-C., Towey, D., & Ding, Z. (in press). A metamorphic testing approach for supporting program repair without the need for a test oracle. Journal of Systems and Software, 126, https://doi.org/10.1016/j.jss.2016.04.002

• Journal Article Type: Article
• Acceptance Date: Apr 3, 2016
• Online Publication Date: Apr 13, 2016
• Deposit Date: Oct 25, 2017
• Journal: Journal of Systems and Software
• Print ISSN: 0164-1212
• Electronic ISSN: 1873-1228
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 126
• DOI: https://doi.org/10.1016/j.jss.2016.04.002
• Keywords: Test suite based automated program repair; Test oracle; Metamorphic testing
• Public URL: https://nottingham-repository.worktribe.com/output/785361
• Publisher URL: http://www.sciencedirect.com/science/article/pii/S0164121216300206
• Contract Date: Oct 25, 2017