Most of the time, software testing is based on comparing the actual behavior of a code or a system with its expected behavior. In his article “The Challenges of Testing in a Non-Deterministic World”, Donald Firesmith explains however that not all software and systems are deterministic, for instance an autonomous adaptive vehicle that uses machine learning software, and that software testing needs to adapt to this context.
Donald Firesmith starts his article by explaining the conditions for performing software testing in a deterministic context. He then introduces the concept of non-determinism, its different types of behavior and how it is present in modern software and systems like robots, IoT or mobile computing where technologies like machine learning influences progressively the behavior and outputs. This means that with non-deterministic systems and software, you can perform the exact same test case multiple times and get different results.
The article also lists some types of the non-deterministic defects: concurrency, performance, reliability, robustness, security, cyclic behavior or questionable output. According to Donald Firesmith, the lack of training and awareness of software testers for non-deterministic software and systems leads to the fact that “Developers and testers do not design, implement, and execute tests designed to uncover non-deterministic defects, making testing less effective and efficient at detecting them. As a result, testing fails to uncover non-deterministic defects resulting in an increase in false positive and false negative test results. ” He then explains the challenges created by this context, like the fact that the success criteria of software test could be a set of acceptable behavior rather than a single specific behavior or that it could be possible to determine success through manual test execution and validation of the results by subject matter experts rather than using test automation.
His conclusion is that “We live and work in a non-deterministic world, and that has significant ramifications on how we need to test our systems and software.”
Read the complete article on https://insights.sei.cmu.edu/sei_blog/2017/01/the-challenges-of-testing-in-a-non-deterministic-world.html
Thank you for sharing such an informative post. I actually am glad about the material you have on your post it really has helped me out a whole lot thanks.