Generative AI in Software Testing

“Right now, people talk about being an AI company. There was a time after the iPhone App Store launch where people talked about being a mobile company. But no software company says they’re a mobile company now because it’d be unthinkable to not have a mobile app. And it’ll be unthinkable not to have intelligence integrated into every product and service. It’ll just be an expected, obvious thing.”- Sam Altman.

If we ponder what Sam Altman said, we know it will soon be true about generative AI. Today, ChatGPT and other conversational LLMs have changed how students learn, how programmers code, how emails are written, and so forth. Generative AI is known for its ability to create new content based on existing training data and is making significant waves across industries. Gartner forecasts that by 2026, over 80% of organizations will have implemented generative AI applications or integrated generative AI APIs into their workflows. Additionally, Gartner views gen AI as a transformative technology of the magnitude that it is comparable to the steam engine, electricity, and the internet.

When we talk about software testing, generative AI does not merely run repetitive tasks; it helps create a new approach altogether to understand, analyze, and test software. You can generate test cases, simulate user behaviors, predict potential failure points, and continuously adapt to new features, which is something traditional testing frameworks and tools struggle to keep up with.

In this article, we will discuss the role of generative AI in software testing with examples and benefits. We will also demonstrate one use case for further clarity.

What is Generative AI?

Let us start with the basic question first.

Generative AI (gen AI) is a type of AI that produces original content (videos, text, code, images, speech, music, data, etc.) based on the user’s input called a prompt. Since it is AI, it works on advanced Deep Learning models to understand the natural language prompts and provide output accordingly. Deep Learning identifies patterns within vast training datasets and replicates them to create new data that resembles human-generated content. This approach runs on neural networks, a type of machine-learning architecture inspired by how the human brain operates and works. Read about the role of AI and Machine Learning in software testing.

Role of Generative AI in Software Testing

Let us move on to software testing and find out how generative AI can help. Here are some areas where generative AI is currently supporting QA:

Test Case Generation

Generative AI can automatically generate test cases based on application requirements or specifications. This feature helps save enormous test scripting time and reduces human error. Through Natural Language Processing (NLP), it can use prompts, documentation, or requirements to create relevant test cases. This capability helps achieve good test coverage that even covers edge cases.

Here is an example of this. Let us assume that we want to search and add the ‘iPhone 16 Pro Max‘ to the Bestbuy app’s shopping cart. The usual traditional approach is to identify and interact with the UI elements through CSS/XPath locators and code the test scripts in some programming language, using Selenium, etc. This will certainly take time and effort, which we all want to save in fast DevOps and Agile environments.

The smarter and easier approach is to use some generative AI-based tool and just let the test automation tool know what you want to test. A prompt can be: “Login to the Amazon app, search iPhone 16 Pro Max, and add to the cart. Verify that the product is added.” That’s it!

Generative AI will take care of the next steps and generate the relevant test case. Another option is to write your own test case in natural languages, For example, you can explicitly write:

enter "iPhone 16 Pro Max" into "What can we help you find today?"

click "submit search"

click "Apple - iPhone 16 Pro Max 1TB - Apple Intelligence - Desert Titanium (Verizon)"

click "Add to Cart"

click "Go to Cart"

click "Checkout"

check that page contains "Place Your Order"

The test case above is an actual testRigor test case. It is an intelligent, Generative AI-powered tool that lets everyone on the team, be it manual testers, BAs, or SMEs, contribute to test creation and execution.

Benefit: It helps maintain excellent test coverage and speeds up the testing process by eliminating the need to code the test automation scripts. Basically, involves everyone on the team in the testing process!

Test Data Generation

Generative AI can create insanely large amounts of unique synthetic test data that mirror the actual real-world data. Using regex expressions or plain English commands in your prompts, you can generate enormous unique test data for email, name, contact number, address, etc. This data is critical for testing scenarios where data privacy or data scarcity is a concern, such as in healthcare or finance.

Benefit: Enables effective and privacy-preserving testing across various test scenarios. Test data is crucial to test that applications perform well with diverse data inputs.

Negligible Test Maintenance

Generative AI does not depend on unstable locators like XPath or CSS selectors. It uses a more intelligent approach based on the visual and functional attributes of the UI elements. So, if the UI structure changes, the AI can still identify the correct elements based on how they appear or behave, reducing the need for frequent manual updates to locators in the test script. This is because Generative AI can understand the context in which elements appear.

Example: On an application’s login page, a button labeled “Sign In” is changed to “Login”. With traditional testing tools or frameworks, you need to change the locators in test scripts every time such changes happen. However, here, the AI can recognize that these labels serve the same function and adjust the test case accordingly. This capability reduces the test script failures due to minor text, layout, and requirement changes. This adaptability means that the test cases are always appropriate and updated as per the software version.

Benefit: This lack of XPath and CSS dependency ensures ultra-stable tests that are easy to maintain. With these intelligent tools, you can save up to 99.5% of maintenance time.

Automated Code Review

Generative AI models trained on vast codebases can help identify code smells, anti-patterns, and potential bugs. These models can act as “assistants” during code reviews, flagging potential issues for developers and suggesting solutions. Read more about AI agents in software testing.

Benefit: It increases code quality by catching issues early in the development cycle, reducing the expensive cost of fixing bugs later in production.

User Interface (UI) Testing Using Vision AI

Generative AI can simulate user interactions within an application to test various UI elements. By generating different input combinations and testing them, it can identify UI inconsistencies, responsiveness issues, and accessibility challenges. For example, Vision AI resolves the testing issues that arise while dealing with dynamic content, frequent UI changes, or complex visual elements. Vision AI addresses these limitations by enabling test automation tools to “view” the application under test (AUT) in the same way you, as a human, would. Instead of depending solely on code-based locators, Vision AI uses computer vision algorithms to recognize and interact with UI components based on their visual appearance, such as shape, color, size, or position.

Here is an example that utilizes Vision AI to validate a natural language statement about the webpage as below:

open url "https://bestbuy.com"

check that page "has a search bar on the top of the page" using AI

Benefit: This enhances UI robustness to make sure users have a smooth and functional experience across devices and screen sizes.

Supports BDD and SDD

You can use the power and advantages of shift left testing with generative AI features. With shift left testing, your team can create test cases early, even before engineers start working on application code using SDD (Specification Driven Development). This intelligent test automation lets you use BDD in its true sense, which is an attempt to express end-to-end tests from an end user’s perspective. With plain English test cases, you can write test cases as the end-user will see them without any implementation (programming) overhead.

Benefit: The application code and automation test cases can be generated using the same product specifications to speed up the entire development and testing process.

Code Generation for Test Scripts

Tools like GitHub Copilot and OpenAI Codex, which work on generative AI, can assist in writing code and even refactoring existing code for better performance or readability. In testing, these tools can create automated scripts, saving testers time and enabling quicker test case scripting. Otherwise, if you want to perform automation testing using natural language commands, you can always go for codeless test automation tools.

Benefit: Reduces manual scripting time and makes code more efficient and easier to maintain.

Natural Language Processing (NLP) for Requirement Analysis

Generative AI can analyze requirements written in natural language and convert them into test cases or executable scripts. It is intelligent enough to understand the intent behind each requirement, and it ensures all conditions are accurately reflected in testing scenarios.

Benefit: Ensures that requirements are directly translated into tests. It reduces misunderstandings and ensures more accurate testing.

Self-Healing Tests

In test automation, generative AI helps create “self-healing” tests that adapt to application or requirement changes without any manual intervention. If the application UI changes, the AI can recognize and adapt the tests to match these changes, reducing considerable maintenance efforts.

Benefit: Minimizes the need for frequent test updates, making test automation more resilient, stable, and cost-effective.

Continuous Testing and Integration

Generative AI supports continuous testing and DevOps practices. It seamlessly integrates with continuous integration/continuous delivery (CI/CD) pipelines through CI/CD tools such as Jenkins, GitLab CI, CircleCI, Buddy, etc. This integration ensures that testing is conducted at every stage of the development cycle. It helps with real-time feedback to the development team to fix bugs and facilitate rapid iterations.

Benefit: It enhances continuous integration and delivery, making it easier to catch defects early in the development cycle and reduce release times.

Generative AI in Software Testing: Use Case

Let us review a use case of generative AI in software testing:

Test AI features

Through generative AI-based test automation tools, you can test real-time user sentiments, positive/negative statements, user intent, etc. These findings will help you act immediately based on that information to solve the customer issue and provide greater user satisfaction. You can test your LLMs and AI using generative AI-based testing. If you see that the real-time user sentiment is negative (e.g., a user is not able to make a payment and book tickets), then you can immediately let customer support take action and aid the user in their user journey.

For example, you can verify whether the user in a customer chatbox has a positive message or not using AI as below:

check that "chat" "contains a positive message" using ai

This testRigor command utilizes AI to analyze the page/screen, in this case, the chatbox (LLM). It then performs complex validations that were previously only possible to do manually. Generative AI makes testing advanced AI features such as LLMs straightforward, simple, and faster. If you are looking for more details, here is an article about Generative AI in software testing.

Conclusion

Generative AI is helping in test case generation, test data creation, SDD, reducing maintenance, CI/CD, AI testing, self-healing, plain English tests, etc. It resolves many challenges associated with traditional testing frameworks or tools. These capabilities, in turn, help achieve greater test coverage, robust test scripts, faster delivery, and high-quality final software products. As we can see, rather than merely automating existing tasks, generative AI introduces a new intelligence layer. This enhances testing processes since it anticipates and adapts to changes, reduces human error, and supports a more intuitive ‘human-like‘ testing process.

These AI agents are helping to keep the software testing as human as possible so that we can test the workflows closer to what our actual end-users will perform.

About the Author

Pragya Yadav is a Content Evangelist with 17+ years of experience in the IT industry. She loves to research, learn, and execute the latest technologies and how they can ease human life. Her creative spark led her to the field of writing, which she thoroughly enjoys!