Leveraging ChatGPT for Effective Software Testing: Opportunities and Challenges

In today’s rapidly changing business landscape, enterprises constantly adopt new models and launch SAAS products to stay competitive. This requires testers to be updated with the latest domain knowledge to ensure they can effectively test the software products being developed.

Knowledge-sharing forums/platforms in software testing neglect the fundamentals, such as testing design techniques for functional testing, testing processes, and test management skills, in favor of technology and tools, leading to a lack of emphasis on essential competencies required for quality assurance. Furthermore, test managers, leads, and testers tend to create new templates and content for each project without reusing existing standard templates or handbooks, leading to extended efforts. The quality of deliverables also varies depending on the testers’ experience level.

Amidst the shift in recent times toward remote work, testers face new challenges when it comes to overcoming these problems with a lack of in-person mentors. By providing real-time guidance and support, Generative AI solutions such as ChatGPT can help testers improve their skills and efficiency in testing activities, ultimately improving their overall competency.

Generative AI, also called AI-generated content or synthetic data, has become increasingly popular in the tech industry due to its ability to produce unique and diverse content. The good news is that the emergence of Generative AI, such as ChatGPT, has simplified the testing process by leveraging advanced machine learning algorithms and natural language processing capabilities.

In this blog post, we will dive into the benefits of ChatGPT for testing and how it helps simplify a tester’s day-to-day activities.

ChatGPT for Software Testing

ChatGPT can significantly help testers across all phases and types of testing. As a generative pre-trained model, it has been learning from vast amounts of data, giving it knowledge of best practices from experiences worldwide. This makes it a valuable resource for suggesting templates and solutions to testers. In addition, ChatGPT can assist various roles in software testing, such as test managers, test leads, and testers.

ChatGPT for All – Test Managers, Test Leads & Testers

For test managers, ChatGPT can offer insights into creating effective test strategies for a project’s requirements. It can generate concise templates or detailed test plans with all the necessary prerequisites and sections, along with action items to get started. Also, ChatGPT can provide high-level or detailed estimates for requirements, including metrics for any type of testing, such as UI, API, performance testing, etc. Test managers can also get help from ChatGPT by describing the project details and allowing the model to suggest potential risks, mitigation, and contingency planning. This can help them streamline their activities across various aspects of test management.

ChatGPT is a tremendous help for testers in generating all necessary prerequisites, test steps, and test data for any specified requirements. The more detailed the prompts are with essential information on the requirements for ChatGPT to process, the better the quality and coverage of test scenarios and cases. Apart from generating test cases, we should harness its power for test scripts design, test coverage, support in selecting testing methodologies, guidance in test data preparation approach, test data generation, guidance in choosing the type of test design, optimal test selection, and analysis of test reports, to name a few.

Tests designed by ChatGPT for COTS products, such as Salesforce, ServiceNow, Guidewire, etc., have been noted to be more efficient and comprehensive. This indicates that the solution is trained on datasets specific to the domain and product features, resulting in better test designs. Utilizing ChatGPT for creating regression test cases and scripts for COTS products can significantly reduce effort, allowing more time and resources to be allocated to other areas, such as exploratory testing.

ChatGPT can create test scripts for various testing tools, including Selenium WebDriver, UFT, TestComplete, Katalon Studio, LoadRunner, SoapUI, JMeter, SilkTest, and more. The scripts are well-commented, written with adherence to best practices, and can save significant time and effort for projects that do not use no-code or low-code platforms. However, minimal efforts would still be required to insert the test scripts in the packages, object identification parameters, and test data.

Another notable feature of ChatGPT in test design is its ability to provide positive and negative test scenarios with varied test data combinations. It can also assist in selecting an optimal test set that achieves maximum coverage.

Selecting appropriate tests for good coverage of requirements can be time-consuming for any release. While we have accelerators in Hexaware that use in-built models to expedite this process, ChatGPT can also streamline this activity.

When to use ChatGPT?

While ChatGPT can support multiple activities in software testing, it being a generative AI, we must consider that the responses may not always be accurate. More examples of such cases are to come below. Hence the solution can be used to support testing activities and not serve as a replacement for it.

Below are a few efficient options on when to use ChatGPT:

1. Getting started quickly: When we are on Day 0 of our activity and require a set of data points to kick-start, ChatGPT can be utilized to bring out a structure.

Examples: Creation of a test plan template with all table of contents, creating reporting templates, generating test scenarios for business workflow, automating test scripts in any tool/language for functionality, and so on.

Prompt Example 1:

Prompt Example 2:

Response from ChatGPT:

2. Review deliverables: In our day-to-day activities, there are times when deliverables need to be submitted on time, and a lack of initial review can lead to multiple iterations of rework. ChatGPT can be effectively used in this aspect by providing details of an already completed deliverable and requesting the model to review and perfect it.

Examples: Provide a set of test scenarios with the requirement description for ChatGPT to find out gaps in test coverage or provide estimates for a requirement and ask ChatGPT to validate the factors and the correctness of the estimations.

3. Augment action items: Treating ChatGPT as a digital buddy supporting any action items that testers carry out will be the most recommended method in using ChatGPT for day-to-day needs. While human intelligence can work on the core activities, the model can help with other tasks that can be time-consuming. ChatGPT’s reasoning skill can also be used to analyze testing data points.

Examples: Provide a list of test cases and request ChatGPT to write additional/missing test cases, generate mock test data, identify redundant or duplicate test cases from a test suite, convert test cases from one format to another, or infer test logs and provide insights on test results.

It is noted that the more descriptive and clear the prompts are, the better the responses will be. Let us see an example of how the details in the prompt can help ChatGPT understand better.

Prompt Example 1 – High-level vague prompt for estimation

This prompt makes ChatGPT respond with a high-level estimation without details. However, it still provides a rough estimate.

Prompt Example 2 – Detailed prompt with the required information for estimation

This prompt helps ChatGPT understand more about the project to be estimated. Phrases such as “Test Manager,” “Estimate for an Automation Testing project,” “Consider test management efforts,” and “Give the estimate in a table format” guides ChatGPT on the actions to take.

The response to this prompt from ChatGPT is elaborate with the information that we asked for; you can check it out for yourself! However, we must consider the response a supportive kick-start rather than a direct solution.

Challenges faced while using ChatGPT for testing

Advantages and Disadvantages of “Context Preservation” Feature

What makes ChatGPT’s test design intelligence interesting is its “context preservation” feature, which allows it to correlate multiple functionalities and provide better test cases in subsequent iterations. However, these capabilities should be thoroughly validated before use, as it sometimes turns out counterproductive.

For example, when an estimation was requested from ChatGPT in the first iteration, a final number of 6200 hours was generated, and we suggested additional factors to be considered. During the second iteration of questioning, the model re-estimated various aspects with the new data coming to 7250 hours of effort; however, it considered 25% of the 6200 hours for regression.

This was because it preserved data from the previous response, which became obsolete as we corrected it later.

Believable incorrect responses

As ChatGPT is a large language model, and as stated by OpenAI themselves, there have been instances where ChatGPT was very confident in its response. However, it was not correct. In another case, for estimation, it did not consider maintenance and parallel execution efforts. When probed, ChatGPT managed to beat around the bush without a pointed response, making it unhelpful.

Conclusion

While Generative AI can help speed up the testing process by generating test assets, it cannot replace the critical thinking and creativity of human testers. Testers must also ensure the generated content is relevant, accurate, and meaningful. Moreover, the tester’s expertise is critical to test management, exploratory testing, and identifying and reporting defects that Generative AI might miss.

Generative AI responds to the same query in many ways due to its nature and based on individuals’ linguistic capabilities and interpretation skills. Hence organizations need to bring enterprise-level usage guidance & prompt engineering mechanism more like a “How-to” guide with thoroughly evaluated prompts to serve testing needs. This will ensure uniformity in the content usage generated by tools like ChatGPT.

In conclusion, Generative AI is an augmentative solution that can help ensure quality and speed and thus simplify the testing process. However, it should not be seen as a replacement for human testers but rather as a tool that can complement and enhance their work.