28. April 2023 By Elif Gürcüoğlu
ChatGPT 101 for QA Engineers
AI will not replace us; it will replace only those who cannot use it in the right way.
Nowadays, IT employees are likely to be concerned about losing their jobs to AI and being replaced by it. What I believe is that while AI is intended to replace manual labor with a more efficient and faster method of doing work, it cannot completely replace the need for human input in the workplace. Just like we all experience with ChatGPT.
The rise of ChatGPT is on everybody’s agenda. Beside the concerns about how it changes the way work has been done so far, it also feeds the curiosity of how far it can go. There is one fact that we have to accept among all the thoughts about it: there is no way other than to learn how we can integrate AI into the way we work.
QA engineers are perhaps the ones who can get the most benefit out of ChatGPT's skills. I believe that in the near future, it will become an indispensable part of their toolbox, maybe even their resume.
In this article, I would like to share some fundamental GPT practices for QA engineers.
Ask ChatGPT to design possible test cases
Being a QA engineer usually feels like solving several probability problems in a day. Resolving more probabilities means better coverage, which is something we always aim for. In other words, the more possibilities a test engineer considers and validates through their tests, the more successful they will be.
But it is not as easy to do as saying it. Discovering all the possible scenarios requires being fully focused. At this point, it is not a bad idea to get support from ChatGPT. As you know, two heads are better than one. If the other head is AI, it is even better. :)
I have to admit, my first try was really impressive. I provided some business rules to ChatGPT and asked for relevant test scenarios. The exact prompt I used was as follows:
Input
“Write as many test cases as you can for this scenario. If you are a new customer and you want to open a credit card account, then there are three conditions. First, you will get a 15% discount on all your purchases today. Second, if you are an existing customer and you hold a loyalty card, you get a 10% discount. Third, if you have a coupon, you can get 20% off today.”
As a result, ChatGPT directly provided nine test cases, each of which was ready to use.
My follow-up question to ChatGPT was about turning the result into a table. The result was much more readable.
Input:
“Show the result in a table.”
Output:
Alright, it is good to see that ChatGPT is really good at probability problems. Apparently, this kind of result can expedite the process of test case design. But something very important is still missing in ChatGPT: the questioning attitude of QA engineers. If I were to design test coverage for these business rules, I would extend the borders by asking “what if” questions. Such as “What if the client is a new user but also has a discount coupon?” The elicitation of the hidden requirements is still something that we need to figure out.
Let ChatGPT guide you while creating Test Dataset
Test Data Design is mostly like finding combinations based on different parameters. QA teams are usually the ones who are responsible for creating these datasets or requesting the proper ones from the authorized business units. A missing test data during the test execution process could be painful from time to time. Since it could block the process, the QA team desires to have the dataset in advance. The following example from ChatGPT proves that when the conditions are fully met, revealing the test dataset is nothing to be concerned about anymore.
Input:
“What is the required test data set to run the cases above?”
Output:
Create test automation scripts as easily as ever
While automated tests make our test executions as fast as possible, indeed it requires a lot of time to write and maintain the scripts. Nevertheless, it is one of the most essential assets in a well-designed QA structure. That’s why test teams should put their focus on shaping the script rather than consuming their time by writing it.
Here, ChatGPT helps us get a template regarding the test we would like to automate. At this time, I picked a use case, which is more specific. I would like to see how good GPT is at test case design for features that require a bit of domain expertise. I moved step by step. First, I asked for the test cases in Gherkin format for a particular screen from a regular airline mobile app.
Input
"What could be the best test cases that should be written in order to test the availability screen of an airline mobile application? Please write the cases in Gherkin format."
Output
GPT was really good at using the Gherkin format. The follow-up question was about converting the first test case from Gherkin syntax to automation script. Since it is a mobile app, I preferred Appium to Java.
Input
“I need to automate scenario 1 in Appium for Java. Could you write an automation test script?”
import io.appium.java_client.AppiumDriver;
import io.appium.java_client.MobileElement;
import io.appium.java_client.android.AndroidDriver;
import io.appium.java_client.android.AndroidElement;
import io.appium.java_client.remote.MobileCapabilityType;
import io.appium.java_client.remote.MobilePlatform;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import java.net.URL;
public class FlightAvailabilityTest {
public static void main(String[] args) throws Exception {
// Set the desired capabilities for the Appium session
DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability(MobileCapabilityType.PLATFORM_NAME, MobilePlatform.ANDROID);
caps.setCapability(MobileCapabilityType.VERSION, "10");
caps.setCapability(MobileCapabilityType.DEVICE_NAME, "Android Emulator");
caps.setCapability(MobileCapabilityType.APP, "/path/to/your/app.apk"); // Path to the APK file of the airline's mobile application
// Initialize the Appium driver with the URL of the Appium server
URL appiumServerURL = new URL("http://127.0.0.1:4723/wd/hub");
AppiumDriver<MobileElement> driver = new AndroidDriver<>(appiumServerURL, caps);
// Navigate to the availability screen
// Assumes that the app has already been launched and is on the home screen
// You may need to modify the locators or steps based on the app's UI structure
// e.g., driver.findElement(By.id("com.example.airlineapp:id/home_button")).click();
// or driver.findElement(By.xpath("//android.widget.Button[@text='Home']")).click();
driver.findElement(By.id("com.example.airlineapp:id/availability_button")).click();
// Enter departure and arrival cities, departure date, and number of passengers
driver.findElement(By.id("com.example.airlineapp:id/departure_city_input")).sendKeys("New York");
driver.findElement(By.id("com.example.airlineapp:id/arrival_city_input")).sendKeys("Los Angeles");
driver.findElement(By.id("com.example.airlineapp:id/departure_date_input")).sendKeys("2023-04-09");
driver.findElement(By.id("com.example.airlineapp:id/passenger_count_input")).sendKeys("1");
// Tap on the "Search" button
driver.findElement(By.id("com.example.airlineapp:id/search_button")).click();
// Verify that a list of available flights is displayed
// Assumes that the flight details are displayed in a list view or similar UI component
// You may need to modify the locators or steps based on the app's UI structure
// e.g., driver.findElement(By.id("com.example.airlineapp:id/flight_list")).isDisplayed();
// or driver.findElement(By.xpath("//android.widget.ListView")).isDisplayed();
if (driver.findElement(By.id("com.example.airlineapp:id/flight_list")).isDisplayed()) {
System.out.println("Test Passed: List of available flights is displayed.");
} else {
System.out.println("Test Failed: List of available flights is not displayed.");
}
// Verify that the flight details include flight number, airline name, departure time, arrival time, and price
// Assumes that the flight details are displayed in separate elements with corresponding IDs or other locators
// You may need to modify the locators or steps based on the app's UI structure
// e.g., driver.findElement(By.id("com.example.airlineapp:id/flight_number
As clearly seen, using GPT in QA engineering can significantly simplify the process of developing automated test scripts. It can help to ensure that the scripts are accurate, comprehensive, and contribute to the overall quality of the product being developed.
The result above provides a clean structure that can be used just at the beginning of creating the final script. Surely, a lot needs to be adjusted. You should update the path of your app, or you definitely still need to update locators, and maybe should add some latency between the steps, etc. However, while we have such a tool, I believe it will make a significant contribution to shortening the duration of automated test design and maintenance work.
Will ChatGPT be our next tester on the team? I don’t think so. All we need to adapt is how we use it the right way. Here are some take-away advice:
- Ask questions in the correct order
- Provide as much detail as possible
- Define the role of GPT
- Describe in what format you want the answer
- Describe the complexity of the answer you want
This is only a small portion of what GPT has to offer QA engineers. It can also be used to analyze test results and provide insights for improving the quality of the product being developed.
As a conclusion, the future of ChatGPT usage in the QA domain seems promising. While GPT trains itself to provide more human-like results, QA teams should train themselves on how to extend use cases like the examples above and how to get quality results.