With automation, regression tests increased from 1 to 50 times. By creating an environment in which we could develop with peace of mind, the way we use time has changed.

Locoguide, Inc.

Mobile app testing

In the user interview, we spoke to MagicPod CEO, Ito, and Locoguide, Inc.
We spoke about a wide variety of matters, including actual use cases and the deciding factors in their selection.

Locoguide, Inc. Company Overview

Locoguide's business is based on "Tokubai," which is a flyer and shopping information service providing information on the best deals at nearby supermarkets and drugstores. The site currently hosts more than 55,000 listings, and more than 17 million people use the site each month. In April 2022, for the first time since the launch of the service in 2013, we plan to renew the logo. We will also focus on developing content and functions that meet customer needs to make shopping, an essential part of daily life, more convenient and enjoyable.

KEY POINTS

There were issues with quality when running all tests manually
The deciding factor in selection of MagicPod was the ability to use in a Cloud environment
Speed of feedback was another deciding factor
By being able to run as many tests as we like, the development environment has become more stable

Locoguide, Inc. Technology HQ, Technology Division, Quality Improvement Group
Mr. Ken Kishi

Issues before MagicPod deployment

I joined the company in December 2020, and since starting up QA, I have mainly been in charge of the “Tokubai” app up to this point. Unit tests and API tests had been automated for both the app and backend prior to my joining the company, but E2E tests for both the app and backend had yet to be implemented. The regression tests are two-week sprints, and as these were done manually until close to the final day, even if defects were found, there was little time to correct them. For that reason, we could not take our time with development, and although minor, some unintentional bugs inevitably crept into the new features that were released. At that point, both the CTO and others in the company began to become aware of the quality issues.

At my previous job, I had worked for a third-party testing company, and been involved in E2E test automation and CI construction. At the point when I felt like I wanted to try my hand at in-house development, I came across Locoguide, and hearing that they wanted to “proceed with everything from regression tests to automation,” I joined the company. As there was no point in automating the test design if there were still issues to be resolved, we first spent about three months preparing regression tests that had been conducted manually and surveying application engineers to identify the issues.

After the regression tests were prepared, the application engineers told us that they “could release the code with peace of mind,” but as we had to assign engineers every time that regression tests were performed, running them manually led to high costs. To create an environment in which they could develop with peace of mind, a system was required in which they could run high-frequency tests during development. We started the process of selecting an automation tool around March 2021. The story up to this point has been summarized on our tech blog, so please refer there for details.

Reference: “First QA team startup record” (Locoguide tech blog)

Circumstances behind the deployment of MagicPod

During the selection process, “writing the code ourselves with Appium” was also a possibility. With Appium, although we tried writing the code ourselves, it needed to be maintained with a device in the office, along with the mothership PC. I had struggled with constructing an Appium environment in my previous job, and as remote work started to become the norm, having one person update the OS, Android SDK, and Xcode every time a new version was released was not realistic. With SaaS systems, the service provider takes care of the management, so we felt that this was a solution appropriate for the situation we faced. We considered tools other than MagicPod, but we could not find any other tools that were strong in terms of mobile app automation that also had a proven track record, so we made the decision to use MagicPod.

I had been at a Test automation study session with Ito, so I knew about the existence of MagicPod before the start of development. I had never actually used it before, so I was allowed to use it on a trial basis first, and I thought it was really good not to have to create your own environment. The fact you can create tests without writing code means that you can build up a good tempo, which makes things a lot easier. When I was using Appium, I used to look at a button and copy the X-Path or id behind it....that was the sort of work involved, but with MagicPod, you can build scenarios intuitively by dragging and dropping UI elements from the screen when creating test cases. After reading the Medley and note tech blogs, the fact that they had a strong track record with other companies was also a strong deciding factors.

MagicPod use cases

Currently, we are using MagicPod for automating regression tests for the “Tokubai” app. We have one regular run for nightly builds triggered by a CircleCI job, and further runs about 4–8 times a day each time pull requests are merged. The run time is approximately 15–20 minutes for Android, and 50 minutes to 1 hour for iOS. If we did it before merging, there would be many blocks, so currently there are no flows where we cannot merge without passing through MagicPod.

The result notifications are provided via Slack and email, so they can be seen by our engineers. However, even if they fail on occasion, this may be because the scroll width is insufficient or loading is slow and times out, so the flow is that first check that I will conduct and, when necessary, I will request support from the engineers. Although I think it would be useful if the engineers used it more directly, none of us have much time, so this is a topic for future consideration. In addition to those related to MagicPod, I would like to hold various QA-related study sessions internally.

By automating regression tests, we can run them as many times as we like without using up engineer man-hours. When this was done manually, this was done once for the whole sprint, but now we have increased it to 50 times. By increasing the number of tests, we no longer need to wait until just before the final day for defects to be discovered, and as the tests are operating constantly, the engineers can develop with peace of mind. On a personal level, I am able to spend more time on QA for new features that are difficult to automate. If we performed regression tests manually, we would still be left with insufficient resources. I am glad that through automation the way that we use time has changed.

Speed of feedback was another deciding factor

Ito: I understand that you recently created a dashboard-like feature.

Kishi: Creating automated tests was not an end in and of itself, and we now feel that we want make improvements through daily maintenance. I made it myself so that I could list the number of runs, times, and results in the hope that it would serve as the basis for this. We are using a library created by DeNA called CIAnalyzer as a reference, and using MagicPod's API to store the data into Google BigQuery in a way that simplifies aggregation. The data stored in the BigQuery is displayed as a table or graph using the BI tool Re:dash.

It is available to everyone through the magicpod-analyzer. The README file contains the usage instructions, so I hope that others can learn how to use it and help the MagicPod community thrive.

Ito: I am very glad to hear that. We often receive requests for a dashboard function, so I would like to keep this in mind for future development.

Kishi: The speed of feedback in cases of requests and inquiries was one reason for choosing MagicPod. The "Tokubai" application is a community-based application, so it is not possible to proceed unless location information is obtained, and functions are limited. So when we had problems while testing functions using location information, your response speed was incredibly helpful for us. Thank you.

Ito: If the motivation of customers using the products is decreased, approaches to automation will also stop. We give high priority to those customers who are having problems with critical issues.

Kishi: I believe that many QA professionals feel that test automation, especially smartphone app automation, has too many obstacles to overcome and they become defensive. Now there are many people who are very knowledgeable about the subject, and there are convenient services like MagicPod. Things that can be automated should be automated, and humans can work on providing value that only humans can provide.