Regression tests that previously took 1–2 man-days to complete are now completed within 3 hours. The secret to nohana increasing its automation rate to 70%
nohana, Inc.
In our user interview, we spoke to the MagicPod CEO, Ito, and nohana, Inc.
We spoke about a wide variety of matters, including actual use cases and the deciding factors in their selection.
Company overview of nohana, Inc.
The mission of the company is to “bring smiles to as many families as possible.” In doing so, the company’s products offerings include “nohana,” which allows parents raising children to easily create photo books from their smartphones, and “nohana New Year’s Cards,” which allows them to create New Year’s cards.
KEY POINTS
- Regression tests that involved 1–2 man-days are now completed in 3 hours
- Speed and responsiveness of support leads to peace of mind
- The key to improving automation is the understanding and cooperation of engineers
- MagicPod was essential in automating nohana
Quality Assurance Group, Service Design Division, nohana Inc.
Left: Yoshiyuki Takeda (Manager)
Right: Kohei Tai
Circumstances behind the deployment of MagicPod
Takeda-san (hereinafter, Takeda): I joined nohana in 2015, but before that I had worked in QA as a subcontractor. Around the time the project was ending, I received an invitation from President Omori, who “wanted me to stay on,” and because I also wanted to “participate in the development of nohana,” I joined the company.
Every time that nohana releases an app, in addition to testing the parts that have changed, we perform regression tests to ensure that the changes have not affected other areas. We had approximately 300 test cases, and when we had to perform this manually for each release, it would consume 1–2 man-days.
It was also very time-consuming to implement this on multiple devices, so we always had the impression that we were wasting too much time. However, although we wanted to automate this, we just procrastinated thinking “we aren’t able to program, and don’t know how to build the environment...”
It was 2017 when I read Ito’s blog and learned about the existence of MagicPod, and thought “this is it!” As it was not clear whether members joining the company thereafter would have programming knowledge and experience, we did not really have the option of “writing test code with Appium,” and so considered that a tool where we could use “no code” or “low code” would be good.
At that time, we were starting to use Vietnamese off-shore operations for QA, and as we had more time to devote to this, we moved forward with a full-fledged deployment. We created a separate Slack channel and had Ito visit our office. If we made an inquiry, they would respond right away, and when we communicated something to the effect of “we would like to add this kind of function,” they would implement it right away, and I remember this being very useful at the time.
*Editor's Note: Currently, another member has been added to the QA team and production is being moved from offshore back to in-house production by the company.
Tai-san (hereafter, Tai): Even though I have barely any programming skills, I find MagicPod very easy to deal with; support is very fast and responds very quickly to my questions, which is very helpful. For MagicPod, in many cases something that seems like a simple question will cause an error, and you will not know what is happening if you have no knowledge of programming. It is very reassuring, therefore, to be able to consult with them at any time and receive support.
MagicPod use cases
Takeda: We are currently running tests equivalent to regression tests on apps in the develop branch every day early in the morning in our Cloud environment. We carry out regression tests on the release branch app in a local environment to check it on the real device before release. This takes about 3 hours.
Previously, we developed an original Slack App called "Magic Hand" using Google Apps Script and requested test runs from my own Slash Command. However, now we can execute it on Bitrise. Local execution is manual from the browser.
The regression tests are run every morning around 4:00 A.M., so the result notification is sent to Slack before we go to the office. If all pass, we can go into work thinking “Yay!”, whereas if an error occurs we know where to start our checks from. That is the general flow.
Tai: We rarely find defects in pre-release regression tests, but they can now be found in the tests run every morning. Just as an example, there were instances when a screen did not render properly and we thought it was simply taking a long time to load. We found out later, however, that it was because the changes made the previous day were no good.
Takeda: By automating what previously took 1–2 man-days for every release, we were able to complete everything, including the work conducted manually, in 3 hours. The implementation of MagicPod has brought about great improvements. The coverage rate of regression tests, which had been about 5% of all test cases in 2017, reached 20% in 2019, and is now approximately 69%, with Android taking the lead.
Tai: Another benefit has been the ease with which engineers are able to update the libraries. In the past, as "we didn't know where it would affect us," QA would be asked to test even a small update. Now, however, it is possible to detect any issues by regularly running tests. The engineers have told us that they “can make updates with peace of mind.”
The secret behind the 70% regression test automation rate
Takeda: With the cooperation of its engineers, nohana was able to advance to close to 70% automation. Although maintenance is the responsibility of the QA team, there may be some parts that cannot be done, or are difficult to complete within the team, thus making the cooperation of engineers essential.
Tai: MagicPod hosts study sessions for engineers and other interested parties, where they introduce the basic functions and allow them to actually experience the product on a “hands-on” basis.
Takeda: The test result notifications are sent to the Slack channel, where they are displayed for the engineers to see. It is not that the engineers need to take any prior action in relation to this. However, they are aware that MagicPod is constantly working. As they cooperate smoothly at the consultation stage, this is very important for deepening understanding internally.
Additionally, currently, at nohana, we are performing development in two-week sprints, and the three members of the QA team carry out reviews in addition to the development team’s review (sprint review). Based on this, we can look at the success rate of the automated tests and analyze which scenarios are unstable, and consider improvements for the next sprint. Android has been automated to a level close to 70%, and it is thought to be difficult to reach a higher level. Recently, we have been focusing on iOS improvements, which we had not made much progress on until now.
Finally...
Takeda: What I would like to relate to those people who are considering deployment is “not to try to implement too much at the initial stage.” I would recommend that they first make just one function and one scenario, make sure that they work properly, and then branch out from there. When reflecting on the actual experience, it seems that automation itself is the goal. I believe that if members are aware of what the automation is for, and they expand the scope of automation little by little, progress in automation can be made with fewer failures.
MagicPod was an essential tool for automating nohana. If we had not encountered it, I think that we would have been forever thinking “I really want to automate this...”, and would not have been where we are today. Services provided by nohana, Inc. include:
nohana Free Photo Book
nohana New Year’s Cards
