---

---

---

Service Test Office, Service Test Center, LY Corporation
Left: Makoto Fukunaga-san (Sticker Test Team Manager)
Right: Yuga Hashimoto-san (Automation Test TF)

---

Circumstances behind MagicPod deployment

Fukunaga-san (hereinafter, Fukunaga): We has about 400 staff members engaged in the testing process, making the testing organization one of the largest centers within the LINE Group. As part of this, there is a department that specializes in testing, as well as a team formed of about 150 people that carries out development and planning on a project-by-project basis, and they are involved in the testing of most of the services provided by LINE.

We deployed MagicPod in March of 2020. In the context of system development, which is becoming larger and faster, in 2019, we began to consider the introduction of test automation tools. This was because we felt that manual testing options alone would not provide the required efficiency and quality, and we needed to prevent testers from losing motivation due to the necessity of repeating simple tasks.

Hashimoto-san (hereinafter, Hashimoto): There had been an increase in defects due to the degradation of existing functions, which was considered problematic for the business. Despite the fact that speed was required, there was a limit to the resources and time available for testing. Therefore, we tried to enhance stability, mainly by automating regression testing.

Fukunaga: After investigating and comparing the tools that currently support Web/iOS/Android, we finally narrowed it down to three possibilities, and after using them on a trial basis, we made the decision to deploy MagicPod. The main deciding factors were as follows:

• Support was provided in a swift and courteous manner
• The expectation that technical assistance will be provided by a well-known development team
• It was even easy for non-engineering staff to use
• There is no limit on the number of tests that can be run, and MagicPod was superior

from a cost perspective. Currently, my team has about 7 or 8 members, and other than Hashimoto and one other who have engineering skills, nearly all of them are non-engineering staff. Although, in the beginning, they did not even know words such as “locator” or “XPath,” now they are actually playing strategic roles.

Hashimoto: Of course, it is preferable if the employees have engineering skills, but as long as we use MagicPod, this is not a problem. I believe the current environment is more and more conducive to the participation of non-engineering staff.

Fukunaga: Hashimoto and the other member with technical ability take charge of the areas where engineering skills are necessary, such as when building a CI environment and when MagicPod needs to be linked with other tools, including in-house tools. This is actually the second time we have tackled automation. Before, we were mainly utilizing Selenium WebDriver and Appium, but when members with specialized skills transferred or retired, they were unable to pass their work on to other members and the team was disbanded.

This time, with the help of MagicPod, we have created a sustainable structure, which has opened up new career paths for existing members within the company. As this had actually produced results, other bases told us that they “wanted to use it too,” and currently we are changing the contract to that for the group as a whole, and it is being used for 30-40 projects.

---

MagicPod use cases

Hashimoto: In our company, we are using MagicPod for projects, and these are being used differently for each project, but in many cases regression tests, which are repeatedly run, are periodically run once a day. The benefit of this is that we are able to detect degradation at an early stage, and this facilitates smooth corrective actions and prevents us from panicking when the deadline starts to draw near. Personally, I am using it to take data when running 24-hour tests designed to test the reliability of the automated tests created using MagicPod.

Fukunaga: As, due to security reasons, it is necessary that the testing environment is kept in-house, MagicPod is run locally. For smartphones, multiple Macs and smartphones can be connected and run together, while in the case of the web, it is run with Docker, which is launched and executed on the in-house server (Linux).

Hashimoto: For one of the iOS/Android projects, MagicPod was able to automate about 30% of the regression testing. By doing this, the testing process was shortened from three days with three people to only two days. With the time created by doing this, we were able to perform some of the manual tests that we had not got around to doing despite feeling that “we should do them,” and this increased our coverage. In another web-related project, 98% of regression tests were automated with MagicPod, and what used to take about 16 hours was reduced to 2 hours. Nearly all regression tests can be performed on a periodic basis, and the feedback from developers has been that “as it detects defects instantly, we can develop proactively and with no worries.” In actual fact, we have been able to increase refactoring pull requests 1.8 times compared to what was possible before automation, and this has contributed to the speed at which we can improve the software.

Fukunaga: In the case of new projects, it is necessary for us to judge where to start from, so in our company, we create a checklist of suitability for automation and set a priority ranking. This helps us to make our own decisions, and be persuasive when we explain to each stakeholder "why we are starting from here."
We referenced case examples from other companies and ISTQB materials when performing the checklist ranking, and combined the wisdom of those who came before us when creating the format.

---

Final deciding factor for choosing MagicPod

Fukunaga: There is something that I would like to ask Ito-san about, but whenever we submit a bug report or make a request, we are told “it will be released this weekend” or “we shall release it next week.” We are always astonished at the fast lead time. Please let me know your secret for being able to respond so quickly to urgent requests.

Ito: We have created and automated a large number of regression tests, and by being able to release the software once all of these pass, we are able to realize a rapid lead time. If the design of the product itself becomes extremely complex, the scope of impact of any correction when fixing the code will be hard to read, so we hire people on who can program in a precise way and prevent the code from breaking down. The design is easy to fix in that, when something happens, we “just need to fix it here.”

It is also the case that a high level of priority is attached to bug reports and requests from our customers. During the implementation stage, in particular, there were several instances where automation stopped due to a decrease in motivation caused by continuous blocked items when the tests were automated. At my previous company, I frequently witnessed colleagues being obsessed with the bugs in the automated testing tools and being heartbroken when automation was not possible. Due to this, I am well aware of the importance of responding quickly, and I think this mindset has pervaded throughout the whole team.

Fukunaga: While listening to you I was reminded of an important factor when choosing MagicPod that I had forgotten just now. The fact that I found Ito’s attitude and stance highly trustworthy was an important point. That was the final deciding factor.

Fukunaga: To continually develop automated testing, it is necessary to constantly battle with error analysis. The fact that you provide solution proposals right away when we face difficulties is extremely helpful. The fact that you tell us the reason why it is happening this way means that we can learn not to repeat the same mistake. It is also very useful for us that you also provide consultation on operations and structure.

Ito: I am glad to hear that. Thank you!

---

---

Finally...

---

Fukunaga: I looked at some published case studies on other companies, and felt that we were definitely in the same boat. I would like to talk a little about “human characteristics.” When engaging in automated testing, it is my impression that simply wanting to take the place of humans and automate (substitute for) what has been performed manually will often cause stumbling blocks along the way. There has recently been a great deal of information provided on test automation, so many people are familiar with the "characteristics of automation;" however, if you also look at "human characteristics" you may find potential needs in areas that you have assumed will always be done by humans.

---

“The 8 human characteristics”

1. Humans are fickle
2. Humans are lazy
3. Humans are inattentive
4. Humans lack perseverance
5. Humans hate monotony
6. Humans are foolish
7. Humans are weak in terms of logical thinking
8. Humans lack decisiveness

*Hidetoshi Takahashi - Human Factors and Reliability. Collection of reports from Human Interface in Computers Symposium, 1983

I think that if humans can delegate the things that humans are not good at or cannot do to automated testing, we can improve the value of the work that ought to be done by us humans and be able to assign the right roles to the right people.

---