How to effectively utilize the "Outsourcing Plan" to automate and solve the issue of tests not finishing and delaying the start of the next development project.

PHONE APPLI Inc.

Mobile app testing Browser testing

For our 13th user interview, our CEO Nozomi Ito interviewed PHONE APPLI Inc.

PHONE APPLI Inc.;

With the vision of "updating work styles through communication reforms," PHONE APPLI Inc.; supports companies promoting work style reforms. Through the development and sales of the communication portal "PHONE APPLI PEOPLE" that strengthens clients’ organizations, and research on work style reforms in the world, PHONE APPLI Inc. aims to transform corporate culture and improve social corporate value.

KEY POINTS

Before the introduction of MagicPod, the situation was dire: "We can't start the next development because the test is not finished"
The decisive factors to implement MagicPod was simple - “permeability” and the “sense of security” provided by MagicPod support
Intuitive to use, with no issues even with first-time operations
MagicPod is a tool that allows you to easily experience successful automation

Left: Dai Matsuyama (Quality Development Manager)
Right: Yuri Sato (Quality Development Automation Team Leader)

Issues before introducing MagicPod

Mr. Matsuyama (hereinafter referred to as Matsuyama): I joined PHONE APPLI at the end of 2019, and back then the testing itself was conducted by a third-party verification company, but the development team was leading the way, so testing was closer to development. The problem was the management and administration part, and it was difficult to check the accuracy of the specifications because the documents were not well organized.
There were a huge number of test cases, and since we were following the waterfall model at the time. We would start testing when the development stage had settled down, but it would take months before the tests were completed. To be honest, I was in a situation where I couldn't start the next phase of development because the tests weren’t finished.

Back then, PHONE APPLI Inc., was in a period of expansion and the development process was about to switch to Scrum, so I thought, ``From now on, people with knowledge of QA and testing will be needed,'' so I joined the company full time and launched Quality Development. Currently, we have a team of about 25 people, including outsourcing.

Ms. Sato (hereinafter referred to as Sato): I joined the company right after Quality Development was launched in 2020. At first, I was in a Scrum, but in 2021, when the automation team was launched, I moved to and am currently working on automation with a team of three.

How MagicPod was introduced to PHONE APPLI Inc.,

Matsuyama: At an early stage after launching Quality Development, I had the feeling that automation was necessary, but it took me a long time to sort things out before I could get started. I think it was about the second half of 2021 when I started to take action little by little.

At first, I was considering open source Selenium, but I chose MagicPod because of the challenges of introducing Selenium and the support I needed to spread it among the team. At that time, MagicPod was the only commercial tool that supported both browsers and smartphone apps, so I was able to make a decision without much hesitation.

Of course, we didn't chose MagicPod by process of elimination. There are other points that I thought were good about MagicPod. One is that it is simple to use. The members of the test team are not necessarily able to write code, so I thought it would be good to be able to use without code – MagicPod fit that criteria. I was also relieved that the support at MagicPod was generous.

In addition to the future potential of MagicPod, the company name was changed from TRIDENT to MagicPod, that seemed like a sign of commitment to me. I also liked the speed with which functions were added. The fact that Mr. Ito was active also led to good expectations.

Ito (CEO of MagicPod): Thank you!

Matsuyama: Then there is the price aspect. I still think about it, but MagicPod service is really cheap. With unlimited test cases and accounts, other test tools cost more. We think that cost performance of MagicPod is also very good.

Dai Matsuyama (Quality Development Manager)

Use Case of MagicPod

Sato: I use MagicPod in practice, and I mainly use it for automated testing of web applications that are used from a PC. About 100 test cases are run every day. The test start time is usually at 1:00 a.m., but it may be moved to daytime due to the execution time of the verification environment.
Currently, our company releases about once a month, so before that, we irregularly conduct tests to check whether the environment has been properly upgraded.

I don't think it's necessary to test every day because there are times when it's not a release verification, but by running tests every day, you'll be able to see things like "failed due to environmental problems" or "just flaky". I think it's good to know whether the failure was really caused by an actual change in the software.

Ito: Certainly, it is an advantage of regular execution. Was the process of introducing MagicPod to your team smooth?

Sato: At first, I didn't know what to make and how to make it, but before I joined, my predecessor had created several test cases, so I started using it as a reference. The operation itself was intuitive without having to look up each one, so I don't remember any troubles while using MagicPod.
However, even now, if I have any problems or questions, I ask in the MagicPod Slack about once or twice a month, and the support replies immediately, which is very helpful. Actually, I'm sorry about my slow replies (laughs). I am always grateful for MagicPod’s support help.

Ito: I'm glad that we are able to solve your problems swiftly! How else is MagicPod used?
Sato: Every morning, we also use it for smoke tests to check whether the verification environment has started up. If it fails, it is set to send a notification to Teams, so that everyone will notice.

Ito: Are there times when they actually don’t start?

Sato: It happens quite a bit. Although there is only one test, it is used in five main verification environments. After that, I sometimes show QA and development members a simple demo that says, "You can automate this kind of thing." Also, although the number is not large, sometimes automation team receives requests from development too.

Ito: There are quite a few companies that struggle to introduce automation because they can't communicate well with the development side, so it's good that the development members are interested. Do QA make requests such as id settings?

Sato: I think people have high hopes for automation. But sometimes it's difficult to find a locator. I will look at the timing, but I would like to convey it as a request from to the development team soon.

Ito: It will improve the efficiency of maintenance, so if you have a chance, please let them know.

Yuri Sato (Quality Development Automation Team Leader)

How to effectively utilize the "Test Automation Outsourcing Plan"

Ito: How has automation progressed since you introduced MagicPod?

Sato: It's still about 30%. I would like to raise it to 100%, but there is simply a shortage of manpower, and the automation team has three people, so we have to interweave reviews, which is taking time. The quality of testing has definitely improved, but the issue is how to balance quality improvement with the amount of man-hours spent on it.

Ito: Earlier you mentioned that there are about 100 test cases, but is there anything you pay attention to during maintenance?

Sato: We have rules for inserting comments while maintenance. Also, checking the results of regular execution is not one person’s responsibility, but we have a rotation system in place to prevent checking to become dependent on one individual.

Ito: It seems to be good because knowledge is shared. What points are checked in test case reviews?

Sato: We are reviewing whether the test conforms to the rules mentioned earlier, whether there are any omissions in perspective, and whether the test can be understood by others.

Ito: I got the impression that your team is not short-sighted and that you are proceeding with automation while looking at the whole picture and the long term.

Matsuyama: First, I think it was good that we were able to use MagicPod's ‘Outsourcing plan’ and proceed with implementing while communicating with the leaders of the third-party verification company. I think it's going according to plan so far.

Ito: It's common to hear that test cases created by a third party cannot be maintained. It seems at PHONE APPLI Inc. you don’t leave everything to the third party but make sure you thoroughly check the test results yourselves, and I feel this has led to great utilization of test assets.

Wrapping Up

Matsuyama: There are various means of automation, and for example, you can use Selenium to automate without spending money. However, there are many people who face trials and tribulations while creating environments and learning how to use them. I myself have experienced that. On the other hand, with MagicPod, you can start small and easy with almost no study required if you don't do anything fancy. I think MagicPod is a good tool that allows you to easily experience successful automation.

Sato: Until I joined this company, I was also interested in automation, but understood the difficulty and couldn't take on the challenge. When I started with automation, I was worried about whether it would work well, but little by little, it's starting to take shape. If you get used to it, you can turn it around and use it every day without worrying about money. MagicPod is a good tool to take a step into automation.