Maintainability is the deciding factor in a “test automation tool” to support product growth. You can realize the value of QA by running a large number of tests.

GLOBIS Corporation

Browser testing

For the user interview, we spoke to MagicPod CEO, Ito, and GLOBIS Corporation.
We spoke about a wide variety of matters, including actual use cases and the deciding factors in their selection.

GLOBIS Corporation Company overview

Since its foundation in 1992, GLOBIS has developed a wide range of businesses based on its vision of “creating and transforming society through the creation of an ecosystem of people, capital, and management-related knowledge.” In terms of “people,” it offers the “GLOBIS management graduate school” as an incorporated educational institute, as well as the corporation-established “GLOBIS Executive School” and “GLOBIS Management School,” the GLOBIS Digital Platform, providing not GLOBIS Corporate Education and e-Learning training within the company, but also the “GLOBIS Manabi-Hodai (learn as much as you like)” fixed-rate video learning service. In terms of “capital,” there is “GLOBIS Capital Partners” a VC for investing and nurturing venture companies, and for “knowledge,” they are promoting their publishing business and the “GLOBIS Chimiroku (knowledge record)” website/application for transmitting information. In addition, to promote creativity and change in society, G1 hosts conferences, while KIBOW provides support for disaster recovery for earthquakes as well as social impact investment.

KEY POINTS

A deciding factor in deploying MagicPod was its high level of maintainability
You can realize the value of QA by running tests
The reverse-triangle testing pyramid was not working
Power of MagicPod community to provide resolution when something is not understood

GLOBIS Corporation GLOBIS Digital Platform
Wataru Yajima (QA Engineer)

Issues before MagicPod deployment

Yajima-san (hereinafter, Yajima): I never thought that I would have the opportunity to meet Ito! It was the book that Ito wrote about Selenium that was what first led to me becoming a QA engineer and learning about “automated testing.”

Ito: Thank you very much. Participating in case study interviews is a very valuable opportunity for me to get honest feedback from users. GLOBIS is very well known for its graduate school of business, but it seems that the GLOBIS Digital Platform department is also developing a wide range of products.

Yajima: Yes, that is right. The GLOBIS Digital Platform is currently offering four products, including “GLOBIS Manabi-hodai”, which is a fixed-rate video learning service for business knowledge, the English version of “GLOBIS Unlimited”, and the “GLOPLA LMS” learning management system. Of these, it is “GLOPLA LMS” that is using MagicPod.

GLOPLA LMS, released in May 2021, is a B2B product. After being deployed by major corporations, it is now being used by several thousand users. When a problem occurs, it affects the company as a whole, so quality control is extremely important. The QA team is part of a development organization known as the Digital Platform department, which is a team consisting of about 15 members, including subcontractors.

Ito: Are each of the products developed in-house?

Yajima: In the past, we contracted them out, but as of around 2017, we started to develop them in-house, and all of the products currently handled by the GLOBIS Digital Platform are developed in-house. The development of GLOPLA LMS started approximately three years ago, and as QA needed to be sped-up into one-week sprints from now on, we deployed an automation tool.

Ito: My impression is that many other companies have a two-week sprint. Releasing every week involves a very high frequency of testing.

Yajima: We chose MagicPod because it allows for fast but reliable sprints with very good maintainability.

Ito: Thank you very much. Did you compare it with other tools?

Yajima: The truth is that we had tried several times to implement automated testing in the past. We looked into using Selenium, but considering the “lack of members with the necessary engineering skills to perform maintenance in a stable manner,” we considered the use of a Cloud service. We tried several automated testing tools at the time, and found the fact that “maintenance could be performed simply” to be a real advantage of MagicPod.

For example, if we implement common steps, it is easy to update the XPath because all you need to do is fix one thing and then the subsequent tests will pass. The self-healing function is really good. Without that functionality, it would take a considerable amount of time to check the tree to see where the problem occurred, to find that XPath has changed, and reproduce the issue again to see if it works. The fact that people without engineering skills can perform this maintenance without needing to write code is really good.

MagicPod use cases

Yajima: In addition to its ease of maintenance, MagicPod has built up a relationship of trust due to its stability. If we make an inquiry when a problem occurs and are told “I will check with the person in charge,” our work will come to a halt. With MagicPod, however, the reply comes back really quickly. There are frequent updates, and high priority functions are often reflected more quickly than expected.

Ito: I am glad to hear that. Many members within the company are concerned about quality because testing tools inevitably tend to produce a certain number of bugs.

Yajima: GLOPLA LMS is a product with a rapidly expanding scale, and we plan to add one or two more automation engineers in July. Since E2E testing is extremely burdensome, ease of maintenance is critically important.

Ito: I see. We will continue to provide you with as much support as we can. At GLOPLA LMS, how often is MagicPod used?

Yajima: GLOPLA LMS is a one-week sprint, so we develop from Monday to Friday, deploy the following Monday and then release on Wednesday. For this reason, tests are run on Mondays and Tuesdays when deploying for staging. We also run tests every day at 11 PM to check stability. There are limits to the problems that humans can discover manually, and I think that an advantage of machines is that they can use the strength of numbers. They are able to find various issues, and the times when I really feel “this is really effective” are increasing.

Ito: Even if you run them every day, do you find “errors in things you made the day before”?

Yajima: Occasionally, yes. “Code that is due for release may be merge-deployed.” There are times when the engineers do not include it in the task, but then with good intentions, we consider a change in the specifications, thinking "it would be better to do this." We are then able to work with customer support before release. It is because we run tests every day, and know the behavior of the previous release that we understand this.

Ito: How do you share the test results?

Yajima: I check the test results and then provide a report to the development team, stating “no issues were found” or “this test failed, but we can ignore this.” The automated tests often find issues before the manual tests, and as the time until drafting the report is shortened, this reduces the likelihood that a bug will be found immediately before release.

By running a large number of automated tests, I think that the site engineers really get a feel for the value of QA. First, with the function tests, we carefully test the new functionality. Then, by thoroughly conducting regression testing, we are able to develop the product together with them as colleagues.

Advantages of having deployed MagicPod

Ito: Is it the normal state testing that you are mainly automating at present?

Yajima: Yes, that is correct. Basically, once a week before release, we create scenarios based on “the expected user behavior,” and run multi-browser test cases one case at a time. We would like to run the irregular state tests as well, but if we increase this too much, the man-hours required for maintenance increase and we cannot cover this.

Ito: I think that irregular state tests are a difficult area. I also say “if possible, it is better to run unit tests,” but this can be troublesome, and is not simple.

Yajima: GLOPLA LMS is using “reg-suite” for visual regression tests, and we check on all of our screens whether any difference has occurred each time.

Ito: That is a good way of doing it. I believe that, in terms of the test pyramid, if the unit and integration level layers are thick, this is effective for E2E as well.

Yajima: I think, if management is conducted properly, the E2E tests at the top can be made thinner. E2E should really just be the final check. We often see the reverse-triangle pattern used, but this requires running a large volume of tests and I don’t think that it is very effective.

Ito: There’s a lot of truth in what you’re saying. Because of the characteristics of MagicPod, that can easily occur, so it is always a challenge to see how we can move toward an ideal pyramid. Is there anything else that makes you glad you chose MagicPod?

Yajima: It has a really good community. I am really enjoying participating in the community. Our community manager is really working hard for us. If there is something that we do not understand, this is nearly always resolved when we ask the community, so we feel like the search cost is low as well. There are times when I am asked in DMs from acquaintances “How do you do this?” and “What do you do if this happens?”, so I can then ask “How do I do this?” in return.

Ito: It happens very quickly! It seems that the community is utilized in ways we didn't even know about. I am really glad, and find it very interesting.

Yajima: MagicPod was originally based on Selenium, so we often see people who are famous in the world of Selenium.

Ito: That is correct. I often find myself thinking “I have seen that person somewhere before.”

Yajima: I saw an example of somebody I worked with at my previous job at Mercari using MagicPod at his new company. I think that these people are working hard on test automation at their respective sites, and then gathering together as a community.

Ito: I am glad to hear that. I hope that it attracts even more interest and that we can make further improvements.

Finally...

Yajima: Recently, there have been a plethora no-code automated testing tools, so I think that there are other “good tools” out there. I believe that MagicPod has the ability to create a framework in which tests are not discarded, but are nurtured and extended.

Even people who are new to automated testing can ask questions in the community, and feel reassured that they will receive replies from members outside of MagicPod as well. Even engineers proficient with Selenium and Appium, who already have the core competencies, will have many avenues for utilizing their engineering skills. I hope that they will take these opportunities.