We implemented MagicPod to empower a single person to oversee the quality of 30 services. This led to a sevenfold increase in test execution frequency and detection coverage, achieving our ideal testing system.
GMO Media, Inc.
In our 22nd user interview, we had the pleasure of speaking with representatives from GMO Media, Inc., joined by Mr. Ito, CEO of MagicPod. We discussed their specific use cases and the key factors behind choosing MagicPod.
About GMO Media, Inc.
Since its founding, GMO Media has expanded its online media business across various sectors, developing and operating services such as rewards-based point sites, gaming platforms, and programming education portals. Additionally, as a solutions provider, GMO Media offers monetization support to partners through services like "GMO Repeatas" for point site development and "Affi Town by GMO" for performance-based advertising.
KEY POINTS
- Initially used OSS testing tools but found limitations with a small team, prompting the shift to SaaS automation tools.
- MagicPod was chosen for its unlimited test executions and ease of account creation.
- Easy test scheduling resulted in a sevenfold increase in both test execution and detection coverage.
- Error detection insights have helped development engineers improve systems.
From left:
Kentaro Nakamura, QC Team, Service Development Department
Masahiko Beppu, Director, Head of Technology Department
Nozomi Ito, MagicPod CEO
Journey to Implementing MagicPod
Mr. Nakamura (hereafter referred to as Nakamura): I joined GMO Media in 2016. In my previous job, I was also in charge of content testing, and I’ve been involved in software testing for about 15 years. At GMO Media, we have three main departments, and I’m responsible for automating the tests of around 30 services across these departments.
Ito (CEO of MagicPod): Managing that much content sounds like a lot of work.
Nakamura: Each team had its own processes and methods, so the challenge was how to standardize everything. When I joined, some teams were already conducting E2E tests, but the tools and languages they used were inconsistent, making it difficult to initially establish a unified testing system. Additionally, the QC (Quality Control) team consisted of just two members, including myself, with the other member having limited testing experience, so as a QC team, we began by sharing our know-how.
After a period where we directly participated in each team as testers, we concluded that the QC team should focus on driving standardization and automation. This led to a full-scale implementation of test automation. We started by creating the test base and implemented it using OSS testing tools, but setting up the environment was difficult, and maintenance was especially challenging.
Ito: So QC refers to overall quality control, and QA focuses on improving the quality of individual services, is that correct?
Nakamura: Exactly. After joining GMO Media, we continued implementing with OSS testing tools, but by 2020, the team became a one-person operation, and I realized that "the workload was too much". This led us to consider SaaS-based test automation tools. After implementing MagicPod, we were finally able to establish the ideal system. Moving forward, I'd like to take on more activities from a QA perspective.
Decisive Factor in Choosing MagicPod
Nakamura: It was in 2021 when we actually started considering test automation tools. Around that time, AI-powered test automation tools were emerging, and we thought it would be useful if AI could handle certain tasks. We compared three tools, including MagicPod. One of the tools was primarily in English, so we eliminated it, and then tested the two tools that were available in Japanese.
What attracted us to MagicPod was its unlimited test executions and unlimited account creation. Additionally, the pricing offered a reasonable entry point for us to start small, which was a significant factor in our decision.
MagicPod runs Selenium in the background, and its UI is similar to Selenium IDE, so it's very user-friendly for Selenium users. Specifically, I really like how the commands are vertically aligned on the left side of the test creation screen. It's incredibly easy to see and use.
I had previously known about Mr. Ito from reading his Selenium book, so when I researched MagicPod and realized it was that Mr. Ito, it was an additional factor in our decision.
Ito: Thank you! Does that mean you started considering it in 2021 and implemented it in 2023?
Nakamura: That's right. Ideally, each team's test engineer would have created their own test cases, but it was challenging to establish that system... So, I ended up creating all the test cases for our content from scratch, which took quite some time. In the end, I created nearly 10,000 test cases. I set them up and managed them using OSS testing tools, but over the past year, we transitioned to MagicPod.
We began full-scale operations in April of this year. My main tasks are giving MagicPod lectures and sharing test results with each team.
Ito: Do the development engineers also use MagicPod?
Nakamura: Currently, I'm the only one actively using it. However, I’ve received feedback from developers and designers saying, “I’d like to try it” and “It looks interesting,” and about 30 people have registered for accounts. I created a simple manual and initiated pair programming sessions with them, focused on running and reviewing the tests I developed. I believe more people will express interest as they continue to use MagicPod.
Utilization of MagicPod
Ito: You mentioned you created nearly 10,000 test cases. How many of those are currently implemented?
Nakamura: Around 70%. For each content, the test cases cover the basic functions of all major pages, from the non-logged-in state to the logged-in state. When we were using OSS testing tools, our coverage was about 60%, but since switching to MagicPod, we’ve increased our coverage by nearly 10%.
We run these tests every night, with the results being notified via Slack. My day at work starts by checking nearly 50 test results, and I reach out to the relevant team members if necessary. We’ve also added quality-focused members from each team to the Slack channel, so sometimes they respond to issues even before I do. I feel like we’ve established a really effective workflow.
Since switching to MagicPod, we have reduced maintenance time by more than half compared to when we were using OSS testing tools. The AI-powered self-healing feature has been a huge help—we simply approve the suggested fixes, and the work is done.
Ito: When you were using OSS testing tools, did you run them every day?
Nakamura: Back then, we ran them manually once a week. We tried to automate it using a different method, but setting up the environment proved too difficult. Since we learned about MagicPod, we decided to go with it. After implementing MagicPod, we were able to easily schedule test runs, resulting in a sevenfold increase in our test execution count and detection coverage.
Ito: Do you encounter any flaky tests that sometimes fail without any actual bugs?
Nakamura: Yes. We occasionally encounter timeout errors due to slow site performance. These are issues on our end, and they pinpoint areas in our development environment that need improvement. For our development engineers, these errors provide valuable insights, so they're incredibly helpful.
Ito: Currently, you're using MagicPod for browser testing. Are you considering mobile app testing as well?
Nakamura: Many parts of our app use web views, so browser testing covers a large portion. However, the purely native parts of the app are still tested manually. Recently, there has been a growing interest in mobile app testing within the company.
Leader's Support Drives Ideal QC Team Operations
Ito: Today, we’re also joined by Mr. Beppu, who heads the technology department. Mr. Nakamura, how did you communicate with him about test automation and the implementation of MagicPod?
Nakamura: Actually, it was Beppu who formed the testing team before I joined GMO Media, and he’s one of the people who’s always had a high awareness of testing and quality. While I regularly reported the results and trends as we worked on test maintenance and automation, his understanding in these areas made the transition to MagicPod much smoother.
Beppu: That’s right. At GMO Media, I believe we maintain a strong focus on quality. To ensure this, it’s best and cost-effective for developers to write the test cases themselves with a clear understanding of the requirements, and then thoroughly test everything before any releases or deployments.
However, with Nakamura being the only member of the test team, the challenge was how to improve efficiency as the number of services continued to grow. I also made sure to maintain open communication, which allowed us to quickly decide on implementing MagicPod.
Nakamura: Our security lead, Irokawa, who has a high awareness of the importance of testing and quality, has also emphasized that E2E testing is crucial from a security standpoint.
Irokawa: Specifically, at GMO Media, we run MagicPod E2E tests as part of our security measures when we perform regular OSS updates. OSS updates affect many areas, so manually checking which modules were impacted one by one isn’t realistic. Therefore, we consider the OSS update complete if the E2E tests show that all main and critical functions are working without any issues.
Nakamura: As a company, we must continuously raise awareness of testing and quality. However, I believe the QC team has been able to operate ideally thanks to the strong support from our technology and security leaders.
In Conclusion
Nakamura: MagicPod's user-friendly interface and functionality make it accessible to everyone. When team members see it for the first time, their usual reaction is, "This looks fun!" The reasonable pricing and ease of implementation were also incredibly helpful.
At GMO Media, we release updates almost daily, including minor version updates, and with MagicPod’s unlimited test executions, we feel confident. Running tests daily to ensure proper operation and quality is incredibly important. I believe MagicPod is the perfect tool for test automation.
GMO Media, Inc.
- Tech Blog: https://zenn.dev/p/gmomedia
- Creator Blog: https://blog.gmo.media/
- Official Website: https://www.gmo.media/
