Reducing Regression Testing Effort by 50% through Automation! A QA Reform That Supports the Quality and Security of Cryptocurrency Trading Services

Coincheck, Inc.

Mobile app testing Browser testing

MagicPod CEO Nozomi Ito interviewed Go Itayama, who leads QA at Coincheck, Inc., about the key factors behind selecting MagicPod and how they have been utilizing it from the implementation phase to the present.

About Coincheck, Inc.

Coincheck, Inc. operates the cryptocurrencies trading services "Coincheck," which has been "Japan's No.1*" downloaded trading app for six consecutive years. The company's mission is "Making Exchange of New Value Easier" by providing better services based on the latest technology and advanced security measures. Coincheck aims to make the "exchange of new value" created by crypto assets such as Bitcoin, Ethereum, and blockchain more easily accessible to its customers.
*Among the Japanese crypto asset trading apps. Data source: AppTweak

KEY POINTS

Before implementation, automated tests were dependent on individuals, and regression testing effort was a challenge
With MagicPod, a system was built where anyone could create, manage, and execute tests
Regular uptime monitoring enables early detection of production environment issues for swift response
API testing was also automated, strengthening a stable system based on the test pyramid
By leveraging frameworks, the QA organization aims to strengthen itself and achieve continuous quality improvement

From left:
Go Itayama – Group Leader, Quality Assuarance Engineering Group, Application Architecture Department
Nozomi Ito – CEO, MagicPod

Itayama: I joined Coincheck in May 2023 and am currently responsible for QA strategy formulation and execution, test process improvement, the implementation and operation of tools such as MagicPod and QualityForward, as well as negotiations with external vendors.

The QAE Group is part of the Application Architecture Division within the development organization. This division is responsible for areas such as login systems, software version upgrades, and infrastructure validation. The group consists of two full-time employees and six contractors, totaling eight members, all dedicated to quality.

I started my career as a debugger for social games, then worked at LY Communications Corporation (formerly LINE Fukuoka) handling QA operations and test automation using Selenium. I had previous experience using MagicPod for automation, which made its introduction at Coincheck a smooth process.

Ito (MagicPod CEO): LY Communications Corporation has participated in our user interviews before—so you were with them back then as well! Thank you for using MagicPod at Coincheck as well. Earlier, you mentioned that the QAE Group specializes in quality assurance—what kind of activities does your team focus on?

Itayama: When I first joined, different teams had varying interpretations of test terminology such as "integration testing" and "system testing," leading to inconsistencies. We worked to standardize test terminology and its usage, ensuring clear definitions and rules.

Beyond that, we focus on test process improvement, involvement in the overall development process, and fostering a quality-driven culture. These are ongoing efforts within the QAE Group.

Adopting MagicPod: The Journey

Itayama: Before I joined, Coincheck adopted a waterfall development process, where a test phase was set after each development stage, and the product was released once testing was completed. At that time, there was no QAE Group, and a third-party verification company handled test design and execution. However, there were several challenges.

The biggest issue was that the test process was not well established, leaving individual developers responsible for test quality. While they were highly aware of testing, it was not systematically managed, resulting in inconsistent testing methods and lack of test result visibility.

Additionally, mobile regression testing had around 1,400 test cases, all executed manually, requiring approximately four person-days per month. This was a significant burden on the team. Although an automated test tool had been introduced before, it had become obsolete due to lack of maintenance.

Ito: What was the issue with the previous automation tool?

Itayama: A key problem was that there was no dedicated personnel to manage it, making systematic operation difficult. Moreover, the tool required specific skills, and the team lacked members with the expertise to maintain it. Additionally, the tool had a limit on execution counts, which contributed to its obsolescence.

Furthermore, QA activities were mainly focused on quality control (QC) during the testing phase, rather than broader initiatives like test process improvement, integration with development workflows (shift-left approach), and fostering a quality culture. To address these issues, we decided to explore a new automated testing tool.

Key Factors in Choosing MagicPod

Itayama: Initially, we considered automating from scratch using Selenium or Appium, but at the time, I was the only full-time member in the QAE Group, so we couldn’t assign a dedicated automation specialist. For that reason, we sought an automation tool with a GUI that allowed easy test case creation and management. Additionally, we looked for tools that offered unlimited test execution at a cost-effective price, making it easier to start small.

Ultimately, the decisive factor in choosing MagicPod was cost. We compared several automated testing tools, and MagicPod stood out as a lower-cost option while offering unlimited executions at a fixed price. Furthermore, the fact that it allowed easy test case creation via a GUI was also a key reason for our decision.

Since I had prior experience using MagicPod at my previous job, I knew that the learning curve was minimal. In fact, not only engineers but also contract-based QA members at our company are using MagicPod, and everyone is able to use it without any issues.

Ito: How do you handle onboarding?

Itayama: MagicPod provides starter guides for both mobile and browser automation, so we have new users go through them first. They experience the entire workflow step by step—setting up a project, creating test cases, launching cloud devices, installing apps, implementing tests, and executing them. With this approach, users can grasp the full workflow within a day without any major issues.

Additionally, MagicPod includes various features to reduce maintenance costs, such as its auto-fix functionality. From my past experience using other automation tools, I learned the importance of maintenance, and MagicPod’s maintenance-friendly features made it a reliable choice for long-term use.

Ito: Thank you! By the way, did you ever consider continuing to use the previously introduced automation tool instead of switching?

Itayama: After considering the cost, we ruled out that option. The previous tool had become obsolete, with only outdated test cases remaining, and the group that had managed it had been disbanded due to organizational changes. Given the situation, we decided it was best to start fresh and select the optimal tool, which ultimately led us to MagicPod.

Current Utilization of MagicPod

Itayama: Currently, we primarily use MagicPod for:

Automating mobile regression tests that were previously done manually
Uptime monitoring of service front pages
Verifying transaction history generation at the turn of each month and checking scheduled nighttime processes

Regarding mobile regression testing, we automated about 700 cases out of 1,400, prioritizing those that were highly important and easy to automate. The automated test cases now cover most of the key areas of our system.

For the remaining 700 cases, some features such as eKYC(identity verification) are difficult to automate, so full automation is not feasible. However, we plan to continue automating whatever is possible moving forward.

Ito: You mentioned that MagicPod is also being used for uptime monitoring of the front page. Can you share more details on how that works?

Itayama: Every hour, we have MagicPod access each page in the production environment and check for any unexpected differences. This helps us quickly detect issues such as broken links or pages that were mistakenly set to private. Previously, these checks were done manually, but automation has significantly improved efficiency.

For mobile regression testing, we run tests every night, and the results are reviewed in the morning when work begins. Initially, I handled these checks myself, but now we have a dedicated automation engineer managing them. If an issue is detected, and it's not due to a specification change, we notify engineers for immediate resolution.

Test scheduling is currently managed using MagicPod’s built-in scheduler. Due to the long test execution times, we can only run tests at fixed intervals for now. However, we aim to optimize test cases, split them into appropriate segments, and run them in parallel to reduce execution time. Ultimately, we plan to integrate automated testing into CI/CD, allowing tests to run automatically with each app build.

Test results are shared not only within the QAE Group but also with developers and other interested team members via a dedicated Slack channel. Currently, 36 members, including myself, are part of that channel.

Ito: How have the development engineers responded to MagicPod?

Itayama: Many have shown positive interest, saying they’d like to try it. Recently, when we switched to a MagicPod plan with unlimited test cases, some development teams requested additional accounts.

However, since developers are already busy with their primary work, only certain teams are actively utilizing MagicPod at this stage.

The biggest impact of MagicPod implementation has been the reduction of regression testing effort. Before introducing MagicPod, regression testing took four person-days per month, but with automation, it has been cut in half to two person-days per month. This reduction was achieved not only through automation but also by eliminating unnecessary test cases and refining the test content.

Additionally, regular uptime monitoring with MagicPod has enabled early detection of issues in the production environment, which has been a major benefit.

For example, a past configuration mistake on an external service caused some Coincheck pages to become private. Fortunately, MagicPod's scheduled checks detected this issue immediately, allowing us to resolve it quickly.

Building a QA Organization That Enhances Technical Expertise and Supports Development

Itayama: One of the unique challenges in testing at Coincheck is handling cryptocurrencies exchange-specific mechanisms such as call auction trading named “Itayose” (grouping orders to determine prices) and circuit breaker (temporary halts in trading during extreme price fluctuations). These involve complex calculations, so testing them requires verifying accuracy across various scenarios. Without domain expertise, creating comprehensive test cases is extremely difficult.

Additionally, we focus on collecting and analyzing test data, evaluating test effectiveness and efficiency quantitatively. Through these efforts, we aim to continuously improve the entire testing process, which is a core mission of the QAE Group.

Ito: I see. So your team's role goes beyond test execution and includes analyzing results and driving continuous improvement. What specific initiatives are you working on?

Itayama: A concept we consider when defining QA engineer roles is the “QM Funnel”. This categorizes QA engineers' expertise into:

TE (Test Engineers): Responsible for test execution
SET/PE (Software Engineers in Test / Pipeline Engineers): Focused on test automation
QA (Quality Assurance): Manages the overall quality assurance process

Currently, our QAE Group leans heavily toward TE, with myself covering some aspects of SET, but we lack senior QA coaching roles.

Moving forward, we aim to enhance each member’s expertise while also fostering cross-functional understanding. Ideally, we want developers themselves to take on QA responsibilities, or alternatively, assign dedicated QA personnel to each development team.

Ito: Are there any specific initiatives for skill development or collaboration with development teams?

Itayama: One major initiative is leveraging the “Test.SFF” framework to visualize team members' testing skills.

This framework allows us to assess strengths, weaknesses, and areas for improvement through self-evaluations and peer reviews, visualized in radar charts. We plan to use these insights for talent development, hiring, and team structuring.

We are also prioritizing API test automation. While E2E testing is important, automating API-level tests is crucial for improving system quality. We are currently testing Postman with Newman for command-line API test execution.

Additionally, we are adopting “TPI Next”—a test process improvement model—to evaluate and refine our test processes objectively. Initially, we will focus on improving upstream processes such as test planning, design, and test data preparation.

Ito: You are truly pushing forward with a wide variety of initiatives.

Itayama: Yes. Our ultimate goal is to build an organization where each member possesses advanced expertise in QA, has a broad perspective on the entire development process, and collaborates with developers to build quality into the product.

Additionally, we aim for the QAE Group to become a resource that provides knowledge and expertise on testing and quality assurance to the entire development organization, supporting and enhancing development as a whole.

Final Thoughts

Itayama: Implementing and sustaining test automation requires time and resources, but choosing the right tool can make a huge difference.

MagicPod is an excellent first step in automation. Its GUI-based test creation and unlimited execution model lower the barrier for many companies, making it an ideal solution for refining and improving test processes.