Test Automation to Reduce Costs and Improve Quality: How DeNA Broke Free from “Manpower Dependency
DeNA Co., Ltd.
MagicPod's CEO, Nozomi, spoke with DeNA Co., Ltd. about the deciding factors in their selection of MagicPod and how they have utilized the platform from introduction to the present day.
DeNA Co., Ltd.
Leveraging the internet and AI, DeNA develops businesses across both the entertainment and social impact domains. The company delivers Delight to the world through a wide range of services, including games, live communities, sports and smart cities, and healthcare & medical.
KEY POINTS
- Ever-increasing regression testing for long-running services has become a business challenge.
- Automating manpower-reliant quality management led to a significant reduction in testing workload.
- After comparing tools against 160 requirements, MagicPod was chosen for its comprehensive features.
- Automating 80% of 1,300 manual test items reduced hours and improved coverage.
- A unique utilization method involves intentionally failing tests to perform differential checks.
From left to right:
Tomohiko Tsuji, QC Group 1, Quality Management Department, IT Division
Kazuya Imamura, QC Group 1, Quality Management Department, IT Division
Nozomi Ito, MagicPod CEO
Mr. Tsuji (hereafter, Tsuji): I have been involved in quality management at DeNA for about 10 years. Initially, I was mainly in charge of content released on Mobage, but now I also handle the platform itself and services other than Mobage, providing cross-departmental support for the implementation of verification technologies and tools.
In our IT Division, besides the QC (Quality Control) teams that handle testing and quality assurance while being involved in projects, there is also an organization called SWET (Software Engineer in Test) that supports testing from an engineering perspective. While SWET primarily provides technical support for the "early stages" of development, such as unit testing, the main mission of our Quality Management Department is to ensure quality in the "later stages" of development, such as E2E testing.
Mr. Imamura (hereafter, Imamura): I am on the same verification technology team as Tsuji, where I am involved in member training, technical support, and maintaining the test infrastructure. I was originally responsible for game testing, but for the past one or two years, I have been mainly in charge of streaming apps and services.
At our company, the level of independence and confidentiality varies greatly from project to project, which makes it difficult to get a bird's-eye view of company-wide tool usage. Amidst these circumstances, one of our key missions is for us in the Quality Management Department to select official tools and promote their adoption to improve quality across the entire company.
Challenges Before Introducing MagicPod
Nozomi (MagicPod CEO): How were test automation tools being used within the company?
Tsuji: DeNA has always been a company that is particular about quality itself. However, in the past, that approach was heavily dependent on "manpower." For example, methods such as "assigning a large number of people to large-scale projects to guarantee quality" were sometimes used, but that wasn't necessarily efficient. With that background, a more efficient approach is now required. There are over 100 people involved in testing within the company, and their skills vary. Some projects proceed using in-house developed tools without using external ones, while other projects use Appium or Selenium, so the use of test automation tools depends on the project.
Nozomi: I heard you were also using another test automation tool before you introduced MagicPod.
Imamura: At the time, we needed to respond to faster release cycles and increase project productivity to provide our services for a longer time. On the other hand, as service operation periods lengthened, the number of regression test cases and their importance only grew, making automation a challenge.
Tsuji: When selecting a tool, the first condition for us was that it had to be no-code. As I mentioned earlier, among the more than 100 people involved in testing, not all of them, including myself, can necessarily read and write code. On top of clearing that prerequisite, the tool we were using before presented the issue of escalating costs due to an increased number of test executions, and at least at that time, MagicPod had superior mobile app testing capabilities, which led us to consider a switch.
The Deciding Factors for Adopting MagicPod
Nozomi: When you were making your selection, did you consider tools other than MagicPod?
Tsuji: At that time, rather than broadly considering new tools, we proceeded with our evaluation from the perspective of how MagicPod compared to our existing tool. There were three main reasons why we decided to adopt MagicPod. First, its cost-performance was excellent. Second, it supported both web and mobile apps, which was a mandatory requirement for us. And the biggest deciding factor was its comprehensive coverage of our functional requirements. MagicPod cleared the most items on the list of about 160 functional requirements we had set, including the ability to bypass multi-factor authentication, which was a major factor in our decision to go with MagicPod.
Nozomi: How long was the period from consideration to the actual migration?
Tsuji: The consideration phase itself took two to three months, and I believe the actual migration work took about half a year. However, we felt confident in the tool at a very early stage, and I recall that our policy to "proceed with the migration" was solidified three months after we started using it. You issued us a trial account, so we were able to proceed with verifying its features in a full-scale environment from an early stage. This allowed us to advance the migration decision and the actual migration work in parallel.
Nozomi: Regardless of a tool's quality, I imagine there must have been some resistance from the team members on the ground, for whom switching is a hassle.
Tsuji: Of course, we heard from the team, "Why change when our current tool is already operating stably?" However, the benefits of cost-performance and feature comprehensiveness were significant, even after accounting for the effort required for migration. The pricing was at a level where we felt we could "just try it for a year," so we were able to make a decisive move with little hesitation.
Nozomi: Did the switch actually proceed smoothly?
Tsuji: There was no major confusion. Thanks to MagicPod's very clear support system and help documentation, even if we stumbled during implementation, we could quickly find a solution. In addition to the easy-to-understand app preparation and authentication settings, we were also helped by the Slack channel where we could directly communicate with support members and receive prompt answers. Of course, since it was a new tool, there were parts we had to relearn, but I feel that overall there were no particularly major problems. The migration is now almost complete.
Imamura: With the change in tools, we're also hearing from the team that "we can now do things we couldn't do before." There was the effort of recreating test scenarios from scratch, but even considering that, I have the impression that the decision to migrate was the right one.
How MagicPod is Being Used
Tsuji: We have issued about 90 accounts since the introduction, but this includes not only active members but also members who are just trying it out. The fact that there is no limit to the number of accounts we can issue is another good thing about MagicPod; while we manage permissions, we can meet the demand from those who "just want to give it a try." Among our active users, there are about four projects that are particularly engaged, and the image is that each has two or three core users. Specifically, we use it for games, live communities, and healthcare-related services. In the gaming-related field, we utilize it for the development of tools like a "dedicated browser" to provide a more comfortable experience for our users, rather than for the game itself. In the case of a typical game, users choose their own playback environment, such as which browser to play on, but since we provide a dedicated tool, we must guarantee the quality of that playback environment itself. Therefore, as we continue to update the tool, regression testing to check for unintentional bugs is extremely important.
Nozomi: Was regression testing done manually before introducing MagicPod?
Tsuji: That's right, and that's where we had a challenge. Our team uses a Scrum framework and has a high development speed, but our regression testing was always done manually, and we were barely able to narrow down the roughly 1,300 test items to about 70. With the introduction of MagicPod, we were able to automate 80% of our test items, which not only reduced the workload for our staff but also dramatically improved our test coverage. The time we gained has also enabled us to pursue activities that further enhance quality, such as increasing the number of test environments, which is a significant achievement. The only tasks still remaining manual are about 200 items that require visual confirmation, such as checking timing.
Imamura: I think we are also making things more efficient, such as by combining multiple related verification items into a single scenario. I often see similar creative efforts in other projects as well.
Nozomi: Is it difficult to use it for testing the game itself?
Imamura: In games, the UI and displayed content change frequently due to events and campaigns, so it's difficult to maintain the tests each time.
Tsuji: To be honest, there are still difficulties in fully utilizing it for regression testing of the game itself, but we are using it in a slightly unusual way to "check if the display is updated correctly." For example, instead of manually checking changes in areas like in-game announcements and event banners where the content changes almost weekly, we use MagicPod. By intentionally causing a test to fail and comparing the screen at that moment, we use it to "efficiently check if the display has been updated to the intended one." However, the drawback is that this method lowers our health score (*) (laughs). This might not be the intended use for MagicPod, but we're making use of it because it's faster and more accurate than doing it manually. * A metric indicating the health of tests in MagicPod.
Promoting MagicPod Internally
Nozomi: How do you handle onboarding for first-time users?
Tsuji: The Quality Management Department doesn't offer anything specific. In the first place, it's so easy to use that our perception is "they'll figure it out if they try it," so we don't feel it's necessary. However, as the ones promoting its adoption, we are naturally concerned about whether it's being utilized effectively, and we do push for its implementation in projects that are considering it.
Nozomi: What kind of challenges do you face when promoting its adoption?
Tsuji: Even in new projects, there are often members who have been involved with MagicPod in other projects, so it's not often that the conversation starts from the very basics of what MagicPod is or how to use it. A bigger challenge is the cultural resistance, something like an "aversion to test automation." For example, this includes preconceived notions like "it's faster to check manually" or the mindset that "there are tasks that should be prioritized over regression testing." However, the overall constitution of the company has also changed significantly over the past few years. In order to continue delivering our services to users for a long time, it has become crucial how we reduce operating costs and generate profit. We are shifting towards leaner, more robust organizational operations, rather than relying on manpower as we once did. In that context, "test automation," which allows for managing costs without sacrificing quality, now holds an even more important position than before. I feel the entire company is shifting its mindset toward automating tasks that would have been handled by sheer manpower a few years ago, now using tools like AI, and embracing the idea that "humans should concentrate on more important decisions." That's why our promotional activities are centered on conveying how crucial regression testing is for continuing to deliver stable quality to our users, and spreading an understanding of the value of automating that arduous work.
Nozomi: In conveying this value to the teams, do you have any personal tips for promotion, like a "certain way of phrasing things that tends to resonate well"?
Tsuji: I do. It might be a slight exaggeration, but I make a conscious effort to share a sense of urgency that "we won't survive in the coming era unless we absorb new technologies like AI." Test automation skills are also linked to an individual's market value. I encourage them by saying, "This is a great opportunity to try a new tool at the company's expense, so let's actively make use of it." Of course, as an administrator, I am always mindful of striking a balance to ensure that using the tool itself doesn't become the goal and impede regular work, or that we don't overlook security risks.
Conclusion
Tsuji: Personally, I strongly feel that rather than large-scale teams, it is the smaller teams, or teams that are struggling with many projects, that should try an automation tool like MagicPod. Through automation, you become able to create a "clear distinction" between tasks that can be left to machines and creative tasks that humans should concentrate on. It's fine to start with a personal goal like "securing my own time." Taking a serious approach to automation can be a significant first step that greatly changes how a team works.
Imamura: I feel that MagicPod, as a no-code tool that can be used in Japanese, excels in ease of use and cost-performance. I think it's a perfect match for projects with few personnel or test teams that handle many small-scale projects but don't want to neglect and want to regularly execute regression testing.
There may be some challenges at the beginning of the implementation, but in our case, we were saved many times by the flexible support system of the Enterprise plan. Therefore, if anyone is hesitating, I recommend that you first try consulting with them.
When you actually implement a tool, it also becomes a good opportunity to review the manual testing process itself, which you had previously taken for granted. I think it's best to experience that "added value" beyond simply replacing manual work.
Photo location: WeWork Shibuya Scramble Square
