All our engineers run tests with MagicPod. The secret to achieving efficiency and quality assurance with a unified understanding across the development team.
KIYO Learning Co., Ltd.
In the 21st edition of our user interviews, we had the opportunity to speak with representatives from KIYO Learning Co., Ltd., alongside Mr. Ito, CEO of MagicPod. We discussed various topics, including specific use cases and the decisive factors for choosing MagicPod.
KIYO Learning Co., Ltd.
In October 2008, the company launched the audio course (now known as "STUDYing"), designed to help users utilize their spare time to acquire qualifications, and was incorporated as "KIYO Learning Co., Ltd." in 2010. With a mission to “revolutionize learning and unlock everyone's limitless potential,” KIYO Learning developed an efficient smartphone-based learning system, enriched its video content, expanded its range of qualifications, and continuously improved quality.
Furthermore, in 2017, the company launched “AirCourse”, a cloud-based corporate training service, positioning KIYO Learning as a leading platform for revolutionizing both individual and corporate education.
KEY POINTS
- Affordable and unlimited execution count were decisive factors in choosing MagicPod.
- Transition was carried out with the understanding of all development engineers.
- Test cases increased from 20 to 100 in a short period.
- Health score feature was utilized to strategize improvement plans.
From left:
Nozomi Ito, MagicPod CEO
Keisuke Tsubone, Lead QA Engineer
Masaki Nakai, Senior QA Engineer
Mr. Tsubone (hereafter referred to as Tsubone): I joined KIYO Learning in March 2023 as the first QA engineer. Our development team currently consists of about 20 people, with Nakai and I as the two QA engineers. My responsibilities include creating test cases and configuring batch runs with MagicPod, as well as designing and executing manual tests for our services, “STUDYing” and “AirCourse”.
I first became involved in testing when I joined Works Applications as a new graduate, where Mr. Ito used to be my senior colleague. I was responsible for testing ERP packages using an in-house tool.
Mr. Ito (CEO of MagicPod): Was it "AUTOTEST"? If so, that was something I developed!
Tsubone: Yes, it was! Thanks to that, I discovered the fun in testing and decided to pursue it professionally, then I changed job to third-party verification. After gaining experience in testing accounting and HR products, I am now in my current position.
Mr. Nakai (hereafter referred to as Nakai): I also work as a QA Engineer in the same group as Tsubone. I joined in February of this year, initially working on creating test cases using MagicPod. Before joining, I worked at a system integrator doing quality assurance and third-party verification contracting, primarily handling manual testing. This is my first experience with automated testing.
Journey to Implementing MagicPod
Tsubone: Before I joined KIYO Learning, a test automation tool was already in use. I continued using it when I first joined, but due to a significant price increase that made renewing the contract impossible, we considered transitioning to a new tool.
Ito: So, the issue wasn't with the tool itself, but rather the price increase.
Tsubone: Exactly. I had just learned to use it, and the development engineers were accustomed to it, so we might have continued using it if it weren't for the price increase. However, at the time, we had about 20 test cases and were running over 500 executions per month, so the tool's execution limit was becoming a constraint. I think we would have reconsidered it eventually.
Decisive Factor in Choosing MagicPod
Tsubone: When we chose our new tool, we compared three options: the existing tool, MagicPod, and another no-code tool. Ultimately, the decisive factors for selecting MagicPod were its “affordability” and “unlimited execution count”.
Additionally, considering future engineer recruits, we required a "no-code tool." The management also preferred a tool developed in Japan due to maintainability concerns.
As a result of switching to MagicPod, we reduced our monthly expenses by over 100,000 yen, and it's great that we can run tests daily without worrying about execution limits. And even though MagicPod incurs additional fees depending on the number of "test cases created," "visual diff checks," "parallel test executions," and "users who can create parallel tests," it is still more affordable compared to other tools.
Ito: Your development engineers were already familiar with the previous tool. Was there any resistance when transitioning to MagicPod?
Tsubone: There wasn't any particular dissatisfaction with the previous tool, so there were questions like, "Why are we changing?" However, when I explained that "the cost would increase significantly from what we’ve been paying," everyone understood the need for the switch.
Additionally, as we moved forward with the transition, we made sure to present it as a "company-driven project," ensuring it didn’t come across as merely "something QA wanted to do." We also held a briefing session for all development engineers, with the approval of the group manager, to ensure everyone understood the transition.
Ito: Mr. Nakai, it was your first time using MagicPod when you joined KIYO Learning. What was your initial impression?
Nakai: I can't write code, but I didn't have any issues using it. My honest impression is that it’s an “intuitive tool.” The steps and features that I’d think of as “nice-to-haves” were all thoughtfully integrated, so I was able to start using it right away.
Utilization of MagicPod
Tsubone: Currently, we use MagicPod for regression testing of "Studying" and "AirCourse." At KIYO Learning, it’s a rule that before any release, all developers must run regression tests on their own work and confirm that all tests have passed.
Ito: Does every development engineer have a MagicPod account and use it individually?
Tsubone: Yes, that’s correct. The developers execute the batch run of automated tests prepared by the QA group and then check the results. If the test fails, they rerun it. But if it fails again, they notify the QA group. Essentially, what our development engineers mainly do is “execute batch runs and check the results.”
Test results are notified via Slack, with separate channels for "Studying" and "AirCourse." When a failure notification comes in, I also try to review the contents as much as possible.
Ito: How frequent do you release updates?
Tsubone: It varies by product. Minor bug fixes are released several times a week. For new major features, development can take several months before release, but including small updates, we’re releasing multiple times a week.
Ito: So, instead of releasing at a fixed time every day, you release updates and run tests in MagicPod as soon as minor fixes are available. About rerunning the tests if there’s a failure—do you rerun all the test cases?
Tsubone: Yes, we do. Since we're still refining our tests as we operate, we encounter about three to five failures per week. Many of these failures are due to flaky tests, and while rerunning usually resolves the issue, we recognize that stability is an ongoing challenge.
In addition, although the development engineers haven't directly pointed it out, I believe that execution time is a concern. Currently, it takes around 25-30 minutes to run about 70 cases. We aimed to “finish within 10 minutes,” so we’ve made efforts to reduce the time, such as increasing the number of parallel executions to the maximum of 10.
Ito: Were you able to utilize the test cases created with the previous tool?
Tsubone: Yes, we migrated about 20 cases from the previous tool, and since implementing MagicPod, we now have about 70 cases running. In total, we’ve now created over 100 test cases. Creating test cases requires multiple test runs, and it would have been difficult to create this many in such a short period without MagicPod's unlimited execution feature.
The main reason we decided to switch tools was due to cost, but switching to MagicPod has been great—we’ve been able to significantly increase the number of test cases and catch bugs that manual testing might have missed. Currently, our test cases focus on the core functions, but we plan to expand them. Thanks to MagicPod, we've achieved both efficiency and quality assurance.
Early Utilization of Health Score Feature
Ito: All your development engineers use MagicPod, but how do you onboard new members?
Tsubone: Our service, "AirCourse," has a knowledge-sharing feature where we post "how-to" guides on using MagicPod. When new members join, we use these articles to help onboard them.
Ito: That's unique to your company and very impressive. You also mentioned getting approval from the group manager to hold a briefing session when transitioning to MagicPod. Was there anything else that made the development engineers so cooperative?
Tsubone: A key factor was that a project to expand automated testing was launched at the same time we transitioned to MagicPod. In fact, we assigned developers to the project to help create the test cases. Specifically, Nakai and I identify "the necessary test cases," assign developers to work on them, and track progress using the project management tool Backlog.
After the project settled down, we focused on improving our test cases, and MagicPod’s Health Score feature has been very helpful. Nakai and I review the Health Scores weekly and strategize improvement plans.
While it was great to have development engineers involved from the start, I think we may have rushed the creation of test cases. Adding too many test cases without ensuring stability led to longer execution times and maintenance challenges. For those just starting out, I recommend not creating too many test cases at once—start with five to ten—and gradually improve them with the help of the Health Score.
In Conclusion
Tsubone: Development is about creating, but what's fun about QA is that we’re appreciated for breaking things. My favorite part of the job is testing with the aim of ensuring the product behaves as designed before it reaches the customer. I'm really grateful that using MagicPod has enhanced my knowledge and experience in automated testing.
MagicPod makes it easy to start automated testing at an affordable cost, so I highly recommend it for those new to automation. When starting, I suggest involving engineers in the process rather than having QA handle it alone.
Nakai: MagicPod is affordable and offers unlimited execution counts, which is a big advantage. It's highly recommended for companies that run tests multiple times each day. The steps are easy to follow, making it simple to get started. I suggest starting small and gradually expand your testing efforts.
KIYO Learning Co., Ltd.
- Corporate Website：https://www.kiyo-learning.com/
- Recruitment Information：https://www.wantedly.com/companies/kiyo-learning/stories
