Implementing MagicPod reduced regression testing by 20–30%, and fostered team-wide skill growth through study sessions and pair programming.

Safie Inc.

Mobile app testing

In our 23rd user interview, we had the pleasure of speaking with representatives from Safie Inc., joined by Mr. Ito, CEO of MagicPod. We discussed their specific use cases and the key factors for choosing MagicPod.

Safie Inc.

With the vision of “Create a better future with intelligent vision”, Safie aims to build a platform that provides cloud-based access to camera footage, easily accessible to everyone, from Japan and around the world. This service empowers customers to leverage video footage for decision-making in areas such as security, employee training, construction management, store operations, and disaster prediction. Recently, Safie has also been developing products that combine video data with AI to create new value.

KEY POINTS

Previously, development team members performed testing just before release
Around 2020, QA team was established and implemented MagicPod for mobile testing
Automation reduced regression testing efforts by 20-30%
Team-wide skill growth in MagicPod was achieved through study sessions and pair programming

From Left to Right:
・Takashi Koyama, QA Engineer
・Jo Morishige, QA Engineer
・Nozomi Ito, MagicPod CEO

Takashi Koyama (hereafter referred to as Koyama): I joined Safie in October 2021. I currently lead a sub-group within the QCD group, where I'm responsible for overall quality assurance. "QCD" stands for "Quality", "Cost", and "Delivery". This means we're not only involved in testing but also participate in projects and the requirements definition phase to enhance development quality. Currently, the QCD group includes around 14 members, including contractors, and is becoming a key part of our 100-member development team.

Jo Morishige (hereafter referred to as Morishige): I joined Safie in March 2023. As a QA engineer, I lead and manage test procedures, guiding testers on how to execute them. I am also focused on promoting automation, from strategic planning to specific scripting methods to improve maintainability and readability. In my previous role, we automated and wrote code using Selenium, but joining Safie has given me my first experience with a no-code tool like MagicPod.

Journey to Implementing MagicPod

Koyama: Before I joined Safie, developers performed regression testing manually once per monthly release. Because testing took place after development was finished, issues were often detected just before release, which frequently led to a hectic release process.

In 2020, our mobile app was renewed, and we launched “Safie Manager”, a web application for centralized management of cloud cameras and user accounts. At that point, it became clear that relying solely on developers for QA was no longer feasible, so we formed a dedicated QA team. We also introduced an automated testing tool specifically for the web viewer.

Safie’s application, “Safie Viewer”, consolidates various features within a single viewer. As the number of features increased, so did the workload for regression testing. This led us to consider automation for mobile testing. However, the test automation tool we used for the web could not meet our requirements for mobile, so we began exploring other options.
We ultimately chose MagicPod because of its:

Unlimited test executions
User-friendly interface for creating test cases
Ease of maintenance and support
Affordable and cost-effective pricing

Ito: Thank you. In Safie's case, your environment involves not just the application itself, but also physical cameras. How do you conduct the testing?

Morishige: Due to the nature of our product, certain aspects, such as video and audio, require human judgment. The video and audio captured by the cameras are beyond the capabilities of automated testing, so these tests are conducted manually. However, for functions that can be managed within the system, like authentication, modifying camera information, and search, we rely on test automation.

Utilization of MagicPod

Morishige: We set MagicPod to run a portion of our regression tests daily between 8 PM to 12 AM. These tests cover the latest iOS and Android app versions in development, as well as released versions prior to manual testing. Running tests daily allows us to detect errors early and provide immediate feedback to the development team. This approach has reduced our regression testing efforts by approximately 20–30%.

Koyama: With MagicPod running, we can add new features without reducing our release frequency and cover tests previously skipped due to risk. Running tests daily provides insights into "changes in the latest version" and "lessons from failures." This approach reduces our workload, allowing us to dedicate more time to process improvements and other projects.

Ito: How do you collaborate with your developers?

Koyama: We share test results on Slack, where some developers review them, but our QCD group is primarily responsible for this. If further investigation is needed, we ask the developers for assistance. Almost all of our mobile app developers have MagicPod accounts, so they can check things directly in MagicPod.

Morishige: Things have stabilized recently, so requests like “we couldn’t identify the cause, please investigate” now happen only about once a week.

Ito: What improvements did you make to achieve this stabilization?

Koyama: When we initially started using MagicPod, we would run regression testing once a month on the latest version of the app, before its release. However, we realized it lacked sufficient testing. So while some rules were still undefined, we prioritized increasing the frequency of test executions and revising our operational processes.

Ito: Was it difficult to identify the cause of issues when you were only testing once a month?

Koyama: Yes, it was. With so many changes between tests, it was hard to pinpoint the cause of issues. Luckily, along with the support of the developers, we established a system that automatically integrates the app with MagicPod after each build, retrieves the latest version, and runs tests.
Running tests daily lets us respond to changes as they occur and quickly detect unexpected bugs from development. Previously, issues would often surface right before release, but now we can identify and fix them during development, which has greatly improved our development process. Initially, the maintenance cost was high and the setup was unstable, but Morishige helped us and stabilized the system.

Morishige: Previously, there were dependencies between test cases, where data used in one test case was also used in another. While some dependencies remain because some of our tests require physical cameras, we've resolved 90% of them.
Additionally, we weren't fully utilizing MagicPod's features for improving maintainability, like shared steps. Now, we consolidated procedures into a single shared step, improving both maintainability and readability. We also held training sessions on how to use MagicPod, as some team members were unfamiliar with concepts like “variables.”

MagicPod Upskill Through Study Sessions and Pair Programming

Ito: How are the study sessions conducted?

Morishige: We involve the entire mobile QA team. When I was assigned to the mobile team, I thought, "Maybe we can improve our overall approach," and that's how it started. Specifically, we followed MagicPod's start guide and had everyone “make actual test cases” using a publicly available site. We also felt that study sessions alone weren’t enough, so we implemented pair programming, where full-time employees and contractors work together to create test cases.

Koyama: We scheduled one hour of pair programming each day for about six months. As a result, I believe “everyone was able to learn the basic concepts.” We knew we had to commit fully to make it work, and looking back, it was a pretty intense effort (laughs).
Our contractors often asked insightful questions, like “Why do we have to use this locator?” Such “why questions” often put me in a tough spot, so I’d quietly research the answers on my own time. Ultimately, it helped solidify my own knowledge.

Morishige: It was interesting to have others point out things I wouldn’t have noticed on my own. Although we haven’t created onboarding materials for new members yet, we still have the study session materials, which we can use. In that sense, I’m glad we held the study sessions.

Ito: It’s impressive that your whole team is so engaged. Do you regularly switch pairs?

Koyama: We didn’t change the pairs for pair programming, but we introduced a rotation system where we "take turns maintaining tests created by others." Generally, we follow the practices Morishige introduced, so it’s usually not difficult. But occasionally, we encounter unique approaches or creative conditionals that provide great learning opportunities.

Morishige: We assign maintenance responsibilities for each test case and rotate these assignments with each release cycle. This prevents a situation where “only one person knows how to maintain a particular test case.”

Ito: Assigning maintenance roles per test is an interesting idea. How do you determine the assignments?

Morishige: We assign teams by function, labeling them with names like “Team Anglerfish”, “Team Rabbit”, and “Team Hippo”. We continually rotate the team members with each monthly release.

Ito: That's a fun way to name the teams (laughs). Regular rotation seems like a good way to avoid dependency on specific individuals, keeping everyone engaged and reduces the risk of anyone forgetting how to handle maintenance.

Koyama: It's essential to set rules to keep testing running smoothly. Another great aspect is that if there are problems, we can always “consult the person who was in charge last month.”

In Conclusion

Morishige: I believe that continuous automated testing is crucial. To achieve this, we must create maintainable test cases. MagicPod offers excellent documentation, providing clear steps toward successful implementation. In fact, even first-time users can create high-quality tests with just a few study sessions, making it really easy to learn.
However, no matter how user-friendly the tool, you won't get the results you want if you don't know what you're doing. In my experience, creating automated tests can be quite isolating. But with MagicPod, there’s an active community where engineers share experiences, and you can ask questions and get reliable answers. It’s great to have such an accessible support environment. By utilizing these resources, I believe users can use the tool more effectively. We’re also eager to contribute to the advancement of the automation field.

Koyama: One of MagicPod's key features is its intuitive UI, which allows even those without specialized programming skills to operate it easily. The pricing is reasonable and even includes a two-week free trial. It's easy to get started, so for anyone interested, I highly recommend giving it a try.