Automated Testing to Eliminate Personnel Dependency and Improve Quality: How Implementing MagicPod Unified Development and QA
Skillnote Inc.
User Interview #25 features MagicPod CEO Nozomi Ito speaking with representatives from Skillnote Inc. about their experience using MagicPod. The discussion covers specific use cases, selection criteria, and more.
Skillnote Inc.
Skillnote is a SaaS startup providing the skill management system "Skillnote" for manufacturing sites. This product digitizes skill maps (competency management charts), traditionally managed with paper, Excel, and stamps, onto the cloud. By visualizing skill data at manufacturing sites, Skillnote enables systematic talent development and placement, contributing to solutions for challenges in the manufacturing industry such as "skill transfer," "multi-skilled worker training," and "rapid workforce readiness."
KEY POINTS
- Although Skillnote was advancing automated testing with Selenide, challenges in dependency on specific personnel and maintenance burden was significant.
- Implementing MagicPod enabled the creation of a system where the entire organization could manage and operate tests without relying on individual skills.
- MagicPod's self-healing and inquiry features contributed to greater testing efficiency and reduced psychological burden.
- Collaborations with frontend teams were strengthened to stabilize tests, such as adding test-specific IDs.
- Future challenges include thorough test case management and clear role division between manual and automated testing.
From the left:
・Louis Stewarta Murata, Frontend Tech Lead
・Naomi Takasawa, Engineering Manager
・Nozomi Ito, MagicPod CEO
Takasawa: I joined Skillnote in 2020. Currently, I manage one of our two scrum teams as an engineering manager, overseeing requirements definition, specification decisions, design, and implementation for new services. Before joining Skillnote, I worked in project management.
Murata: I joined in 2022 and serve as the frontend tech lead in the same scrum team as Takasawa. Before Skillnote, I gained experience as a frontend engineer and development team leader.
Currently, Skillnote's development organization consists of two scrum teams and a VPoE office, totaling 20 members. About six months ago, we adopted a two-week sprint scrum development cycle, with product releases occurring every two months.
Takasawa: Previously, developers were individually responsible for quality assurance through to release. About three years ago, a QA engineer joined, and we began building a formal QA structure. At one point, our infrastructure engineer also joined the QA and SRE (Site Reliability Engineering) team. However, with the transition to scrum development, QA functions were absorbed into each scrum team.
Ito: Does this mean that each scrum team now includes not only developers performing tests but also non-programming members like QA engineers, handling both manual and automated testing?
Takasawa: Exactly. We're advancing automation with a focus on enabling developers to perform more fundamental testing. Rather than assigning QA to specific members, we are transitioning to a model where the entire team shares QA responsibilities.
Background of MagicPod Implementation
Takasawa: Our automation efforts began about three years ago when a QA engineer built automated tests using Selenide (an open-source UI testing framework based on Selenium WebDriver). However, the testing process became overly dependent on this engineer. Due to their heavy workload, maintaining the tests became increasingly difficult. Additionally, the automated tests could only run on their local environment and were not integrated into the CI/CD pipeline, making it challenging to keep up with release cycles and raising concerns about quality assurance.
Ito: So even running the tests was a challenge. Which areas did you mainly automate?
Takasawa: We focused on automating end-to-end (E2E) testing. Ideally, automated tests should be run daily or with every deployment, but at that time, we could only execute them once before our bi-monthly release. As a result, the creation of new tests couldn’t keep up with feature development, and many existing features remained untested.
At the time, my involvement was limited to reviewing Selenide code; I didn’t write the test code myself. About a year ago, discussions began on how we could better manage automated testing as an organization and move forward with automation using some kind of tool.
The immediate trigger was a proposal from our VPoE, Ando, but even before that, the team had been voicing concerns that maintenance would become unsustainable. Ando’s proposal aligned perfectly with these concerns, pushing us to take action.
Ito: Did any critical issues occur after releases due to this situation?
Takasawa: Fortunately, no major issues occurred. However, as our team grew and more features were added, we strongly felt that our personnel-dependent E2E automated tests had reached their limit. This led us to act proactively to prevent future problems.
Ito: Did you consider migrating the Selenide test environment to the cloud?
Takasawa: We did consider it, but after consulting with the QA engineer, we concluded that the maintenance burden of Selenide itself was too high. At the time, our QA engineer was handling many other tasks and couldn’t allocate enough resources to maintain Selenide.
We also tried to involve more members in maintaining the tests, but this placed a significant mental and time burden on them, causing resistance within the team. As the product grew, the frequency of design changes in Skillnote increased, and Selenide couldn’t keep up. Therefore, we prioritized creating an environment that could easily adapt to frequent design changes, reduce maintenance burdens, and eliminate dependency on individual engineers, leading us to adopt MagicPod.
Key Factors in Choosing MagicPod
Ito: How did you proceed with selecting and implementing a tool?
Takasawa: Once the project was launched, the lead person in charge, Murata, and I were assigned. The three of us began by listing potential automated testing tools and applied for trials with each provider.
We had two main criteria for selection.
The first was cost. Skillnote is still a growing company, and some well-known tools were simply too expensive. In contrast, MagicPod was reasonably priced when limiting the number of test cases created, and its flat-rate pricing with unlimited execution was very appealing.
The second criterion was ease of use. We anticipated situations where QA engineers who don't write code would need to manage and maintain tests. Therefore, it was essential that the tool be easy to maintain and allow users with varying skill levels to create automated tests comfortably. MagicPod's low learning cost was particularly attractive in this regard.
Ito: After actually trying MagicPod, what were your impressions?
Takasawa: I skimmed through the manual and tried using it. I was surprised at how intuitively I could proceed with test steps just by predicting, "If I click here, it will probably do this." I didn’t expect to operate it so smoothly after just about 15 minutes of learning.
Murata: I remember feeling the same way. I barely read the manual, but I could naturally create what resembled a test case. It felt like mistakes were acceptable, so I was more motivated to explore and interact with it through trial and error.
How MagicPod is Utilized
Takasawa: After deciding to implement MagicPod, we proceeded with migrating regression tests that had been created with Selenide. Currently, automated tests run daily at 8:00 PM in the staging environment, enabling early detection of unintended regressions. Resolving the situation where "automated tests only run before releases" was a significant achievement of implementing MagicPod.
Additionally, we use MagicPod to run tests after system-wide tasks like database version upgrades, which can have widespread impacts. Before major updates, we now hear team members saying, "With MagicPod, we can proceed with confidence," which shows how much the team relies on it.
Murata: We recruited a wide range of members to help with the migration, including QA engineers and infrastructure engineers who don’t write much code. Since then, we’ve been regularly calling on the entire development team to expand the range of automated tests. As a result, what was once a personnel-dependent maintenance task for automated tests has become something anyone can handle without needing specialized skills. This is also a significant outcome of implementing MagicPod.
Takasawa: On the other hand, although we aimed to expand automated testing for existing features and introduce automation from the early stages of new feature development, we haven’t fully achieved this yet. This issue isn’t due to MagicPod but rather reflects challenges in our overall development process. We’re currently exploring ways to integrate automated testing into our development workflows without slowing down our pace.
Ito: How many people are currently using MagicPod?
Takasawa: All 20 of our development members have MagicPod accounts, and everyone has used the tool at least once. However, about 10 of those members handle daily operations. These core members rotate responsibilities, reviewing test results daily and performing maintenance as needed. Specifically, to align with our two-week scrum development cycles, senior members pair up with newer members, and they rotate responsibilities every two weeks.
Murata: We use Microsoft Teams as our communication tool, so test results are delivered to Teams where everyone can check them. If the rotation members find something difficult to handle, they can consult within their scrum team or ask the entire development team, allowing for flexible responses.
Ito: Has MagicPod helped detect issues that prevented problems before they occurred?
Takasawa: Yes, quite a few. In the past year alone, MagicPod has detected about ten issues that developers hadn't noticed.
Ito: I’m glad to hear it’s been helpful. Are there any challenges you’re currently facing?
Takasawa: The main challenge is expanding the coverage of automated test cases. Previously, we asked available members to help, but due to organizational changes and staff turnover, the process has stalled. We’re now reconsidering how to proceed.
Ito: Do you have a system in place to manage all test cases and prevent test coverage gaps, like identifying which screens lack certain test cases?
Takasawa: That’s exactly one of our current challenges. We used to maintain a comprehensive list covering manual tests, automated tests, and those run with MagicPod to cover all E2E tests. However, the quality of this management was problematic. For example, we mistakenly thought some screens were automated with MagicPod when they weren’t, leading to missed manual tests. This made us realize the need for stricter E2E test case management. We’re now working on clarifying which tests are done manually and which are handled by MagicPod to improve test case management quality.
Additionally, some team members feel that E2E test results are more unstable than expected. We recognize the need to explore measures to improve stability.
Ito: E2E tests are inherently more unstable compared to unit or integration tests due to various external factors. However, you can improve test stability by organizing test-specific IDs for element identification and using features like the shared step function to standardize frequently changing parts, enhancing maintainability. Still, cases like “the test passed after a re-run without any changes” can occur. We’ll continue improving MagicPod to reduce such cases and make testing more stable.
Murata: We’re actively working on organizing test-specific IDs right now. By leveraging MagicPod’s analytics feature, we can confirm that the number of unstable locators is decreasing, and we’re steadily making progress.
Recommended Features of MagicPod
Ito: Are there any specific MagicPod features that you find particularly useful or that you like?
Takasawa: First and foremost, the automatic repair feature has been incredibly helpful. It’s common for us to add div elements when making design changes. In such cases, when the automatic repair feature is enabled, it can easily handle changes even if they affect about ten test cases. This has been a tremendous help. When applying automatic repairs, we always review and approve the modifications, and every time we do so, I’m reminded of how useful this feature is. We recently benefited from it again.
Another feature I find excellent is the inquiry feature. It’s so well-designed that I think we could reference it for Skillnote as well. When you click the inquiry button, not only is all the necessary information automatically attached, but there’s also a well-prepared format for entering the inquiry details. This makes it very easy to ask questions without worrying about what to write. We use this feature quite often.
Additionally, the cross-browser testing feature is incredibly effective. Our recommended environments for Skillnote are Chrome and Edge. Although, due to certain circumstances, we’ve temporarily stopped testing on Edge, we previously used MagicPod to run tests on both browsers. In reality, even though both browsers were officially supported, we couldn’t manually test both Chrome and Edge thoroughly. We plan to reorganize our environment soon to resume Edge testing with MagicPod, just like before.
Murata: I also find the inquiry feature very helpful. When I don’t understand something about how to operate the tool, I can easily ask, “Excuse me, I don’t understand this part.” This ability to ask questions right away when I’m stuck has significantly reduced my mental burden. I think this is a great feature of MagicPod.
Personally, I also like the “Wait until element appears” feature. Initially, I didn’t know about this feature and was using commands like “wait for X seconds.” However, that approach often didn’t work as intended. I realized that it’s necessary to have processing that waits until a specific element appears.
Takasawa: I completely agree! I also like the “Wait until the element becomes active” feature.
Murata: At Skillnote, we frequently change elements from an inactive to an active state. Using the “Wait until the element becomes active” feature allows us to accurately capture this state change and execute the test correctly, which I find very convenient. These waiting-related features are ones I use regularly.
Takasawa: I only learned about these waiting features after seeing Murata use them. I was surprised and thought, “There’s such a convenient feature?” and immediately incorporated it into the test cases I was in charge of. As I mentioned earlier, since we rotate responsibilities in line with our two-week scrum cycles, we often look at other members’ test cases and learn from how they write them, helping us make better use of MagicPod.
Final Thoughts
Takasawa: We’ve successfully turned what was once a personnel-dependent E2E automated testing process into an organizational asset, and I truly believe implementing MagicPod was the right decision. We focused on no-code tools to avoid relying on individual members’ skills, and MagicPod allowed us to reach a state where anyone could start using it with minimal effort. The ease of learning was extremely valuable to us.
However, after actually operating it, I realized that improving the quality of automated testing requires fundamental engineering skills, especially in design and coding. Looking back, I think we should have dedicated more time and resources initially. Specifically, after implementation, we should have allocated some engineering resources to carefully design the overall automated testing process and create sample test cases that could serve as guidelines for the team. Had we done that, the rollout to the team would have been smoother.
Murata: I also feel that introducing MagicPod was a great decision. As a member who was actively involved in promoting its adoption, I feel a strong personal connection to its implementation and operation. I believe it has had a positive impact on the members involved.
On the other hand, because MagicPod makes it so easy to create test cases, there’s a risk of creating them without clearly defining their purpose or the expected results. Moving forward, I think we need to manage each test case more carefully by clarifying questions like “What is this test verifying?” and “What is the expected result?” This is a challenge we need to tackle in the future.
Additionally, I believe collaboration with frontend engineers is essential for stabilizing tests. For example, if test-specific IDs aren’t appropriately assigned, the tests can become unstable. But just having one frontend engineer who pays attention to this can greatly improve the quality of testing.
Takasawa: Back when we were using Selenide, we discussed adding test-specific IDs, but due to individual decisions driven by limited time and resources, the initiative never moved forward. Now that automated testing has become an organizational asset and everyone is engaged, we’ve finally been able to prioritize organizing test-specific IDs. I think implementing MagicPod was the catalyst for this shift.
Murata: For those considering adopting MagicPod, I recommend confirming in advance how much organizational understanding there is for automated testing and how much engineering resources can be allocated. By doing so, I believe you can achieve a smoother implementation and stabilize testing more effectively.
Skillnote Inc.
- Official Note: https://note.com/skillnote
- Skillnote Product Division, Executive Officer VPoE Ando's Note: https://note.com/daisuke_ando
