Test Automation Cuts Full Test Cycle from 3 Months to 2 Weeks: How PORTERS Built a Quality Assurance System with MagicPod and Offshore Teams

PORTERS Co., Ltd.

Browser testing

MagicPod CEO Nozomi Ito spoke with PORTERS Co., Ltd. about what led them to choose MagicPod and how they have been using it from implementation through to today.

PORTERS Co., Ltd.

With the vision of "contributing most to employment worldwide through technology," PORTERS Co., Ltd. offers the cloud service "PORTERS" for the staffing and recruitment industry, both domestically and internationally. The platform has been adopted by over 2,200 companies, primarily staffing and recruitment agencies across 12 countries. The company also publishes and operates PORTERS MAGAZINE, a human resources strategy support publication.

KEY POINTS

  • Full testing required 3 months, causing development and update bottlenecks
  • Unlimited test executions and ease of operation were the deciding factors in choosing MagicPod
  • Early implementation struggles were overcome by revisiting test design with an external partner
  • Full test cycle reduced to approximately 2 weeks, cutting costs by approximately 5 million yen per run
  • Built a self-sustaining offshore team operation, achieving continuously running QA
From left: Takahiro Yamauchi, Product & Service Unit Mitsuhiro Oishi, Product & Service Unit, General Manager Nozomi Ito, MagicPod CEO

From left:
Takahiro Yamauchi, Product & Service Unit
Mitsuhiro Oishi, Product & Service Unit, General Manager
Nozomi Ito, MagicPod CEO

Mitsuhiro Oishi (hereafter Mitsuhiro): I joined the company in 2008, when the development team had about 10 engineers. In 2012, we launched PORTERS — a matching system specialized for staffing and recruitment, and have continued updating it ever since.

PORTERS is a SaaS platform that centrally supports the core operations of staffing businesses, from job order management and candidate management to hiring progress tracking and revenue management. Because we continuously expand our feature set to accommodate the industry's complex workflows, figuring out how to build an organization that balances development speed with quality has been my overarching challenge.

Testing was originally handled by the development engineers themselves. Around 2011, our CTO at the time joined and established a QA team, and as the service grew, our quality assurance structure gradually took shape. For test automation specifically, that has been Takahiro's domain.

Takahiro Yamauchi (hereafter Takahiro): I joined PORTERS as a contractor about two years ago. I've been involved in testing for around 20 years — as a QA chief, team lead, and sometimes as a tester. On top of that I had been driving automation at my previous company as well. However, the tool we introduced there would sometimes fail to run scenarios we had created, and other team members were struggling with the same issues, so I had developed the impression that automated testing was something that just didn't work reliably.

Challenges Before Implementing MagicPod

Takahiro: Actually, before joining PORTERS, I was involved in PORTERS' testing as a member of an external company. I wasn't directly testing the PORTERS service itself, but even then, looking at the sheer volume of features, I thought, "the people testing all of this must have it rough." When I actually joined as a contractor and got inside, it was exactly as I had imagined. (laughs)

Mitsuhiro: Our QA operations are currently handled by VNEXT, a Vietnamese offshore company. They manage everything from test case creation to manual test execution. But as PORTERS' feature set kept growing, the time required for testing became a serious problem at a certain point.

The moment we felt the most pressure was during middleware version upgrades. Version upgrades are essential for maintaining strong security, but when a full test cycle takes three months, it means no new releases can go out during that entire period. Our biggest concern was development coming to a halt when we wanted to be delivering more value to our users — we had reached a point where manual testing simply wasn't sustainable anymore.

On top of that, there were areas where a programming error could lead to information leaks — such as email sending and external data integrations. We had a strong desire to automate testing for those areas to bring regression defects as close to zero as possible.

Three or four years ago, our engineers had actually tried building automated tests using Selenium. With features expanding as the service grew and manual testing falling behind, everyone pitched in to write code and we made it through that period. But once the engineers returned to development work, there was no bandwidth left for maintenance, and the automation ultimately never became a sustainable operation.

Why and How MagicPod Was Selected

Nozomi Ito (hereafter Nozomi): With those challenges in mind, how did you go about selecting a tool?

Mitsuhiro: We compared about three options, and the deciding factors were the number of test executions and ease of implementation and operation.

On the features side, MagicPod has no limit on the number of test executions. Because we release frequently to deliver value to our customers as quickly as possible, and because we need to run tests with every security update, we determined that any cap on executions would make it impossible to keep up.

On the operations side, our experience with Selenium had taught us that we needed to avoid tools so complex that only a handful of people could use them. Knowing that MagicPod could realistically take root on the ground and be continuously maintained was another major reason for choosing it.

We completed the selection in about a month and have been using it since November 2023. We started by having the VNEXT team study MagicPod and building test cases for a subset of features such as email-related functionality — however, at that initial stage we weren't able to get it running smoothly. So we went looking for an external partner to help formalize our automated testing operations, reached out to about three companies, and ultimately brought in Verisurv.

Verisurv has deep expertise in MagicPod, and what really made the difference was that they said they would thoroughly share their knowledge with the VNEXT team as well.

Takahiro: After Verisurv came on board, I also started engaging with MagicPod in earnest. Watching their approach to knowledge sharing and maintenance, I found it completely different from the tools I had used before. It was easy to build in, easy to read, easy to verify. I thought, "this is really good."

Using MagicPod

Mitsuhiro: Verisurv supported us for about six months, during which we built test cases starting from the most critical features. That structure has now found its footing, and the VNEXT team is able to operate autonomously. Takahiro manages things from the Japan side, with a VNEXT team of 13 people on manual testing and 4 on automated testing.

Because the manual and automated teams both sit within VNEXT, we're able to create a strong feedback loop — defects caught by automated tests get fed back into the manual test cases, and conversely, bugs found during manual testing of new features get incorporated into the automated test scenarios. That mutual integration has been hugely valuable.

Takahiro: Execution runs mainly overnight and during daytime hours on weekends, which are times when team members aren't working — running 2 instances in parallel. The number of test cases has grown considerably, so even with parallel execution it takes about three days to complete a full cycle. The QA team checks the results first, and if there are any new errors or scenarios that stopped, I get notified. I then review the results on my end as a double-check to make sure nothing was missed.

One thing that's particularly helpful from an operational standpoint is how much easier the test results are to read. Screenshots are captured at the point of failure, so you can immediately see what was being done, where, and what happened. I also find it very useful that when a test encounters an error, you can choose whether to stop there or continue running — that flexibility matters a lot in practice.

For example, if execution stops because of a minor error like a slight text difference, you have no way of knowing whether everything after that point is fine or not. Being able to continue running through an error and check all subsequent behavior in one go is a significant operational advantage.

Mitsuhiro: When we ran the numbers, we had roughly 30,000 test cases in total. Running all of them as manual tests cost approximately 6 million yen in labor per cycle. By incorporating automated testing, we've been able to bring that down to approximately 1 million yen — a reduction of roughly 5 million yen per full test run.

On top of that, the full test period has been cut from three months to approximately two weeks, which means development no longer has to stop during testing — and that impact is enormous. The system has also been reliably catching defects, detecting multiple regression issues, which has been a huge help.

Takahiro: A little while after I joined, I asked the manual testing team, "How long would it take to do a full test right now?" They said, "Six months." Now, with the exception of some areas like admin screens, we've reached a state where we can cover nearly all the features a standard user would interact with.

The execution environment is also flexible — simply changing the base URL lets us switch between production and staging, so we can check for regression defects in the customer-facing environment or verify that there are no regressions outside of new features in the development environment. Automated testing has also become well understood within the development team. We now regularly get messages asking, "How soon could you complete a full cycle if we requested it now?"

Mitsuhiro: There's been a change in terms of knowledge concentration as well. PORTERS has accumulated over a decade of features, and even current development team members regularly encounter things like, "I didn't know this feature existed." Automated testing covers those areas that tend to get overlooked, and whenever a regression occurs, we add a test case so the test suite grows over time. Having a system where, even as team members change, things like "this area slipped through the cracks" become less likely to happen. I think that's genuinely valuable.

Nozomi: Are there any practices or approaches you've developed to keep daily operations running smoothly?

Mitsuhiro: Actually, we've intentionally chosen not to use the notification feature. Every day, the VNEXT team members go directly to check the results themselves to see whether any errors have occurred. Automated testing will always have things that don't go perfectly, so rather than relying on notifications, having dedicated people doing daily checks is something we think is essential to keeping the operation running continuously.

Takahiro: There's also a specific approach we've developed for handling the quirks of automated testing. For example, in tests that verify email body content, MagicPod performs strict comparison down to invisible line break differences, which can cause failures. But in practice, there's no visible impact on the customer-facing screen. For defects that only appear in automated testing, rather than forcing changes to the scenario, we've adopted the practice of letting those run as known failures. We believe it's important not to get too caught up in errors that have no customer impact, and to make judgment calls with flexibility.

From "Automate for Now" to Sustainable Test Operations

Nozomi: You mentioned that there was a period early on after implementing MagicPod where things didn't get off the ground smoothly. What specifically made it difficult?

Mitsuhiro: At first, we tried to directly replace manual tests with automated ones. We were building in granular checks like "did this text change or not?" But what matters in automated testing isn't that — it's whether the features are functioning correctly.

When you're checking areas where minor text differences don't matter, execution becomes a burden and failures pile up to the point where it stops being useful. We handed things off without getting that sorted out, which is why we felt it would be difficult to scale.

On top of that, this was before Takahiro had joined, so we didn't have test design expertise to begin with. VNEXT wasn't in a position to lead in that area either, and we felt we'd hit the limits of what we could do on our own — that's what led us to reach out to Verisurv. Once Verisurv came on board, they reviewed things from the ground up, starting with the fundamental question of how to approach building test cases, and that knowledge accumulated within the VNEXT team.

Takahiro: We ourselves didn't have deep expertise in MagicPod at that point either, so the guidance from Verisurv was genuinely educational. They taught us things on a near-daily basis — such as how to restructure test procedures to parallelize multiple test cases and shorten the execution period, and how to ensure test case updates are made reliably without anything slipping through.

Shortly before now, we expanded the automated testing team within VNEXT from 2 members to 4. The knowledge that the original 2 had gained from Verisurv was passed on smoothly to the new members, and they got up to speed without any significant issues. The fact that knowledge is now properly handed down is something I think is directly attributable to the hard work we put in during the initial setup period.

Nozomi: Were there any challenges on the product side in terms of element identification?

Mitsuhiro: PORTERS is built with React, so IDs are auto-generated and can change, and users can freely configure screen fields, meaning what appears where can change day to day. But MagicPod handles this well — it's working without issues. We haven't had to ask the development team to make any special accommodations so far.

Takahiro: Occasionally we get a message saying, "There's an element we can't capture, so we'll handle it with coordinate-based targeting," but it happens very rarely.

Closing

Takahiro: I believe that regardless of the system, the testing man-hours required for regression testing are never trivial. Automated testing is a highly effective solution for addressing that. Getting from zero to one is challenging, as we've discussed. But once you reach that point, it becomes an incredibly powerful asset. MagicPod in particular is easy to use and has excellent maintainability, so I can recommend it with confidence. It's been a huge help for us, and we hope to continue using it for a long time to come.

Mitsuhiro: We had our struggles with automated testing in the past, but now Takahiro manages it and VNEXT's dedicated team runs it every day. Being able to build this structure of "always keeping it running" has been truly significant. Automated testing will always produce errors, so the key is checking daily and keeping it in a consistently working state. That's what makes it actually succeed. I shudder to think about where we'd be without it now.

Being able to run tests as part of everyday processes rather than only before releases has allowed us to treat testing not as a special activity, but as an integral part of development. In the SaaS world, working software is a baseline expectation — if there are defects, customers simply can't use the product. Being able to ensure quality through testing and release with confidence is, I think, enormously valuable.


Reference:
Verisurv Case Study Interview
PORTERS Co., Ltd. case study interview now published

PORTERS Co., Ltd.