MagicPod is like a second QA process! The “peace of mind of repeated runs” increases the development speed.

Aldagram Inc.

Mobile app testing Browser testing

For the 10th user interview, we spoke to MagicPod CEO, Ito, and Aldagram Inc.
We discussed various topics, including actual use cases and the decisive factors when making a selection.

Company Profile of Aldagram Inc.

Aldagram is a startup established in May 2019 with the mission “Unlock Your Value.” They launched a product named “KANNA” that increases site productivity in the construction, real estate, manufacturing, and other non-desk industries worldwide, and unlocks valuable capabilities of these sites globally.

KEY POINTS

Aldagram developers could see a future where performing manual testing alone would lead to failure
The decisive factor for implementation was the ability to run the program as many times as required
Peace of mind leads to faster development
How to build cooperative relationships between development and QA

Aldagram Inc.
Left: Yosuke Obata (Engineer)
Right: Shigenobu Sano (QA Team)

Circumstances behind the implementation of MagicPod

Obata-san (hereafter, “Obata”): We introduced MagicPod in February of this year, and after a trial period, we signed an official agreement around March. I joined the company in December 2021, and after hearing about the development environment and observing the release process, I felt that it would be difficult to continue with only manual testing.

Sano-san (hereafter, “Sano”): I joined the organization as the first QA member in October 2021. KANNA has web and mobile apps (iOS/Android), which are always tested manually before release. We would run regression tests the day before release. If I or the PdM found that a correction was necessary at that stage, the release would be delayed by a day or two.

Obata: KANNA focuses on providing an easy-to-use UI/UX, even for non-desk workers. This is ensured through manual testing prior release. As the number of functions increases, the number of test items and the time required for the test also increase proportionally. As delays in finding and fixing bugs would affect the release, the other members of the team, including COO Atsumi, had to be aware of the issues in advance. For this reason, I proposed the introduction of E2E automated testing. This was agreed upon as a solution to ensure that we would not need to compromise on either quality or release frequency. Therefore, we started the process of selecting tools at the beginning of the year.

Reasons for choosing MagicPod

Obata: At my previous job, I was running E2E automated tests using Capybara. However, I felt it was an expensive process as we did not have a QA team. Although it was good that Aldagram could ask QA professionals, including Sano-san, to design tests for us, we had a scarcity of engineering resources. Thus, we considered a low-code tool to be the best solution.

Although we should have compared a number of tools during the selection process, we were not aware of MagicPod at the time, so we only tried the other tools that we already knew about. The operability of the tool was reasonably good, but we decided not to implement it due to the fee structure, and decided to try MagicPod after we looked into other tools.

We defined the three criteria for selection as being “easy to use and operate,” “capable of being run repeatedly,” and “capable of writing E2E for mobile apps.” When I tried using MagicPod, my impression was that engineers can easily understand it. It had many items that could be manipulated and raw logs that could be viewed without it being overly complicated.

Ito (MagicPod CEO): Thank you very much. What kind of scenarios did you specifically think of in terms of it “not being overly complicated?”

Obata: For example, when test results lack “idempotency*”. In particular, “sometimes the test fails after running it several times,” and you can view the log by yourself and investigate “why the test is failing” and “what MagicPod is trying to do that is failing” by yourself. Of course, MagicPod also provides support, but I also have peace of mind, as I will be able to help Sano-san if he faces any difficulties.
(*"Idempotency" refers to achieving the same result regardless of whether an operation is performed once or multiple times.)

Additionally, you can run MagicPod an unlimited number of times, and write tests for mobile apps. I think the greatest strength of automated testing is the ability to run it repeatedly. Furthermore, this can be done without any substantial increase in the cost, thereby making MagicPod a cost effective testing option.

Sano: After actually using it, I was impressed with how I could write tests smoothly without having any coding knowledge. I was particularly impressed by its self-healing function. An English version of KANNA was released in July of this year, and when language corrections were made in Japanese and English languages, MagicPod provided us with a notification that “the wording has changed here,” which would not have been caught by a manual QA. I felt that “this was corrected by a global team engineer,” and I feel like it is like running a second QA process, or QA is always present, which I found really helpful.

MagicPod Applications

Sano: Currently, we are running it regularly once a day; early in the morning on weekdays. My routine is such that, once the morning meeting is over, I check my Slack notifications, and if something has returned an error I check “where the error has occurred.”

Obata: It is impossible to manually test everything every day, so MagicPod is a big help. As introduced in the Tech Blog, this chart shows the development flow using the automated E2E testing that we are currently aiming for.

E2E testing is used to ensure that the system works as intended with existing functions, while testing of new functions and UX perspectives will be done manually. If all tests pass, it will be released for testing in the main environment and new E2E automated tests shall be added. We believe this will allow manual testing to focus on the essential elements even as the number of functions increases.

Sano: Whereas, originally, all QA was done manually, now about 60–70% can be done with E2E automated testing. If we did all of our current QA testing manually, it would take more than 7 hours, but with MagicPod, we have been able to reduce that time to about 3–4 hours.

If E2E runs well, we can see the results in the periodic run before regression, so we can tell development to "fix this" as soon as possible. This also allows for time for exploratory testing. It was really nice to have less to "reschedule the day before release."

Obata: I think MagicPod, which can be run many times, has helped us to improve the speed of development by giving us the peace of mind that the test is running comprehensively. We plan to add mobile app testing in the future. As mobile behavior inevitably varies from device to device, MagicPod allows you to run tests on a wide range of devices. We believe that we will be able to become aware of bugs that occur only on specific devices before release.

How to build cooperative relationships between development and QA

Ito: As you mentioned earlier about idempotency, are you consciously rewriting the locator with the aim of obtaining the same results no matter how many times the same test is run?

Obata: In the case of KANNA, the front-end is written in React; however, as we did not initially include data-testid, there was some instability when it came to auto-detecting elements (e.g. buttons). So all the engineers worked together and went around adding data-testid to the code. The code is now very stable.

Sano: Engineers sometimes suggest things that I had not thought of, saying, "If you organize it in this way, it will be more stable.” I think it is because we could cooperate in this way that we were able to move smoothly from the deployment stage.

Ito: It is much easier to work when you have the cooperation of the engineering team. Is it because Obata-san is looking at both aspects?

Obata: Yes, I think that is partly the reason. I like to stand in the middle and try to think of ways for both sides to work more efficiently. I also wrote code-based E2E in my previous job, so I have an understanding of the level of difficulty involved. When it breaks, he/she doesn't know where to fix it, even if he/she is an engineer, not having written E2E code. That is why I think it is a good idea to allocate some of the work to QA personnel, and tell them that “when things don’t go well, the engineers will provide support.”

All of our engineers are very cooperative and understand when we explain the importance of the project. Sano-san also understands when we ask him to identify elements using data-testid, so I feel that communication costs can be kept low for both parties.

Ito: I am often told by QA people that they are struggling to get engineers to cooperate with them, so I think it is wonderful that you have built such a cooperative relationship. Would you say that this represents the culture of Aldagram as a whole?

Obata: I think so. Since we are creating a service for non-desk workers, we are very conscious of the need to create a UI/UX that is easy for everyone to use. Therefore, even before I joined the company, I think there was a strong awareness of the need to write high quality code” and to “not spare any man-hours when doing so.” I think such a culture is born from COO Atsumi's strong sense of diligence. The COO himself still participates in most of the QA work himself, in parallel with the various other tasks that he performs.

Sano: Of all the services I have been a part of in the past, I have never known a COO to be so directly involved as this one. I think that if the top management has this awareness, it will also permeate to the other employees.

Finally...

Obata: E2E testing is an important part of maintaining high quality, but doing it manually is not always the best option. By automating the areas that can be automated, we can perform tests that can only be tested manually in greater depth, and thus increase the frequency of releases.

MagicPod has an unlimited number of runs, and this makes it a tool that can take full advantage of the strengths of automated testing. In addition to being automated, as in the case of the self-healing function, there is also the advantage that you notice changes more easily than when merely performing a visual check. I believe that we can make even better products by successfully combining automatic and manual testing.

Sano: Even though I am not an engineer, MagicPod is easy to use even for me. Having said that, even though I took the lead in implementing it, I think there were some areas that did not go well along with the maintenance of data-testid. A major element in the success of the project is how we involve the engineers. At Aldagram, as we worked as a team, we could cover almost half of the test cases within six months. I think that it is good to enlist the support of the whole company when moving forward, so the QA representative does not have to worry about the project on his own.