How I helped an OEM to decrease time to production by a factor of 10.

For almost a year I worked for a client, the biggest manufacturer in the world when it comes down to Arcade games. You will probably this product, since you can find these machines in almost all arcades, fairs and theme parks all over the world. There’s a big chance you have already played with it, possibly under the pressure of your son or daughter.

Although they might look a bit simple from the outside, a Claw game is actually a pretty complex piece of machinery. There are several different hardware and software components that need to interact to get a smooth gaming experience. In the current version, where Claw games are only playable when you are physically in front of the machine, the software bit was relatively straightforward. That is because there is a single board computer that does not need any input from the outside world. But this would soon change.

From Classic Claw to Online Claw Game

The client came to us with its vision; arcade gamers should be able to play their favorite arcade games, wherever they are, whenever they feel like playing. The catch is, they didn’t want to make a digital replica of their claw machine (eg in the metaverse), but rather they wanted to enrich the physical machines with IoT capabilities, so that gamers could play with the real physical machine, from the comfort of their home.

To enable this, we were asked to help the internal R&D team of the client to “digitalize” the classic Claw Game. We would need a cloud-platform, were gamers could browse and play games. We would call it an online arcade. We would also need the Claw machines to be able to communicate with the cloud platform, and the gamers should be able to see what they are doing, hence the need for a low-latency streaming solution.

Development to Delivery

During the initial development phase, little attention was paid to future delivery. My first mission was to get the cloud-platform up and running, and make sure the CI/CD flow supported continuous delivery. The engineers who were working on the Claw Machine’s (to add the IoT capabilities) hadn’t really heard about continuous delivery; some experiments went undocumented, they would not use an automated process for building their software, version control was minimal, etc.

After a few months, when the cloud-platform reached a certain maturity, I switched my focus to try to understand how the engineers were working on the Claw machines, since their pace was much slower than that of the cloud developers. I wondered what was happening, so I tried to listen as good as I possibly can to the engineers. They told me they had issues to get up to speed, because they were struggeling with getting one Claw functionally working, before even thinking about delivery and scaling.

Pretty early in the process it was noticable that there was a discrepancy between the pace that the management envisioned, and the actual speed that the engineers were capable of maintaining.

Even though the assembly line included a lot of manual steps, the operators were very skilled up until the point the machine was assembled. Then, when booting the machine for the first time, is where their manual mindset slowed down the process. Machines came from the assembly line and where sitting unused in the warehouse, because the engineers couldn’t keep up with provisioning.

While the production line is was a relatively stable, long running process, the provisioning line was not, because machines are produced in batches. The team responsible for provisioning goes through high and low intensity phases. On the first try, provisioning and configuring one machine took about 2 full days, from the moment it rolled from the assembly line, until it was connected with the client’s cloud platform and ready to operate.

Managing expectations, and setting clear goals

At some point I asked the Product manager what he thought would be an acceptable pace. By that time he was painfully aware that it took 2 working days to get 1 machine ready.

The manager told me that they wanted to provision 200 more machines in the near future, and that he needs to have us 10 machines ready every day. I felt that this would be virtually impossible, given the current throughput. 2 times 200 is 400, the amount of working days we would need to deliver to our client (assuming nothing would go wrong during the provisioning, which is ehm, quite unrealistic).

Going from two days for 1 machine to 10 machines in one day, how should we go about that? I remember this discussion with the manager very well. The solution he was thinking of was to add more people: if one engineer needs 2 full days for 1 machine, then you need 2 engineers for 1 machine / day, 4 to provision 2 / day, and so on. That quickly became a very costly thought experiment.

I had to convince him that it would be smarter to spend more time on automating the process using 1 test machine, then franticly trying to provision the 200 machines with all hands on deck. At first, with every day going by, the product manager feared his deadline coming closer without machines getting provisioned. You can imagine the tension!

Speeding up the process

I proposed to make an effort to thoroughly understand this provisioning process, to see what we could automate, to be able to speed up the process. I had to sit together with people from the assembly line, people from quality control and of course hardware and software engineers, to get them all to cooperate towards this clearly defined goal of 10 machines / day.

First I listed all the steps that needed to be followed to do the provisioning in extreme detail. At some point we had tens and tens of small steps, that were all seemingly important.

Then I would try to provision a machine myself, using my own list. This way I experienced first hand what every step was doing, and whether it was even a relevant step.

On the next run, I would start automating the low hanging fruit, manual steps that were very easy to automate. For example, every Machine gets a cryptographic certificate to authenticate with the cloud platform. In the beginning this certificate was generated and distributed manually. With every extra step automated, the atmosphere got less tense, because the strategy started to pay of.

After some time we managed to shorten the process form 2 to 1 days per machine. We only needed 200 more working days to accomplish the mission! With everyones involvement, it took us 5 months to shorten the provisioning process down to 20 minutes per machine. So with 1 month to go, we managed to increase the throughput so well, that we would meet the deadline and even be relaxed about it!

In fact, to deliver the 200 machines, it took us 200 hours of automation and 40 hours of provisioning, instead of 400 days.

Conclusion

If your organization doesn’t have a culture of automation, it will be a challenge to go from manual to automatic. But if you manage to inspire your teams to evolve the mindset, you will reap enormous benefits. Come talk to us if you have a project in need of automation.

Want to experience first hand how it feels to play a digital Claw? Head over to Gamestate Now and try to catch a Batman or 2!