Use This 5 Step Process to Quickly and Affordably Create Turnkey Vision Systems that Actually Work

Some of the most heartbreaking experiences of my professional career have been when I visited a factory for the first time and I noticed a vision system in the corner, not being used, gathering dust. These were full, turnkey “solutions” that the factory paid for, but, for various reasons, were abandoned.

I always ask the engineers at the factory what caused the system to be abandoned. There are a variety of causes, but one common reason is that the system worked "well enough" during the acceptance testing but didn't actually work well enough in everyday practice to be trusted in the manufacturing process.

After more than ten years of experience in this industry, I can now see why this keeps happening: these systems were developed using images acquired in the integrator’s workshop, not in the factory, and only a small data set was used! 

The Old, "Acceptance Test” Method

For those of you who aren’t familiar with the typical practices of automation systems development and commissioning, I’ll explain the usual steps that the turnkey vision system integrator takes: 

1. Design and build the system at the integrator’s workshop

2. Painstakingly acquires images of a few hundred samples one-by-one

3. Write the vision algorithm based on these sample images

4. Perform an acceptance test at the integrator workshop using a small number of samples (sometimes the same ones that were used for development)

5. Commission the system and repeat the acceptance test at the factory, again, using a small number of samples

6. If the system didn’t pass the acceptance test, allow the developer to “tweak” the system and test it again until it passes

When you look at it spelled out like that, it seems obvious that it’s not a good way to create a useful machine. The sample sizes are too small to ensure that the system will perform with a high degree of accuracy. The integrator’s workshop isn’t the same environment as the factory (differences in dust concentration, vibration, material handling, etc.). New types of defects can develop in the future that weren’t represented in the original sample set. And, allowing the vision developers to run the acceptance test a second time gives them an opportunity to “roll the dice” a second time, increasing the probability that the acceptance test will pass without actually ensuring that the system is sufficiently accurate.

But why do so many factories and integrators still do it that way? Maybe it’s because it’s they way that they’re used to doing their contracts for other automation systems. Maybe it’s because there’s a business expectation that the system is “supposed to work” immediately once it’s installed in the factory. Frustratingly, this mindset leads to many problems.

This about it: do you really believe that a system integrator can develop a robust vision inspection system using only 500 samples, when that same system will test 500,000 samples on its first day after commissioning. That’s not an unrealistic number– it’s less than 10 samples per minute over a 16 hour period.

Also, factories almost always discover new things about their product after the vision system is installed. Why is everyone always surprised when this happens? If it’s the first time that you’ve ever acquired high resolution images of 100% of your products, then you should naturally expect that you’re going to discover new defects or anomalies.

Acquiring images at the factory is almost always different than acquiring the images at the developer’s workshop. Factories can be dirty and hot. The ambient lighting conditions can be different. Nearby heavy machinery can shake the system. Many of these problems can be addressed with good system design (enclosures, strobe lighting, vibration dampers, etc.), but the developer can’t be totally confident that they work until the system is used on-site.

It doesn’t take a lot of errors before the factory loses confidence in the system and decides to abandon it. From what I’ve seen, if the operators start to notice that 2% to 5% of the products are incorrectly being passed or failed, then factory leadership will decide to turn the system off and start inspecting by human eye.

You might wonder why the vision system integrator doesn’t just fix the system once they discover that it’s not working as well as they thought? Well, frankly, it’s because the integrator already got paid after the final acceptance test! The factory would have to make a new contract with the integrator, which greatly increases cost. At this point, there’s a loss of trust between the factory and the integrator and the factory doesn’t want to hire the integrator again. So, the machine just gets turned off and moved into storage.

The New, “Big Data” Method

Based on these observations, what do I believe is the best approach for developing a vision inspection system? Use the Big Data Method.

1. Design and build the system at the integrators workshop, but only to acquire the images.

2. Install the system at the factory and set it up to acquire images. Not every image needs to be saved, it could probably be set up to randomly acquire 1 to 10% of the images. This could still be 5,000 to 50,000 images per day or more.

3. Tag the images either defective or non-defective. To do this, you have a few options (note that this should be done by humans using software assistance):

   1. Have the vision system integrator do it (this is probably the most expensive)

   2. Have one of your in-house engineers or technicians do it

   3. Hire temp workers

   4. Hire a data set labeling service such as Amazon Sage Maker Ground Truth (typically only 2 to 8 cents per image).

   5. Use a combination of all of these techniques

4. Develop the vision inspection algorithms using conventional machine vision or machine learning/deep learning techniques. Every time the developer makes changes to the algorithm, then it can be tested on the image data set that is saved on file. Improve the algorithm until the batch testing shows that the algorithm works sufficiently to the factory’s requirements.

5. Deploy the software with the updated algorithm to the system that’s already installed at the factory.

At this point, you will have more confidence in the system than you can ever have. There is no better way to be sure that the system works than by testing a large data set that was acquired on-site at the factory during normal production. 

Notice that there is no need to perform traditional acceptance tests. The results from the batch tests are more than enough to prove that the inspections are working. Skipping the acceptance tests will save the client time and money because integrators will not need to take time preparing for them and frankly, they are not very valuable to begin with.

Of course, this approach may not work for every situation. Sometimes, a factory won’t be able to allow the acquisition-only vision system to be installed because it might impede the manufacturing process. Sometimes, the manufacturing process itself is not possible without a working vision system. But, I’m telling you, whenever it is possible to perform a project using this approach, you will get better results, and, like a cherry on top, you can often get those results in less time and for a lower cost.

And your vision system won’t end up collecting dust in the corner.