Applying Defect and Root Cause Analysis to Tuning Check Recognition with AI
OrboGraph will soon be launching a website refresh with the official market announcements around OrbNet AI and OrboAnywhere 4.0. Check recognition is a process which is not perfect, but the gap of unread items is closing. As technology approaches reading all legible items, it is important to look closely at “defect” items which did not pass the acceptable thresholds.
In this blog post, we discuss how defect analysis along with root cause analysis are complementary methodologies to optimally tune a system in a test environment or field deployment. This process is also used to improve the core AI models.
What is defect analysis? A definition from technopedia.com:
Defect analysis is part of the continuous quality improvement planing in which defects are classified into different categories and are also used to identify the possible causes in order to prevent the problems from occurring. It helps projects to identify how issues can be prevented and in reducing or eliminating significant numbers of defects from being injected into the system.
To resolve defects, you need to do a Root Cause Analysis. Evocon explains in detail the American Society for Quality (ASQ) 6 step Root Cause Analysis Process that is highlighted in the chart below:
Note that defect analysis is actually a common process incorporated in Six Sigma training. There are multiple ways to identify and investigate defects, including Fishbone diagram, Five Whys, Flowchart, Pareto Chart, and Scatter Diagram.
What is a Check Recognition Defect?
In the case of amount recognition, check images or snippets of the amount are selected for defect analysis if the amount field was not read correctly or was not read at a high enough score. If you recall, there are items which have “low scores” assigned by the system. These lower scored items have a higher probability of failure than high score items. Typically we create a ROC curve, which shows the read rate/misread performance at difference scores. The concept of a ROC curve is illustrated below.
Note that this diagram shows “false negative”on X axis. For read rate/misread analysis, we use “misread” on X axis.
Low score items under a certain threshold are considered defects. Defects in this process are typically identified as having several root causes, hence the term: root cause analysis.
- Reference errors: There are many circumstances which cause an item to have an incorrect dollar amount assigned to the final, balanced amount updated for posting. It could be caused by a keying error or 3rd party software misread at the bank of first deposit. Additionally, software applications many times “force balance” work in the case of low dollar errors. This can cause a final posting value to be incorrect. This incorrect amount value would then be used as the “reference value,” causing an incorrect read in the test set. Defects identified as reference errors are typically experienced in texting as well as AI model training.
- CAR/LAR discrepancies: This issue can be a relatively high percentage of items (.5%) where the courtesy amount and legal amount are not matching. This is typically overcome with a human decision overriding the value of one field over another; i.e. LAR over CAR, or in some cases, vice versa. From a system standpoint, how does a classification model score an item with a “real life” CAR/LAR discrepancy? Should the LAR always override the CAR? What if the LAR is difficult to read, and CAR is most prevalent? On occasion, the cents can be omitted from the LAR field and a check writer. The point is, during the final analysis of results, the CAR/LAR discrepancy needs to be a factor in the final review of defects.
- Field Detection Limitations: There are unique situations where the OrbNet AI field detection model is not able to locate the field location. This may cause either an unread item or incorrect recognition amount. To address this, the tester may need to use OrboTool definitions to lock on to the field based on a unique MICR line.
- AI Model Errors: Simply put, no AI technology is perfect yet. There may be cases where the data set used to train the AI model was not comprehensive enough, causing an incorrect “classification” of an amount on a check. In this case, root cause analysis can help future AI retraining.
- Other: As we expand the capabilities of OrbNet AI, there will be situations within the defect population which were unexpected. These item level commonalities should be reported to OrboGraph during testing if possible.
As we embark on a new era of check recognition using OrbNet AI technology, a defect-oriented approach is recommended to optimize the system. With the appropriate logging capabilities, this process can actually streamline tuning and optimization!
This blog contains forward-looking statements. For more information, click here.