A step-by-step process to achieve 99% autoverification in a core clinical chemistry laboratory at a tertiary care cancer center
Debdatta Das 3, Subhosmito Chakraborty 2
Discipline of Quality Assurance, Tata Trusts, Mumbai, India
Discipline of Laboratory Medicine, Tata Medical Center, Kolkata, India 
ABSTRACT
Autoverification is a process whereby clinical laboratory results are released without manual intervention. It uses predefined computer rules such as instrument error flags, interference indices, reference ranges, analytical measurement range (AMR), critical values, and delta checks to govern release of results. Autoverification is performed using a middleware. We present the implementation process and TAT improvement data at our hospital. This study focuses on the TAT of routine clinical chemistry metabolic panel from 2016 to 2017. After design of the verification rules, the team for implementation consisting of the biochemist, technologist, IT personnel, and the quality managers were defined and the process implemented. 
The TAT data was gathered and analyzed using the R software. A significant reduction in TAT of about 50% was observed in the study and >75% of the tests were subjected to autoverification. Future inclusion of other platforms and tests are currently in the pipeline. 
. 
Key words: algorithms; TAT; AMR; automated data processing; software; patient safety; 

1.	Introduction											Autoverification is a process whereby clinical laboratory results are released without manual human intervention . Autoverification uses predefined computer rules to govern release of results. The clinical laboratory is a part of health systems under the increasing pressure by users and administrators to increase their productivity in order to respond efficiently to the increased volume of patients, optimizing costs and professional time. Autoverification rules helps to take decisions based on instrument error flags (e.g. short sample, possible bubbles, or clot), interference indices (e.g. hemolysis, icterus, and lipemia), reference ranges, analytical measurement range (AMR), critical values, and delta checks (comparison of current value to previous values, if available, from the same patient).
Rules may also define potentially absurd (physiologically improbable) values for some analytes and additionally may control automated dilutions and conditions for repeat analysis of specimens. Autoverification is commonly performed using the laboratory information system (LIS) and/or middleware software that resides between the laboratory instruments and the LIS.
Autoverification can greatly reduce manual review time and effort by laboratory staff, limiting staff screen fatigue caused by reviewing and verifying hundreds to thousands of results per shift. Ideally, autoverification allows laboratory staff to focus manual review on a small portion of potentially problematic specimens and test results.However, improperly designed autoverification can lead to release of results that should have been held, potentially negatively impacting patient management.
There is a guideline document produced by the Clinical and Laboratory Standards Institute (CLSI) on autoverification of clinical laboratory test results which focuses on the process for validating and implementing autoverification protocols. In this study, we present data on autoverification used in a clinical chemistry core laboratory in a tertiary care cancer center.

2.	Objective												This study aimed to evaluate the implementation of the routine biochemistry tests autoverification process and its impact on the safety of patients. 

3.	Experimental design, materials and methods								A core laboratory within the Department of Laboratory Science provides clinical chemistry testing for both outpatient and inpatient services. This study focuses on the clinical chemistry division from 1/1/2016 to 30/11/2017.An outline of the Quality process improvement schedule based on DMAIC (Design, Measure, Analyze, Improve, Control) methodology is provided in Table 1. The project team consisted of thirteen-members representing managers, clinical biochemists, front line staff and others. The process metrics and benchmarks/targets were established during the “Design and Measurement” phases. Various process maps including Fig. 1 which outlines the patient Verification rules were designed in the connectivity software, based on manual validation criteria for laboratory professionals, according to the guidelines of the Clinical and Laboratory.

Following steps involved for implementation of AV process
Master algorithm template development - All reagent-specific (or test-specific) AV algorithms must be derived from an analyzer-specific algorithm, which must be derived from a master algorithm. The master algorithm provides the high-level description that will act as the blueprint for the entire AV system.

Tool selection and implementation- It is common for laboratories to invest in information technology platforms before developing the requirements for their function. The AV tool must be able to meet the needs of the laboratory as outlined in the algorithms. A single system for the entire laboratory reduces the cost of implementation, interfacing, and training, and the overall cost of service.
Translation of algorithms into rules. One of the greatest difficulties of implementing AV is translating the algorithms into code that the tool can use to perform AV.

Exceptions review process- One hundred percent autoverification is unachievable, and managing the AV exceptions (results that do not pass AV) can take a lot of time. Therefore, spend time understanding and optimizing the exceptions review process. Even when a laboratory isn’t performing AV, an optimal review process with the proper information system can dramatically improve error rates, TAT etc. A perfect opportunity to implement a new exceptions review process is during AV implementation.

Substantiation- Testing and validating rules can be time-consuming, but it’s critical to test every rule each time an algorithm is updated. After running quick tests on all rules to test functionality, document each test case and store the documentation electronically. Each test case should include analyzer printout, audit trail (from AV tools), and patient report.. AV revalidation must be performed regularly, with the frequency dependent on the laboratory’s regulating agency

Scale- Any level of autoverification will result in significant process improvement. Regular reassessment of AV must be part of the laboratory’s continuous improvement processes. The process improvement gained from implementing AV will make it possible for the laboratory to dedicate resources to AV. A balance must be maintained, however, because higher levels of autoverification become harder to achieve as the rates asymptotically approach 100 percent.

To define the turnaround time data a range total 15154 test results were collected in 2016 & 2017. These data were extracted from HMS then tabulated and calculated further
The autoverification percentage was established, and non-conforming product (NCP) percentages were estimated before and after the procedure. Pilot tests were also performed in different days so as to adjust the process. 

Table-1: Summary of activities by phase of the AV project
Phase 	Description	Activities	Relative start time (Duration)	
Define 	Most elements of project planning were carried out during this phase of the work	•	Draft Project Charter
•	Draft & finalize Schedule
•	Select/Prepare project team
•	Construct top level process map
•	Construct top level SIPOC a diagram
•	Finalize Project Charter	2 weeks
	
Measure	This phase involved defining, evaluating, and implementing
a system for measuring the AV process.
	•	Construct process maps for AV
•	Select metrics
•	Develop/Implement monitoring system
•	Begin data collection
•	Evaluate the measurement system	7 weeks
	
Analyze	This phase involved developing AV benchmarks and targets; and analyzing and interpreting data to inform decisions on improvements	•	AV Value Stream Analysis
•	Determine AV benchmark
•	Perform AV variance analysis
•	Perform Root cause analysis
•	Analyze requirements and process drivers summarize analyses
	2 weeks
	
Improve	This phase involved development and implementation of
new AV process.
	•	Prioritize improvement opportunities
•	Design new AV process
•	FMEA for new AV process
•	Implement new AV process
•	Examine early data from new
•	Process and optimize parameters
•	Feedback meetings with Technical staff
	6 weeks
	
Control	This phase involved verification of improvements and development of a control plan to maintain the new AV
process.
	•	Confirm/validate new AV process
•	Develop and implement SOPs and monitoring plan
•	Assign a monitor
•	Approve of deliverables
•	Project closeout and review	6 weeks	
a.	SIPOC (Suppliers, Inputs, Process, Outputs, Customers).
b FMEA (Failure Modes and Effects Analysis).
c SOP (Standard Operating Procedures).
E.W. Randell et al. / Data in Brief 18 (2018) 1740–17491742

Fig. 1. Top level process map describing the AV work flow. This flowchart identifies actions done by the automated analyzer, the middle ware software (Instrument Manager), the Technologist, and laboratory/hospital information system (LIS/HIS).[ E.W. Randell et al. / Data in Brief 18 (2018) 1740–1749]

Fig. 2.  Decision tree for tests held as extreme results (A) and delta checks (B). [ E.W. Randell et al. / Data in Brief 18 (2018) 1740–1749]

4.	Results												Measuring key metrics before and after implementation is critical to understanding return on investment and monitoring for continuous improvement. Turnaround time, technologist utilization, and error rates are three metrics that should be understood and tracked. It is relatively uncommon for many laboratories to have access to all of these statistics but important for them to be obtained. TAT is a relatively easy metric to obtain from laboratory information systems, but technologist utilization (amount of time a technologist spends performing duties) is generally unknown. Error rates can be obtained by determining the number of corrected reports (due to releasing or data-entry error) per number of patients that are run on the analyzers where AV will be performed.
99% of automated routine biochemistry parameters autoverification were achieved, and subsequently from the audit of 23 months.
Reduction of non-conforming products (NCP) during the authorization phase roses from 1.2 hrs to 0.10 min and the overall TAT decreased from 2hr 67 min to 1 hr 7min from the beginning to the end of the process. 

Fig 3: Break down of various aspects of TAT prior to autoverification

Fig 4: Break down of various aspects of TAT after autoverification

From both the figures it is clearly evidenced that there is a sharp decrease in overall TAT by 1 hr and significantly authorization took less than 30 min after introduction of autoverification rules. Our workflows allow for critical value autoverification, provided no other rules are violated

5.	Discussion											Autoverification of laboratory test results is an essential component of increasing efficiency within the clinical laboratories. Despite the importance of autoverification, there is relatively scant literature published on practical application within a clinical laboratory over the course of years.					In this report, we present our experience with autoverification in a busy automated core chemistry laboratory in a tertiary care cancer center. The autoverification rules gradually evolved, with a steady increase in autoverification rate to the current rate of 99.5%. The high rate of autoverification is driven in large part by the highest volume tests or test panels (e.g. routine chemistry), which all have autoverification rates exceeding 90.0%. Some tests in our study currently have autoverification rates under 90%; however, these tests comprise a small fraction of the total test volume.			To our knowledge, there is very little published data on autoverification of critical values. Our workflows allow for critical value autoverification, provided no other rules are violated. Critical values still require provider notification and subsequent documentation; however, communication of these results by laboratory staff is facilitated by the provider often seeing the autoverified value prior to the call. For example, the emergency treatment center and intensive care units in our medical center use electronic displays or dashboards that continuously display patient data in restricted staff areas. Phone calls to document the critical value proceed more quickly when the laboratory test result has already been seen.												Despite the advantages of autoverification, there are potential negatives that can arise. The validation of autoverification is time-consuming and attention to detail is paramount. Even the most thorough validation plan can miss unexpected instrument error flags or other rare events. It also not possible to test every conceivable combination of rules.										Informatics support is critical to successful implementation and maintenance of autoverification. The most common problems interfering with autoverification would be interruptions of network, LIS, AV software, and/or the interfaces between these systems. Another primary reasons were cited was information system incapable of performing complex rules and results review; and lack of autoverification experience and an established AV implementation process									In our institution, we have maintained both production and shadow middleware servers in different geographic locations, in addition to using separate test servers for initial testing of middleware rules without compromising the production system. This has reduced period of times the systems are down. The other risk with autoverification, and indeed with increased automation in general, is reduction of staff (both in number of staff and in the mix of level of training and experience) to such a degree that the staff cannot handle downtimes or other challenges without severe compromise of turnaround time.						One benefit of computer rules is to catch rare events that could elude manual verification. 		Our experience suggests that successful and continued use of autoverification requires investment in personnel and training over the course of years. Validation of autoverification rules requires high attention to detail. Rules should be based on published evidence and analysis of assays. The use of autoverification also does not obviate need for careful quality control. Lastly, close collaboration between the clinical laboratory and computing services is the key for ongoing success.

6.	Conclusion												The autoverification rules, along with instrument interfaces to the AV software and the LIS for which these rules depend, presented in this manuscript were developed over the course of a number of years and would not be possible without the hard work of staff. The autoverification process enabled to reduce the variability associated with human intervention, therefore the professional is able to focus on the pathological report analysis, reducing the risk of errors, TAT improvement and advocating greater importance on patient safety.

REFERENCES

1)	Building and Validating an Autoverification System in the Clinical Chemistry Laboratory: Laboratory Medicine, Volume 42, Issue 11, November 2011, Mu-Chin Shih, MT, Huey-Mei Chang, MS, Ni Tien, MS, Chiung-Tzu Hsiao, MS, Ching-Tien Peng
2)	Autoverification in Clinical Laboratory: Pitfalls and Challenges: Dr. Mohd Jamsani Mat Salleh: 24 Jun 2022
3)	Autoverification in a core clinical chemistry laboratory at an academic medical center: Matthew D. Krasowski1 , Scott R. Davis1 , Denny Drees1 , Cory Morris1 , Jeff Kulhavy1 , Cheri Crone1 , Tami Bebber1 , Iwa Clark1 , David L. Nelson1 , Sharon Teul1 , Dena Voss1 , Dean Aman2 , Julie Fahnle2 , John L. Blau1: Received: 17 December 2013 Accepted: 13 February 2014 Published: 28 March 14.
4)	A model to establish autoverification in the clinical laboratory: Deniz Ilhan Topcu 1, Ozlem Gulbahar 2, PMID: 33831387:DOI: 10.1016/j.clinbiochem.2021.03.018
5)	Autoverification: lessons drawn from a core lab in 2022 Issues, ARTICLES, August 2022
6)	Design and evaluation of a LIS-based autoverification system for coagulation assays in a core clinical laboratory: Zhongqing Wang1,2, Cheng Peng3, Hui Kang4, Xia Fan4, Runqing Mu4, Liping Zhou4, Miao He2 and Bo Qu: Wang et al. BMC Medical Informatics and Decision Making (2019) 19:123 https://doi.org/10.1186/s12911-019-0848-2
7)	A step-by-step process to 95% autoverification in 2015: Bill Marquardt, C(ASCP), MBA, CSSBB Issues, ARTICLES, December 2015
8)	Strategy for 90% autoverification of clinical chemistry and immunoassay test results using six sigma process improvement: Edward W. Randell a b, Garry Short c, Natasha Lee a, Allison Beresford a, Margaret Spencer c, Marina Kennell a, Zoë Moores a, David Parry a b: https://doi.org/10.1016/j.dib.2018.04.080
9)	S.H. Park, S.Y. Kim, W. Lee, S. Chun, W.K. Min, New decision criteria for selecting delta check methods based on the ratio of the delta difference to the width of the reference range  can be  generally applicable  for each  clinical  chemistry  test item, Ann. Lab. Med. 32 (5) (2012) 345–354.
10)	J. Lee, S.Y. Kim, H.J. Kwon, H.K. Lee, Y. Kim, Y. Kim,  Usefulness  of biological variation in  the establishment of  delta check limits, Clin. Chim. Acta 463 (2016) 18–21.