Quality control validation for a veterinary laboratory network of six Sysmex XT‐2000iV hematology analyzers

Abstract Background Quality control (QC) validation is an important step in the laboratory harmonization process. This includes the application of statistical QC requirements, procedures, and control rules to identify and maintain ongoing stable analytical performance. This provides confidence in the production of patient results that are suitable for clinical interpretation across a network of veterinary laboratories. Objectives To determine that a higher probability of error detection (P ed) and lower probability of false rejection (P fr) using a simple control rule and one level of quality control material (QCM) could be achieved using observed analytical performance than by using the manufacturer's acceptable ranges for QCM on the Sysmex XT‐2000iV hematology analyzers for veterinary use. We also determined whether Westgard Sigma Rules could be sufficient to monitor and maintain a sufficiently high level of analytical performance to support harmonization. Methods EZRules3 was used to investigate candidate QC rules and determine the P ed and P fr of manufacturer's acceptable limits and also analyzer‐specific observed analytical performance for each of the six Sysmex analyzers within our laboratory system using the American Society of Veterinary Clinical Pathology (ASVCP)‐recommended or internal expert opinion quality goals (expressed as total allowable error, TEa) as the quality requirement. The internal expert quality goals were generated by consensus of the Quality, Education, Planning, and Implementation (QEPI) group comprised of five clinical pathologists and seven laboratory technicians and managers. Sigma metrics, which are a useful monitoring tool and can be used in conjunction with Westgard Sigma Rules, were also calculated. Results The QC validation using the manufacturer's acceptable limits for analyzer 1 showed only 3/10 measurands reached acceptable P ed for veterinary laboratories (>0.85). For QC validation based on observed analyzer performance, the P ed was >0.94 using a 1‐2.5s QC rule for the majority of observations (57/60) across the group of analyzers at the recommended TEa. We found little variation in P fr between manufacturer acceptable limits and individual analyzer observed performance as this is a characteristic of the rule used, not the analyzer performance. Conclusions An improved probability of error detection and probability of false rejection using a 1‐2.5s QC rule for individual analyzer QC was achieved compared with the use of the manufacturers' acceptable limits for hematology in veterinary laboratories. A validated QC rule (1‐2.5s) in conjunction with sigma metrics (>5.5), desirable bias, and desirable CV based on biologic variation was successful to evaluate stable analytical performance supporting continued harmonization across the network of analyzers.


Objectives:
To determine that a higher probability of error detection (P ed ) and lower probability of false rejection (P fr ) using a simple control rule and one level of quality control material (QCM) could be achieved using observed analytical performance than by using the manufacturer's acceptable ranges for QCM on the Sysmex XT-2000iV hematology analyzers for veterinary use. We also determined whether Westgard Sigma Rules could be sufficient to monitor and maintain a sufficiently high level of analytical performance to support harmonization.
Methods: EZRules3 was used to investigate candidate QC rules and determine the P ed and P fr of manufacturer's acceptable limits and also analyzer-specific observed analytical performance for each of the six Sysmex analyzers within our laboratory system using the American Society of Veterinary Clinical Pathology (ASVCP)-recommended or internal expert opinion quality goals (expressed as total allowable error, TE a ) as the quality requirement. The internal expert quality goals were generated by consensus of the Quality, Education, Planning, and Implementation (QEPI) group comprised of five clinical pathologists and seven laboratory technicians and managers. Sigma metrics, which are a useful monitoring tool and can be used in conjunction with Westgard Sigma Rules, were also calculated.

Results:
The QC validation using the manufacturer's acceptable limits for analyzer 1 showed only 3/10 measurands reached acceptable P ed for veterinary laboratories (>0.85). For QC validation based on observed analyzer performance, the P ed was >0.94 using a 1-2.5s QC rule for the majority of observations (57/60) across the group of analyzers at the recommended TE a . We found little variation in P fr between manufacturer acceptable limits and individual analyzer observed performance as this is a characteristic of the rule used, not the analyzer performance.

| INTRODUC TI ON
Quality goals have been established for clinical laboratory procedures based on biological variation data or expert opinion, [1][2][3][4][5] and analytical equipment that achieves these quality goals will generate results that are suitable for clinical decision-making. 6 Once analyzers have been evaluated and optimized to meet quality goals, statistical quality control (QC) aims to maintain performance within those goals. Statistical QC relies on one or more control rule applied to the results generated from regular analyses of quality control materials (QCM). For statistical QC to be effective in achieving its purpose, the selected rule(s) must be sensitive and specific for the identification of deteriorating analytical performance. That is, statistical control validation is a necessary step in the design and implementation of statistical QC rule(s).
In a network of veterinary laboratory hematology analyzers that have undergone a harmonization process, 7 statistical QC performs an additional role in maintaining harmonization and allowing for continued interchangeability of results and common reference intervals within the veterinary network. In the same way that QC validation supports the effectiveness of QC to ensure the stability of a single analyzer, QC validation is a necessary step in the design and implementation of QC rules for a network of harmonized analyzers.
The sensitivity and specificity of QC are reflected by the probability of error detection (P ed ) and probability of false rejection (P fr ), respectively. The P ed is a measure of the frequency with which a control rule would cause analytical runs to be rejected when results contain errors beyond the inherent imprecision. Ideally, error detection should be set to 100% for medically significant errors; however, error detection at ≥90% is considered sufficient. 6 Conversely, P fr is a measure of the frequency with which analytical runs are rejected when there is no apparent reason or issue. The goal is to have the highest possible P ed to ensure that medically important errors are not missed and a low P fr (≤5%) 6 to reduce the waste cost and efficiency impact on patient sample volume, reagent use, and result turnaround time.
This optimizes the efficiency and capability of statistical QC as a tool for the demonstration of stable laboratory system performance. The analytical performance capabilities of the analyzer inform the choice of the number of control materials and statistical rules selected, which then determines the P ed and P fr achieved. This step-by-step process results in a quality control validation.
Laboratories using statistical QC often employ the Westgard rules, which are denoted by a shorthand notation. For example, a 1-2s rule means that control limits encompass two standard deviations (SD) on each side of the observed mean. A 1-3s refers to a rule that is set at ±3 SDs, and the rule is violated when one measurement exceeds ±3 SDs from the QCM mean. The numeral 2 placed in front of a rule, such as 2-2s, indicates the rule is violated when two consecutive control measurements or two single measurements across two QCMs are outside the 2SD control limits. 6 Six Sigma process-improvement methodology has been applied to clinical laboratory analyses such that performance capability (bias and imprecision) relative to TE a can be represented on a numeric scale as a "sigma metric" (σ). Sigma metrics and QC rules have been combined in Westgards Sigma Rules, 6 such that sigma metrics can, in turn, determine whether a simple single QC rule or a more complicated collection of rules is required. A sigma metric ≥6 for an analytical method indicates <3.4 errors/defects per million results, 8 defined as world-class performance, and the implied low bias and imprecision mean that these methods are easily controlled with simple QC rules and a low number of QCM measurements. It would take a large deterioration in performance to cause a clinical error, and such a large shift could be readily detected with a simple QC rule.
In fact, for measurands with sigma metrics ≥6.0, acceptable P ed and P fr can be achieved using 1 or 2 control measurements. Conversely, measurands with lower sigma metrics require a multirule and/or larger number of control measurements and may or may not be able to achieve acceptable P ed and P fr . 9 Commercially available software called EZRules3 (Westgard QC, https://www.westg ard.com/store/ softw are.html) allows the user to explore and select QC rules based on the total error quality goal and observed analytical performance as well as the P ed and P fr that can be achieved. 8 It has been acknowledged that a P ed of ≥0.85 can be used 10 for point of care testing (POCT) in veterinary medicine or for hematologic analyses, where the use of a single QCM is preferred. The use of a single hematology control material is traditional in the UK but a P ed ≥0.90 cannot be achieved with a 1-3s rule and a single QCM data point can only achieve a P ed ≥0.85. 11,12 This paper describes a network of six harmonized Sysmex XT-2000iV hematology analyzers across five locations, 7 which required a QC approach that could ensure the maintenance of individual Conclusions: An improved probability of error detection and probability of false rejection using a 1-2.5s QC rule for individual analyzer QC was achieved compared with the use of the manufacturers' acceptable limits for hematology in veterinary laboratories. A validated QC rule (1-2.5s) in conjunction with sigma metrics (>5.5), desirable bias, and desirable CV based on biologic variation was successful to evaluate stable analytical performance supporting continued harmonization across the network of analyzers.

K E Y W O R D S
harmonization, hematology, quality control, sigma metrics, validation, Westgard rules analyzer performance and supported continued harmonization. In deciding which QC approaches to evaluate, the following preferences and constraints had to be taken into consideration. First, the use of a simple control rule (e.g., 1-2.5s, 1-3s) rather than a multirule and a single level of QCM was preferred due to the simplicity of training and evaluation and based on traditional laboratory practices and economics. Second, the choice of QCM was limited because a third-party control material was not available for the Sysmex analyzer, only a manufacturer-provided QCM. It was felt that the potential disadvantages of a single level of QCM could be mitigated since all blood smears undergo additional nonstatistical quality control via microscopy by a fully trained technician to validate the automated results prior to a pathologist's review of laboratory data and correlation with clinical findings before results are reported.
The authors set out to discover whether an effective QC approach supporting network harmonization could be successfully implemented, given these constraints and preferences. This study addressed the following objectives: 1. Determine that a higher P ed can be achieved using QC rules validated for each individual analyzer than that achieved using the manufacturer's acceptable limits for the control limits.
3. Assess the use of sigma metric evaluation as part of the QC approach in conjunction with the validated QC rules appropriate for each analyzer, with a goal of using it to monitor when instrument servicing is needed to maintain a high level of instrument performance.
4. Assess whether analyzer performance criteria established in a previous study (bias <3%, achievement of desirable biologic variation-based goals for CV and bias, and 0.33CV I goals, with sigma metrics >5) would be confirmed in this study as useful contributions to an overall QC approach.

| Hematology analyzers
The data from six Sysmex XT-2000iV analyzers (Sysmex Corporation, Kobe, Japan) located in five laboratory locations in the UK (n = 4) and Ireland (n = 1) were evaluated. Analyzers were designated analyzers 1, 2, 3, 4a, 4b, and 5. This network of analyzers had previously undergone optimization and successful harmonization. 7 Analyzer 1 was designated the reference analyzer.

| Quality control material
The QCM used for these evaluations was

| Evaluation of manufacturer's acceptable limits as control rules using EZRules3
The manufacturer's acceptable limits for the hematologic measurands were evaluated using data from 1 month of QCM results from analyzer 1. The width of the manufacturer's acceptable limits for a measurand was divided by the standard deviation (SD) for the reference lab (analyzer 1) to determine the number of SDs contained within this range.
That number was divided by two to determine the SDs on each side of the target mean and was then rounded to the nearest QC rule available within EZRules3 to evaluate the performance of the control rule most closely representative of the manufacturer's acceptable limit.
The control rule was then assessed using EZRules3 to determine the P ed and P fr that are possible using the startup QC design, which is a program that allows the user to follow a series of prompts, including the manufacturer's target mean as the decision-level concentration, ASVCP recommendations, 3 and/or internal expert opinion for total allowable error (TE a ) as the chosen quality requirement, and the number of QCMs set to n = 1. For those measurands, where no ASVCP recommendation for TE a was available (PCT and RETIC number), expert opinion goals from an internal working group composed of pathologists and technicians were used (5 and 7 persons, respectively). The expected instability setting in the software was set to off.

| QC validation-evaluation of control rules based on observed individual analyzer performance
The QC validation for each of the six Sysmex instruments, customized for observed performance, was determined using approximately 1 month of QC data (March 2020), as reported previously. 7 We used EZRules3 to plot the observed imprecision and observed bias (calculated from the manufacturer's target mean) generated from daily repeated QC measurements for each analyzer and measurand to determine individual analyzer QC rules and associated P ed , P fr , and sigma metrics.
The candidate rules evaluated using the manual selection option were 1-2.5s, n = 1 and 1-3s, n = 1. A P ed ≥0.90 can theoretically be achieved with a 1-2.5s rule and a single QCM if performance is sufficiently good.

| Review of criteria for measurands with poor performance
Findings from a previous study 7 indicated that Sigma metrics >5, bias <3%, and measurands achieving desirable biologic variation goals for CV and bias demonstrated confident stable analytical performance.
These criteria were reviewed and compared with the findings of this study, which confirmed that harmonization was maintained during the course of the study.

| QC validation for manufacturer's acceptable limits
The results of the QC validation for the manufacturer's acceptable limits for the QCM using the observed SD for analyzer 1 are summarized in Table 1. Acceptable criteria for P ed (>0.85) were met for only three measurands (HGB, PCT, and WBC). Acceptable P fr (≤0.05) was achieved for all 10 measurands using the available control rules that most closely represented the manufacturer's acceptable limits.

| QC validation based on observed analyzer performance for six Sysmex analyzers
Tables 2-4 summarize the results of the QC validation for two control rules (1-2.5s and 1-3s) based on observed analyzer performance (March 2020) for the selected measurands on six Sysmex analyzers.
Only three measurands did not achieve a P ed ≥0.85; RBC and PLT on analyzer 4a using a 1-3s rule (n = 1), and PCT using both QC rules, 1-2.5s and 1-3s (n = 1). All other measurands achieved a P ed ≥0.85 with either control rule across the six analyzers. P fr ≤0.05 was achieved for all 10 measurands for each of the six Sysmex analyzers and both candidate control rules.

| Evaluation of QC rules
The 1-2.5s rule offered the highest P ed for all measurands for the group of Sysmex analyzers based on the observed analyzer performance and yielded P fr s of only 1% (Tables 2-4). The manufacturer's QC limits achieved acceptable P ed for only three measurands (Table 1).

| Sigma metric monitoring
A monthly review of sigma metrics showed three measurands that performed <5.5 sigma failed to achieve quality goals for P ed for one (RBC and PLT) or both QC rules (PCT). On further investigation, the quality goal index (QGI) 13 demonstrated that observed imprecision was implicated for two measurands, while imprecision and bias were implicated for one measurand. The same analyzer (4a) accounted for all three measurands with poor performance, which was compared with the other analyzers, where performance was >5.5 sigma. (see Tables 2-4).

| Review and optimization of criteria for measurands with poor performance
Previous criteria established for stable analytical performance indicated that measurands with a bias <3% achieved desirable biologic variation-based goals for CV, bias, and 0.33CVi goals, with sigma metrics >5 supported analytical stability and harmonization. In this study, we noted that a sigma metric >5.5 rather than 5 demonstrated analytical stability. P ed was achieved when the observed CV and Bias achieved desirable biologic variation goals, as previously reported. 7 Imprecision limits to achieve acceptable P ed (>0.85) for RBC (see Table 2, analyzer 4a) using the 1-2.5s rule; if desirable bias based on biologic variation was applied (1.761%), then the observed CV % must be ≤1.45 to achieve a P ed >0.94. Similarly, for PLT on analyzer 4a (Table 3), the Sigma metric was 5.13, and for the 1-2.5s rule, we could determine that if desirable bias was achieved based on biologic variation (5.17%), then the observed CV % must be ≤2.60% (3.72%).
Therefore, the 1-3s rule was not satisfactory, the 1-2.5s rule was satisfactory at 0.86 P ed , and the other analyzers achieved a P ed >0.94.

| DISCUSS ION
To the authors' knowledge, this is the first reported QC validation for a network of Sysmex analyzers used in veterinary hematology and ongoing harmonization within a laboratory system. The quality goal used to determine the P ed and P fr was TE a provided by the ASVCP 3 or internal expert opinion (RETIC and PCT). QC validation, using the manufacturer's acceptable limits and observed analytical performance of analyzer 1, showed that only 3/10 measurands (HGB, WBC, and PCT) achieved an acceptable P ed with the recommended TE a or expert opinion in the case of PCT (Table 1)   Notes: Number of observed SDs in the manufacturer's acceptable range was divided by two to determine the number of SD on each side of the manufacturer's target mean. This was used to determine the closest control rule available for selection in the EZRULES3 program for evaluation. If the number of SD within the acceptable range/2 is much less than the closest available control rule, the width of the acceptable range will be reduced compared with the manufacturer's acceptable range, resulting in overestimation of the P ed (PCT). If the SD within the acceptable range/2 is much greater than the closest available control rule, the width of the acceptable range will be greater than the manufacturer's acceptable range, resulting in underestimation of the P ed (RDW-CV).

RETIC
Abbreviations Notes: All instruments used the same QC material lot number. The number of quality control materials is one (n = 1). P ed <0.85 is highlighted by bold type.
Abbreviations: CV, coefficient of variation; P ed , probability of error detection; P fr , probability of false rejection; PCT, plateletcrit; PLT, platelet count; QC, quality control; TE, total error; TE a , total allowable error.
derived from groups of instruments rather than individual instruments. For the manufacturer's limits that failed to achieve an acceptable P ed , error detection was as low as >0.01, using the closest available QC rule 6,8,15 of 1-5s or 1-6s. In comparison, the QC validation used the observed performance of individual analyzers for all 10 measurands, achieving a P ed >0.94 with the 1-2.5s QC rule and a P ed >0.85 with the 1-3s QC rule for the observed performance of analyzer 1, resulting in much higher error detection.
We determined that a higher P ed could be achieved using Q C rules validated for each individual analyzer rather than the use of the manufacturer's acceptable limits. We determined that we could achieve an acceptable P ed , >0.94 for 95% of observations and P fr ≤0.05 for all observations for our network of Sysmex analyzers. We failed to meet the P ed criteria (>0.85, n = 1) for 3/60 (5%) of observations (analyzer 4a for RBC, PLT, and PCT) but were able to achieve the required P ed >0.85 using either 1-2.5s and/or 1-3s QC rule, n = 1, for all other measurands. It is noteworthy that the performance of analyzer 4a improved following instrument service and was able to achieve the quality goals (data not shown). This is compared with the manufacturers' P ed , where only two measurands achieved >0.94 (Table 1). We are satisfied that the QC validation from the observed analytical performance offers the most consistent probability of error detection ≥0.85 for the largest number of measurands and, therefore, greater confidence in quality control and the likelihood of network stability. Using the manufacturer's acceptable limits does not provide sufficient P ed for most measurands to have peace of mind regarding detection of unstable instrument performance.
The probability of false rejection was within P fr criteria (<0.05) for analyzer 1 using both manufacturer's limits and observed analytical performance. Overall, the individual analyzer validation allowed us the ability to achieve an error detection ≥0.85 and low levels of false rejection, resulting in more satisfactory QC using the 1-2.5s rule.
The 1-2.5s rule proved to be the optimal solution for one level of control material as the P ed was >0.94 for the majority of observations, whereas a P ed of 0.88 was the highest value achieved using the 1-3s rule at the recommended TE a . This was achieved with little variation in P fr between the rules used to assess individual analyzer validation since P fr is considered a function of the rule used and the number of QCM and is not based on analyzer performance.
For the measurand (PCT) that failed to meet acceptable criteria for P ed using both QC rules (1-2.5s and 1-3s), it was considered whether the TE a based on expert opinion was too stringent; however, as the remaining analyzers performed optimally, declining performance was more likely. Moreover, on closer evaluation, the large CV (5.31%) and sigma metric <6 (σ = 4.56) suggest that for this measurand, the instrument performance was deteriorating and required attention, which was reflected in both QC rule failures.
Sigma metrics were applied as an additional performance monitoring tool and in conjunction with QC validation. The use of sigma metrics as a performance indicator is well documented. 16 distinct from the other analyzers. Biologic variation data were not available for this measurand, but from our data, if bias was 0%, an observed CV ≤7.0% is required to meet the quality goal (40%).
These observations demonstrate that, depending on the error budget, a low % CV with a high bias may still achieve the quality goal and vice versa, but they must meet desirable goals based on biologic variation. The use of sigma metrics in conjunction with the QC rules (Westgard Sigma rules) highlighted analytical instability for analyzer 4a compared with the other 5 analyzers and technical attention was required for those measurands performing at <5.5 sigma.
The measurands that were easily controlled were >5.5 sigma.
These measurands had results with desirable biases and CVs based on biologic variation and could achieve high P ed and low P fr at the specified TE a . By meeting these criteria, these measurands were controlled using a simple QC rule. This is more cost-effective 16,18 and less labor-intensive for the technician, offering a degree of confidence to the clinical pathologist interpreting the results in a multisite, multi-analyzer environment. For these measurands, a 1-2.5s rule was considered the best candidate rule for ongoing use. This validation study is a much-needed step in a harmonization process, ensuring that our network of analyzers is comparable. We know that analytical variability exists between instruments of the same model 7,19 ; therefore, by implementing validated QC rules and monitoring analytical performance using Westgards Sigma Rules, 20 we are controlling this aspect of variability. The benefits gained from harmonization using maximally efficient high P ed and low P fr control approaches include a reduction of network costs, uniformity of standard operating procedures, unified quality management policy, unified training and proficiency, less rerun waste, and increased uniformity of turnaround times across laboratories.
Measurands more difficult to control were those <5.5 sigma, which had lower error detection (<0.85 P ed ) and higher observed CV (>3.5%) compared with better performing measurands. In most cases, these measurands failed to meet the desirable CV based on biologic variation (Tables 2-4). In review of the criteria for identifying a need for analytical and technical attention, previous findings 7 suggest that a bias >3%, failure to meet desirable biologic variation goals, and sigma metrics <5 were immediate triggers for poor performance requiring servicing. However, this study suggested some modifications to those criteria because some measurements showed suboptimal performance (PLT, RBC, and PCT for analyzer 4A) when sigma metrics were <5.5 rather than <5. For those example measurands with biological variation data available (RBC, HGB, HCT, WBC, and PLT), those that failed to meet desirable biologic variation goals with a sigma metric <5.5 warrant further investigation as they were poorly controlled with a 1-2.5s and/or 1-3s rule.
A more complicated multirule could be adopted in these cases, but in the interest of keeping QC simple and not too laborious for the technicians and network quality Managers, our recommended approach is to monitor the sigma metric, observed bias %, and observed CV %. We are confident that using (1) the 1-2.5s QC rule, (2) ensuring that the desirable biologic variation-based quality requirements for bias and CV are met, and (3)  Some academic work has been published on QC validation of hematology analyzers, but this has predominantly been in human medicine. 21,22 In veterinary medicine, biochemistry analyzer QC validation has encompassed more of the focus. 14,15 Early work by Freeman and Gruenwaldt, 23 as well as some comparative work, [24][25][26] has created a good foundation for our harmonization study and emphasized that QC validation should be a requirement for veterinary laboratories, which is further supported in this study.

| CON CLUS IONS
We recommend QC validation of individual hematology analyzer performance to achieve high P ed and low P fr rather than the use of the manufacturer's acceptable QC limits for hematology in veterinary laboratories. We validated 57/60 observations using the 1-2.5s and/or 1-3s QC rule; however, for optimal P ed , we applied the 1-2.5s

D I SCLOS U R E
The authors have indicated that they have no affiliations or financial involvement with any organization or entity with a financial interest in, or in financial competition with, the subject matter or materials discussed in this article.