A Bayesian Framework to Optimize Performance of Pre-Hospital Stroke Triage Scales

Article information

J Stroke. 2021;23(3):443-448
Publication date (electronic) : 2021 September 30
doi : https://doi.org/10.5853/jos.2021.01312
aDepartment of Radiology, University of Calgary Cumming School of Medicine, Calgary, AB, Canada
bCalgary Stroke Program, Department of Clinical Neurosciences, University of Calgary Cumming School of Medicine, Calgary, AB, Canada
cHotchkiss Brain Institute, University of Calgary Cumming School of Medicine, Calgary, AB, Canada
dDepartment of Neuroradiology, University Hospital Basel, Basel, Switzerland
eCerebrovascular Center and Department of Neurology, Seoul National University Bundang Hospital, Seoul, Korea
fDepartments of Neurology, Radiology, and Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, Netherlands
gDepartment of Neurology, Albert Schweitzer Hospital, Dordrecht, Netherlands
Correspondence: Mayank Goyal Department of Radiology, University of Calgary Cumming School of Medicine, 1403 29 St NW, Calgary, AB T2N 2T9, Canada Tel: +1-403-944-3379 Fax: +1-403-270-7907 E-mail: mgoyal@ucalgary.ca
Received 2021 April 10; Revised 2021 June 8; Accepted 2021 June 9.

Dear Sir:

Early identification of patients with stroke due to large vessel occlusion (LVO), potentially eligible for thrombolysis and endovascular therapy.(EVT), is important for patient triage and management [1,2]—such as deciding whether to directly transport a given patient to an EVT-capable comprehensive stroke center.(CSC). Various pre-hospital triage tools for acute stroke have been developed for this purpose. Clinical evaluation scales are most commonly used, as they can be adopted by paramedics after some basic training and require no specialized equipment [3]. Below, using simulated data for the USA, we examine the strengths and weaknesses of current pre-hospital triage tools through a Bayesian lens, and discuss areas where our efforts should be directed to improve acute stroke triage.

What are we trying to accomplish with pre-hospital triage of stroke with suspected LVO?

Stroke triage tools are subject to sensitivity-specificity trade-offs like all tests. In general, a screening test needs high sensitivity, whereas a confirmatory test in positively screened patients needs high specificity. Most clinicians would think of LVO stroke as being a not-to-miss condition, in which case the primary goal of triage tools would be to maximize sensitivity. A policymaker’s perspective might differ: with scarce CSC resources, the priority may be to minimize false-positive LVO rates even if some LVOs are not flagged early for transport to CSCs. Advantages and disadvantages of both these paradigms are shown in Supplementary Table 1.

If we take the “mile-high” view of all strokes in the USA, approximately 795,000 Americans have a stroke annually [4]. Of these, around two-thirds (approximately 517,000) seek ambulance transport and are potentially eligible for pre-hospital triage [5,6]. Most of these patients are clustered towards the lower end of the severity spectrum (Figure 1), where the probability of LVO is low. Of course, the National Institutes of Health Stroke Scale (NIHSS) score is not actually known at the pre-hospital stage, but let us use it here to illustrate the spectrum of stroke severity. On a population level, the median severity for ischemic stroke patients per NIHSS is around 2 to 3 [7-9], likely shifted to around 5 among those seeking emergency medical attention [10]. But with each step up the NIHSS, the probability of LVO rises (Figure 1A), particularly getting into a moderate range of severity (NIHSS ≥10) [11]. Even though the percentage of patients with LVO may be lower at intermediate NIHSS scores, the overall number of patients in the middle range is considerably higher, and thus, the total number of EVT-eligible patients in that range will likely be higher than at the extreme of stroke severity (Figure 1B). When measuring the performance of pre-hospital triage tools, it is therefore important how they perform not only in the extremes of stroke severity, but also in the middle of the range.

Figure 1.

The occurrence of vessel occlusions in patients with ischemic stroke presenting with different degrees of stroke severity per the National Institutes of Health Stroke Scale (NIHSS) score. (A) The proportion of patients with and without large vessel occlusion (LVO), in each NIHSS score category, among all patients undergoing computed tomography or magnetic resonance angiography. These graphs are based on re-analyses of data reported from the Bernese Stroke Database [11]. (B) The estimated number of patients with ischemic stroke in the USA, seeking emergency medical attention each year, who are likely to have LVO for a given NIHSS score. These estimates were obtained by applying published data from the Bernese Stroke Database to stroke statistics from the American Heart Association [4]. In (B), the corresponding NIHSS score for different cut-offs on the Los Angeles Motor Scale (LAMS) are also shown, as roughly estimated from work by the Field Administration of Stroke Therapy–Magnesium (FAST-MAG) investigators on the distribution of vessel occlusions for different scores on the LAMS and NIHSS..

What are the characteristics and limitations of current pre-hospital triage tools?

Current prediction scales have low sensitivity (range 38% to 67%) for LVO—given their superficial focus on gross deficits—but high specificity (80% to 93%) [12-17]. A meta-analysis estimated that with a positive LVO prediction test, the LVO probability could be 50% to 60% depending on LVO prevalence in the population, but the probability with a negative test could still be ≥10% [13]. Thinking of patient-centered care, such a low sensitivity means that treatment might be withheld/delayed from patients who might still benefit from it, which is suboptimal, especially when considering that most pre-hospital tools are used in relatively wealthy, industrial countries with sufficient healthcare resources.

For instance, if we apply the Los Angeles Motor Scale (LAMS) to all emergently presenting stroke patients in the USA to decide whom to transport to CSCs (Figure 1B) [18], we would direct over one-third (n=188,000) to CSCs if using the conventional LAMS cut-off of ≥4, but more than one in five of these patients will not actually have a LVO, and we would still end up missing over one-quarter of all LVOs (n=59,000) (Supplementary Table 2). If we only consider patients with stroke severities in the intermediate range, the situation is even worse (Supplementary Table 3)—around 45% of positively screened patients would not have a vessel occlusion, but now we would be missing more than half of the patients with vessel occlusions (n=28,000).

Where might our efforts be focused to improve pre-hospital triage tools?

It can be helpful to adopt a Bayesian approach and think about how much current pre-hospital scales can shift our diagnostic confidence about whether a patient has an LVO. Although the sensitivity and specificity of pre-hospital scales for LVO likely varies by stroke severity (Table 1), most studies only report overall operating characteristics [18]. In the extremes of stroke severity, a positive test does little to shift the high pre-test probability of LVO (Figure 2), because the more disabling the stroke deficits, the more evident the diagnosis will be even to a layperson without any specialized knowledge. The patients/caregivers themselves may call in to say they are having a stroke. For example, with NIHSS ≥20, the pre-test LVO probability is around 94%, and a positive LAMS simply shifts this by around 3% to a post-test probability of 97%, meaning the test is far less useful in this range. Whereas pre-hospital scales still offer a common language for paramedics to evaluate deficits, one might argue that with such intuitively severe strokes, the best destination for the patient is almost always the CSC.

The value of pre-hospital triage tools likely differs by stroke severity

Figure 2.

Pre- and post-test probability of large vessel occlusion (LVO) when using the Los Angeles Motor Scale (LAMS) with its typical cut-off of LAMS ≥4. Of course, the National Institutes of Health Stroke Scale (NIHSS) is not typically determined in the pre-hospital setting, but we show it here as a quantification of the manifest stroke severity observed by the pre-hospital team. We also show what the post-test probabilities would be with hypothetical pre-hospital technologies that achieve a slightly better positive likelihood ratio (+LR) of 3.00 (vs. 2.17 for LAMS) and a much better +LR of 19.00, including at the mildest end of the severity spectrum, to which current scales are likely insensitive. Such technologies would be especially valuable if ongoing trials of endovascular therapy (EVT) in low-NIHSS populations are positive.

In contrast, at the mildest end of stroke severity, current scales have little to offer as they will almost always be negative. This leaves us with the intermediate severity range (e.g., NIHSS 5–10). Here, the pre-test LVO probability is 35% to 40%, so a positive LAMS moves us to a post-test probability of 55% to 60%, resulting in a substantial shift of 20%. Thus, we are left with a “sweet spot” or “Goldilocks zone” in the middle of the severity spectrum where a pre-hospital tool could make a tangible difference to patient management by identifying a stroke that a human being may be uncertain about (Table 1). This is particularly important given the high overall number of patients in the middle range of symptom severity.

Looking ahead: can alternative triage technologies solve this problem?

One possibility to improve stroke triage with existing tools is a probabilistic, rather than a dichotomous approach to LVO detection: rather than classifying patients as either LVO+ or LVO–, the entire spectrum of the score could be translated into an estimated “LVO probability.” Simulation models could then individually determine the optimal triage pathway for a given patient with a certain LVO probability under local geographic circumstances in real-time, e.g., via an app, while also taking into account transport times and workflow efficiency at local primary stroke centers and CSCs [19,20]. Time from onset would be an important consideration, since patients arriving earlier are not only more likely to be thrombolysis-eligible, but also tend to have more severe deficits [21].

Furthermore, alternative pre-hospital technologies have been developed that rely on direct visualization of the occluded vessel (e.g., computed tomography angiography in mobile stroke units, transcranial Doppler) or indirect signals of impaired cerebral activity like electroencephalography [22,23]. Since these technologies do not rely on clinical deficits, they may detect LVOs in even mild strokes for which current scales are insensitive. This will become very important if ongoing EVT trials in low-NIHSS patients with LVO (like ENDOLOW, NCT04167527 and MOSTE, NCT03796468) are positive, as stroke systems will then need to accurately flag these patients (~38,000 LVOs with NIHSS ≤5 annually in USA) (Figure 1B). For example, even a slightly better technology that achieves a positive likelihood ratio (+LR) of 3.00 would boost the LVO probability from 10% to 15% for a minor stroke with NIHSS ≤3 to 30% to 35%; a much better technology achieving both sensitivity and specificity of 95% (+LR 19.00) would further boost post-test probability to 75% to 80% (Figure 2). To succeed in practice, such technologies will also need to offer an appropriate combination of costs, practicability, and ease of use.

Ethical statement

As the estimates and figures in this paper were derived from published data from prior studies, no ethical approvals or patient consents were required for this work.

Conclusion and outlook

There are many ongoing efforts to develop better technologies for pre-hospital triage of patients with suspected vessel occlusions [22], but where might our limited resources be best directed when seeking to test and improve these technologies? Firstly, future studies should focus on the ability of pre-hospital tools to identify patients with intermediate stroke severity. Secondly, stroke systems have to decide what aspect of pre-hospital triage to prioritize in their environment: sensitivity e.g., when there are sufficient resources to avoid missing EVT-eligible patients; or specificity e.g., when there is resource scarcity at CSCs. Triage tool cut-offs should be adjusted accordingly. There is a need for high-quality data on how pre-hospital scales perform across the spectrum of stroke severity to help us better understand their strengths and weaknesses when seeking to apply them in care pathways, while we continue working on alternative triage technologies. Such new technologies may have maximal impact by achieving higher sensitivity and specificity for LVO among patients with low-to-medium stroke severity, particularly if EVT begins to play a bigger role in patients with milder LVO-related strokes.

Supplementary materials

Supplementary materials related to this article can be found online at https://doi.org/10.5853/jos.2021.01312.

Supplementary Table 1.

Sensitivity/specificity trade-off in acute ischemic stroke triage depending on the symptom severity cut-off

jos-2021-01312-suppl1.pdf
Supplementary Table 2.

Implications of using different LAMS cut-offs for triage of all suspected acute strokes

jos-2021-01312-suppl1.pdf
Supplementary Table 3.

Implications of using different LAMS cut-offs for triage of suspected acute strokes of intermediate severity

jos-2021-01312-suppl1.pdf

Acknowledgements

Mayank Goyal reports consulting fees from Medtronic, Stryker, Microvention, and Mentice; and has a patent for Systems of stroke diagnosis licensed to GE Healthcare. Johanna M. Ospel is supported by the Julia Bangerter Rhyner Foundation, University of Basel Research Foundation, and Freiwillige Akademische Gesellschaft Basel. Beom Joon Kim, Nima Kashani, Martijne H.C. Duvekot, and Bob Roozenbeek have nothing to disclose. Aravind Ganesh reports membership in editorial boards of Neurology, Neurology Clinical Practice, and Stroke; speaker honoraria from NHS Health Education England; consulting fees from MD Analytics, MyMedicalPanel, Adkins Research Group, and Genome BC; research support from The Rhodes Trust, Wellcome Trust, the University of Calgary, Alberta Innovates, the Canadian Cardiovascular Society, and the Canadian Institutes of Health Research; stock/stock options from SnapDx, TheRounds.ca, and Advanced Health Analytics (AHA Health Ltd); and has a provisional patent application (US 63/024,239) for a system to deliver remote ischemic conditioning or other cuff-based therapies.

References

1. Saver JL, Goyal M, van der Lugt A, Menon BK, Majoie CB, Dippel DW, et al. Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a metaanalysis. JAMA 2016;316:1279–1288.
2. Emberson J, Lees KR, Lyden P, Blackwell L, Albers G, Bluhmki E, et al. Effect of treatment delay, age, and stroke severity on the effects of intravenous thrombolysis with alteplase for acute ischaemic stroke: a meta-analysis of individual patient data from randomised trials. Lancet 2014;384:1929–1935.
3. Zhao H, Coote S, Pesavento L, Churilov L, Dewey HM, Davis SM, et al. Large vessel occlusion scales increase delivery to endovascular centers without excessive harm from misclassifications. Stroke 2017;48:568–573.
4. Benjamin EJ, Muntner P, Alonso A, Bittencourt MS, Callaway CW, Carson AP, et al. Heart disease and stroke statistics-2019 update: a report from the American Heart Association. Circulation 2019;139:e56–e528.
5. Xirasagar S, Tsai MH, Heidari K, Hardin JW, Wu Y, Wronski R, et al. Why acute ischemic stroke patients in the United States use or do not use emergency medical services transport?: findings of an inpatient survey. BMC Health Serv Res 2019;19:929.
6. Adeoye O, Lindsell C, Broderick J, Alwell K, Jauch E, Moomaw CJ, et al. Emergency medical services use by stroke patients: a population-based study. Am J Emerg Med 2009;27:141–145.
7. Ganesh A, Luengo-Fernandez R, Wharton RM, Gutnikov SA, Silver LE, Mehta Z, et al. Time course of evolution of disability and cause-specific mortality after ischemic stroke: implications for trial design. J Am Heart Assoc 2017;6:e005788.
8. Reeves M, Khoury J, Alwell K, Moomaw C, Flaherty M, Woo D, et al. Distribution of National Institutes of Health stroke scale in the Cincinnati/Northern Kentucky Stroke Study. Stroke 2013;44:3211–3213.
9. Brown DL, Lisabeth LD, Garcia NM, Smith MA, Morgenstern LB. Emergency department evaluation of ischemic stroke and TIA: the BASIC Project. Neurology 2004;63:2250–2254.
10. Wolters FJ, Paul NL, Li L, Rothwell PM, ; Oxford Vascular Study. Sustained impact of UK FAST-test public education on response to stroke: a population-based time-series study. Int J Stroke 2015;10:1108–1114.
11. Heldner MR, Zubler C, Mattle HP, Schroth G, Weck A, Mono ML, et al. National Institutes of Health stroke scale score and vessel occlusion in 2152 patients with acute ischemic stroke. Stroke 2013;44:1153–1157.
12. Nguyen TT, van den Wijngaard IR, Bosch J, van Belle E, van Zwet EW, Dofferhoff-Vermeulen T, et al. Comparison of prehospital scales for predicting large anterior vessel occlusion in the ambulance setting. JAMA Neurol 2021;78:157–164.
13. Smith EE, Kent DM, Bulsara KR, Leung LY, Lichtman JH, Reeves MJ, et al. Accuracy of prediction instruments for diagnosing large vessel occlusion in individuals with suspected stroke: a systematic review for the 2018 guidelines for the early management of patients with acute ischemic stroke. Stroke 2018;49:e111–e122.
14. Vidale S, Agostoni E. Prehospital stroke scales and large vessel occlusion: a systematic review. Acta Neurol Scand 2018;138:24–31.
15. Antipova D, Eadie L, Macaden A, Wilson P. Diagnostic accuracy of clinical tools for assessment of acute stroke: a systematic review. BMC Emerg Med 2019;19:49.
16. Mazya MV, Berglund A, Ahmed N, von Euler M, Holmin S, Laska AC, et al. Implementation of a prehospital stroke triage system using symptom severity and teleconsultation in the Stockholm Stroke Triage Study. JAMA Neurol 2020;77:691–699.
17. Duvekot MH, Venema E, Rozeman AD, Moudrous W, Vermeij FH, Biekart M, et al. Comparison of eight prehospital stroke scales to detect intracranial large-vessel occlusion in suspected stroke (PRESTO): a prospective observational study. Lancet Neurol 2021;20:213–221.
18. Noorian AR, Sanossian N, Shkirkova K, Liebeskind DS, Eckstein M, Stratton SJ, et al. Los Angeles Motor Scale to identify large vessel occlusion: prehospital validation and comparison with other screens. Stroke 2018;49:565–572.
19. Holodinsky JK, Williamson TS, Demchuk AM, Zhao H, Zhu L, Francis MJ, et al. Modeling stroke patient transport for all patients with suspected large-vessel occlusion. JAMA Neurol 2018;75:1477–1486.
20. Venema E, Lingsma HF, Chalos V, Mulder MJ, Lahr MM, van der Lugt A, et al. Personalized prehospital triage in acute ischemic stroke. Stroke 2019;50:313–320.
21. Matsuo R, Yamaguchi Y, Matsushita T, Hata J, Kiyuna F, Fukuda K, et al. Association between onset-to-door time and clinical outcomes after ischemic stroke. Stroke 2017;48:3049–3056.
22. Walsh KB. Non-invasive sensor technology for prehospital stroke diagnosis: current status and future directions. Int J Stroke 2019;14:592–602.
23. Sergot PB, Maza AJ, Derrick BJ, Smith LM, Berti LT, Wilcox MR, et al. Portable neuromonitoring device detects large vessel occlusion in suspected acute ischemic stroke. Stroke 2021;52:1437–1440.

Article information Continued

Figure 1.

The occurrence of vessel occlusions in patients with ischemic stroke presenting with different degrees of stroke severity per the National Institutes of Health Stroke Scale (NIHSS) score. (A) The proportion of patients with and without large vessel occlusion (LVO), in each NIHSS score category, among all patients undergoing computed tomography or magnetic resonance angiography. These graphs are based on re-analyses of data reported from the Bernese Stroke Database [11]. (B) The estimated number of patients with ischemic stroke in the USA, seeking emergency medical attention each year, who are likely to have LVO for a given NIHSS score. These estimates were obtained by applying published data from the Bernese Stroke Database to stroke statistics from the American Heart Association [4]. In (B), the corresponding NIHSS score for different cut-offs on the Los Angeles Motor Scale (LAMS) are also shown, as roughly estimated from work by the Field Administration of Stroke Therapy–Magnesium (FAST-MAG) investigators on the distribution of vessel occlusions for different scores on the LAMS and NIHSS..

Figure 2.

Pre- and post-test probability of large vessel occlusion (LVO) when using the Los Angeles Motor Scale (LAMS) with its typical cut-off of LAMS ≥4. Of course, the National Institutes of Health Stroke Scale (NIHSS) is not typically determined in the pre-hospital setting, but we show it here as a quantification of the manifest stroke severity observed by the pre-hospital team. We also show what the post-test probabilities would be with hypothetical pre-hospital technologies that achieve a slightly better positive likelihood ratio (+LR) of 3.00 (vs. 2.17 for LAMS) and a much better +LR of 19.00, including at the mildest end of the severity spectrum, to which current scales are likely insensitive. Such technologies would be especially valuable if ongoing trials of endovascular therapy (EVT) in low-NIHSS populations are positive.

Table 1.

The value of pre-hospital triage tools likely differs by stroke severity

Symptom severity at presentation Frequency Predicted performance of current clinical triage tools* Added value of a clinical triage tool for decisions Added value of a new pre-hospital technology highly sensitive and specific for LVO Rationale for added value or lack thereof
Severe stroke symptoms Relatively uncommon Moderate sensitivity Low Low Pre-test probability for LVO is very high
High specificity Diagnosis of major stroke is obvious even to an untrained observer
Intermediate severity Common Poor sensitivity High High Intermediate pre-test probability for LVO
Moderate specificity Patients with intermediate stroke severity are common
Mild stroke symptoms Common Poor sensitivity Low Potentially very High Pre-test probability for LVO is very low
Poor specificity Current triage tools are likely insensitive to mild strokes
If low-NIHSS trials of EVT are positive, then a pre-hospital technology able to identify such mild LVO-related strokes may be very valuable

LVO, large vessel occlusion; NIHSS, National Institutes of Health Stroke Scale; EVT, endovascular therapy.

*

The comparative performance of pre-hospital scales is yet to be systematically studied across different gradations of stroke severity.