External Validation of the ELAPSS Score for Prediction of Unruptured Intracranial Aneurysm Growth Risk
Article information
Abstract
Background and Purpose
Prediction of intracranial aneurysm growth risk can assist physicians in planning of follow-up imaging of conservatively managed unruptured intracranial aneurysms. We therefore aimed to externally validate the ELAPSS (Earlier subarachnoid hemorrhage, aneurysm Location, Age, Population, aneurysm Size and Shape) score for prediction of the risk of unruptured intracranial aneurysm growth.
Methods
From 11 international cohorts of patients ≥18 years with ≥1 unruptured intracranial aneurysm and ≥6 months of radiological follow-up, we collected data on the predictors of the ELAPSS score, and calculated 3- and 5-year absolute growth risks according to the score. Model performance was assessed in terms of calibration (predicted versus observed risk) and discrimination (c-statistic).
Results
We included 1,072 patients with a total of 1,452 aneurysms. During 4,268 aneurysm-years of follow-up, 199 (14%) aneurysms enlarged. Calibration was comparable to that of the development cohort with the overall observed risks within the range of the expected risks. The c-statistic was 0.69 (95% confidence interval [CI], 0.64 to 0.73) at 3 years, compared to 0.72 (95% CI, 0.68 to 0.76) in the development cohort. At 5 years, the c-statistic was 0.68 (95% CI, 0.64 to 0.72), compared to 0.72 (95% CI, 0.68 to 0.75) in the development cohort.
Conclusions
The ELAPSS score showed accurate calibration for 3- and 5-year risks of aneurysm growth and modest discrimination in our external validation cohort. This indicates that the score is externally valid and could assist patients and physicians in predicting growth of unruptured intracranial aneurysms and plan follow-up imaging accordingly.
Introduction
The prevalence of unruptured intracranial aneurysms in the adult population is around 3% [1], and the number of incidentally found intracranial aneurysms is increasing [2]. After diagnosis of an unruptured intracranial aneurysm, the risk of aneurysm rupture has to be carefully weighed against the risk of complications from preventive aneurysm treatment. In many patients, the risk of treatment complications from preventive treatment is estimated to outweigh the risk of aneurysm rupture. In a number of these patients, the initial treatment of choice is conservative management with follow-up imaging [3]. Follow-up imaging is often recommended because some aneurysms grow during follow-up, indicating aneurysm instability with an increased risk of rupture [4,5]. If aneurysm growth is detected during follow-up, preventive treatment should be reconsidered [3]. However, current international guidelines lack clear recommendations regarding the optimal timing for follow-up imaging [6,7].
The ELAPSS (Earlier subarachnoid hemorrhage, aneurysm Location, Age, Population, aneurysm Size and Shape) score was developed to guide clinicians in predicting absolute 3- and 5-year growth risks per aneurysm, according to six simple patient- and aneurysm characteristics (Table 1) [8]. The total score ranges from 0 to 40, with a higher score indicating increasing 3- and 5-year growth risks. The score had accurate calibration and modest discrimination. However, since prediction models often perform worse in new data than in the development data, a prediction score is preferably externally validated before it is implemented in clinical practice [9]. Therefore, we performed a multicenter study to externally validate the ELAPSS score.
The ELAPSS score for prediction of risk of aneurysm growth
Methods
Study population
We obtained individual patient data from 11 cohorts of patients with unruptured intracranial aneurysm(s) from the following geographic regions: Japan (Toyama, Tokyo, and Otawara), Finland (Tampere, Kuopio, and Turku), the Netherlands (Nijmegen, Amsterdam, and the Hague), France (Toulouse), United Kingdom (Glasgow), Germany (Mannheim and Düsseldorf) (Supplementary Table 1). The Kuopio University Hospital also participated in the study to develop the ELAPSS score [8], but for the validation study a more recent and not previously analysed patient cohort was used.
Inclusion and exclusion criteria
Following the criteria of the previous study [8], we included cohorts with consecutive patients ≥18 years with ≥1 unruptured saccular intracranial aneurysm with ≥6 months of radiological follow-up by means of magnetic resonance angiography, computed tomography angiography, or calibrated digital subtraction angiography. Extradural aneurysms, fusiform aneurysms, or aneurysms that were part of an arteriovenous malformation were excluded.
Data collection
The following variables were retrieved by the local investigators. At time of aneurysm detection: date of first imaging, age, sex, history of subarachnoid hemorrhage, aneurysm location, and aneurysm size and shape; and during follow-up: occurrence of aneurysm growth, date of first imaging with aneurysm growth, date of last imaging without aneurysm growth, occurrence of rupture, date of rupture, and date of last imaging assessment. Aneurysm size was assessed by the local investigators or retrieved from radiologic reports on 0.1 to 1 mm scales, depending on local facilities. The local investigators also assessed aneurysm shape. Irregular aneurysm shape was defined as multiple lobes, blebs or wall protrusions.
Outcome assessment
Aneurysm growth was defined as growth in one direction of ≥1 mm or an indisputable change in aneurysm shape on follow-up imaging. Because aneurysm growth is thought to be an irregular process with periods of stability followed by periods of slow or rapid growth, it is not possible to determine the exact time of aneurysm growth [10]. The time of aneurysm growth was estimated as the time point halfway between the last imaging prior to aneurysm growth and the first imaging with aneurysm growth. Follow-up data for patients were censored at the time of aneurysm growth, at the time of the last imaging assessment before surgical or endovascular treatment or rupture, or at the time of the last imaging assessment during follow-up.
Approval
The Institutional Review Board of the University Medical Center Utrecht decided that no formal approval was needed to conduct this study.
Statistical analysis
Data were missing on aneurysm size for 13 aneurysms (<1%). These aneurysms were excluded from the analysis. The original regression equations were applied to calculate 3- and 5-year growth risks for each aneurysm [8]. We assessed calibration with calibration plots and discriminatory performance of the model with the c-statistic with 95% confidence interval (CI). Calibration at 3 and 5 years was examined by dividing patients in quintiles according to their predicted risk. The mean predicted risk per quintile group was subsequently plotted against the observed risk per quintile group. Calibration over time was assessed across risk groups that were predefined as low-risk (0 to 9 points on the ELAPPS score), intermediate-risk (10 to 19 points), and high-risk (≥20 points). Analyses were performed in R version 3.4 (R Foundation for Statistical Computing, Vienna, Austria). Results are reported in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [11].
Results
Between February 1996 and May 2017, a total of 1,072 patients with 1,452 aneurysms and 4,268 aneurysm-years of follow-up were followed. Median follow-up time was 2.0 years (interquartile range, 1.1 to 4.1). Patient- and aneurysm characteristics at time of aneurysm detection in the development and validation cohort are shown in Tables 2 and 3. Patients in the validation cohort were younger than patients in the development cohort and were less often Japanese.
Aneurysm growth was observed in 199 aneurysms (14%) in 155 patients (14%). Growth rates were comparable to those in the development cohort (Figure 1). Aneurysm rupture occurred in 14 patients (1.3%). In 10 of these 14 patients (71%) with aneurysm rupture, growth was detected before rupture.
Cumulative risk of aneurysm growth in development and validation cohort.
Calibration at 3 years showed that the ELAPSS score slightly underestimated growth risk, in particular in the lowest risk quintile, but overall observed risks were within the range of the expected risks (Figure 2A). For calibration at 5 years again observed risks were within the range of the expected risk, but some underestimation occurred in patients in the lowest-risk quintile, and an overestimation occurred in the highest risk quintile (Figure 2B). Calibration across ELAPSS risk categories over time was accurate, but 5-year growth risk was somewhat underestimated in the low- and intermediate-risk categories (Figure 3). Supplementary Figure 1 shows observed growth risks in the development and validation cohort according to ELAPSS risk categories. Higher growth risks were observed in the low- and intermediate-risk categories in the validation cohort compared to the development cohort.
Calibration plots: predicted and observed (A) 3-year and (B) 5-year growth risk. Dots represent observed probabilities and bars the corresponding 95% confidence intervals, grouped according to quintiles of predicted probabilities.
Calibration of the ELAPSS (Earlier subarachnoid hemorrhage, aneurysm Location, Age, Population, aneurysm Size and Shape) score according to low-, moderate-, and high-risk categories. Cumulative observed risk (solid line) and predicted risk (dotted line) across risk categories of the ELAPSS score (low, ≤9; moderate, 10 to 19; high ≥20).
The c-statistic at 3 years was 0.69 (95% CI, 0.64 to 0.73), compared to 0.72 (95% CI, 0.68 to 0.76) in the development cohort, and at 5 years 0.68 (95% CI, 0.64 to 0.72), compared to 0.72 (95% CI, 0.68 to 0.75) in the development cohort.
Discussion
In this validation study, we found that the ELAPSS score showed accurate calibration for 3- and 5-year risks of aneurysm growth, comparable to that in the development study [8]. Discrimination was modest but also comparable to that of the development study, making the score overall an externally valid tool with modest discriminative power.
Our findings are in line with a recent study that found increasing ELAPSS score to be associated with aneurysm growth in a cohort of 353 patients with 431 unruptured intracranial aneursyms [12]. The reason for the underestimation of the 5-year growth risk for low-risk aneurysms in our validation cohort may be in part because median follow-up time in the validation cohort was only 2.1 years, which limits reliability of the Kaplan-Meier estimated observed probabilities at 5 years. Underestimation could also result from factors not included in the model. Several aggregate data meta-analyses have identified risk factors for aneurysm growth in addition to the risk factors included in the ELAPSS score, such as sex, hypertension, smoking, and aneurysm multiplicity [13,14]. However, in the development of the ELAPSS score, sex and hypertension were discarded as predictors because of their limited predictive value. Aneurysm multiplicity carries a higher patient risk of growth, but this is probably because of the multiple hit risk [15]. Smoking is an important risk factor for aneurysm growth that should be weighed in clinical decision-making, but could not be included in the present prediction model because of lack of data in the parent cohorts of the development study. Since patients may start or quit smoking during follow-up, or increase or decrease the number of cigarettes per day over time, the exact impact of smoking on aneurysm growth is difficult to assess.
This study has some limitations that need to be addressed. First, because management of the patients was not according to a pre-specified protocol, indication for and timing of aneurysm follow-up were determined by the local treating physicians, and thus varied between and within the cohorts, as it was the case in the ELAPSS development study. Moreover, aneurysms with a very high growth risk were probably treated and therefore did not contribute to follow-up. Similarly, patients with a limited life expectancy due to advanced age or comorbidity often do not undergo follow-up imaging. Therefore, this risk score can only be applied to patients in whom it is decided to perform a wait-and-scan policy. Second, measurements were done by the local investigators of the participating centers, mostly without double data entry or evaluation of interobserver agreement, again similar to how this was done in the study underlying the development of the ELAPSS score. Furthermore, we included different imaging modalities. These are limitations for the outcome measurement, but on the other hand increase the external validity of our results.
It remains to be investigated if the discriminative performance of the model can be improved by including other parameters. Examples may be smoking status, population differences not accounted for by the model, novel imaging parameters like gadolinium enhancement of the aneurysm wall on magnetic resonance imaging [16], or other, still unknown risk factors. Also, in order to formulate recommendations on the ideal time intervals for repeated imaging, a cost-effectiveness analysis should be performed. As shown in a recent study [17], no follow-up imaging may be indicated for small, low-risk aneurysms.
Conclusions
The ELAPSS score showed accurate calibration for 3- and 5-year risks of aneurysm growth and modest discrimination in our external validation cohort. This indicates that the score is externally valid and could assist patients and physicians in predicting growth of unruptured intracranial aneurysms and plan follow-up imaging accordingly.
Supplementary materials
Supplementary materials related to this article can be found online at https://doi.org/10.5853/jos.2019.01277.
Patient- and aneurysm characteristics of included cohorts
Cumulative observed growth risks in development and validation cohort according to low-, moderate-, and high-risk categories of the ELAPSS (Earlier subarachnoid hemorrhage, aneurysm Location, Age, Population, aneurysm Size and Shape) score. Cumulative observed growth risk in validation cohort (solid line) and development cohort (dotted line) across risk categories of the ELAPSS score (low, ≤9; moderate, 10 to 19; high ≥20).
Notes
The authors have no financial conflicts of interest.