Machine learning models for predicting hepatocellular carcinoma development in patients with chronic viral hepatitis B infection

We collected two datasets, one for derivation of the machine learning models and the other for validation of the model performance, as follows:

Derivation dataset was used for developing the model and internally evaluating the model performance. We retrieved patient demographics, International Classification of Diseases (ICD)-10 diagnosis codes, and laboratory data and radiology reports through the Hospital Information System (HIS) and Radiology Information System (RIS) of the King Chulalongkorn Memorial Hospital, Bangkok, Thailand between January 2008 and December 2020.

Patients with CHB were identified using the ICD codes B18.0, B18.1, B18.8, or B18.9 or a positive test for hepatitis B surface antigen (HBsAg). HCC diagnosis was defined by ICD codes C22.0, C22.7, or C22.9. We also manually reviewed patients' electronic medical records to verify the diagnosis of HCC as well as the dates of HBV infection and HCC diagnosis. Patients diagnosed with HCC within 6 months after the date of HBV diagnosis were excluded because it was possible that HCC existed prior to the date of the patient being diagnosed with HBV infection. The presence of cirrhosis was retrieved from upper abdominal ultrasound reports or ICD codes K74.0, K74.6 (Table 1). Other retrieved laboratory data were hepatitis C virus (HCV) and HIV infection, hepatitis B and C viral loads, hepatitis B e antigen (HBeAg) and antibody (anti-HBe), AFP level, liver and renal function tests, blood glucose, complete blood counts, and coagulation studies. Data from all follow-up visits starting from the date of HBV infection diagnosis to the date of HCC diagnosis or the end of study period (December 2020) were used for constructing the predictive models. The primary outcome was development of HCC within the next 6 months from each visit.

Another dataset obtained from the Faculty of Medicine Siriraj Hospital, Mahidol University, was used for externally validating the performance of the predictive models. This dataset was obtained by a case–control method. Patients with CHB who developed HCC served as cases. For controls, we identified CHB patients who had not developed HCC, then matched with cases in a 1:1 ratio by the time period of visit at the Hepatology clinic. The data were manually reviewed and abstracted from medical records by 2 experienced hepatologists. The same inclusion and exclusion criteria of the derivation dataset were applied to the validation dataset.

We trained Extreme Gradient Boosted (XGBoost) models using clinical, laboratory, and radiologic parameters in each follow-up as inputs to predict HCC development within 6 months after each index follow-up.

A total of 28 features were included as “set of all features”, consisting of the following features: sex, age, quantitative HBV viral load, Hepatitis B e Antigen (HBeAg), anti-HBe, anti-HCV, qualitative HCV viral load, anti-HIV, AFP, total bilirubin (TB), direct bilirubin (DB), aspartate transaminase (AST), alanine aminotransferase (ALT), alkaline phosphatase (ALP), albumin, globulin, hemoglobin (Hb), white blood cell count (WBC), absolute neutrophil count, absolute lymphocyte count, platelet count, international normalized ratio (INR), blood urea nitrogen (BUN), creatinine (Cr), fasting plasma glucose (FPG), HbA1C, presence of cirrhosis, and liver steatosis status.

We acknowledged that selecting only some important features as model inputs may enable more practical use of the models by clinicians, as the set of all features may not always be completely available during the follow-up visit, and focusing on only important features can save time and effort of clinicians in inputting data into the models. Thus, we further identified features that had a high correlation with the outcome of interest (i.e., HCC development) determined by Pearson correlation coefficient in the derivation dataset. Among features that exhibited multicollinearity, we kept only a single feature that had the highest correlation with the outcome. As a result, 3 of the most highly correlated features, namely TB (Pearson correlation coefficient 0.095), ALP (0.120), and INR (0.130), along with 4 clinically important A Priori features (age [0.047], sex [0.025], liver cirrhosis [0.075], and liver steatosis [0.040]), were included as a “selected feature set” (7 features in total) (Table 1).

Considering that serum AFP is an important feature associated with HCC development [13], the AFP test is not always available, especially in hospitals located in remote areas; we therefore developed two model variants: a model with AFP as an input feature and a model without AFP as an input feature, in order to enable the model to be applied in a situation when the AFP test is not performed. To summarize, the following notations are used in the remaining of this article (Figure 1).

Figure 1.

Flow diagram of the study. AF^A, represents the model using a set of all features, with AFP, as inputs. AF^N, represents the model using a set of all features, without AFP, as inputs. SF^A, represents the model using a set of selected features, with AFP, as inputs. SF^N, represents the model using a set of selected features, without AFP, as inputs; HCC: Hepatocellular carcinoma.

All analyses were performed using Python version 3.7.3 (Python Software Foundation, Delaware, Beaverton, Oregon, US). XGBoost library implementation of Gradient Boosted Trees was used to construct the model. Machine learning deeply learned all training data and identified relationships between the data and the diagnosis of HCC, which might not be detected by traditional statistics, then developed conditions and rules for HCC predictive models. Using the derivation dataset, we performed 10-fold cross-validation to estimate the performance of the AF ^A, AF^N, SF ^A, and SF^N models. In each fold, 90% of the dataset were used for training and 10% were used for performance evaluation. Data split in each fold was independent at patient level. Sensitivities, specificities, and area under receiver operating characteristic curve (AUROC) along with 95% confidence intervals (CIs) of the models for predicting HCC development were estimated. For each ROC, a cutoff point that yielded maximum sensitivity was used to obtain the sensitivity and specificity of the model. AUROC on the derivation dataset evaluated by 10-fold cross-validation was reported as mean ± standard deviation (SD) of AUROCs across all 10 folds. Next, the performance of the AF^A and SF^N model was evaluated on the external validation dataset. AUROCs on the external validation dataset was reported along with 95% CIs. The two-tailed statistical significance level (α) was set at 0.05.

Because the data were retrospectively abstracted from medical records, some input features were inevitably missing. To evaluate how the number of missing input features affected the model performance, we further evaluated the model performance using a set of selected features on subsets of follow-ups having certain allowed number(s) of missing input features. For example, in the subset of follow-ups having 2 allowed missing input features, follow-ups having 0, 1, and 2 missing input features were included. Of note, all follow-ups had at least 4 features, that is, sex, age, cirrhosis, and liver steatosis status, as these features were always available in the medical records. Therefore, the maximum numbers of allowed missing features were three for SF^N (INR, ALP, TB) and four for SF^A (AFP, INR, ALP, TB) (Figure 1).

The derivation dataset comprised 2,382 patients with 15,187 follow-up visits. There were 1,331 (55.1%) males with a mean age of 51.0 (SD: 14.7) years. Cirrhosis and HCV co-infection were present in 609 (25.6%) and 65 (2.7%) patients, respectively. There were 11,837 follow-ups with AFP values and 3,350 follow-ups without AFP values. During a median follow-up time of 18 months (interquartile range [IQR]: 52), 117 (4.9%) patients developed HCC. Of the 117 HCC patients of whom 91 (77.8%) had information on HCC stage available, 26 (28.6%) patients were diagnosed HCC at Barcelona Clinic Liver Cancer (BCLC) stage 0, and 44 (48.4%), 10 (11.0%) and 11 (12.1%) were diagnosed at BCLC stage A, B, and C, respectively (Table 2).

The dataset comprised 162 CHB-infected patients with 564 follow-ups. In this dataset, 104 (64.2%) patients were male and the mean age was 58.0 (SD: 13.6) years. Cirrhosis was present in 57 (35.2%) patients. None had HCV coinfection. There were 355 and 209 follow-ups with and without AFP values, respectively. During a median follow-up time of 11.2 (IQR: 8.7) months, 57 (35.2%) patients developed HCC; and 13 (22.8%), 32 (56.1%), 10 (17.5%), and 2 (3.5%) patients were diagnosed at BCLC stages 0, A, B, and C, respectively (Table 2).

Performance of the models on the derivation dataset are demonstrated in Table 3. In the models with a set of all features, AF^A achieved a better sensitivity and specificity than AF^N, with sensitivities of 0.634 (95% CI: 0.559–0.708) and 0.553 (95% CI: 0.476–0.630); and specificities of 0.836 (95% CI: 0.830–0.842) and 0.786 (95% CI: 0.779–0.792), for AF^A and AF^N, respectively. A similar trend was detected for the models using a set of selected features as inputs, however, with less magnitude of difference in performance between the models with and without AFP, that is., SF^A and SF^N had a sensitivity of 0.683 (95% CI: 0.611–0.755) and 0.658 (95% CI: 0.585–0.732); and a specificity of 0.756 (95% CI: 0.749–0.763) and 0.744 (95% CI: 0.737–0.751), respectively. Overall, the models with all features achieved lower sensitivities but higher specificities than the models with selected features. Among the four models, AF^A achieved the highest mean ± SD AUROC of 0.786 ± 0.113, followed by AF^N (0.731 ± 0.089), SF^A (0.727 ± 0.097) and SF^N (0.707 ± 0.088) (Table 3).

In contrast to the derivation dataset, AF^A achieved a lower sensitivity than SF^N, that is, 0.634 (95% CI: 0.522–0.746) and 0.690 (95% CI: 0.583–0.798), respectively. Specificities of the two models were relatively similar, that is, 0.657 (95% CI: 0.615–0.699) and 0.651 (95% CI: 0.609–0.693), for AF^A and SF^N, respectively. The AUROC of AF^A and SF^N were 0.646 (95% CI: 0.585–0.706) and 0.663 (95% CI: 0.605–0.721), respectively (Table 3).

We evaluated the AF^A and SF^N models on subsets of follow-ups having varied numbers of missing input features (Table 4). Without missing features, AF^A and SF^N achieved sensitivities of 0.833 (95% CI: 0.535–1.000) and 0.889 (95% CI: 0.684–1.000). We found that the sensitivities of both models progressively decreased by the numbers of missing features, that is, AF^A achieved sensitivities of 0.720 (95% CI: 0.544–0.896), 0.698 (95% CI: 0.560–0.835), 0.647 (95% CI: 0.533–0.761), and 0.634 (95% CI: 0.522–0.746) when the maximum numbers of missing features were 1, 2, 3, and 4, respectively. Likewise, SF^N achieved sensitivities of 0.727 (95% CI: 0.575–0.879), 0.708 (95% CI: 0.580–0.837), and 0.690 (95% CI: 0.583–0.798) when the maximum numbers of missing features were 1, 2, and 3, respectively. These observations indicated that the models achieved better sensitivities with lower numbers of missing input features.