Development and Validation of a Clinical Risk Model for Predicting Malignancy in Patients with Thyroid Nodules

³ Cancer Research Center, Institute of Cancer, Avicenna Health Research Institute, Hamadan University of Medical Sciences, Hamadan, Iran

*Corresponding author: Erfan Ayubi, Cancer Research Center, Institute of Cancer, Avicenna Health Research Institute, Hamadan University of Medical Sciences, Hamadan, Iran. Email: aubi65@gmail.com

EXTENDED ABSTRACT

Background

Thyroid cancer is the most common malignancy of the endocrine system [1], and its incidence has increased in many populations. It has been projected to become one of the four most common cancers among adults aged 20-49 years by 2040 [2]. Incidence trends vary by country, sex, and histological type [3], and the burden has also increased in North Africa and the Middle East, including Iran [4, 5]. Although thyroid cancer generally has a favorable prognosis, metastatic disease can markedly reduce five-year survival, and metastatic risk depends on nodule characteristics, histological type or subtype, and cancer stage [6-9]. Fine-needle aspiration cytology (FNAC), interpreted using the Bethesda System for Reporting Thyroid Cytopathology, is the principal method used to evaluate thyroid nodules [10-12]. However, FNAC may be less reliable in some large nodules or follicular-pattern lesions, and definitive assessment of suspicious lesions may require postoperative histopathology [13, 14]. Multivariable models and machine-learning (ML) algorithms combining demographic, biochemical, ultrasonographic, and clinicopathological information have been used to predict malignancy and stratify risk [15-21]. The present study aimed to develop and internally validate a clinical risk model for thyroid nodule malignancy using demographic, clinical, and Bethesda data and to compare its performance with several ML algorithms.

Methods

This analytical cross-sectional study included 650 patients with thyroid nodules who underwent thyroidectomy after referral to the specialized endocrinology clinic in Hamadan, western Iran, between 2014 and 2022. The mean age was 42.36+/-13.45 years, and 86.15% were female. All patients underwent fine-needle aspiration followed by thyroidectomy; cytology was interpreted using the Bethesda system and postoperative histopathology was classified as benign or malignant. Candidate predictors were age, sex, education, first-degree family history of thyroid disease, thyroid functional status, clinical symptoms, nodule focality, nodule size (<=4 cm or >4 cm), cervical adenopathy, and Bethesda category. Descriptive variables were compared using the chi-square test, and significant characteristics were entered into multivariable logistic regression as a generalized linear model (GLM). Internal validity was assessed using 1,000 bootstrap resamples. Discrimination, calibration, and clinical usefulness were evaluated with the area under the receiver operating characteristic curve (AUC), calibration plot and Brier score, and decision curve analysis (DCA), respectively; the final model was displayed as a nomogram. GLM performance was compared with linear discriminant analysis (LDA), random forest (RF), neural network (NN), support vector machine (SVM), and k-nearest neighbors (kNN). Data were split into 80% training and 20% testing sets, with repeated 10-fold cross-validation for tuning. Accuracy, kappa, sensitivity, specificity, precision, negative predictive value, F1 score, AUC, and area under the precision-recall curve (AUPRC) were calculated. Analyses used STATA 14 and R 4.1.3, with P<0.05 considered significant.

Results

Among the 650 operated patients, 280 (43%) had benign pathology and 370 (57%) had malignant pathology. Younger age groups, female sex, first-degree family history of thyroid disease, cervical adenopathy, unifocal nodules, and the FN/SFN and SM Bethesda categories were more frequent among patients with malignant disease (P<0.05). In the multivariable GLM, patients aged 45-60 years had lower odds of malignancy than those aged <=30 years (OR=0.37, 95% CI: 0.18-0.75; P=0.006), and multifocal nodules had lower odds than unifocal nodules (OR=0.36, 95% CI: 0.19-0.67; P=0.001). Cervical adenopathy increased the odds of malignancy (OR=3.89, 95% CI: 1.67-9.06; P=0.002). Relative to benign Bethesda cytology, the odds increased for ND/US (OR=4.08), AUS/FLUS (OR=4.33), FN/SFN (OR=9.50), and SM (OR=97.69), with the corresponding confidence intervals and P values shown in Table 1. The malignant Bethesda category was excluded from multivariable analysis because its observed malignancy risk was 100%.

The bootstrap-validated multivariable logistic model achieved an AUC of 0.89 for discriminating malignant from benign nodules. Calibration was reported as relatively good, with a Brier score of 0.12. DCA indicated that the nomogram incorporating the significant predictors provided greater clinical net benefit than use of the Bethesda system alone. The nomogram illustrates how individual predictors can be combined; for example, the source article reports that a man younger than 30 years with cervical adenopathy and an FN/SFN Bethesda result would receive an overall score of approximately 13, corresponding to an estimated malignancy probability of about 40%. In the held-out testing data, no single algorithm showed a uniform advantage across every performance measure. LDA produced the highest accuracy (0.78) and kappa (0.55), whereas GLM had the highest AUC (0.84) and AUPRC (0.86). GLM sensitivity, specificity, precision, and negative predictive value were 0.78, 0.75, 0.81, and 0.72, respectively, with an F1 score of 0.79. For LDA, the corresponding values were 0.76, 0.80, 0.84, and 0.71, with an F1 score of 0.79. RF, NN, SVM, and kNN showed somewhat lower overall performance on several measures, although differences among the six approaches were generally small. These findings were consistent with the authors' interpretation that conventional GLM and LDA remained competitive with the evaluated ML algorithms when using the available demographic, clinical, and Bethesda predictors.

Table 1. Characteristics of participants with thyroid nodules and results of the multivariable logistic regression model.

ND/US: nondiagnostic/unsatisfactory; B: benign; AUS/FLUS: atypia of undetermined significance/follicular lesions of undetermined significance; FN/SFN: follicular neoplasm/suspicious of follicular neoplasm; SM: suspicious of malignancy.

Conclusion

A clinical risk model combining demographic, clinical, and Bethesda information showed good internal discrimination and calibration for predicting malignancy in patients with thyroid nodules. Bethesda category was the strongest predictor, while age, nodule focality, cervical adenopathy, family history, and thyroid functional status also contributed to risk estimation. GLM and LDA showed the strongest overall performance among the six algorithms, but no method was uniformly superior across all metrics. Interpretation should consider the limited predictor set, absence of TNM, biochemical and molecular data, possible sparse-data bias, and lack of external validation. The model may support treatment planning and management after validation in larger external populations.

Keywords: Machine Learning, Malignancy, Thyroid Nodule, Validation

REFERENCES

1. Seib CD, Sosa JA. Evolving Understanding of the Epidemiology of Thyroid Cancer. Endocrinol Metab Clin North Am. 2019;48(1):23-35. PMID: 30717905 DOI: 10.1016/j.ecl.2018.10.002
2. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2018;68(6):394-424. PMID: 30207593 DOI: 10.3322/caac.21492
3. Miranda-Filho A, Lortet-Tieulent J, Bray F, Cao B, Franceschi S, Vaccarella S, et al. Thyroid cancer incidence trends by histology in 25 countries: a population-based study. Lancet Diabetes Endocrinol. 2021;9(4):225-34. PMID: 33662333 DOI: 10.1016/S2213-8587(21)00027-9
4. Pizzato M, Li M, Vignat J, Laversanne M, Singh D, La Vecchia C, et al. The epidemiological landscape of thyroid cancer worldwide: GLOBOCAN estimates for incidence and mortality rates in 2020. Lancet Diabetes Endocrinol. 2022;10(4):264-72. PMID: 35271818 DOI: 10.1016/S2213-8587(22)00035-3
5. Nejadghaderi SA, Moghaddam SS, Azadnajafabad S, Rezaei N, Rezaei N, Tavangar SM, et al. Burden of thyroid cancer in North Africa and Middle East 1990-2019. Front Oncol. 2022;12:955358. PMID: 36212501 DOI: 10.3389/fonc.2022.955358
6. Wang Y, Guan Q, Xiang J. Nomogram for predicting central lymph node metastasis in papillary thyroid microcarcinoma: A retrospective cohort study of 8668 patients. Int J Surg. 2018;55:98-102. PMID: 29803769 DOI: 10.1016/j.ijsu.2018.05.023
7. Chung SR, Baek JH, Choi YJ, Sung TY, Song DE, Kim TY, et al. The relationship of thyroid nodule size on malignancy risk according to histological type of thyroid cancer. Acta Radiol. 2020;61(5):620-628. PMID: 31554409 DOI: 10.1177/0284185119875642
8. Parameswaran R, Shulin Hu J, Min En N, Tan WB, Yuan NK. Patterns of metastasis in follicular thyroid carcinoma and the difference between early and delayed presentation. Ann R Coll Surg Engl. 2017;99(2):151-154. PMID: 27659362 DOI: 10.1308/rcsann.2016.0300
9. Wu MH, Lee YY, Lu YL, Lin SF. Risk Factors and Prognosis for Metastatic Follicular Thyroid Cancer. Front Endocrinol. 2022;13:791826. PMID: 35299967 DOI: 10.3389/fendo.2022.791826
10. Baloch ZW, LiVolsi VA, Asa SL, Rosai J, Merino MJ, Randolph G, et al. Diagnostic terminology and morphologic criteria for cytologic diagnosis of thyroid lesions: a synopsis of the National Cancer Institute Thyroid Fine-Needle Aspiration State of the Science Conference. Diagn Cytopathol. 2008;36(6):425-37. PMID: 18478609 DOI: 10.1002/dc.20830
11. Kant R, Davis A, Verma V. Thyroid Nodules: Advances in Evaluation and Management. Am Fam Physician. 2020;102(5):298-304. PMID: 32866364
12. Linhares SM, Handelsman R, Picado O, Farrá JC, Lew JI. Fine needle aspiration and the Bethesda system: Correlation with histopathology in 1,228 surgical patients. Surgery. 2021;170(5):1364-8. PMID: 34134896 DOI: 10.1016/j.surg.2021.05.016
13. Haugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid. 2016;26(1):1-133. PMID: 26462967 DOI: 10.1089/thy.2015.0020
14. Aliyev A, Aliyeva I, Giammarile F, Talibova N, Aliyeva G, Novruzov F. Diagnostic accuracy of fine needle aspiration biopsy versus postoperative histopathology for diagnosing thyroid malignancy. Endocrinol Diabetes Metab. 2022;5(6):e373. PMID: 36149057 DOI: 10.1002/edm2.373
15. Gu J, Xie R, Zhao Y, Zhao Z, Xu D, Ding M, et al. A machine learning-based approach to predicting the malignant and metastasis of thyroid cancer. Front Oncol. 2022;12:938292. PMID: 36601485 DOI: 10.3389/fonc.2022.938292
16. Liu WC, Li ZQ, Luo ZW, Liao WJ, Liu ZL, Liu JM. Machine learning for the prediction of bone metastasis in patients with newly diagnosed thyroid cancer. Cancer Med. 2021;10(8):2802-11. PMID: 33709570 DOI: 10.1002/cam4.3776
17. Lu J, Liao J, Chen Y, Li J, Huang X, Zhang H, et al. Risk factor analysis and prediction model for papillary thyroid carcinoma with lymph node metastasis. Front Endocrinol. 2023;14:1287593. PMID: 38027220 DOI: 10.3389/fendo.2023.1287593
18. Zhang TT, Zeng J, Yang Y, Wang JJ, Kang YJ, Zhang DH, et al. A visualized dynamic prediction model for survival of patients with geriatric thyroid cancer: A population-based study. Front Endocrinol. 2022;13:1038041. PMID: 36568078 DOI: 10.3389/fendo.2022.1038041
19. Slabaugh G, Beltran L, Rizvi H, Deloukas P, Marouli E. Applications of machine and deep learning to thyroid cytology and histopathology: a review. Front Oncol. 2023;13:958310. PMID: 38023130 DOI: 10.3389/fonc.2023.958310
20. Liu W, Wang S, Ye Z, Xu P, Xia X, Guo M. Prediction of lung metastases in thyroid cancer using machine learning based on SEER database. Cancer Med. 2022;11(12):2503-15. PMID: 35191613 DOI: 10.1002/cam4.4617
21. Kourou K, Exarchos KP, Papaloukas C, Sakaloglou P, Exarchos T, Fotiadis DI. Applied machine learning in cancer research: A systematic review for patient diagnosis, classification and prognosis. Comput Struct Biotechnol J. 2021;19:5546-55. PMID: 34712399 DOI: 10.1016/j.csbj.2021.10.006
22. D'Andréa G, Gal J, Mandine L, Dassonville O, Vandersteen C, Guevara N, et al. Application of machine learning methods to guide patient management by predicting the risk of malignancy of Bethesda III-V thyroid nodules. Eur J Endocrinol. 2023;188(3):lvad017. PMID: 36799885 DOI: 10.1093/ejendo/lvad017
23. Liu X, Medici M, Kwong N, Angell TE, Marqusee E, Kim MI, et al. Bethesda Categorization of Thyroid Nodule Cytology and Prediction of Thyroid Cancer Type and Prognosis. Thyroid. 2016;26(2):256-61. PMID: 26563459 DOI: 10.1089/thy.2015.0376
24. Cibas ES, Ali SZ. The 2017 Bethesda System for Reporting Thyroid Cytopathology. Thyroid. 2017;27(11):1341-1346. PMID: 29091573 DOI: 10.1089/thy.2017.0500
25. Gweon HM, Son EJ, Youk JH, Kim JA. Thyroid nodules with Bethesda system III cytology: can ultrasonography guide the next step?. Ann Surg Oncol. 2013;20(9):3083-8. PMID: 23700214 DOI: 10.1245/s10434-013-2990-x
26. Mileva M, Stoilovska B, Jovanovska A, Ugrinska A, Petrushevska G, Kostadinova-Kunovska S, et al. Thyroid cancer detection rate and associated risk factors in patients with thyroid nodules classified as Bethesda category III. Radiol Oncol. 2018;52(4):370-6. PMID: 30265655 DOI: 10.2478/raon-2018-0039
27. Xi NM, Wang L, Yang C. Improving the diagnosis of thyroid cancer by machine learning and clinical data. Sci Rep. 2022;12(1):11143. PMID: 35778428 DOI: 10.1038/s41598-022-15342-z
28. Yaghoobi Notash A, Yaghoobi Notash A, Omidi Z, Haghighat S. Prediction of lymphedema occurrence in patients with breast cancer using the optimized combination of ensemble learning algorithm and feature selection. BMC Med Inform Decis Mak. 2022;22(1):195. PMID: 35879760 DOI: 10.1186/s12911-022-01937-z
29. Mourad M, Moubayed S, Dezube A, Mourad Y, Park K, Torreblanca-Zanca A, et al. Machine Learning and Feature Selection Applied to SEER Data to Reliably Assess Thyroid Cancer Prognosis. Sci Rep. 2020;10(1):5176. PMID: 32198433 DOI: 10.1038/s41598-020-62023-w
30. Borzooei S, Briganti G, Golparian M, Lechien JR, Tarokhian A. Machine learning for risk stratification of thyroid cancer patients: a 15-year cohort study. Eur Arch Otorhinolaryngol. 2024;281(4):2095-2104. PMID: 37902840 DOI: 10.1007/s00405-023-08299-w