Evaluating machine learning algorithms: The role of sample size and quality
By: Przemysław Kupidura and Michał Szkibiel
References
- Belgiu, M. and Drăguţ, L. (2016). Random forest in remote sensing: A review of applications and future directions. ISPRS Journal of Photogrammetry and Remote Sensing, 114:24–31, doi:10.1016/j.isprsjprs.2016.01.011.
- Bigdeli, A., Maghsoudi, A., and Ghezelbash, R. (2023). A comparative study of the XGBoost ensemble learning and multilayer perceptron in mineral prospectivity modeling: a case study of the Torud-Chahshirin belt, NE Iran. Earth Science Informatics, 17(1):483–499, doi:10.1007/s12145-023-01184-4.
- Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, COLT92, page 144–152. ACM, doi:10.1145/130385.130401.
- Budach, L., Feuerpfeil, M., Ihde, N., Nathansen, A., Noack, N., Patzlaff, H., Naumann, F., and Harmouch, H. (2022). The effects of data quality on machine learning performance. arXiv preprint arXiv:2207.14529.
- Burkholder, A., Warner, T. A., Culp, M., and Landenberger, R. (2011). Seasonal trends in separability of leaf reflectance spectra for Ailanthus altissima and four other tree species. Photogrammetric Engineering and Remote Sensing, 77(8):793–804, doi:10.14358/PERS.77.8.793.
- Chen, T. and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794. ACM, doi:10.1145/2939672.2939785.
- Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37–46, doi:10.1177/001316446002000104.
- Cracknell, M. J. and Reading, A. M. (2014). Geological mapping using remote sensing data: A comparison of five machine learning algorithms, their response to variations in the spatial distribution of training data and the use of explicit spatial information. Computers and Geosciences, 63:22–33, doi:10.1016/j.cageo.2013.10.008.
- Duro, D. C., Franklin, S. E., and Dubé, M. G. (2012). A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery. Remote sensing of environment, 118:259–272, doi:10.1016/j.rse.2011.11.020.
- Figueroa, R. L., Zeng-Treitler, Q., Kandula, S., and Ngo, L. H. (2012). Predicting sample size required for classification performance. BMC Medical Informatics and Decision Making, 12(1), doi:10.1186/1472-6947-12-8.
- Fu, Y., Shen, R., Song, C., Dong, J., Han, W., Ye, T., and Yuan, W. (2023). Exploring the effects of training samples on the accuracy of crop mapping with machine learning algorithm. Science of Remote Sensing, 7:100081, doi:10.1016/j.srs.2023.100081.
- Ghayour, L., Neshat, A., Paryani, S., Shahabi, H., Shirzadi, A., Chen, W., Al-Ansari, N., Geertsema, M., Pourmehdi Amiri, M., Gholamnia, M., Dou, J., and Ahmad, A. (2021). Performance Evaluation of Sentinel-2 and Landsat 8 OLI Data for Land Cover/Use Classification Using a Comparison between Machine Learning Algorithms. Remote Sensing, 13(7):1349, doi:10.3390/rs13071349.
- Halevy, A., Norvig, P., and Pereira, F. (2009). The Unreasonable Effectiveness of Data. IEEE Intelligent Systems, 24(2):8–12, doi:10.1109/mis.2009.36.
- Hand, D. J., Christen, P., and Kirielle, N. (2021). F*: an interpretable transformation of the F-measure. Machine Learning, 110(3):451–456, doi:10.1007/s10994-021-05964-1.
- Koppaka, R. and Moh, T.-S. (2020). Machine Learning in Indian Crop Classification of Temporal Multi-Spectral Satellite Image. In 2020 14th International Conference on Ubiquitous Information Management and Communication (IMCOM), page 1–8. IEEE, doi:10.1109/imcom48794.2020.9001718.
- Kupidura, P., Kępa, A., and Krawczyk, P. (2024). Comparative analysis of the performance of selected machine learning algorithms depending on the size of the training sample. Reports on Geodesy and Geoinformatics, 118(1), doi:10.2478/rgg-2024-0015.
- Kupidura, P. and Niemyski, S. (2024). Analysis of the effectiveness of selected machine learning algorithms in the classification of satellite image content depending on the size of the training sample. Teledetekcja Środowiska, 64.
- Labatut, V. and Cherifi, H. (2012). Accuracy measures for the comparison of classifiers. arXiv preprint arXiv:1207.3790.
- Li, X., Chen, W., Cheng, X., and Wang, L. (2016). A Comparison of Machine Learning Algorithms for Mapping of Complex SurfaceMined and Agricultural Landscapes Using ZiYuan-3 Stereo Satellite Imagery. Remote Sensing, 8(6):514, doi:10.3390/rs8060514.
- Liu, J., Zuo, Y., Wang, N., Yuan, F., Zhu, X., Zhang, L., Zhang, J., Sun, Y., Guo, Z., Guo, Y., Song, X., Song, C., and Xu, X. (2021). Comparative Analysis of Two Machine Learning Algorithms in Predicting Site-Level Net Ecosystem Exchange in Major Biomes. Remote Sensing, 13(12):2242, doi:10.3390/rs13122242.
- Mansor, N. S., Awang, H., Malami, S. T. S., Zolkafli, A., Taiye, M. A., and Maulana, H. (2024). Support Vector Machine for Satellite Images Classification Using Radial Basis Function Kernel Method, page 301–312. Springer Nature Singapore, doi:10.1007/978-981-99-9589-9_23.
- Maxwell, A., Strager, M., Warner, T., Zégre, N., and Yuill, C. (2014a). Comparison of NAIP orthophotography and RapidEye satellite imagery for mapping of mining and mine reclamation. GIScience and Remote Sensing, 51(3):301–320, doi:10.1080/15481603.2014.912874.
- Maxwell, A., Warner, T., Strager, M., Conley, J., and Sharp, A. (2015). Assessing machine-learning algorithms and imageand lidar-derived variables for GEOBIA classification of mining and mine reclamation. International Journal of Remote Sensing, 36(4):954–978, doi:10.1080/01431161.2014.1001086.
- Maxwell, A. E. and Warner, T. A. (2015). Differentiating minereclaimed grasslands from spectrally similar land cover using terrain variables and object-based machine learning classification. International Journal of Remote Sensing, 36(17):4384–4410, doi:10.1080/01431161.2015.1083632.
- Maxwell, A. E., Warner, T. A., Strager, M. P., and Pal, M. (2014b). Combining RapidEye Satellite Imagery and Lidar for Mapping of Mining and Mine Reclamation. Photogrammetric Engineering and Remote Sensing, 80(2):179–189, doi:10.14358/pers.80.2.179-189.
- Mousavinezhad, M., Feizi, A., and Aalipour, M. (2023). Performance Evaluation of Machine Learning Algorithms in Change Detection and Change Prediction of a Watershed’s Land Use and Land Cover. International Journal of Environmental Research, 17(2), doi:10.1007/s41742-023-00518-w.
- Powers, D. (2007). Evaluation: From precision, recall and F-factor to ROC, informedness, markedness and correlation. Technical report, Techn. Rep. SIE–07–001, Flinders University, Adelaide, Australia.
- Ramezan, C. A., Warner, T. A., Maxwell, A. E., and Price, B. S. (2021). Effects of Training Set Size on Supervised Machine-Learning Land-Cover Classification of Large-Area High-Resolution Remotely Sensed Data. Remote Sensing, 13(3):368, doi:10.3390/rs13030368.
- Raudys, S. and Jain, A. (1991). Small sample size effects in statistical pattern recognition: recommendations for practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3):252–264, doi:10.1109/34.75512.
- Richards, J. A. and Jia, X. (2006). Remote Sensing Digital Image Analysis: An Introduction. Springer Berlin Heidelberg, doi:10.1007/3-540-29711-1.
- Seydi, S. T., Kanani-Sadat, Y., Hasanlou, M., Sahraei, R., Chanussot, J., and Amani, M. (2022). Comparison of Machine Learning Algorithms for Flood Susceptibility Mapping. Remote Sensing, 15(1):192, doi:10.3390/rs15010192.
- Shang, M., Wang, S.-X., Zhou, Y., and Du, C. (2018). Effects of Training Samples and Classifiers on Classification of Landsat- 8 Imagery. Journal of the Indian Society of Remote Sensing, 46(9):1333–1340, doi:10.1007/s12524-018-0777-z.
- Shih, H.-c., Stow, D. A., and Tsai, Y. H. (2018). Guidance on and comparison of machine learning classifiers for Landsat-based land cover and land use mapping. International Journal of Remote Sensing, 40(4):1248–1274, doi:10.1080/01431161.2018.1524179.
- Sim, J. and Wright, C. C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Physical Therapy, 85(3):257–268, doi:10.1093/ptj/85.3.257.
- Sobieraj, J., Fernández, M., and Metelski, D. (2022). A Comparison of Different Machine Learning Algorithms in the Classification of Impervious Surfaces: Case Study of the Housing Estate Fort Bema in Warsaw (Poland). Buildings, 12(12):2115, doi:10.3390/buildings12122115.
- Volke, M. I. and Abarca-Del-Rio, R. (2020). Comparison of machine learning classification algorithms for land cover change in a coastal area affected by the 2010 Earthquake and Tsunami in Chile. Natural Hazards and Earth System Sciences Discussions, 2020:1–14, doi:10.5194/nhess-2020-41.
- Zhao, Z., Islam, F., Waseem, L. A., Tariq, A., Nawaz, M., Islam, I. U., Bibi, T., Rehman, N. U., Ahmad, W., Aslam, R. W., Raza, D., and Hatamleh, W. A. (2024). Comparison of Three Machine Learning Algorithms Using Google Earth Engine for Land Use Land Cover Classification. Rangeland Ecology and Management, 92:129–137, doi:10.1016/j.rama.2023.10.007.
- Zheng, W. and Jin, M. (2020). The Effects of Class Imbalance and Training Data Size on Classifier Learning: An Empirical Study. SN Computer Science, 1(2), doi:10.1007/s42979-020-0074-0.
Language: English
Page range: 1 - 13
Published on: Jul 28, 2026
Published by: Warsaw University of Technology
In partnership with: Paradigm Publishing Services
Publication frequency: 2 issues per year
Keywords:
Related subjects:
© 2026 Przemysław Kupidura, Michał Szkibiel, published by Warsaw University of Technology
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.