Comparison of outlier detection approaches in a Smart Cities sensor data context

Zafeirelli, Sofia; Kavroudakis, Dimitris

doi:10.2478/ijssis-2024-0004

Abstract

This study examines outlier detection in time-series sensor data from PurpleAir low-cost sensors in Athens, Greece. Focusing on key environmental parameters such as temperature, humidity, and particulate matter (PM) levels, the study utilizes the Interquartile Range (IQR) and Generalized Extreme Studentized Deviate (GESD) methods on hourly and daily basis. GESD detected more outliers than IQR, most of them in PM, while temperature and humidity data had fewer outliers; applying filters before outlier detection and adjusting alpha values based on time scales were crucial, and outliers significantly affected spatial interpolation, emphasizing the need for spatial statistics in smart city air quality management.

References

Aix, M. L., Schmitz, S., & Bicout, D. J. (2023). Calibration methodology of low-cost sensors for high-quality monitoring of fine particulate matter. Science of the Total Environment, 889(February). https://doi.org/10.1016/j.scitotenv.2023.164063
Search in Google Scholar Back to article
Al Samara, M., Bennis, I., Abouaissa, A., & Lorenz, P. (2022). A Survey of Outlier Detection Techniques in IoT: Review and Classification. Journal of Sensor and Actuator Networks, 11(1). https://doi.org/10.3390/jsan11010004
Search in Google Scholar Back to article
Alvear-Puertas, V. E., Burbano-Prado, Y. A., Rosero-Montalvo, P. D., Tözün, P., Marcillo, F., & Hernandez, W. (2022). Smart and Portable Air-Quality Monitoring IoT Low-Cost Devices in Ibarra City, Ecuador. Sensors, 22(18), 1–17. https://doi.org/10.3390/s22187015
Search in Google Scholar Back to article
Amini, M. H., Arasteh, H., & Siano, P. (2019). Sustainable Smart Cities Through the Lens of Complex Interdependent Infrastructures: Panorama and State-of-the-art (pp. 45–68). https://doi.org/10.1007/978-3-319-98923-5_3
Search in Google Scholar Back to article
Aslan, M. E., & Onut, S. (2022). Detection of Outliers and Extreme Events of Ground Level Particulate Matter Using DBSCAN Algorithm with Local Parameters. Water, Air, and Soil Pollution, 233(6). https://doi.org/10.1007/s11270-022-05679-6
Search in Google Scholar Back to article
Ayadi, A., Ghorbel, O., Obeid, A. M., & Abid, M. (2017). Outlier detection approaches for wireless sensor networks: A survey. Computer Networks, 129(2), 319–333. https://doi.org/10.1016/j.comnet.2017.10.007
Search in Google Scholar Back to article
Barkjohn, K. K., Gantt, B., & Clements, A. L. (2021). Development and application of a United States-wide correction for PM2.5 data collected with the PurpleAir sensor. Atmospheric Measurement Techniques, 14(6), 4617–4637. https://doi.org/10.5194/amt-14-4617-2021
Search in Google Scholar Back to article
Becnel, T., Sayahi, T., Kelly, K., & Gaillardon, P. E. (2019). A recursive approach to partially blind calibration of a pollution sensor network. 2019 IEEE International Conference on Embedded Software and Systems, ICESS 2019. https://doi.org/10.1109/ICESS.2019.8782523
Search in Google Scholar Back to article
Bi, J., Wildani, A., Chang, H. H., & Liu, Y. (2020). Incorporating Low-Cost Sensor Measurements into High-Resolution PM2.5 Modeling at a Large Spatial Scale. Environmental Science and Technology, 54(4), 2152–2162. https://doi.org/10.1021/acs.est.9b06046
Search in Google Scholar Back to article
Blázquez-García, A., Conde, A., Mori, U. & Lozano, J. A. (2021). A Review on Outlier/Anomaly Detection in Time Series Data. ACM Computing Surveys, 54(3). https://doi.org/10.1145/3444690
Search in Google Scholar Back to article
Braei, M., & Wagner, S. (2020). Anomaly Detection in Univariate Time-series: A Survey on the State-of-the-Art. ArXiv, abs/2004.00433
Search in Google Scholar Back to article
Chen, L. J., Ho, Y. H., Hsieh, H. H., Huang, S. T., Lee, H. C., & Mahajan, S. (2018). ADF: An Anomaly Detection Framework for Large-Scale PM2.5 Sensing Systems. IEEE Internet of Things Journal, 5(2), 559–570. https://doi.org/10.1109/JIOT.2017.2766085
Search in Google Scholar Back to article
Cieplak, T., Rymarczyk, T., & Tomaszewski, R. (2019). A concept of the air quality monitoring system in the city of Lublin with machine learning methods to detect data outliers. MATEC Web of Conferences, 252, 03009. https://doi.org/10.1051/matecconf/201925203009
Search in Google Scholar Back to article
Cleveland, R. B., Cleveland, W. S., McRae, J. E., & Terpenning, I. (1990). STL: A Seasonal-Trend decomposition Procedure Based on Loess. Journal of Official Statistics, 6(1), 3–73.
Search in Google Scholar Back to article
Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368), 829–836. https://doi.org/10.1080/01621459.1979.10481038
Search in Google Scholar Back to article
Csaji, B. C., Kemeny, Z., Pedone, G., Kuti, A., & Vancza, J. (2017). Wireless Multi-Sensor Networks for Smart Cities: A Prototype System with Statistical Data Analysis. IEEE Sensors Journal, 17(23), 7667–7676. https://doi.org/10.1109/JSEN.2017.2736785
Search in Google Scholar Back to article
Dancho, M. & Vaughan, D. (2022). Anomalize: Tidy Anomaly Detection. R package version 0.2.2. https://cran.r-project.org/package=anomalize
Search in Google Scholar Back to article
Ding, Z., Mei, G., Cuomo, S., Li, Y., & Xu, N. (2020). Comparison of Estimating Missing Values in IoT Time Series Data Using Different Interpolation Algorithms. International Journal of Parallel Programming, 48(3), 534–548. https://doi.org/10.1007/s10766-018-0595-5
Search in Google Scholar Back to article
Elbaz, K., Hoteit, I., Shaban, W. M., & Shen, S. L. (2023). Spatiotemporal air quality forecasting and health risk assessment over smart city of NEOM. Chemosphere, 313(December 2022), 137636. https://doi.org/10.1016/j.chemosphere.2022.137636
Search in Google Scholar Back to article
Elmenshawy, D., & Helmy, W. (2018). Detection Techniques of Data Anomalies in Iot: A Literature Survey. International Journal of Civil Engineering and Technology (IJCIET), 9(12), 794–807. http://www.iaeme.com/IJCIET/issues.asp?JType=IJCIET&VType=9&IType=121.
Search in Google Scholar Back to article
Esnaola-Gonzalez, I., Bermúdez, J., Fernández, I., Fernández, S., & Arnaiz, A. (2017). Towards a semantic outlier detection framework in wireless sensor networks. ACM International Conference Proceeding Series, 2017-September, 152–159. https://doi.org/10.1145/3132218.3132226
Search in Google Scholar Back to article
Fan, Z., Feng, H., Jiang, J., Zhao, C., Jiang, N., Wang, W., & Zeng, F. (2020). Monte Carlo optimization for sliding window size in Dixon quality control of environmental monitoring time series data. Applied Sciences (Switzerland), 10(5). https://doi.org/10.3390/app10051876
Search in Google Scholar Back to article
Feenstra, B., Collier-Oxandale, A., Papapostolou, V., Cocker, D., & Polidori, A. (2020). The AirSensor open-source R-package and DataViewer web application for interpreting community data collected by low-cost sensor networks. Environmental Modelling and Software, 134(August), 104832. https://doi.org/10.1016/j.envsoft.2020.104832
Search in Google Scholar Back to article
Fenger, J. (1999). Urban air quality. Atmospheric Environment, 33(29), 4877–4900. https://doi.org/10.1016/S1352-2310(99)00290-3
Search in Google Scholar Back to article
Kelly, K. E., Whitaker, J., Petty, A., Widmer, C., Dybwad, A., Sleeth, D., Martin, R., & Butterfield, A. (2017). Ambient and laboratory evaluation of a low-cost particulate matter sensor. Environmental Pollution, 221, 491–500. https://doi.org/10.1016/j.envpol.2016.12.039
Search in Google Scholar Back to article
Krishnamurthi, R., Kumar, A., Gopinathan, D., Nayyar, A., & Qureshi, B. (2020). An overview of iot sensor data processing, fusion, and analysis techniques. Sensors (Switzerland), 20(21), 1–23. https://doi.org/10.3390/s20216076
Search in Google Scholar Back to article
Liang, C. J., & Yu, P. R. (2021). Assessment and improvement of two low-cost particulate matter sensor systems by using spatial interpolation data from air quality monitoring stations. Atmosphere, 12(3). https://doi.org/10.3390/atmos12030300
Search in Google Scholar Back to article
Liu, H., Shah, S., & Jiang, W. (2004). On-line outlier detection and data cleaning. Computers and Chemical Engineering, 28(9), 1635–1647. https://doi.org/10.1016/j.compchemeng.2004.01.009
Search in Google Scholar Back to article
Liu, W., Jiang, H., Che, D., Chen, L., & Jiang, Q. (2020). A real-time temperature anomaly detection method for iot data. IoTBDS 2020 - Proceedings of the 5th International Conference on Internet of Things, Big Data and Security, IoTBDS, 112–118. https://doi.org/10.5220/0009410001120118
Search in Google Scholar Back to article
Loo, B. P. Y., & Tang, W. S. M. (2019). “Mapping” Smart Cities. Journal of Urban Technology, 26(2), 129–146. https://doi.org/10.1080/10630732.2019.1576467
Search in Google Scholar Back to article
Lovén, L., Peltonen, E., Pandya, A., Leppanen, T., Gilman, E., Pirttikangas, S., & Riekki, J. (2019). Towards EDISON: an edge-native approach to distributed interpolation of environmental data. Proceedings -International Conference on Computer Communications and Networks, ICCCN, 2019-July(May). https://doi.org/10.1109/ICCCN.2019.8847121
Search in Google Scholar Back to article
Ma, L., Gu, X., & Wang, B. (2017). Correction of outliers in temperature time series based on sliding window prediction in meteorological sensor network. Information (Switzerland), 8(2). https://doi.org/10.3390/info8020060
Search in Google Scholar Back to article
Merello, P., García-Diego, F.-J., & Zarzo, M. (2014). Diagnosis of abnormal patterns in multivariate microclimate monitoring: A case study of an open-air archaeological site in Pompeii (Italy). Science of The Total Environment, 488–489(1), 14–25. https://doi.org/10.1016/j.scitotenv.2014.04.068
Search in Google Scholar Back to article
Nilson, B., Jackson, P. L., Schiller, C. L., & Parsons, M. T. (2022). Development and evaluation of correction models for a low-cost fine particulate matter monitor. Atmospheric Measurement Techniques, 15(11), 3315–3328. https://doi.org/10.5194/amt-15-3315-2022
Search in Google Scholar Back to article
Ogasawara, E., Martinez, L. C., De Oliveira, D., Zimbrão, G., Pappa, G. L., & Mattoso, M. (2010). Adaptive Normalization: A novel data normalization approach for non-stationary time series. In Proceedings of the International Joint Conference on Neural Networks. https://doi.org/10.1109/IJCNN.2010.5596746
Search in Google Scholar Back to article
Pastorio, A. F., Spanhol, F. A., Martins, L. D., & De Camargo, E. T. (2022). A Machine Learning-Based Approach to Calibrate Low-Cost Particulate Matter Sensors. Brazilian Symposium on Computing System Engineering, SBESC, 2022-November. https://doi.org/10.1109/SBESC56799.2022.9964983
Search in Google Scholar Back to article
Pereira, F. C., Gonçalves, A. M., & Costa, M. (2023). Short-term forecast improvement of maximum temperature by state-space model approach: the study case of the TO CHAIR project. Stochastic Environmental Research and Risk Assessment, 37(1), 219–231. https://doi.org/10.1007/s00477-022-02290-3
Search in Google Scholar Back to article
Pinder, R. W., Klopp, J. M., Kleiman, G., Hagler, G. S. W., Awe, Y., & Terry, S. (2019). Opportunities and challenges for filling the air quality data gap in low- and middle-income countries. Atmospheric Environment, 215(June). https://doi.org/10.1016/j.atmosenv.2019.06.032
Search in Google Scholar Back to article
PurpleAir. (2022). Download Data with the Sensor Data Download Tool. https://community.purpleair.com/t/download-data-with-the-sensor-data-download-tool/316
Search in Google Scholar Back to article
Rosner, B. (1983). Percentage points for a generalized esd many-outlier procedure. Technometrics, 25(2), 165–172. https://doi.org/10.1080/00401706.1983.10487848
Search in Google Scholar Back to article
Samara, M. Al, Bennis, I., Abouaissa, A., & Lorenz, P. (2022). A Survey of Outlier Detection Techniques in IoT: Review and Classification. Journal of Sensor and Actuator Networks, 11(1), 4. https://doi.org/10.3390/jsan11010004
Search in Google Scholar Back to article
Sayeed, M. S., Abdulrahim, H., Abdul Razak, S. F., Bukar, U. A., & Yogarayan, S. (2023). IoT Raspberry Pi Based Smart Parking System with Weighted K-Nearest Neighbours Approach. Civil Engineering Journal, 9(8), 1991–2011. https://doi.org/10.28991/CEJ-2023-09-08-012
Search in Google Scholar Back to article
Schilt, U., Barahona, B., Buck, R., Meyer, P., Kappani, P., Möckli, Y., Meyer, M., & Schuetz, P. (2023). Low-Cost Sensor Node for Air Quality Monitoring: Field Tests and Validation of Particulate Matter Measurements. Sensors, 23(2), 1–29. https://doi.org/10.3390/s23020794
Search in Google Scholar Back to article
Sharma, B., Sharma, L., & Lal, C. (2019). Anomaly Detection Techniques using Deep Learning in IoT: A Survey. Proceedings of 2019 International Conference on Computational Intelligence and Knowledge Economy, ICCIKE 2019, 146–149. https://doi.org/10.1109/ICCIKE47802.2019.9004362
Search in Google Scholar Back to article
Stavroulas, I., Grivas, G., Michalopoulos, P., Liakakou, E., Bougiatioti, A., Kalkavouras, P., Fameli, K. M., Hatzianastassiou, N., Mihalopoulos, N., & Gerasopoulos, E. (2020). Field evaluation of low-cost PM sensors (Purple Air PA-II) Under variable urban air quality conditions, in Greece. Atmosphere, 11(9). https://doi.org/10.3390/atmos11090926
Search in Google Scholar Back to article
Tahmasseby, S. (2022). The Implementation of Smart Mobility for Smart Cities: A Case Study in Qatar. Civil Engineering Journal, 8(10), 2154–2171. https://doi.org/10.28991/CEJ-2022-08-10-09
Search in Google Scholar Back to article
Yigitcanlar, T., Li, R. Y. M., Inkinen, T., & Paz, A. (2022). Public Perceptions on Application Areas and Adoption Challenges of AI in Urban Services. Emerging Science Journal, 6(6), 1199–1236. https://doi.org/10.28991/ESJ-2022-06-06-01
Search in Google Scholar Back to article

Comparison of outlier detection approaches in a Smart Cities sensor data context

Abstract

Paradigm

My account