A Parallel Processing Framework for Big Data Analytics in Smart City ESG Compliance

As smart cities evolve, numerous IoT (Internet of Things) sensors are emerging, which, evidently, generate a data deluge, and managing them in real time, in compliance with ESG (Environmental, Social, and Governance) standards, is becoming increasingly difficult. Today, collecting Big Datasets is easy, but their classic, sequential processing often cannot keep up with the need for real-time information. This study examines the extent to which Python’s parallelism frameworks, specifically multiprocessing, Joblib, and Scikit-Learn, can accelerate advanced data analytics while also keeping the stability of predictive models. Based on a synthetic dataset of 50 million rows and real-world EPA (Environmental Protection Agency) air quality datasets, this paper uses a Divide-and-Conquer strategy to evaluate Random Forest ensemble models in scenarios with 1, 4, and 8-core configurations. The obtained results demonstrate that parallelization can reduce execution time by over 60%. However, performance remains restricted by memory limitations, particularly the memory wall and parallel overhead, and the administrative costs of executing tasks in parallel. In localized hardware settings of 16 GB RAM, the 8-core configuration showcases a sublinear speedup and a lower efficiency than the 4-core case, a situation explained by the inter-process communication overhead and memory bus saturation. The stability of the MAE (Mean Absolute Error) across all configurations confirms that accelerating the process does not compromise the model accuracy. These results set a direction for the organizations looking to optimally use digital resources, indicating the optimal core configuration for real-time information monitoring of data from IoT sensors or the Internet.