Efficient Storage and Retrieval of Big data in Smart Cities Using Probabilistic Data Structures

Abstract

The exponential growth of data production from technologies like IoT and cloud com- puting has led to the emergence of big data. In smart cities, big data plays a crucial role in capturing, processing, and analyzing information from diverse sources, enabling data-driven decision-making and optimization of resources. By leveraging big data analytics, city plan- ners can gain valuable insights to address urban challenges and enhance services in areas such as transportation, energy, and public safety. However, the responsible use of big data also requires addressing the concerns related to efficient storage and retrieval of such kind of big data. Collaborative technologies play a vital role in the data life-cycle of smart cities, encompassing gathering, processing, storage, retrieval, analysis, and decision-making. How- ever, traditional data processing methods struggle to handle the massive volume and high velocity of data, as well as the challenges of semi-structured and unstructured data, data pri- vacy, security, and real-time processing. Probabilistic Data Structures (PDS) have emerged as a promising solution to address these issues in smart city applications. PDS can be effec- tively employed across various domains within smart cities, such as health-care, transporta- tion, environment, energy, and industry, enhancing data management and decision-making processes. The purpose of this study is to investigate the significance of PDS in the evolution of smart cities. It focuses on the formation of smart cities, present research status, issues, solutions, existing PDS applications, research gaps, and future directions. This thesis focuses on two key concerns originating from the different natures of data, namely data volume, velocity, and variety. The research seeks to provide significant insights using PDS on handling and utilizing diverse forms of data successfully in the context of smart cities by recognizing and addressing these problems. The thesis's aim of providing innovative solutions for managing big data in smart cities using Probabilistic Data Structures(PDS). This addition emphasizes the thesis's focus on fostering improved urban governance and community well-being. First, the health-care industry has witnessed a transformation in the process of remote patient monitoring and diagnosis as a result of the integration of technologies such as the Internet of Medical Things (IoMT) and apps hosted in the cloud. The process of data analytics does, however, provide a number of obstacles, including those pertaining to the transfer, mining, access, and storage of data. This thesis explores problems in the design of the IoMT and suggests solutions using Bloom filters (BE), a PDS that may be implemented at several tiers. The critical issues to maintain sustainability in health-care like authentication, intelligent scheduling of devices, removing redundant data on the final layer, and improving access times for stored data have been addressed. BF-based solutions are proposed at different layers of IoMT (Edge-Fog-C10ud). The other issue is that the geographic distribution of the world's metropolitan population is changing significantly, resulting in an increase in traffic rule violations and congestion in smart cities. The detection of anomalous vehicles is a serious problem that calls for solutions that operate in real-time in order to improve road safety. This research introduces a novel approach utilizing a persistent bloom filter for storing and retrieving vehicle data, with a significant impact on road safety in urban areas. By employing individual bloom filters based on different time intervals, the method enables efficient temporal membership searches on massive datasets, paving the way for enhanced data analysis and improved road safety measures. Urban traffic congestion poses a significant challenge for smart city infrastructure, necessi- tating innovative solutions for efficient traffic management. This chapter introduces an in- telligent traffic management system that leverages machine learning-based probabilistic data structures (PDS) to enhance traffic flow and reduce congestion. By integrating Bloom Fil- ters, Count-Min Sketch, and HyperLogLog with advanced ML models like Support Vector Machines (SVM) and Gradient Boosting Machines (GBM), the system efficiently handles high data volumes, processes real-time traffic information, and provides predictive traffic management. The deployment on AWS ensures scalability and effective resource utilization, making this solution a robust and cost-effective approach for modern urban environments. The proposed framework and approaches have been validated through experiments on real datasets, showcasing their effectiveness in authentication, smart device scheduling, data re- dundancy reduction, improved access times, query efficiency, and storage space optimiza- tion. This research contributes to enhancing urban safety and road network management in smart cities, bridging the gap between academia and real-world implementation. In conclusion, the use of PDS in smart city applications, such as BF, plays a crucial role in tackling the issues of huge data storage, retrieval, and analysis. This research provides valuable insights into the potential application of PDS in diverse smart city domains. It highlights the benefits and challenges associated with utilizing PDS and proposes avenues for further exploration and advancement in this field. The findings contribute to the grow- ing body of knowledge and offer recommendations for future research and development to harness the full potential of PDS in smart city applications.