| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| en:iot-open:introduction:introduction_to_data-related_design_questions_of_iot [2023/11/20 23:33] – pczekalski | en:iot-open:introduction:introduction_to_data-related_design_questions_of_iot [2023/11/23 18:08] (current) – pczekalski |
|---|
| ====== Data Management Aspects in IoT ====== | |
| |
| | ====== Data Management Aspects in IoT ====== |
| | {{:en:iot-open:czapka_m.png?50| General audience classification icon }}\\ |
| Data management is a critical task in IoT. | Data management is a critical task in IoT. |
| Due to the high number of devices (things) already available, that is tens of billions. Considering the data traffic generated by each of them through, e.g. sensor networks, infotainment (soft news) or surveillance systems, mobile social network clients, and so on, we are now even beyond the ZettaByte (ZB 2^70, 10^21 bytes) era. | Due to the high number of devices (things) already available, that is tens of billions. Considering the data traffic generated by each of them through, e.g. sensor networks, infotainment (soft news) or surveillance systems, mobile social network clients, and so on, we are now even beyond the ZettaByte (ZB 2^70, 10^21 bytes) era. |
| From a technological perspective, the main aspects of dealing with IoT data management are: | From a technological perspective, the main aspects of dealing with IoT data management are: |
| |
| * **data source**: data generation and production is a relevant part of IoT, involving sensors probing the physical system. In a cyber-physical-social system view, such sensors could be virtual (e.g. software) or even human (e.g. citizens, crowdsensing). The main issues in data production are related to the type and format of data, heterogeneity in measurements and similar issues. Semantics is the key to solving these issues, also through specific standards such as Sensor Web Enablement and Semantic Sensor Network, | * **Data source** - data generation and production is a relevant part of IoT, involving sensors probing the physical system. In a cyber-physical-social system view, such sensors could be virtual (e.g. software) or even human (e.g. citizens, crowdsensing). The main issues in data production are related to the type and format of data, heterogeneity in measurements and similar issues. Semantics is the key to solving these issues through specific standards such as Sensor Web Enablement and Semantic Sensor Networks. |
| * **data collection/gathering**: once data are generated, these should be gathered and made available for processing. The collection process needs to ensure that the data collected are defined and accurate so that subsequent decisions based on the findings are valid. Some types of data collection include census (data collection about everything in a group or statistical population), sample survey (collection method that provides for only part of the total population), and administrative byproduct (data collection is a byproduct of an organization’s day-to-day operations). Usually, wireless communication technologies such as Zigbee, BlueTooth, LoRa, Wi-Fi and 3G/4G networks are used by IoT smart objects and things to deliver data to collection points, | * **Data collection/gathering** - once data are generated, these should be gathered and made available for processing. The collection process needs to ensure that the data collected are defined and accurate so that subsequent decisions based on the findings are valid. Some types of data collection include census (data collection about everything in a group or statistical population), sample survey (collection method that provides for only part of the total population), and administrative byproduct (data collection is a byproduct of an organization’s day-to-day operations). Usually, wireless communication technologies such as Zigbee, BlueTooth, LoRa, Wi-Fi and 3G/4G networks are used by IoT smart objects and things to deliver data to collection points. |
| * **filtering**: is a specific preprocessing activity, usually performed at data source or data collector (IoT) nodes (e.g. motes, base stations, hotspots, gateways), aiming at cleaning noisy data, filtering noise and not helpful information, | * **Filtering** - is a specific preprocessing activity, usually performed at data source or data collector (IoT) nodes (e.g. motes, base stations, hotspots, gateways), aiming at cleaning noisy data, filtering noise and not helpful information. |
| * **aggregation/fusion**: to reduce bandwidth before sending data to processing nodes, these are further elaborated, compressed, aggregated and fused (sensor/data fusion) to reduce the overall volume of raw data to be transmitted and stored, | * **Aggregation/fusion** - to reduce bandwidth before sending data to processing nodes, these are further elaborated, compressed, aggregated and fused (sensor/data fusion) to reduce the overall volume of raw data to be transmitted and stored. |
| * **processing**: once data are adequately collected, filtered, aggregated, and fused, they can be processed. Processing can be local and remote and usually includes preprocessing activities to prepare data for actual processing. Local processing, when possible, is mainly tasked with a fast, lightweight computation on edges (Edge computing) and in the Fog layer, wherever possible, quickly providing results and local analytics. More complex computations are usually demanded to remote (physical or virtual) servers provided by local nodes (e.g. communication servers, cloudlets) in a Fog computing fashion or by Cloud providers as virtual machines hosted in data centres. This kind of computation can also involve historical data, providing global analytics, but hardly meets time-constrained applications and real-time requirements, | * **Processing** - once data are adequately collected, filtered, aggregated, and fused, they can be processed. Processing can be local and remote and usually includes preprocessing activities to prepare data for actual processing. Local processing, when possible, is mainly tasked with a fast, lightweight computation on edges (Edge computing) and in the Fog layer, wherever possible, quickly providing results and local analytics. More complex computations are usually demanded to remote (physical or virtual) servers provided by local nodes (e.g. communication servers, cloudlets) in a Fog computing fashion or by Cloud providers as virtual machines hosted in data centres. This kind of computation can also involve historical data, providing global analytics, but hardly meets time-constrained applications and real-time requirements. |
| * **storage/archive**: remote servers are also used for permanently storing and archiving data, making these available for further processing, even to third parties. The database is often used for that, mainly based on distributed, NoSQL key-store technologies to improve reliability and performance, | * **Storage/archive** - remote servers are also used for permanently storing and archiving data, making these available for further processing, even to third parties. The database is often used for that, mainly based on distributed, NoSQL key-store technologies to improve reliability and performance. |
| * **delivering/presentation/visualization**: processing activity results must then be delivered to requestors and users. These have to be, therefore, adequately organized and formatted, ready for end-users. IoT data visualization is becoming an integral part of the IoT. Data visualization provides a way to display this avalanche of collected data in meaningful ways that clearly present insights hidden within this mass amount of information, | * **Delivering/presentation/visualization** - processing activity results must then be delivered to requestors and users. These have to be, therefore, adequately organized and formatted, ready for end-users. IoT data visualization is becoming an integral part of the IoT. Data visualization provides a way to display this avalanche of collected data in meaningful ways that clearly present insights hidden within this mass amount of information. |
| * **security and privacy**: data privacy and security are among the most critical issues in IoT data management. Good results and reliable techniques for secure data transmission, such as TLS and similar, are available. This way, IoT data security issues mainly concern (([[https://developer.ibm.com/articles/iot-top-10-iot-security-challenges/|Top 10 IoT security challenges]])) **securing IoT devices**, since they are usually resource-constrained and therefore do not allow to adopt traditional cryptography scheme to data encryption/decryption. **Data privacy and integrity** should also be enforced in remote storage servers, anonymizing data and allowing owners to properly manage (monitoring, removing) them while ensuring availability. Indeed, security and privacy issues vertically span the whole IoT stack. A promising technique to address IoT security issues, attracting growing interest from both academic and business communities, is blockchain (([[https://www.iota.org/|IOTA: A permissionless distributed ledger for a new economy]])). | * **Security and privacy** - data privacy and security are among the most critical issues in IoT data management. Good results and reliable techniques for secure data transmission, such as TLS and similar, are available. This way, IoT data security issues mainly concern (([[https://developer.ibm.com/articles/iot-top-10-iot-security-challenges/|Top 10 IoT security challenges]])) **securing IoT devices**, since they are usually resource-constrained and therefore do not allow to adopt traditional cryptography scheme to data encryption/decryption. **Data privacy and integrity** should also be enforced in remote storage servers, anonymizing data and allowing owners to properly manage (monitoring, removing) them while ensuring availability. Indeed, security and privacy issues vertically span the whole IoT stack. A promising technique to address IoT security issues, attracting growing interest from both academic and business communities, is blockchain (([[https://www.iota.org/|IOTA: A permissionless distributed ledger for a new economy]])). |
