Building a Scalable Data Management Framework


Long Huynh

Oral Defence Date: 



SCI 241


Professors Puder and Petkovic


In environmental monitoring, collecting and interpreting sensor data are the prominent requirements. Researchers often use highly specialized sensors to measure environmental conditions. These sensors are typically off-the-shelf products that are ready to be deployed in the field. However, archiving, viewing, or analyzing data produced by these sensors still poses significant challenges. One prominent difficulty is the heterogeneity of the data. Each sensor typically generates data in a proprietary format. The heterogeneity of the sensor data complicates the development of an application that can be extended in the future to support new data formats. To solve this problem, we have devised a so-called Data Management Platform (DMP) that developers can use to help them build flexible and extensible data management systems for scientists and researchers. The DMP implements generic components to provide the base for any data management system. Developers can then implement the functionalities required for their system and add those functionalities to the platform in a plug-and-play fashion. This is achieved by dividing the system into two different types of components, data producers and data consumers. The data producers' responsibility is to get data into the system. Once data is in the system, the data consumers' responsibility is to consume the data. One data consumer might upload the data to a database for archiving and another data consumer might display the data on a chart for user analysis. This encapsulation of data producers and data consumers allows the components to be independent of each other. This also decouples all the components allowing them to be plugged into or removed from the system without affecting any other part of the system. A prototype of the DMP has been implemented and it is currently being used in the NetBEAMS web application which is online and operational at http://www.netbeams.org.


Data Management Framework, Scalable, Plug-and-play, Plug-n-play, NetBEAMS, Sensor network, Plug-in, Data Producer, Data Consumer


Long Huynh