Multicast for Parallel Stream Processing
Aizaz HakroOral Defence Date:
Friday, March 9, 2007 - 16:10Location:
Prof. Marguerite Murphy, Prof. Dragutin Petkovic
Scientific data is huge in size and requires substantial processing in real-time to get useful results. The goal of this project is to design, implement and evaluate infrastructure for the collection, archiving and retrieval of multi-modal streams of scientific data. This project presents and evaluates a prototype implementation solution, Ultimate Stream, which allows real-time delivery of scientific data to a virtually unlimited number of computers. No special hardware or middleware other than standard TCP/IP Multicast over a local subnet is required. Several independent tasks can be done in parallel to achieve very high effective computing power without expensive hardware. The support for virtually unlimited number of nodes is achieved through the use of IP Multicast as the network protocol. Use of IP Multicast allows computers to join and leave at their will without affecting the entire system. Frame level data shredding and reconstruction technology is used to avoid retransmissions for consistency and synchronization. The system is flexible enough to work with a number of different types of data. Our prototype is operational and initial performance studies on the MouseCam implementation demonstrate that it can be used as a backbone inside a private subnet to transfer large amounts of data to multiple nodes in real-time for subsequent processing in parallel.