Historical data storage for large scale sensor networks

Wireless sensor networks are rapidly finding their way through a plethora of new applications like precision farming and forestry, with increasing network scale, system complexity, and data rate. While scalable MAC and routing protocols for sensor networks have been well addressed in recent years, the scalability of the back-end storage architecture has been largely overlooked. As a result, current storage and retrieval architectures usually lead to an excessive I/O cost when it comes to improving the scalability and responsiveness of the system. In this paper, we present a scalable backend storage and retrieval architecture to support very large volumes of real-time measurements from wireless sensor networks. In particular, our contribution provides: (i) a database partitioning and structuring scheme coupled with a double-buffering technique to reduce the end-to-end delay while minimizing the processing power, and (ii) an optimized historical measurement data query format tailored for superior performance in terms of data retrieval responsiveness. Through a realistic emulator for large scale sensor network, we evaluate this storage and retrieval system to illustrates its delay and I/O benefits in both high and low traffic rate scenarios. The evaluation guides our design of an adaptive design, that applies batch insert method for smaller deployments to reduce insertion delay, and double-buffering for larger deployments to reduce I/O cost and avoid saturation, at the cost of higher delay.