Design, implementation, and measurement of a crowdsourcing-based content distribution platform

Content distribution, especially the distribution of video content, unavoidably consumes bandwidth resources heavily. Internet content providers invest heavily in purchasing content distribution network (CDN) services. By deploying tens of thousands of edge servers close to end users, CDN companies are able to distribute content efficiently and effectively, but at considerable cost. Thus, it is of great importance to develop a new system that distributes content at a lower cost but comparable service quality. In lieu of expensive CDN systems, we implement a crowdsourcing-based content distribution system, Thunder Crystal, by renting bandwidth for content upload/download and storage for content cache from agents. This is a large-scale system with tens of thousands of agents, whose resources significantly amplify Thunder Crystal’s content distribution capacity. The involved agents are either from ordinary Internet users or enterprises. Monetary rewards are paid to agents based on their upload traffic so as to motivate them to keep contributing resources. As far as we know, this is a novel system that has not been studied or implemented before. This article introduces the design principles and implementation details before presenting the measurement study. In summary, with the help of agent devices, Thunder Crystal is able to reduce the content distribution cost by one half and amplify the content distribution capacity by 11 to 15 times.