Much work have been done in activity recognition using wearable sensors organized in a body sensor network. The quality and communication reliability of the sensor data much affects the system performance. Recent studies show the potential of using RFID radio information instead of sensor data for activity recognition. This approach has the advantages of low cost and high reliability. Radio-based recognition method is also amiable to packet loss and has the advantages including MAC layer simplicity and low transmission power level. In this paper, we present a novel wearable Radio Frequency Identification (RFID) system using passive tags which are smaller and more cost-effective to recognize human activities in real-time. We exploit RFID radio patterns and extract both spatial and temporal features to characterize various activities. We also address two issues - The false negative issue of tag readings and tag/antenna calibration, and design a fast online recognition system. We develop a prototype system which consists of a wearable RFID system and a smartphone to demonstrate the working principles, and conduct experimental studies with four subjects over two weeks. The results show that our system achieves a high recognition accuracy of 93.6 % with a latency of 5 s.