Network embedding models automatically learn low-dimensional and neighborhood graph representation in vector space. Even-though these models have shown improved performances in various applications such as link prediction and classification compare to traditional graph mining approaches, they are still difficult to interpret. Most works rely on visualization for the interpretation. Moreover, it is challenging to quantify how well these models can preserve the topological properties of real networks such as clustering, degree centrality and betweenness. In this paper, we study the performance of recent unsupervised network embedding models in Web service application. Specifically, we investigate and analyze the performance of recent random walk-based embedding approaches including node2vec, DeepWalk, LINE and HARP in capturing the properties of Web service networks and compare the performances of the models for basic web service prediction tasks. We based the study on the Web service networks constructed in our previous works. We evaluate the models with respect to the precision with which they unpack specific topological properties of the networks. We investigate the influence of each topological property on the accuracy of the prediction task. We conduct our experiment using the popular ProgrammableWeb dataset. The results present in this work are expected to provide insight into application of network embedding in service computing domain especially for applications that aim at exploiting machine learning models.