Cyber-situational awareness in the presence of encryption

Maintaining cyber-situational awareness is a critical requirement for effective threat intelligence. However, the ubiquitous presence of encryption across numerous protocols makes it ever more challenging for organizations to monitor traffic for security purposes. This paper presents the results of analyzing encrypted traffic and its metadata to provide intelligence on the communication channel. In this study, we aim to 1) analyze and decipher the protocols of TLS and IPSec concentrating on how the session key is negotiated, and 2) analyze the ciphertext of symmetric algorithms, looking for patterns or non-randomness, which by specification, should be non-observable. We demonstrate that we are able to probabilistically identify participating parties in communication, identify signature of Suite-B algorithms (AES-GCM-256), recognize cipher text in near real-time, identify encrypted data in open channel, uncover flaws in cipher modes, and identify unknown and proprietary ciphers.