By Securitynik on 2023-08-09 10:45:10As a leader in a SOC at a Managed Security Services Provider (MSSP), leading multiple teams, it is always interesting to see how new analysts may freeze when they hear the communication is encrypted. What many of these new analysts do not know, is in some cases, you may be able to decrypt this communication. This post provides guidance to these new Analyst, to reduce their fear about being able to decrypt Transport Layer Security (TLS). Do keep in mind, while some of the TLS communication can be decrypted, there are many organizations which do not decrypt it. As a result, let's take the opportunity to learn about TLS (some people still say SSL), starting with 1.0 before we even transition to the latest version (at the time of this writing) 1.3. A little theory is needed, before attempting to decrypt this communication. Before going further, let's understand why we might or might not want encryption. Here is a client and server session using ncat. And here is the tcpdump output of that session.
With that out of the way, let's get to the theory ....
TLS is responsible for providing authentication, encryption and integrity.Authentication: This is where each party verifies who the other claim to be. In many cases, you are going to find the client verifying (authenticating) the server. You can also have mutual authentication, where the server also verifies (authenticates) the client. Encryption: This is where the data is encrypted as it flows on the wire. for example, let's say we had a file with the contents www.securitynik.com. Now we want to encrypt this content.
After encryption rather than www.securitynik.com, we see
Obviously, at this point there is not much we can make of the encrypted content. At this point, even if the communication channel is unencrypted, we can still pass this file on the wire without anyone being able to read the original content easily.
Below we setup ncat to listen for the incoming file on the server side and then the client uses the encrypted file as input.
When we look on the wire, there is a hint that this communication is encrypted byt nothing much for us to interpret via cleartext.
Because we used symmetric encryption, we only need the same key/password to decrypt the file on the receiver side
To decrypt, we rerun the command and add the "-d" option. We also change the input file to be the encrypted file while creating a new out file with the decrypted content.
Awesome, we recovered the file while getting a better understanding of symmetric encryption. Integrity: Validation that the data is not lost, damaged, tampered or falsified in any way. Normally, when we talk about integrity, we are looking at it from simply the hashing function such as MD5, SHA-*, etc. Let's hash the string "www.securitynik.com"The server responds to a "Client Hello" with a "Server Hello", assuming that that it finds an acceptable set of algorithm. We've discussed 1, 2 and 3 as it relates to the Client Hello. No need to repeat here. Starting from bullet 4. 4: Handshake Type: We see handshake type 2 which is the Server Hello. 5: GMT Unix Time: This represents the current time and date in seconds, since January 1, 1970 according to the client's internal clock. Unlike the Client Hello, we do see the true date and time when this negotiation was started from the Server Hello perspective. 6: Random Bytes: Generated by the server and must be different and independent from the Client.Hello.random, whichis part of the Client Hello. 7: Session ID Length: These 32 bytes represents the value of the session ID. 8: Session ID: If this was an attempt to reuse an existing ID and the server is willing to establish the connection, then the "Session ID" sent via the Client Hello would be reused here. However, above, we saw the Client Hello had a Session ID of 0. As a result, the value in this field represents a new Session ID. It is also possible for the server to respond with an empty Session ID, meaning this session will not be cached and thus cannot be resumed. 9: Cipher Suite: Where as the Client Hello had 27 cipher suites above, the server hello will have one. This is the one selected from the client's list chosen by the server. If the session was resumed, this will be the value from the resumed session. The choice made in this case is TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) 10: Compression Method: null (0). No compression is being used here. 11: Extensions Length: 24. While there are 24 extensions, like the Client Hello extensions, we are not going to cover those. Looking at handshake type 11 - Server Certificate.The Server Key Exchange message is sent directly after the server sends its certificate message and is used by the client to communicate the premaster secret. It is sent by the server only when the server certificate does not contain enough data to allow the client to exchange a premaster secret. So while our server certificate above uses RSA for authentication, we can see the Server Key Exchange message is instead using "EC Diffie-Hellman Server Params". One of the reasons for using the Server Key Exchange message is if the RSA_EXPORT public key is longer than 512 bits. I know the server certificate is using RSA but I did not notice anything specific for RSA_EXPORT. Anyhow, the public key for this cert is 2048 bits. Hence longer than 512 bits. The client may use an RSA public key to encrypt the premaster secret or a Diffile-Hellman Public Key to complete the key exchange ("with the result being the premaster secret") Closing out, we see the Server Hello process completed, with the "Server Hello Done" message. This message is sent to state that the server is done sending messages to support the key exchange. The client can now proceed with its phase of the key exchange process. The server now waits for the client's response. Looking at the Client Key Exchange messageAfter the Change Cipher Spec, a finished message is sent. This verifies that the key exchange and authentication process were successful. This is the first protected message which uses the negotiated algorithms, keys and secrets. Once a peer has sent its Finished message and received and validated the Finished message from its peer, it may then begin to send and receive data over this connection. Wrapping this up by looking at the server's Change Cipher Spec and Finished messages.Finally, we see the application data is encrypted. Note, while the data is encrypted, we are still able to see that the protocol being encrypted is HTTP2. Here is that full sequence of packets.As we dig into TLS 1.3, we will not review every field but will try to identify the difference between TLS 1.0 while learning about 1.3. We already know the 3-way handshake has to be completed, so no need for us to repeat that step.We already learned from the Client Hello that fields such as version, session_id, compression_method, etc., are deprecated. No need to touch on them again. Keeping it simple. 1: Handshake Type: Server Hello in response to the Client's Hello. 2. Cipher Suite: The value selected represents the chosen cipher suite from the Client's Hello message. As we see in this list, the server choose the first value. Considering the client already sent its key_share and the server has now chosen the algorithm, the server can then generate it's key_share which it will send to the client. At the same time, the server can use the client's key_share and its key_share to generate the symmetric key needed for encryption. 3. Key_share: The server calculates its Key_share value based on the first item in the client's list of supported_groups. 4. supported_versions, identify this as TLS 1.3. 5. The communication transition to encrypted. Not much for us to do beyond here. This is where however, the Server Certificate, etc., is sent for the client to authenticate the server. Looking at frame 8, we see encrypted application data and the protocol being HTTP.So as seen above, even without decrypting, we are able to see the domain the client is going to. This is exactly the feature that my tool pktItel.py (https://github.com/SecurityNik/pktIntel) uses. Let's decrypt from a few perspectives. First let's decrypt using the SSLKEYLOGFILE. To decrypt from the browser or from a command line tool such as curl that supports SSL decryption, we need to set the SSLKEYLOGFILE environment variable. Let's do this!First I create environment variable via the export command. Next I validated the SSLKEYLOGFILE environment variable was created by using the echo command. When the variable was created, it pointed to a file. When I ran ls, we see the file does not currently exists. If this was setup correctly, the file will be created automatically, once they (pre-)master secrets are written to it. Setup TShark to capture the traffic.Now if a threat actor is able to see the string, it can be modified and the hash updated. This way, when the recipient receives the packet and compute the hash, it will match what the threat actor wanted. To mitigate this, Keyed-Hashing for Message Authentication is used via HMAC. With HMAC along with the hash, we use a password. TLS consists of a Record and a Handshake Protocol. These are also called layers. The Record Protocol provides security and reliability. Security is handled via symmetric cryptography (DES, AES, RC4, etc.) for data encryption. Reliability/Message integrity, is handled via hashing algorithm such as SHA, MD5, etc. The Record protocol may also be used without encryption. When a new connection begins, the Record layer connection, hash and compression states are all set to null. Data fragmentation/reassembly into manageable blocks, optional compression/decompression, application of MAC and encryption/decryption prior to or after transmission is all done by the Record Protocol. The Handshake protocol allow the server and client to authenticate each other, negotiate the encryption algorithm and cryptographic keys before any data is transmitted or received. While the Record Protocol uses symmetric algorithms for encryption, the Handshake Protocol uses asymmetric (Public Key Cryptography: RSA, DSS, etc.). Authentication can be made optional but is generally required. Once the channel is encrypted, the data sent by the two endpoints, is only visible by those two endpoints. The TLS protocol does not hide the length of the traffic of the data being transmitted. Do keep in mind that the TLS endpoints may pad the length to reduce the effectiveness of traffic analysis techniques. To get an understanding of the TLS communications for both TLSv1.0 and TLSv1.3, we will target my domain: www.securitynik.com. If you are wondering why not TLSv1.2 also, well that falls within 1.0 and 1.3. My expectation is if you understand 1.0 and 1.3, then you have a good understanding of what is in the middle. Interestingly, above does not show information about the versions of TLS enabled. I lost interest in trying to refine it :-D . Let's get this information another way. Copy and paste the output into a text editor such as vi, so you can see the actual output I am interested in.
TLSv1.0 Was launched in January 1999 via RFC 2246.
Understanding TLSv1.0 through packet analysis. Let's setup tcpdump to capture the traffic of interest. Note below shows the captured traffic also.
Here is the curl command line to force curl to use TLSv1.0
Reviewing the PCAP file.
As TLS run above TCP (in this example), first, we see the TCP 3-way handshake
Next packet to come is the "Client Hello"
Breaking the "Client Hello" down (well some of it).
The hello messages are used to exchange security capabilities and enhancements between the client and server. The client's hello message must be responded to by the server's hello message. If there is no response, the connection will fail. The client and server hello messages, establishes the protocol version, session ID, cipher suites and compression methods to be used.
While the client normally will first send its hello as shown above, in TLSv1.0 the server may send a "Hello Request" asking the client to begin the negotiation process anew.
Both the client and server also generate their respective random values ClientHello.random and Server.Hello.random.
While there is a lot in there, we are going to focus on the most important items at this time, for the purpose of this conversation.
1: Content Type: Because TLS was designed to allow extensions to the protocol, new content types can be added. If the other side of a TLS implementation receives a record type it does not understand, it should ignore it.
Four content types are defined by default:
* Handshake Protocol
* Alert protocol
* Change Cipher Spec Protocol
* Application Data Protocol
2: Version: For TLSv1.0 record layer version information shows ... well ... TLS 1.0. However, looking at the brackets we see 0x0301. TLS 1.0 is a minor modification to SSL 3.0.
3: Length: The length of the TLS client protocol coming next. In this case, the handshake. It has a length of 177 bytes. This fragment length should not exceed 2**14.
4: Handshake Type: Client Hello (1): There are various handshake types, such as hello_request, client_hello, server_hello, certificate, server_key_exchange, certificate_request, server_hello_done, certificate_verify, client_key_exchange and finished.
Above, we see the type is Client Hello
5: Length: 173: There are 173 bytes coming beyond this
6: Version: TLS 1.0 (0x0301) : This represents the TLS handshake protocol version. This shows TLS_1_0 is being used .
7: GMT Unix Time : This represents the current time and date in seconds, since January 1, 1970 according to the client's internal clock. Interestingly, these internal clocks are or required to be set correctly. Well that is good to know. If you look closely above, it says Sep 8, 2081.
8: Random: A 28 byte value generated by a secure random number generator.
9: Session ID Length: When not 0 as is shown above, this session ID identifies a connection between two peers for which the client wishes to reuse the security parameters used to create the original session. This identifier may have been from a previous, current or another currently active session. Considering the third option, clients may create several independent secure connections without repeating the full handshake. In this case, the client wishes to start a new connection
10: Cipher Suites Length: This represents the length in the number of bytes occupied by all the cipher suites the client is sending to the server. In this case we have 54.
11: Cipher Suites: We see 27 cipher suites being passed by the client. These are cryptographic algorithms supported by the client and passed to the server in the client's order of preference. Meaning "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)" is what the client wishes to use first. The cipher suites passed defines the key exchange algorithm (ECDHE), the authentication algorithm (ECDSA), the bulk encryption algorithm, including the secret key length (AES_256_CBC) and the hashing algorithm (SHA). The server will choose one from the list. If no acceptable cipher suite is found by the server, a failure alert will be sent and the connection closed.
12: Compression Methods Length: The number of bytes occupied by the Compression Methods. In this case, we see 1 byte.
13: Compression Methods (1 method): This represents how many compression methods are defined.
14: Compression Method: Currently, no compression method is defined. Hence null (0). If there were multiple, like the cipher suites, they would be placed in the order of preference.
15: Extensions Length: I will not focus on the extension at this time. However, currently there are 78 defined.
With that understanding of the key fields within the TLS Client Hello. Let's look now at the server hello.
For the Client Hello, our Tshark filter was "handshake.type == 1", for the Server Hello, our handshake type will be 2.
Above shows a sequence (chain) of 3 certificates. The sender's www.securitynik.com certificate comes first. Hence, from to top bottom, the certificate for the domain www.securitynik.com followed by the intermediate CA (GTS CA 1D4) and the root CA (GTS Root R1). Each following certificate certifies the one preceeding it.
The server must send its certificate, providing the agreed upon key exchange method is not anonymous. The server certificate should always be sent.
The certificate provided is generally an X.509v3 certificate. Note, the client may also send a certificate upon request from the server. In this scenario, this is not being done.
Looking at the Server Key Exchange.
The Client key exchange message is always sent by the client. If RSA is used, the client generates a 48 bytes premaster secret. This premaster secret will be encrypted with the certificate RSA public key and transmitted to the server. This premaster secret is used to generate the master secret.
In this example, we see Diffie-Helman is being used for the Client Key Exchange. Do remember, the server sent its Diffile-Hellman parameters during its key exchange. The Diffie-Hellman parameters will allow each side to agree on the same premaster secret.
The size of the master secret will always be 48 bytes in length. The length of the premaster secret varies based on the key exchange algorithm.
When RSA is used for authentication and key exchange, the premaster secret is 48 bytes. For Diffie-Hellman the negotiated key size is used as the premaster secret. This is then converted into the master secret.
The client also sends it Change Cipher Spec message
What that understanding in place, time to move to TLSv1.3
Released via RFC 8446.
TLS 1.3 represents a major revision of the TLS protocol. While TLS 1.3 has significant changes, the basic capabilities for previous TLS version has been retained.
Some key objectives of TLS 1.3 are:
* Remove unused and unsafe features of older TLS version
* Include strong security analysis in the design
* Improve privacy by encrypting more of the protocol
* Reduce the time needed to complete the handshake.
* All handshake messages after the Server Hello are now encrypted. There is also the ability to encrypt extensions messages which were previously sent in the clear via the EncryptedExtensions message.
TLS 1.3 also has a new set of cipher suites that are exclusive to it. These cipher suites are based on Authenticated Encryption with Associated Data (AEAD).
With the exception of the messages which are needed to establish a shared secret, the TLS handshake is encrypted.
Additionally, items such as renegotiation, generic data compression, Digital Signature Algorithm (DSA) certificates, static RSA key exchange and key exchange with custom Diffie-Hellman (DH) groups have been removed. TLS 1.3 also supports forward secrecy.
The TLS1.3 handshake should be looked at as having 3 phases.
1. Key Exchange
Selection of cryptographic parameters and establishment of a shared key.
2. Server Parameters
Establish other handshake parameters such as application layer protocol support, whether the client should be authenticated, etc. This phase is encrypted.
Authenticate the server and optionally the client, provide key confirmation and handshake integrity. This phase is also encrypted.
Capture TLSv1.3 Traffic.
This is the default for curl. Let's still force it.
1: Similar to TLS 1.0, we see the record layer reports a version of TLS 1.0 (0x0301).
2: In the Handshake Protocol, we see version 1.2 (0x0303) for TLS 1.2. In reality, this version field is considered legacy in TLS 1.3. TLS 1.3 instead leverages the "supported_versions" extension for the client to indicate its preferred version. We see the "supported_versions" extension number 9 above. At a minimum, the ClientHello message will contain a "supported_versions" extension. Only if this extension exists, then the servers will attempt to negotiate TLS 1.3
3: Random Bytes: One of the things this is used for is protection against replay for TLS 1.3 1-RTT data.
4: Session ID: The session_id field is also considered legacy in TLSv1.3. As we saw with TLSv1.0, the ability to resume a session existed. The value we see above would be from a pre TLSv_1.3 session. TLSv1.3 does not support session resumption.
5: Cipher Suites (4 suites): This is the list of symmetric cipher suites supported by the client, the length and the hash algorithms to be used. As with TLSv1.0, these are in the order of client preference.
6: Compression methods are also considered legacy in TLS 1.3
7: Extensions: TLS 1.3 makes extensive use of extensions. TLS 1.3 makes some extensions mandatory, since functionality has moved into these extensions, so as to preserve compatability with previous versions of TLS. If a server does not recognize an extension, the server should just ignore it.
8: supported_groups: The client uses this group to indicate the named groups the client supports for key exchange. As with the cipher suites and the signature algorithms, it is ordered by most preferred to least preferred. In this list, we have x25519 as the first item.
9: signature_algorithms: This identifies the signature algorithms maybe used in digital signatures. In this example, the signature_algorithms also refers to the signatures appearing in certificates as their is no "signature_algorithms_cert" extension. If the client requires the server to authenticate itself via a certifiates, then the client must send this signature_algorithms extension. If the server is authenticating via a certificate but the client has not sent this extention then then server MUST abort this handshake with a "missing_extension".
9: supported_versions: As mentioned in 2, this is used by the client to indicate which version of TLS is supports and by the server to specify the version it is using.
10: key_share: This identifies the endpoint cryptographic parameters. The Key exchange inforamation is determed by the group and its corresponding definition. We see here that the server has generated it's "key_share" based on the group x25519 which was the first group identified in 8 above. This key_share represents the client thinking it knows what cipher suite the server is going to choose based on the list the client provides. In this case we see one key_share.
A combination of the Client Hello and Server Hello messages are use to determine the shared keys.
With that understanding of the Client Hello, taking a quick glance at the Server Hello message.
With all of this understanding now out of the way, let's look to decrypting this communication.
Before we decrypt anything, if I said to you, "we could see traffic occurring over port 443 without decrypting", you may say that is not possible because we know port 443 traffic is normally encrypted. This is correct. However, what we should notice above in our earlier learning, is that when this connection is starting up, it is in clear text. There is also an extension called "Server Name Indication extension"
Launching curl ...What are those values? See the reference section. Basically those are the secrets used to encrypt the application data or the handshake, etc. Note, because we set the SSLKEYLOGFILE environment variable, if we open our browser and it has been compiled with this option, then similarly the connections your browsers make, will result in information be written to this file. Hence, I am not going to repeat this with a browser. You can test it for yourself. How do we decrypt this? Let's try to look at the protocol hierarchy of the undecrypted traffic with TShark.
Revisiting SSLKEYLOGFILE, we see.
Looking inside the file we see one entry for our curl session;
Clearly, beyond the "tls" we are unable to conclude what type of traffic is following TLS. Let's fix that by leveraging the sslkeylog.file we created via the SSLKEYLOGFILE environment variable. We need to overwrite the TShark configuration by telling it to use the sslkeylog.file as input for decryption.
Whow! We see now we can see all the application layer data as HTTP2. What can I do with this now? Well if I wanted to do signature based analysis we could. Let's say we are aware of the threat actor's attack tool contains a string/signature "securitynik". Do note, you can feed the same file to Wireshark and it can also decrypt the traffic.
Let's first reassembled the session by following the TLS stream.
We should recognize immediately, we are able to see the plaintext. Remember, this should have all been encrypted.
Let's use grep as our signature based detection tool, looking for the string/pattern "securitynik"
There we go, we decrypted the traffic running over TLS 1.3.
Ok! At this point, you are probably saying but Nik I work mostly in a Windows environment. How do I do this in Windows? Let's address this concern.
Setting and verifying the environment variable on Windows11 under the user profile.
Before setting the environment variable, let's verify that Microsoft Edge browser is not running. Sometimes this seems to cause an issue if we add the environment variable while edge is running.
Getting the process information via both cmd and powershell. Just to show different ways.
Imagine that, all I did was start-up my computer. Did not open a browser but I have 7 instances of MSEdge running. Geez!!!
Back to regular programming. Kill those processes
Set the environment variable
Validate in a Powershell or CMD a new windows the environment variable has been set
With everything in place, let's use Invoke-WebRequest in Powershell, similar to what was done in curl.
What is funny in this situation, is this action did not create the file. I was very much surprised by this. Maybe more testing on my part in the future.
However, if I open MSEdge, this file gets created and populated.
If I run invoke-webrequest in another Terminal while edge is running it appends the new keys to the file. This invoke-webrequest will require more testing as my results are inconsistent. Not interested in this right now as everything works from within MSEdge.
Let's wrap up this post by looking at decrypting via a proxy and a browser.
In this case I am using Burp Suite Community Edition for my proxy
------ BurpProxy On image ------
Validate the proxy is listening via ss
Configure the browser to use the proxy.
In my browser, I try to go to https://www.securitynik.com and I see the request is intercepted by the proxy.
At this point, some people may say "Advanced" and "Accept the Risk and Continue" the attempt to accept the error. That could be a bad move. Let's do so however for the purpose of this blog post. With the risk being accepted.
Forward the intercepted request within the proxy ...
... and now we see in Burpe also the response in clear text. Because we forwarded the request.
While I captured the traffic from my blog, the takeaway for you is, once a proxy is in place, then you should be aware that the traffic can not only be seen but also manipulated. This is true, even if TLS is in use.
But what about HTTP Strict Transport Security (HSTS)?
HSTS is a response header sent by the server, stating that the site should only be accessed using HTTPS. HSTS can be used to mitigate the risk of man-in-the-middle types of attacks.
The browser will reports that the site supports HSTS and thus if we wanted to access that site, we would have to create an exception.
windows - What is the meaning of the values of the Protocols field from Get-TlsCipherSuite Output? - Server Fault
protocols - What is the significance of the version field in a TLS 1.1+ ClientHello message? - Information Security Stack Exchange
scapy-ssl_tls/scapy_ssl_tls/ssl_tls.py at eb6df1c940527e65e3f16c662836d7382ba4ace0 · tintinweb/scapy-ssl_tls (github.com)
Decrypt SSL traffic with the SSLKEYLOGFILE environment variable on Firefox or Google Chrome using Wireshark (f5.com)
Authenticated Encryption with Associated Data (AEAD) | Tink | Google for Developers HMAC-SHA1 in bash - Stack Overflow How To Encrypt Files With OpenSSL (linuxwebdevelopment.com) encryption - How to use OpenSSL to encrypt/decrypt files? - Stack Overflow TLS Versions: What They Are and Which Ones Are Still Supported? (cheapsslsecurity.com)