Post migration issue- Exchange 2013 to Domino- TLS negotiation

I would like to share my recent experience with email queue pileup towards to Domino infrastructure from my Exchange 2013 gateway server. 

This exchange 2013 server is recently built gateway server and we noticed the email queue pileup towards to our Domino infrastructure just after the promotion of  from Exchange 2010 to Exchange 2013.

We  noticed a lot of TLS related events triggered on the Exchange 2013, so it gives us a hint that it could be due to the TLS negotiation related but we are clueless, as you all know how difficult to narrow down TLS related negotiation issues.

As a preliminary troubleshooting we have enabled protocol logging (to enable how to enable protocol logging, please refer this article) on the domino connector on Exchange 2013 server and we noticed the below TLS negotiation error.

Later we compare this with netmon and see the same error reflect in the protocol logs.

We engage MS support to trouble-shoot and they found out  SSL 3.0 client settings were causing the issue. Based on the troubleshooting, I assume that Lotus notes on the environment does not understand a cipher suites which is our Exchange is sending , hence the SSL handshake is not getting completed which in turn caused the email flow issues when sending emails from Exchange 2013 server to Lotus Domino.

The below table showing how the SSL settings on the server

Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0

		Before rectified the issue	After rectified the issue
SSL 3.0	Client	DisabledByDefault: 00000001 Enabled: 00000000	Deleted
	Server	DisabledByDefault: 00000000 Enabled: ffffffff	DisabledByDefault: 00000000 Enabled: ffffffff

Server had SSL3 Client keys set to disable which restricted the server to use SSL3 and hence the mails were queued because TLS handshake/Negotiation failure.

Hence we have removed the SSL 3.0 Client Key from the server and it allows the server to clear the queue towards to Domino.

Lessons Learned from this:

•   DisabledbyDefault and Enabled controls whether we want to enable or disable the server to use the protocol.
•   If the registry key/sub keys aren’t present, server is allowed to use the Protocol for communication.
•   Server had SSL3 Client keys set to disable which restricted the server to use SSL3 and hence the mails were queued because TLS handshake/Negotiation failure.
•  Deleting the Client key allowed the server to use the SSL3 protocol and mail flow started working fine.
•   0xffffffff in hexadecimal effectively means “all values other than 0”. But either that, or decimal 1, should be equally effective in turning that setting on. So having SSL 3 client set to disabled by default=0 and enabled=1(decimal) or 0xffffffff (hexadecimal) would have been the best, most consistent way to approach this rather than delete the client key

What is TLS?

Transport Layer Security (TLS) is a standard protocol that is used to provide secure web communications on the Internet or on intranets. It enables clients to authenticate servers or, optionally, servers to authenticate clients. It also provides a secure channel by encrypting communications. TLS is the latest version of the Secure Sockets Layer (SSL) protocol.

Reference:

https://blogs.technet.microsoft.com/samdrey/2014/10/17/vulnerability-in-ssl-3-0-poodle-attack-and-exchange-2010-or-exchange-2013/

http://blogs.technet.com/b/exchange/archive/2015/07/27/exchange-tls-amp-ssl-best-practices.aspx

https://technet.microsoft.com/en-us/library/security/3009008.aspx

https://support.microsoft.com/en-us/kb/245030