From Azure to Firebase: Could not establish trust relationship for the SSL/TLS secure channel.

UPDATE (16/Apr/2014):  Following a lead from the Firebase Support it looks like the problem could be inside Azure for all SSL, since "https://www.google.pt".GET(); also doesn't work.

Just had a really weird scenario happen to me in the last couple hours, which could be somebody hacking Azure (but I think there is a more benign explanation)

The new version of TeamMentor (currently in 3.4.1 RC0) has a really cool real-time log/activity log viewer which uses Firebase to push data and pull data (from a 'configured TM server' into 'multiple browser-based viewers').

For a while all was good (both locally and in Azure), but in the last couple hours, I noticed that the 'data push' stopped working (i.e. my test version of TM running on Azure was not pushing Activities, DebugMsg and RequestUrls into the assigned Firebase account).

Here is what the viewer looks like (with new messages not being received):

In order to debug the problem, I opened up the CSharpREPL (part of TM admin features), and wrote a simple script to replicate the API_Firebase process used to check if the firebase server is online. This is basically a GET request to a random Firebase area, which expects to get 'null' string back (for example if the firebase AuthToken provided was invalid we would get a different message)

As expected the script failed execution

Looking at the TM Debug logs (which will will show core Firebase errors))

... we can see that there was an 'Error in getUrlContents: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.'

Which is basically .NET saying 'Hey I cannot verify the SSL connection with this server'

This can happen for couple reasons:

1. The  Firebase SSL certs have been updated and somehow Azure is failing to update them (the certs could had been updated as a mitigation to the OpenSSL Heartbleed Bug (read server side memory anonymously) vulnerability)
2. Azure (or the .NET Framework used in Azure) is putting extra demands on the SSL connection (which Firebase SSL is not matching)
3. There is 'somebody' proxying the SSL connection (and doing it badly since they don't have a root cert to be able to proxy the Firebase domain without being noticed). This would be the worse case scenario, and I really don't want to think too much about it and its implications.

What is interesting is that when I connect directly to that particular URL, I get no SSL errors in Chrome

... and if run TM locally (on Cassini or IIS) I also get no SSL errors (i.e. the Firebase data push works as expected).

Finally to confirm that the problem is indeed the SSL connection, I used the FluentSharp.CoreLib.API.Web.Https.ignoreServerSslErrors(); C# API to disable .NET's SSL verification:

With that change, the execution of the GET request to the Firebase test URL returns 'null':

... and the firebase activity logs will be back in action (note that the latest activity entry shows that the current user (me) is viewing the User_View Tbot page)

... so will the real-time Debug Logs (note the .info() message from the REPL script is shown in the log viewer)

So definitely there is something weird going on in Azure. 

I will ping the Firebase support team with a link to this post and hopefully they will be able to figure out what is happening.