How TeamMentor creates SHA256+PBKDF2 password hashes and stores them in XML files

In the 3.3. version of TeamMentor there was a significant change in how the user's password hash is submitted and stored.

In version 3.2. we used a SHA256 hash of “username+password” (created either on the client (browser) or server) which was stored in the user’s xml file (which is a serialization of the in memory user-object).

Although we never stored the user’s password on disk (in fact in most cases we never even sent it to the server), due to advances in processing power and Credentials Brute Force Attacks, in 2013 that is not a secure way to store password anymore.

After some threads (see TM stores passwords insecurely issue), it was agreed that a solution based on PBKDF2 should be used.

The problem with SSL is not performance, its management

In a chat with Michael Hidalgo about SSL, he mentioned the posts Overclocking SSL (from Google), Dispelling the New SSL Myth (from F5) and Still not computationally expensive (Google guy responding to F5).

No SSL on Azure WebSites (maybe in May 2013?), and shy SSL deployments are so hard

As with the dozen comments in the SSL and Windows Azure Web Sites thread, at SI we are also in the same situation, where we want to host a couple TeamMentor sites on Azure, but can’t do it properly without SSL.

Really nasty bug created by different behavior of WCF Security Principal in Azure (vs Locally in IIS or Cassini)

A couple days ago, I spent most of one day 'bashing my head against a brick wall'  due to an authorization bug that only happened in Azure!

Here are the rough screenshots I took when I was debugging it (the extra logging entries where added to help me understand what was going on (in Azure and locally))

Webinar: Why Software is Still Insecure

Ed and Charles (from SI) are doing a webinar on Feb 20th about why (in most cases) the software industry have not (yet) figured out how to integrate security into the software development lifecycle (in a repeatable, measurable and visible way).

I couldn't agree more and I’m looking forward to hearing what they have to say :)

You can read more about it and register here

The power of Static Analysis to create solid code (in this case JSLint)

I just spent some time using JSLint inside WebStorm cleaning up and refactoring the TeamMentor’s GlobalVariables.js file, so that It shows the much desired green box (top right)

Is the TeamMentor Development/SDL team as good as it gets? (from a security point of view)

As we're having another internal (which should be public but is a topic for another post) debate/email-thread at SI/TM about the 'best way to handle user passwords/hashes', I was thinking "is this (TM SDL) as good as it gets?"

Here is what we have today (regarding TeamMentor's SDL and Team):

Why does windows Azure need to '0wn' my GitHub Account?

While creating an Azure website (part of TeamMentor CI) I tried to connect Azure with GitHub and got this request:

IIS permission problem on TM deployment ('race condition' like)

We're having an 'interesting' IIS deployment problem in one of TeamMentor live servers (running IIS 7.5)

After installing an TM library (using the technique show in the Installing New Libraries via control panel video) I was faced with this:

Interesting Encoding problem with TeamMentor Japanese translation

Here an interesting prob I had to deal today with the first pass of TeamMentor's translation into Japanese.

The translation phase was quite sweet. We sent the TeamMentor Xml files file to the translation company, which they were able to plug into their system, and once completed, they send us back a zip with the translated xml files :)

That zip was then uploaded (via drag and drop) into a TM test server (see video of that workflow here) and it all worked .... except for the metadata and filters:

Netflix “Resilience Engineering” FOSS Library and Security

Really powerful stuff (and to-read material):

• Netflix Open Sources “Resilience Engineering” Code Library  - news article
• Reddit thread about the article linked above
• Fault Tolerance in a High Volume, Distributed System and Making the Netflix API More Resilient - detailed NetFlix blog posts
• Performance and Fault Tolerance for the Netflix API - August 2012 - speaker deck presentation
• Netflix / Hystrix - GitHub repo

Windows Shatter attacks (research links)

Following on my research on Win32 Messaging and the O2's Util - Windows Handles Viewer tool, I just started the day by (re)reading about the Windows Shatter attacks (since it used Windows Messages)

Using SAST technology on Android to detect SSL implementations issues

Arvind called my attention to this pdf Why Eve and Mallory Love Android: An Analysis of Android SSL (In)Security which (looks like) is a great read and it includes this really interesting paragraph:

8. COUNTERMEASURES
8.1 OS SolutionsMalloDroid 

Installation Protection. (page 59)

MalloDroid could be integrated into app installers, such as Kirin [9], to perform static code analysis at install time. This analysis performed directly on a phone could warn of potentially unsafe applications. Users would then have to decide whether they wish to install the app irrespective of the warning.

I ABSOLUTELY agree with this!!!!!

There is a point where the complexity level of what is being implemented is so high and there are so many moving parts to take into account (or to do them consistently in deployment) that we need a way to codify what is (or should) happen.

Interesting SSL challenge between Dev/QA and production

So as we have another small issue with SSL in of TeamMentor's live server (see Problem loading content via SSL in docs.teammentor.net) I'm trying to figure out a way to make SSL deployment more robust.

At the moment there is already a setting in TM to force SSL redirects for all requests. But one issue that exists, is that normally during Dev and QA we don't test on SSL sites, which means that this setting tends to be disabled. Yes this something that we should be changing, but I really like the current light 'dev environment setup' that TM currently has, and adding SSL to it will increase its complexity.

This is actually one of those cases where the 'individual' solution(s) are easy and understood. The problem is in getting them to work in a streamlined and 'check-list-driven' mode. And that is one of the big challenges of Security. Every change as side-effects and ensuring that all moving parts happen at the same time and in the right order is a big challenge.

So when we (the security-side) say 'you should run all your site in SSL', we should understand that there is a lot more to it than just flip a switch in a webserver. See Etsy's Scaling User Security post for some of the challenges that they had when moving to SSL (and I also agree with their decision that the control of SSL should rely on the application and not on the infrastructure).

Another big problem that exists is getting accurate feedback on what is going on. Our UnitTest coverage in TM is not as good as it should, and we still don't have a live website monitoring with 'is it still ok' service (which is something we need to look into next)

I'm thinking about ways to address this in code:

• Detect if the current server supports SSL (and don't allow SSL redirects if that is not working)
• Allow the enable and disable of SSL via a special 'admin only' REST command (which will be a challenge to do if the whole site is currently redirecting to SSL)
• Create a special 'Current Stats' REST command that provides information about the current server
• Find if there is a way to access IIS logs from TM so that we can use TM to detect that something is going wrong with IIS (for example the 404 errors in the current problem are IIS one, and they never reach the .NET Http pipeline
• Create a set of SAST rules specific for each release so that these 'check-list' items can be checked in a programatically fashion (and part of the deployment workflow)

Opposite of 'Security as TAX'

In the multiple articles tagged with Security as TAX I tried to show examples where security and 'secure coding' doesn't add (direct) value to an application/developer and is something the devs/companies have to pay in order to publish/release an application.

They are usually cases where the app was working ok in the 'insecure' state and the fixes had some nasty side effects (or where hard to implement).

But as I am planning to write a couple example of where security DOES add value (think: Amazing presentation on integrating security into the SDL), I was wondering what is the opposite of 'Security as TAX':

• Security as Bonus or just Security Bonus
• Security as Added-Value
• Security as a Good Fairy
• Security as Friend
• Security NOT as TAX
• Security as TAX Refund
• OWASP's Security
• Security Zen
• Beautiful Security
• Application Security done in a way that will not drive Developers crazy
• Here something from your best friend: Security
• SecDDev  - see SecDDev - Security Driven Development 
• Invisible Security or Security Invisible - see "Making Security Invisible by Becoming the Developer's Best Friends" presentation 

Humm... none of them works very well...

Let me know if you have any good ideas

Amazing presentation on integrating security into the SDL

Nick Galbreath  (@NGalbreath)  has published the great presentation he delivered at OWASP's AppSec USA last week.

You can read it on his Rebooting (secure) (web) software development with Continuous Deployment blog entry or use the embedded  slide-share viewer below:



There are lots of great ideas and concepts in there, but for me the slide that really describes what we are trying to do (and how we have to solve the 'software security problem') is this one:

"If we want to fix Security .... we have to fix Development" 
"If we want to fix Security .... we have to fix Development" 
"If we want to fix Security .... we have to fix Development" 
.... (write/say 100x until internalized)
"If we want to fix Security .... we have to fix Development" 

One of the reasons why driving security changes and making code-fixes is so hard, is because security doesn't live in isolation and it is 100% depended on the development process that exists on the other side.

What I like about Nick's pragmatic approach is that he is showing (with real examples) that when there is a slick, fast and effective SDL (with daily pushes to production), security is much easier to embed and there is a much better architecture to 'inject security' into the SDL (and to understand the side effects of those security changes)

The good news is that we 'the security dudes' have such a good reputation with developers, and they trust us so much, that we are the best guys to drive this change.............

.... I can just hear the developers calling the security teams and say "....Hey we want to change how we develop our applications/websites, can't you come over and tell us what to do?  ... Since you've been trying to 'tell us what to do' forever ...  you must have good solutions for how to create the type of Development environment that Nick is talking about'..."

....yeah ..... right :(


Related posts:

• SecDDev - Security Driven Development 
• 'About' page broken due to ClickJacking protection 
• Couple XSS issues and XSS-By-Design (in TeamMentor) 
• Security evolution into Engineering Productivity 
• My presentation at OWASP AppSec Brazil: "Making Security Invisible by Becoming the Developer's Best Friends"  or "Making Security Invisible by Becoming the Developer's Best Friends" presentation 
• What are the challenges with SAST that don't need a better engine 
• Why should a developer care about security training? 

SecDDev - Security Driven Development

I was thinking today about the areas where security (and secure development) can add value to the SDL and hit on the concept of Security-Driven-Development (SecDDev)

SecDDev is security coding activities driven by security focused developers (which is much more productive than the current security as TAX model we have today)

The way I got into this idea was that I was thinking about one of the TeamMentor (TM) refactorings that I would like to see happening, namely the need to create a 'read-only' version of TeamMentor for sites like TeamMentor.net

This would be a version of TM that has no ability to change the Libraries and Article's content, which is a great way to achieve security, since it is harder to exploit a feature when the code is not there :)

What is interesting on this scenario ('Read-Only TM') is that the business requirement ('have a rock-solid version of TM hosted in TeamMentor.net that can be sold to customers') doesn't NEED this isolation! It possible (and easier) to deploy a version that has the full (very powerful) TM editing capabilities and hope that those features are not exploited, or that none of the editors and admin users delete anything (see missing iOS Library incident). This lack of business requirement,  means that unless there is an additional driver to create this 'read-only' version, it won't happen (which explains why we end-up with the bloated 'security-vulns-rich' applications we have today).

Now there is a good business case for 'application isolation and robustness' but those tend to be fuzzy concepts which are hard to measure. For example, in this kind of activities, there is a massive lack of external recognisable metrics, where the site looks and behaves the same before and after the refactoring (until it is able to sustain an exploit/mistake or massive traffic spike). In fact these changes tend to introduce massive side effects (see 'About' page broken due to ClickJacking protection)

And this is where the idea of SecDDev occurred to me where I was thinking 'if only I had a security focused development guy/team that would help me in doing this refactoring'.

Since that person (the Security-Driven-Developer) would have has as part of his job spec the "improvement of TM's security" (where that is just one of the (many) roles and responsibilities that I have in TM's SDL) he/she would be much more focused into making it happen.

Creating a read-only version of TM, which in practice means the isolation of the read-code and the edit-code into separate sites/areas which can be easily removed, is a good example of an important-but-not-urgent activity that would make a massive difference in TM's overal security architecture.

To complicate matter worse, it is hard to make this type of code changes. Even after all my efforts to develop TM in a modular architecture, the technologies and frameworks used (Html, Javascript, jquery, .Net) promote/reward tight integration and inter-dependencies (for example it would be 'security by obscurity' to remove the edit-code from the front-end, but leave it all on the webservices back-end).

This is why when you look at most apps, the code is always in 'development activities blocks' instead of 'security focused blocks'

Another area that SecDDev would have to focus a lot is on UnitTests, since they would struggle to make/propose any meaningful change without the support and cushion of a solid Unit/Integration test suite.  And anybody that gives me a working UnitTest for TM, is somebody that is adding value to me as a developer! (in fact, SI will even pay for those Unit Test :) )

Another interesting 'political' problem happens when the product managers/owners have a lot of power and want to 'lock-in the customer' into a product's technology. For example, a real secure way to create a 'read-only' version of TeamMentor, would be to remove 99% of its code (on both client and server sides) and built a flat-file version (i.e. pure HTML) protected by a simple OAuth-based authentication/authorisation layer (which could even be done as an external service). But this would mean that the 'TeamMentor' product would 'disappear' and all that would be left would be the Content. The good news (for TeamMentor) is that since SI's center of gravity is on the 'Content + Services side' there are no problems in following what is the best engineering route, but when you look at why some applications/websites are so bloated (and don't like to allow the exporting of its data), you will usually find that it happens due to political/strategic decisions instead of engineering ones)

Finally, where are we going to get those SecDDevs from? Well Mark has a good idea for that, see: Let's make this happen: "Investing in Developing Software Security Talent"

'About' page broken due to ClickJacking protection

We're just about to release TeamMentor 3.2 when a final round of QA noticed that the About page was not working:

As you can see by the screenshot and the issue opened (Fix about page which was broken due to ClickJacking protection) this was caused by the fixed applied to the ClickJacking vulnerability reported in TeamMentor.

So here is another nice example the TAX we developers have to pay due to security changes (the app was working great before the fix).

Also, note how the 'security vulnerability' information that I received made no reference to the problems that would be caused by applying the fixes!  

Saying 'Enable the X-Frame-Options' is much easier than saying 'Enable the X-Frame-Options and here are the site effects on YOUR app of that change' 

And this is the key message that I try to give to security professionals in my "Making Security Invisible by Becoming the Developer's Best Friends" presentations. 

Providing information about a security vulnerability and some pointers on how to fix it, is not good enough!!!! Since that is JUST a small part of what is needed to fix that issue. 

Usually more important is 'What are the side effects of applying that fix'

For another similar example see the Couple XSS issues and XSS-By-Design (in TeamMentor) post. 

Security #Fail UX Wifi Connection

So I'm on a local cafe and to access the wifi I have to enter a 13 UpperCase+LowerCase+Number password (given to me from the counter)

This means that it takes me an additional minute and 500% more brain power to login than I would want (and was expecting to). That minute might not sound a lot (and extra use of the brain), but when you are in the zone and with a good mojo, it can be the difference between an idea being implemented and being lost.

This is what the login (and password) looks like (at least they had the common sense to NOT mask the password):

Couple XSS issues and XSS-By-Design (in TeamMentor)

The security review of TeamMentor (TM) has raised a couple XSS issues which seem (so far) to require user-intervention, where the user (or admin) are required to inject the payload in order to trigger it (i.e. they are attacking themselves).

Here is a small description of the issues and how to reproduce them (this really should be a script):