Dynamically patching ASP.NET code in real-time? (Why don't WAF vendors do it?)

I was asked today "...Do you think it's feasible to dynamically patch code in real-time? Think agent on a host instead of a waf. Think real patches instead of virtual ones..." and my short answer is:

Yes it is! (if you looking for the technical details, skip to the middle of the post)

I have done that several times in the past, for example in Sharepoint where I wrote a PoC that applied a 'real-time' C# patch (without touching the disk) into a live/running instance of MOSS.

So if it is possible, "...why are all the WAF vendors still trying to operate on the data stream? There is so much more you could so by hooking the runtime...".

There are a number of reasons why the WAF vendors don't want to get into this world:

1. The WAF vendors (just like the SAST ones) are still trying to ignore the Applications they are supposed to protect. By that I mean that their model is based in protecting ALL types of Apps, and not a Specific Application. Which means that they are not very good at trying to understand how an Application actually works. For example they want to ignore the need to consume:
• an Spring/Structs configuration file 
• an static analysis of an ASP.NET MVC application,
• the directory listing of all Aspx pages
• the current HttpModules mappings/behaviour
• etc...
2. The WAF vendors are still selling an 'infrastructure security' product vs an 'application security' product.
3. Very few (if any) WAF vendor has an effective way for it to 'talk' with an Application and issue/receive commands (for example via an agent)
4. Dynamically patching is hard to do and requires a lot of customisation and Unit-Testing environments (see below)
5. The WAFs vendors don't believe that there is a market for this type of capabilities, so they are not investing on it (every time I ask them for this, I get the answer: "Its only you (Dinis) that wants this"). Of course that it is hard for a client to buy something that doesn't exists, but (just like in the SAST world) the clients are also to blame here since they should be putting pressure on the WAF vendors to do this.

Before I continue, I want to say that I am a BIG fan of WAFs and I think they provide a lot of value on the Virtual Patching world. I'm not really covering DAST+WAF on this post, but that is also a great workflow, and if you are into that world , you have to take a look at Ryan Barnett's amazing work with ModSecurity: See Dynamic DAST/WAF Integration: Realtime Virtual Patching and OWASP ModSecurity Core Rule Set Project 

Now from a client point of view, would it make sense to do Dynamic Patching and be able to fix vulnerabilities/bugs in a very selective, low touch and focused way!

Absolutely!

And the security guys, who usually have tantrums at the idea of doing dynamic patching (or WAF rules), are the ones not responsible for real-world applications with lots of moving parts, separate development teams and heavy deployment models (and even for the guys who have very nice and slick CI environments, I would argue that there are tons of cases where it will make a lot of sense to start a security fix process with a real-time patch).

I have done this before (live patching) using some pretty cool O2 Platform Foo, and what I found what that the dynamics of the 'conversation' between security guy and development team, were completely changed.

Here is the scenario, "...there is a security vulnerability in an application where it is possible to invoke from the GUI a feature inside a web control that is vulnerable to command injection..."

Without changing the code (which was not available since this was from a 3rd party component), the options were:

• a) remove the control from the website (not acceptable solution from the business point of view)
• b) detect the use of that control (on WAF or HttpModule) and redirect the user to a error page , since the vulnerability was in the main control's GUI, this was as good as option a) 
• c) try to detect the code in transit and modify it - which was kinda doable, but made really hard by the sheer amount of webpage code, the dynamic nature of Javascript code used to build the control and the fact that this would need to be done on page rendering and page submission (there were a couple other factors with the HTML code created that made this really difficult).

BUT, when we added the capability to do a real-time .NET patch (where we can 'modify' any method on any class), this is what happened:

• Suddenly, as the guy providing the advise, I started thinking about:
• what is the 'best place in the code' to put the patch?
• should I do input or output encoding/validation? (since its a case-by-case answer)
• what are the side-effects of my patch?
• how can I make the user experience as good as possible (where there are no GUI side effects)?
• how to correctly/safely handle the cases where the dangerous functions (to be fixed) are used in a completely OK way by internal code?
• If you look at the questions I was asking myself, I was starting to think 'as a developer' (remember that we usually just tell them what is wrong, or to use API XYZ to fix it)
• As I wrote my patch, I found the need to have a set of unit-tests that covered the scenarios I was trying to fix. Because without them, how could I show the difference between 'This is an exploitable issue' vs 'Here is the app with the same user experience but without the exploitable scenario'
• Once I had my Patch+UnitTests and gave them to the developers, the first thing they did was to do QA testing on the app running with the patched code (which surprised me the first time it happened, although it's pretty obvious in hindsight). 
• Why did they do this? Well because they could! Suddenly, the security dude was basically saying "Hey here is a version of your site that 'should' behave like the previous version, but has a number of security fixes on it" . Naturally the first thing the developers wanted to do was to confirm that the app 'still behaved' like the original (non patched version)
• And when they were happy that the proposed fixes had no side effect on the real app, they started to re-code my 'patch' in such a way that the behaviour of their new code and my 'dll injected' patch was the same (using the provided UnitTests to confirm it)

So technically speaking how can we 'dynamically patch' an ASP.NET application?

Let's say that we wanted to do what this guy asks for in this StackOverflow question: Programmatic MSIL injection 

Well, it depends where we are and how much control we have over the app's deployment:

• In memory app, post JIT: This is the worse case scenario since the code we want to change has already been compiled/Jitted and already lives as unmanaged code:
• Hook a debugger and automatically: breakpoint on the method, call patch, continue execution (this works quite well and unless the original app is very performance intensive, you would be surprised at its effectiveness). What is cool about this hook technique is that we can write the patch in C#
• Hook the method via C++/Detours type injection - If the patch can be written in C++/Assembly this process is easier, if you want to write the patch in C# the process is much more complex
• In memory app, pre JIT: If we can get our code to run before the code to fix has been Jitted (which might mean that you need to disable Native Image precompilation) you can do an In-Memory MSIL patch which:
• is smaller than the original code: easier since you can do it via a simple memory copy (i.e. replace the original IL bytes with the 'patched' bytes)
• is larger than the original code: harder since you will need to jump/call another method (ala detours)
• In memory, on app that has been instrumented and allows dynamic function hooking: easy, but it's not very common to find an app set-up like this
• UPDATE: Fortify's RTA provides some of these capabilities (see follow-up post Any real-world Fortify RTA case studies out there? )
• In memory, on app that is running with the .NET profiler enabled: easy, but this has even worse performance implications than running with the debugger (and has the side effect that the app NEEDS to start under the profiller, where we can hook and detach a debugger into a live process)
• Before load, without access to source code (not signed code): If you have access to the dlls you want to patch, and can modify them:
• Use AOP like PostSharp:  This is a great way to do this since you get a LOT of plumbing provided by the the AOP framework (specially the method cross-cutting capabilities)
• Direct MSIL patch on disk: Of which the Mono Cecil libraries are amazing
• ILDASM decompile, patch IL,  ILASM compile: This should work, unless the original dll creator did unusual IL compilation or manipulation (like Microsoft with some of their core assemblies)
• ILSPY decompile into C#, patch C# code,  Compile with VS: This depends a lot on the original code, and by now things tend to get quite messy
• Before load, with access to source code  (not signed code): There are a number of options available here, of which the one I find more interesting is to use a Roslyn type refactoring to create the 'hook' functions that contain the patch (note that techniques like AOP are much easier to do with the code)
• Before load, with signed code: Now if you want to modify an assembly which has been signed and used by other assemblies (who were compiled against the original signed assembly, not the patched one), one technique I've used in the past was to patch the CLR directly (i.e. the actually engine) to disable Strong Name Verification (it's actually a 2 byte patch: ret true; ). Note that by now (v4.0 of the .Net framework), there are probably better ways to load these 'patched' assemblies into memory that have the wrong strong name

I'm sure there other ways / techniques that could be used (like trying to do in memory Mocking and AOP), but the ones above are the ones I have used successfully :)

Note that I was using the O2 Platform when creating the PoCs described here :) (in fact there are a number of  current O2 modules and capabilities that were created during these engagements)

I will strongly advise anybody trying to do this, to use O2's REPL Scripting, or have a similar REPL environment. Since you will go crazy if you have to compile and press F5 every time you want to run your code/script/patch :)