O2 .NET SAST Engine: MethodStream and CodeStrams for a WebService Method

One of the hidden Gems of the O2 Platform is its SAST engine that is able to create what its called MethodSteams and CodeSteams (from C# code).

MethodSteams are a code representation of an entire call-tree, i.e. one file that contains the original method and all the methods it calls (recursively).

CodeSteams are all the possible 'taint follow' paths that exist on a particular MethodSteam

Here are they in action in the HacmeBank's source code:

This is how we have to show security vulnerabilities to developers (in real time as they are created)

I posted a PoC today that represents my vision for O2 and what I have been trying to do for the past 5 years.

You can see the video at Real-time Vulnerability Creation Feedback inside VisualStudio (with Greens and Reds) where every time the user makes a change to the code there is an auto-compilation (using Roslyn's C# compiler) and a SAST scan (using Cat.NET)

What I like the most about this, is that I now get to think about 'the best workflow to present developers the security guidance they need'.

Although this PoC is quite agressive (I do a compilation and scan on every keystoke which is a bit OTT), here is another video that shows a bigger compilation+scan on save: Real-Time C# Solution Compilation and Security Scanning (using Roslyn and Cat.NET) 

What do you think?

Real-time Vulnerability Creation Feedback inside VisualStudio (with Greens and Reds)

Here is a pretty cool PoC of a Real-time Vulnerability Creation Feedback inside VisualStudio where I show how a number of security vulnerabilities (XSS, SQL Injection, Path Transversal, Unvalidated Redirects) are detected as soon as they are created.

I'm trying to find out the best visual way to present this info to the user, and Greens/Red are usually a good indication of good/bad stuff (also note how the squiggly lines show the traces (i.e. the places where there is a vulnerability)

If you want to comment on this PoC or idea, please use this reddit thread 

Related posts:

• Real-Time C# Solution Compilation and Security Scanning (using Roslyn and Cat.NET) 
• Running Cat.NET SAST Scanner outside VisualStudio 
• 1 line to compile, create and execute: O2 Script to use Roslyn to Dynamically compile and execute a method 
• New Reddit Community for Cat.Net 
• Using/Consuming Cat.Net's engine inside the O2 Platform (and outside VisualStudio) 
• Using Roslyn to Load and (quickly) Compile C# Solution files (outside VisualStudio) 
• Video: Real time Vulnerability Scanning using Cat.Net and Roslyn (SAST) 

Roslyn NerdDinner compilation errors (on June 2012)

Using the Util - Roslyn - Compile Solution and view errors.h2 O2 Platform Script/GUI, I was able to quickly try to compile the latest version of NerdDinner (just drag and drop the NerdDinner.sln file: 

As you can see by the screenshot above, there are the current errors, some of which are a bit weird.

Here are all the errors in an easier way to consume:

Real-Time C# Solution Compilation and Security Scanning (using Roslyn and Cat.NET)

On the Real time Vulnerability Scanning using Cat.Net and Roslyn (SAST) example, the compilation and scanning happened on a single file, which was cool, but not that realistic.

The next step was to be able to compile and scan an VS Solution file (*.sln), like one from the 'vulnerable by design' SuperSecureBank site.

In the video below, I use Roslyn's project loading and monitoring capabilities to trigger a compilation every time there is a change made to a solution file. And if the compilation is ok, a security scan (using Cat.NET) is triggered (both steps done in a couple seconds):

Running Cat.NET SAST Scanner outside VisualStudio

One of the big limitations of Cat.NET was its need to be executed inside VisualStudio.

Yes VisualStudio is one of the places where we want to use Cat.NET's engine and GUIs, but it is also very important to be able to run it as a stand-alone tool (and to inject it into other .NET applications).

Here is a video that shows a GUI/script I created with the O2 Platfrom, which shows the execution of Cat.NET GUIs + Scanning engine outside VisualStudio, and a couple extra features added.

REPL C# Editor to edit another REPL C# Editor ( i.e. 'Script the Script')

Here is a practical C# example of what can be done with the Type Confusion capabilities added to the O2 Platform.

What is happening in this GUI is that the REPL script on the right is given as a parameter (via the script variable) the REPL gui on the left (which is of type ascx_Simple_Script_Editor);

The problem is that the ascx_Simple_Script_Editor object comes from a dynamically compiled dll (with ascx_Simple_Script_Editor.cs) and the object from the left cannot be casted into the same type on the right.

The solution is to do a Type Confusion cast, which has no direct side effects since the objects are the same.

Here is the code that builds this GUI:

Here is the code that is running on the right-hand-side REPL environment:

To see a practical example of this ideas, next time you are inside an O2 C# REPL Editor, go to the context menu and chose the menu item: Utils -> Script this Script

C# Type Confusion - tricking the compiler to view a String as a StringBuilder (or a TextBox as a Label)

One of the less known security implications of the fact that the .NET verifier is disabled when executing code in FullTrust (which is the default) is the fact that the type-safety of the code executed is mainly enforced by the compiler and not (as expected) by the CLR (which is the case when the verifier is enabler).

'Type Confusion' happens when we are are able to cast an Class A object into a Class B type,  without triggering an compilation or runtime error. Under normal behaviour, we would get an compilation error since Class A and Class B are not compatible classes.

We are basically trying to do this (which breaks 'type safety'):

String a = "";
StringBuilder b = (StringBuilder) a;

Here is a C# code sample that shows a number of examples around an specific Type Confusion scenario:


Extension method used:


Code that creates the DLL which does the type confusion:


Here is a different example where I was able to cast an TextBox as a Label:

Although we can can cast any object into any type, if the memory values don't align, we will end up crashing the runtime :)

For example, casting a TextBox as a ToolStripTextBox

Triggered this error:

And this one:

So yeah, we have to be careful, but I already have a number of places I can use this technique inside the O2 Platform (for example when reusing dynamically compiled objects which from the CLR point of view are created in different assemblies, and are treated as incompatible)

Related Posts:

• REPL C# Editor to edit another REPL C# Editor ( i.e. 'Script the Script') 

Using the System.Windows.Forms.Form ClientSize property to fix its different sizes on different windows 7 themes

Here is a really annoying C#/Windows GUI behaviour created by the fact that current Windows 7 theme affects the height of a control (namely is main Panel).

Using O2, here is a script that creates a simple form:


The idea is that this should create a popup window with a Form with width of 400 and height of 40 and a panel inside it that always have the same size  (ie the width and height values of the inner panel should be consistent across multiple versions of Windows).

But that doesn't happen

This is what originally happened in Windows XP (or Windows 7 when we have the 'Windows Classic' Theme):

And this is what happens when we use the Windows 7 Basic theme (note the difference in size of the inner panel).

If you don't think that this is a problem, look at the differences between these two O2 GUIs (which is the script Util- Stop all (but current) O2_XRules_Database processes.h2)

and

The solution is to use the ClientSize property to ensure that we have the right size

Which will give us a consistent  panel size on both Windows Themes:

and

This fix has been added to the latest version of O2 Platform (with some O2 GUIs showing in a slightly bigger window, but at least the size of the main panel is consistent across windows themes)

There is also a new extension method that allows the easy modification of a Form's ClientSide value (in a thread safe way):

example of using this extension method:

C# GUI to View Processes Open Handles (Files, RegKeys, etc...) just like SysInternal's ProcessExplorer

How many times have you tried to find which file was open in one of your currently open processes?

One of the few reasons I use SysInternals ProcessExplorer is to use it 'Find Handle' capabilities, which allows me to find the processes that are locking a particular file.

I always wanted to do this from C# (and O2), and this weekend, based on the code from the StackOverflow's what process lock a file answer, I was able to build an O2 script that collects information about all open handles in the processes the current user has access to (you will need admin privs to look at all running processes)

You can see the scripts created here, and below is a video that shows the Tool - View Open Handles for all Processes.h2 in action (note the ability to quickly search on all open file handles)

Note 1: this script was written in C# using a couple Win32 DLLImport methods and can be executed by non-admins on userland (somehow I was under the impression that this type of handle information was only available: a) via the kernel, or b) if we hook all processes and got the handle data from via 'dll injection-code-execution')

Note 2: If it hangs while getting the process' handles, sometimes it helps to kill the process (for example TGitCache.exe) that is causing trouble (see the last entry on the LogViewer)  There is also quite a lot of memory allocs going on in the conversion of unmanaged data into C# objects, so this script will leak a lot of memory (i.e. you will not be able to use that process for a long time :) )

Removing an Event from a WinForm ListView control using reflection

The objective was to remove an event from a WinForm control that we don't have the source code for (and can't recompile and remove the event on directly on the source code).

This problem happened originally when dynamicaly consuming Cat.NET's guis (outside VisualStudio) and there were a couple controls (like a ListView) that hooked event handlers that triggered functions that had DTE dependencies (which triggered an exception since we were running Cat.NET outside visualstudio (and the DTE2 object was null).

There doesn't seem to be an easy way to do this (and google didn't find a good solution) so using O2's powerful reflection APIs I was able to find a solution which is now available as these extensionmethods:



Here is a document that shows how these extension methods were created:

Using Roslyn - Refactoring AddingMethodToClass (while creating an API)

Here is a step-by-step refactoring activity where I used of the Roslyn example methods (the AddingMethodToClass which shows how to "replace a class inside a namespace") and rewrote it using O2's Roslyn APIs (adding a bunch of Extension methods on the way) 

This is what the refactored code looks like:

This is what the original code looks like:


This is a smaller version of the refactored code:

Here is a document that shows how the refactored code was created:

1 line to compile, create and execute: O2 Script to use Roslyn to Dynamically compile and execute a method

Using the code sample from Loading an assembly generated by the Roslyn compiler here is how to write it using O2's Roslyn API.

Version 3 (doing it all in 1 line : ) )

Version 1 (replicating the original script with some error detection):

Code Samples

Version 1:


Version 2 (simpler with no error handling):


Version 3 (in one line):

Simple O2 script to show Roslyn's Ast Error detection capabilities

Here is an simple O2 script that shows Roslyn's C# AST parsing capabilities in action, including how to show squiggly lines in the source code on the locations of those errors.

Note how fast the whole process is, and how Roslyn is able to handle broken C# files (my plan is to use this inside O2's REPL environment in order to give much better error messages when the AST is broken)

Video:

O2 Script:gist
https://gist.github.com/2903012

New Reddit Community for Cat.Net

I just created a new Cat.Net community (discussion forum) at Reddit.

You can access it at http://www.reddit.com/r/CatNet/ and if you are interested in this topic, please subscribe to it :)

To seed this community, I added my recent Cat.Net-related posts. and here is what it looks like:

For reference, here is the current introduction text (to be rewriten once there is more activity on this Reddit community)

"Since Microsoft stopped supporting its static analysis engine there is no place to talk and discuss about this engine and what can be done with it.

There is a lot of Cat.Net related activity happening (for example at the OWASP O2 Platform project), so the objective of this Community is to capture and enhance it"

I am a recent Redit user, but the more I used it the more I like it

 ... starting to think of creating a Reddi't community for the O2 Platform... :)

Using/Consuming Cat.Net's engine inside the O2 Platform (and outside VisualStudio)

A key component of the Real time Vulnerability Scanning using Cat.Net and Roslyn (SAST) PoC is the 'Real Time' part :)

The only way to achieve that type scanning speeds was to find a way to directly consume (i.e. access) Cat.Net scanning engine directly (i.e. in-process), instead of the command line (this will also be critical when we try to scan large projects, which will need to sliced, partially scanned and have its results glued/merged).

The document below shows how I did that and the evolution of the scripts.

Note that by these techniques, we have access to the entire Cat.Net scanning engine and can easily control its rules and execution workflow :)

In addition to the sheer fast-prototyping/development speed that we get with O2's REPL Scripting environment, one the key advantages of doing this type of development on top of the O2 Platform is the number of Static-Analysis-focused tools/APIs that already exists. For example note how I was able to easy consume/transform the Cat.Net findings into O2Findings (which can then be consumed/manipulated by the multiple O2Finding's viewers and tools)

asd

Using Roslyn to Load and (quickly) Compile C# Solution files (outside VisualStudio)

Now that we can use Roslyn to compile (and create assemblies) in real-time the next step is to use it to open up and consume C# Solution files (something that I've always wanted to do properly without needing to use VisualStudio or deal with the xml format used by the *.sln files).

Weird Roslyn.Utilities.SerializableDataStorage casting issue when using new version

While writing a couple Roslyn scripts I started having a weird problem when consuming the June 2012 CTP dlls.

I was having the exception:

Unable to cast transparent proxy to type 'Roslyn.Utilities.SerializableDataStorage'. Unable to cast transparent proxy to type 'Roslyn.Utilities.SerializableDataStorage'.

3 pdfs on how I added Roslyn support to the O2 Platform (lots of Roslyn Script examples)

As you saw on Video: Real time Vulnerability Scanning using Cat.Net and Roslyn (SAST) and Video: Using O2's Roslyn-based C# REPL Script environment I was able to add a good number of Roslyn capabilities to O2.

Video: Real time Vulnerability Scanning using Cat.Net and Roslyn (SAST)

Here is a pretty cool PoC of creating an environment for developers that helps them to know when they create a security vulnerability (in real time).

Video: Using O2's Roslyn-based C# REPL Script environment

Roslyn is a new Microsoft's C# compiler written in C# (with also a VB flavour)

It is actually quite a spectacular piece of technology and in a way there is a LOT of similarities between the REPL environment I created in the O2 Platform and the one Roslyn is trying to do. In fact I'm thinking of moving some of O2's scripting capabilities from the current SharpDevelop parser to this new Roslyn one since their object model seems to be MUCH more powerful (of course that I'm bias, but I think that the O2's REPL environment as a couple killer features missing in current version of Roslyn REPL, so let's create the best-of-both-worlds REPL environment :) )

I've started using Roslyn in O2, and the first thing to do was to create an REPL environment which you can see in action in this video.

I am documenting the 'scripts evolution' (as drafts), and here are two I just created:

• Using Roslyn (first tests).rtf  (series of scripts that ended up with the Roslyn-based REPL enviroment shown above)
• Using Roslyn (consuming Static and Sematic data).rtf (execution of the scripts from the high-recommended Whitepaper: Roslyn Project Overview )

I think there is HUGE potential with this technology, and one area that I'm specially interested in is doing Security-focused Static Analysis, with integration with TeamMentor and Cat.NET (see Video: Injecting TeamMentor into Cat.Net running inside VisualStudio )

You can get Roslyn from O2 (it will install the package from NuGet), but if you want to try the visual studio integration you need to get the Microsoft “Roslyn” October 2011 CTP

If you want to comment this post or video, please use this reddit thread

Some Roslyn References and articles:

• http://msdn.microsoft.com/en-us/roslyn  (main site)
• http://social.msdn.microsoft.com/forums/en-us/roslyn (forums)
• Two draft docs published by Roslyn team 
• REPL and Code Semantics: http://tinyurl.com/3kprrlk
• Scripting APIs: http://tinyurl.com/3ezykda
• Channel 9 videos:
• The Roslyn Project - Meet the Design Team 
• Going Deeper with Project Roslyn: Exposing the C# and VB compiler’s code analysis 
• http://blogs.msdn.com/b/visualstudio/archive/2011/10/19/introducing-the-microsoft-roslyn-ctp.aspx
• http://blogs.msdn.com/b/csharpfaq/archive/2011/12/02/introduction-to-the-roslyn-scripting-api.aspx
• http://visualstudiomagazine.com/articles/2011/11/16/the-roslyn-scripting-api.aspx

Light Table gets 300k in funding

Congratulations to the Light Table guys, they just got 300k in funding via Kickstarter.

As I mentioned in  Light Table - Amazing PoC of IDE I really think it is a great idea and I wish the them best of luck.

Some comments:

• This shows that it is possible to gather funding for software development projects (I did tried a half-baked attempt in 2010 which didn't really go anywhere: New funding model for O2's Development)
• It shows that there is a lot of desire for better IDEs since the main ones we have (VisualStudio and Eclipse) are really showing off their age (and size)
• That said the reddit comments on the 300k anoucement don't start in a very positive way
• It is a great way to build a community and get a list of interested parties. I would say that the current list of backers (all 7317 of them) is as valuable (if not more) as the 300k raised
• Although there is an OpenSource element of it, they also have a commercial license, which might have helped with the pledges since there is something tangible given in return (notice the different types of pledges)
• The money is going directly to the developers (from the community) which could be a good model for OWASP since it would be compabible with the concept of Why OWASP can't pay OWASP Leaders 

Big Security challenges with creating APIs for US Gov agencies

So Barack Obama Directs All Federal Agencies to Have an API 

Here is the White house memo (pdf) which mandates the implementation of  "Digital Government: Building a 21st Century Platform to Better Serve the American People" (pdf).

The good news it that at least security and privacy seems to be taken into account (with it's own chapter and focus)

I haven't read the document but after a skim, it looks like there is more focus on the non-secure-application-development 'security side' of these APIs.

And this could be an issue, since creating APIs is usually done by exposing internal systems or WebServices, which will now need to have much higher level of security than before (when they were connected to much less hostile environment).

I also like the use/focus on Privacy, since that will be a good way to drive coding and application changes.

This is a great opportunity for OWASP community to be involved since there is going to be a lot of API developers out there that could do with some help

Video: Injecting C# DLLs into Managed (C#) and Unmanaged (C++) processes

I was finally able to inject O2's REPL scripting environment into other processes (both managed and unmanaged)

Originally I was just trying to inject a managed dll into another .NET process, but the 4.0 .NET Framework will load up a CLR when it is not there, so this technique will also work on unmanaged (i.e. C++ based) applications.

Here is what happens in the video below:

• Inside O2,  run the Util - Inject O2 into other processes.h2 script (which will provide a nice 'Injection GUI')
• on the left there is a TreeView with a list of all current processes that have MainWindowHandle != 0 (a requirement at the moment)
• on the right there is a TreeView with a list of the loaded modules (i.e. dlls) from the selected process
• if you right-click on the process list you will be able: 
• bring the process window to front, 
• refresh the process list and 
• do the injection into the selected process
• Open LinqPad using the link from the 'Actions Panel' (on the top)
• After LinqPad opens, select it on the process list and click on the 'Inject O2 into Process' link (top left)
• An O2 REPL environment will apear. A number of scripts are dynamically created, showing that we now have complete control over the LinqPad GUI, by for example:
• injecting a new Tab with another O2 REPL Editor
• making the entire GUI pink or azure :)
• Close LinqPad and back in the Util - Inject O2 into other processes.h2 click on the 'Open Calc' link to open Calc.exe (an unmanaged windows app (i.e. no CLR))
• After Calc opens, select it on the process list and click on the 'Inject O2 into Process' link (top left)
• A come more scripts are written to show that we are indeed inside the Calc.exe process.

This open up a LOT of interesting integration possibilities, and it is a much more solid solution that the use used before which required O2 dlls to be copied into the target's application folder (see Injecting O2 into IBM Rational AppScan Standard and Creating a Pink version of IBM Rational AppScan Standard)

Related Posts:

• Video: Injecting compiled C# script into a Managed Process (using Snoop API) 
• Video of O2 Platform v4 Installer and quick demo of its scripting tools 
• Video: Installing O2's VisualStudio 2010 Add-In 
• Video: NodeJS O2 REPL environment (using node.exe) 

ConsolR Compilify and Roslyn

ConsolR is a really nice web-based C# execution environment, here is their description:

ConsolR enables you to execute C# code againt a running .NET 4.0 web application's app domain through a browser. Once installed, ConsolR will automatically configure itself during application start and is accessible through the "/consolr" path. This enables an interactive console session against for instance an application's production environment.

ConsolR is based on the Compilify project which provides the base REPL environment

Compilify uses the Microsoft's Roslyn engine which looks like a very powerful C# code analysis and manipulation (in fact from this Roslyn document I get the feel that I should rewrite O2's REPL Script environment using it)

Note: to install the VisualStudio I had to upgrade to the latest version of NuGet (which in VS 2011 SP1 can only be done after installing this HotFix)

Here it is in action:

Humm.... I wonder how much work it would be to convert this into an web version of O2 REPL environment :)