How does O2 compare with LinqPAD

(just got this question on this reddit thread, and here is my answer) 

There are a number of similarities, specially in the capability of quickly running C# scripts. Most of the differences come from the fact that they have are different approaches and development histories. LinqPAD (I think) was developed to help improving the execution and visualization of LINQ Queries, where the O2 Platform (and its REPL environment) was created to help to script application security analysis, and eventually O2 evolved into a full scripting and application development environment.

I can't really talk for LinqPAD (since I'm not part of it development team), but here is where I think O2 Platform provides extra value:

• O2 tries to integrate with as many tools as possible, in fact you can consume LinqPAD from O2:
  • Here is a video that shows how to dynamically script LinqPAD's GUI from O2's REPL: Video: Injecting C# DLLs into Managed (C#) and Unmanaged (C++) processes
  • Here are two LinqPAD related scripts: the first one downloads LinqPAD and the 2nd starts LinqPAD inside the current AppDomain and adds a new Menu to it
  • For reference here is a List of 3rd Party tools and APIs that O2 Supports
• O2 is Open Source and has AutoCompletion support (which LinqPAD has on the paid-for-version)
• O2's scripts are designed to create mini-tools. I.e. the idea is that once you finished a script you package it as a stand-alone script (with *.h2 extension) or an .exe. In fact the O2 Platform, is basically a collection of tools, GUIs, technology, and APIs developed using this C# REPL environment. For example the main O2 REPL environment exposes a WinForm's Panel Control which is 'feed' to the script so that it can be used while developing a GUI (with 1 line change to use Form Control and make it a stand-alone tool))
  • here is a simple example: Solving the problem of 'path too long error' using O2 tool
  • here are a bunch of examples based on Microsoft's Roslyn: Multiple Roslyn based tools (all running Stand-Alone outside VisualStudio)
  • If you like videos, see: 39 O2 Platform videos with 12k YouTube views
  • Most of O2.Platform's functionality is made of scripts, which are stored in the O2.Platform.Scripts folder which is available in this Git repository . See also O2 Platform Blog for a large number of script examples.
• One core idea of the O2 REPL environment is that it should be used everywhere, since it provides real time feedback/coding-loop while developing or testing new ideas. For example I have already used it for:
  • Browser automation (with extended support for WatiN and .NETs Browser control)
  • Graph automation (via GraphSharp )
  • .NET Process Injection (via Snoop)
  • VisualStudio Extension development and coding
  • Roslyn compilation and refactoring
  • SharpDevelop, NRefactory and Mono.Cecil code driven analysis (see O2's .NET Static Analysis scripts)
  • Java static analysis via IKVM
  • JavaScript analysis via JInt
  • DSL and Custom Parsers via Irony Parser
  • IronPyhon integration and scripting
  • Managed Debugger scripting (via Mdbg, Sun Of Strike and SoSNET)
  • Git Support via NGit
  • and more: LinqPAD, NUnit, AppScan Standard, NetSparker, Chrome, TeamMentor, LessMSI ,etc....
• The REPL environment can also be used to expose (or debug) objects on applications being developed. There is an 'Script_Me(...)' extension method that open up a new WinForms window with the 1st parameter (the 'this' value) being 'feed' to the REPL environment.
• In fact I develop O2 in O2 :) , and I frankly find VisualStudio really painful to use (and slow). Although now that there is an O2 VisualStudio Extension that allows the REPL of VisualStudio itself, this might improve. You can download it from : O2 Platform for VisualStudio 2010
  • .... I really like the concept of '...the IDE is the Application...' where you also have real time REPL environments in IDEs or tools that provide rich funcionality
• Using this REPL environment I also developed this pretty cool PoC Real-Time Vulnerability Feedback in VisualStudio which is closer to how I think we need to provide security information to developers (when compared with what we do now which is an out-of-band process)
• Included in the O2's REPL environment is an Fluent API, which Is called Fluent# and is available at CodePlexor NUGet (note that these are a bit out of date with the current distribution of O2). This API really makes the O2's REPL environment powerful since you can execute quite complex tasks with only a few lines of code (its still C# , but there is extended use of C# Extension Methods to reduce the amount of code needed). You will see the Fluent# APIs in action, on most (if not all) O2 scripts.
  • Of Fluent# APIs and Extension Methods, I would stand out the Reflection and the WinForms APIs. The reflection one will give you easy access to any .Net assembly, class, method, property, field, enum (on disk or in memory) regardless of its public/private/internal status. The WinForms one will give you a thread-safe (very important feature) way to create and manipulate WinForm controls (if you look at the code behind most O2 scripts (including the C# REPL environment) you will see this API in action
• Another powerful capability that you get from the O2 REPL environment is that the return value is viewed in a PropertyGrid which allows for a very organic growth of the script under development (although this might be a case of the workflow I use with O2 since LinqPAD result window is also quite mature and powerful)
• A very powerful capability of O2's REPL environment which is hard to understand until experienced, is its data consumption and analysis capabilities. Due to .Net's powerful serialization and XML support, it is possible to script (and visualize) multiple data-transformations (with all steps stored as .NET Serialized objects)
• There are many more features that exist in O2's and its REPL environment, and the best way for me to help you is if you have a particular problem that you are trying to so solve, please post the details here and I'll point you to the right direction (or to an existing script)

I would like to stress that that I don't view O2 as a competitor / replacement for LinqPAD!

If anything I want to make LinqPAD better and be able to integrate it's capabilities with other tools. My objective with O2 is not to invent all wheels, but to use the best possible technology to solve the problem at hand.

For example here is what I think LinqPAD does better: * it is more focused and mature * there is a better GUI to provide examples (we're trying to create a similar cook-book style for O2) * the tabbed interface for scripts allows for multiple script development in the same GUI * the GUI looks better, is more complete and shows its age (in a good way) * there are a couple more scripting options * built-in visualizers for LINQ queries * better help and support website

Finally, since the O2 Platform and its C# REPL environment are Open Source (https://github.com/o2platform), if you have some cycles we could do with some help :)

Multiple Roslyn based tools (all running Stand-Alone outside VisualStudio)

Over the past months I have been using Rolsyn in a number of scenarios, creating in the process multiple O2 Platform tools which might be useful to new Roslyn users.

Here are the stand alone exes, all about 9Mb since they contain the Roslyn dlls and the O2 REPL Script environment as embedded resources.

• PoC - Roslyn C# ScriptEngine Execute v1.0.exe  - provides a REPL for Roslyn
• Util - View Roslyn AST Graph v1.0.exe - good to see the AST tree created by Roslyn
• Util - Exe Creator (using Roslyn Compiler v1.0.exe - Nice tool to create stand alone exes
• Util - Roslyn - Compile Solution and view errors v1.0.exe - compile entire solutions
• Util - Roslyn - OnSave Compile Solution (with REPL) v1.0.exe - adds a REPL for Roslyn objects
• PoC - Roslyn refactor on InitializerExpressions v1.0.exe - shows an refactoring example
• Util - Roslyn AST Refactoring via REPL Script v1.0.exe - gui to develop refactoring scripts
• Util - Roslyn Cat.Net- Compile and Scan Solution v1.0.exe - shows integration of code compile and security scanning (Cat.NET will be download the first time this tool is executed)

The titles of these tools should give a good clue on what they do, but if you want me to expand on them (or how the script works), just add a comment to this post with your request/questions.

Note that you can run all these scripts using the latest version of the O2 Platform (just search for 'Roslyn' scripts).

If you are trying to get your head around Roslyn, the best one to try is the first one (PoC - Roslyn C# ScriptEngine Execute v1.0.exe) which is a nice REPL environment. Here is a post I wrote a while back that shows this tool in action: Video: Using O2's Roslyn-based C# REPL Script environment

Solving the problem of 'path too long error' using O2 tool

Download: Util - View O2.Platform.Scripts file length v1.0.exe
View: Script Source Code

---

Amazingly in 2012 windows still complains for paths bigger than 240 chars. Now that O2 can be easily installed from a single exe, there is a normal case where O2 will be started from path that is a big long (lets say more than 100 chars). This created a problem since the O2 temp dir would be created from that  path , and there were a number of scripts that (since I like to use verbose file names) would break the 240 char limit (which would make the unzip of the O2.Platform.Scripts folder to fail).

So to help me solve this, I wrote a quick tool that gives me a quick view of what files are bigger than a particular size.

Here is the tool in action before I renamed a bunch of O2 files:

As you can see there were 198 files whose path was bigger than 120.

This tool (which you can download from here or see the source code below), has a number of nice features:

• Slider to select the depth to filter the loaded files
• Loads files once, which gives a nice real-time fell when moving the slider
• Allows drag n' drop of folders to load all files from it
• Auto filters/removes files in .git directories
• Context menu (on the file list) to open the parent folder of the selected file and to reload the files

Here is a screenshot of what the O2.Platform.Scripts folder looks like now, after a number of script renames (which took me a couple minutes to do):

Script Source Code:

Please test: Stand-Alone version of O2 Platform and C# REPL environment

I just published a stand-alone version of the main O2 Platform GUI and its C# REPL environment, and if would be great if you took them for a test-drive before a wider release (namely the rest of the OWASP community).

You can download the O2 GUI (and its scripts) from: http://tiny.cc/O2Platform   (this is the same you would get if you installed O2 from GitHub)

You can download the C# REPL environment from http://tiny.cc/REPL  and read about it at this post: C# REPL Script Environment 

Let me know how it goes, and if have issues please use the 'Report Bug' feature from the Help menu (which will send an email to me with your logs)

Thanks

C# REPL Script Environment (new page)

I just added a new top level page to this blog with details about the C# REPL Script Environment that is the key to O2 Platform's powerful scripting capabilities. Please read it for more details

You can download the latest version from: http://tiny.cc/REPL and this is what it looks like:

repl.it great online REPL Environment

Just found http://repl.it/ and it looks really good.

They provide an online REPL for 16 of languages, and their UI feels really nice (specially the part of providing code samples)

Interestingly they don't support C#,  so maybe there is a good opportunity to use O2 to enable that :)

The source code of that website is at GitHub: https://github.com/replit

Ed on SecurityInnovation's focus on Application Security

Yeah it's Marketing but its actually quite a nice description of SI (Security Innovation) focus on Application Security.



One key concept are the 3 pillars of Secure SDL: Standards, Education and Assessment:














Here are a couple more videos on these 3 pilars (click on the each pillar to see them)

Good progress on VisualStudio Extension and couple more issues

Following from Adding an item to VisualStudio's Error List Window I've made quite a lot of progress on my VisualStudio Extension efforts, and have now created some really nice APIs and PoCs (if you want a sneak preview you can get it from the VisualStudioGallery: O2 Platform for VisualStudio 2010)

Here is where I'm am at the moment: 

• My implementation is based on the Package and MEF objects (and the services I can get from there)
• I now have a full REPL environment inside VisualStudio, so I can code VisualStudio dynamically :) (i.e when I'm working on my Extension I don't need to recompile the Extension assembly everytime I make a code change or want to try some DTE , MEF or WPF action)
• I'm able to get an DTE object by using: packageObject.GetService(typeof(EnvDTE.DTE))  (I use a similar technique to get and ErrorListProvider, IVsUIShell, OleMenuCommandService objects)
• In this case the packageObject implements Package and is marked with the [PackageRegistration(UseManagedResourcesOnly = true)]   attribute
• Once I had the ErrorListProvider object (created using ErrorListProvider(packageObject) ) , I was able to add items to it 
• To create windows I used the packageObject.CreateToolWindow(typeof(WindowPane_WPF), index++) ; 
• The WindowPane_WPF is a class I added which implements ToolWindowPane and sets its Content to a vanila WPF grid (I also have a WindowPane_WinForms which sets the content to a System.Windows.Forms.Integration.WindowsFormsHost control
• In order for this to work, I had to preregister them on packageObject classes with:
• [ProvideToolWindow(typeof(WindowPane_WPF)     ,  MultiInstances = true,  Style=VsDockStyle.Linked, Orientation = ToolWindowOrientation.Top, Window=EnvDTE.Constants.vsWindowKindOutput)]
• [ProvideToolWindow(typeof(WindowPane_WinForms), MultiInstances = true, Style = VsDockStyle.Linked, Orientation = ToolWindowOrientation.Top, Window = EnvDTE.Constants.vsWindowKindSolutionExplorer)]
• A number of problems I had initially where caused by threading issues. My scripts run a separate thread and there where a number of cases where the request needs to be executed from the main VisualStudio thread (although I usually got weird COM errors which didn't point that way). I dealt with this problem by creating an extension method that used the Dispatcher object from System.Windows.Application.Current
• I also had issues getting the Events I subscribed via the DTE.Events.BuildEvents callbacks, until I pinned the object DTE.Events.BuildEvents to a static variable in my class
• I'm adding menus dinamically using dte.CommandBars["MenuBar"] (since I didn't want to hard code the menus in my extension dll)
• I'm using the [ProvideAutoLoad(UIContextGuids80.NoSolution)]  on my packageObject to get a callback on VisualStudio load
• I implemented a couple wrappers on some MEF exports which give me easy access to the WPF TextEditor: 
• IWpfTextViewCreationListener (allows the manipulation of Code content and add new Adornments)
• IWpfTextViewMarginProvider (allows adding a new margin or section to the Code provider)

Here are the issues/questions I currently have:

• How to get a live instance of the AddIn object. I currently have a DTE object, but don't know where to get the EnvDTE.AddIn
• This is the object that is passed as the addInInst parameter of the OnConnection method from the VsConnect : IDTExtensibility2, IDTCommandTarget  (I needed that in order to be able to correctly set the variables used by consumed add-ins
• How to control the position of a new ToolWindowPane when created programatically.
• If you noticed from the ProvideToolWindow attribute above, I use it to define the place where new instances of this window are created. My issue in that at the moment I don't seem to be able to set them programatically (specially the target Window GUID).
• Faster way to add menu items. The dte.CommandBars works ok, but it still takes about a couple secs to do it (and it hangs the GUI) so, I was wondering if there was another way. I have access to the main WPF VisualStudio window, so maybe that will be faster. Note that I don't need to preregister these items as MenuCommands, I just want to get Action callback when they are clicked
• Is the  [ProvideAutoLoad(UIContextGuids80.NoSolution)] the best way to get some code to run on VisualStudio load?