Loading, parsing and consuming unstructured data (i.e. password hashes from google)

Following Generating an small MD5 Rainbow Table in C#,  the objective of this post is to show how to consume unstructured data (to be used for example on a fuzzing brute-force login attempt, which you will see on the next post)

Now that we have a way to resolve MD5 hashes, we need a couple hashes to test it out.

Unfortunately (for the affected users) there are a HUGE amount of password hashes out there. For example, take a look at the http://www.exploit-db.com/google-dorks/9/ page:



Clicking on the 2nd one (shown above in red) take us to this page:



… which links to this Google results page: https://www.google.com/search?q=ext:sql%20intext:@hotmail.com%20intext%20:password&safe=active



… with the first hit in the 791 list of results returned from Google being:



… which contains a large number of user IDs, username, hashes, emails and names.

WTF Moment: I know this shouldn’t shock me these days, but ...

In this case what we are after is to extract the username and hashes from this list, so let’s go into an O2 Platform REPL environment, and download this file:



Note: in cases like this (where we first need to load some data) it is better to use the o2Cache object, so that there is only one load (per O2 process):



To get an idea of the quality of the data in the lines, lets see how many split sizes we have:



In this case there are 10 lines that will not split to a constant amount



... and 358 lines splitting into 9 blocks.



Lets look at the first line (note that the results in the Output window are sorted alphabetically):



Here is how to get a clean hash:



With the username being the 2nd value:



Here is a Dictionary indexed by Username:



In cases like this, it is better to use the Items class (part of FluentSharp.Core) since it is serializable:



Another option is to save the data as a \tab delimited file (so that it can be consumed by our next script)



Using LINQ

If you are into LINQ queries, here is another way this data can be sliced:



And since the FluentSharp.WinForms TableList, supports LINQ queries, here is a nicer way to visualize the data:



… specially when we add the rest of the data collected:



(note how now it is very easy to detect the odd entry that has a bad format)

Source code example #1) Code that creates the GUI shown above (with the user account details and hashes)

//var topPanel = "{name}".popupWindow(700,400);
var topPanel = panel.clear().add_Panel();

var rawData = "rawData".o2Cache<string>(()=>"http://reflets.info/hcsr.gov.sy_users.sql".GET());

var lines = rawData.fix_CRLF().lines();

var userMappings = 
    (from line in lines
     let splittedLine = line.remove("'","(",")").split(",")     
     select new {
                   userId    = splittedLine.first(),
                   username  = splittedLine.second(),
                   hash      = splittedLine.third(),
                   email     = splittedLine.fourth(),
                   firstName = splittedLine.fifth(),
                   lastName  = splittedLine.value(5),
                   flag1     = splittedLine.value(6),
                   falg2     = splittedLine.value(7) });

topPanel.clear().add_TableList("User Mappings")
                .show(userMappings);
return userMappings;

Resolving hashes using ‘small MD5 Rainbow Table’:

Since the objective is to resolve the passwords, lets now load up the small Rainbow Table we created in the previous post:



It better to load the heavy ones (with the 40+ Mbs) into an o2Cache object (using a Dictionary<string,string> since it has much better performance that Items)



Final step is to use this rainbowTable object to resolve the hashes previously mapped:



This will create this table using the ‘(10x numbers - 6x length) M5_for_10_chars_with_6_depth.txt’ 44Mb file:



and this table using the ‘(10x numbers and 26x letters  - 4x length) M5_for_36_chars_with_4_depth.txt’ 65Mb file:



Which basically means that our simple rainbow tables  was able to ‘decrypt’  88 (64 + 24) MD5 passwords (yes, we are not decrypting it, just brute-forcing it).

Question: Is anybody notifying these users?

As you can see it is quite easy to calculate these passwords, and it would be nice (if you are the affected user) to be notified that this is happening (specially for users that reuse passwords on multiple sites)


Source Code example #2)  Code that creates the GUI shown above (with the passwords resolved)

//var topPanel = "{name}".popupWindow(700,400);
var topPanel = panel.clear().add_Panel();

var rawData = "rawData".o2Cache<string>(
                ()=>"http://reflets.info/hcsr.gov.sy_users.sql".GET());

var rainbowFolder = @"C:\Users\o2\AppData\Roaming\OWASP_O2_Platform_5.3\8_12_2013\MD5_Hashes";

var rainbowTable = "rainbow_2".o2Cache<Dictionary<string,string>>(
                        ()=>{
                                // this file finds 64 passwords
                                var md5Hashes_File = rainbowFolder.pathCombine("(10x numbers - 6x length) M5_for_10_chars_with_6_depth.txt"); 
                                // this file finds 24 passwords
                                //var md5Hashes_File = rainbowFolder.pathCombine("(10x numbers and 26x letters  - 4x length) M5_for_36_chars_with_4_depth.txt");
                                var items = new Dictionary<string,string>();
                                
                                foreach(var line in md5Hashes_File.fileContents().lines())             
                                {
                                    var splittedLine = line.split("\t");
                                    items.add(splittedLine.first().upper(), splittedLine.second());
                                }
                                return items;
                            });

var lines = rawData.fix_CRLF().lines();
var pwdCount = 0;
var userMappings = 
    (from line in lines
     let splittedLine = line.remove("'","(",")").split(",")     
     let hash         = splittedLine.third().upper()
     let password      = rainbowTable.hasKey(hash) ? rainbowTable[hash]
                                                         .removeLastChar()
                                                         .removeLastChar()+"**" : ""
     let foundIndex      = password.empty() ? "" :  (++pwdCount).str()
     select new {
                   userId     = splittedLine.first(),
                   username   = splittedLine.second(),
                   hash       = hash,
                   password   = password,
                   foundIndex = foundIndex,
                   email      = splittedLine.fourth(),
                   firstName  = splittedLine.fifth(),
                   lastName   = splittedLine.value(5),
                   flag1      = splittedLine.value(6),
                   falg2      = splittedLine.value(7) });

topPanel.clear().add_TableList("User Mappings")
                .show(userMappings);
return userMappings;