Here are some comments on the comments made on the .. thread:
@Andrew: you touched on a very important point which is the importance of the 'operator' (i.e. the knowledgeable user). As per the points you make, I really think that we need to take into account its impact.
For example, we should measure the following:
A) can a tool find Issue XYZ using every trick in the book? (i.e. tool + operator + customization)
B) can the tool be customized so that It can find that issue XYZ without requiring an operator?
C) once customized can the tool find similar issues (to XYZ) automatically
D) can the tool find issue XYZ (and similar) automatically without any customization
You comment was not boring at all :) Are you interested in helping on creating this test/evaluation environment?
@Romain: 1. Yes I am focusing on 'traditional' web frameworks (Java, .NET, PHP, Python, Ruby, etc..) so I did not mention the C++ ones
2. we are going to have to agree to disagree on this one :) . In fact in your answer you already provided a couple examples of why public and pear-reviewed test cases are so important :)
3. Yes NIST (via SATE) has been trying to improve visibility into this space (and thx for the link to the report which I will read the report properly http://samate.nist.gov/docs/NIST_Special_Publication_500-279.pdf), as we move along this process I really want to make sure that all the great work done already by you (and others at NIST) are taken into account and used. My plan is to use the O2 Platform to address some of the issues you had (for example having pragmatic ways to compare the different tools results and allow the use of multiple sets of data (like results created using advanced customization and results created using NO customization))
----
Also, I just found this comment from Neil MacDonald For Static Application Security Testing, Frameworks Matter which is absolutely spot on! Frameworks ARE the key issue on Static Analysis, because without it, the tools have no idea what is really going on:
from Neil's post: "...Bottom line: if a static analysis tool doesn’t have an accurate understanding of the underlying framework, it can’t generate an accurate model and the analysis will be incomplete, resulting in at least false negatives. This is true for any of the languages a SAST tool may scan, not just Java and this is true of any SAST tool – whether analyzing source code, byte code or binaries.
When a SAST vendor says they support “Java” or any other language, dig deeper...."
