Saturday 27 February 2016

Is Quality is a measure of how successful a product is in what it is SUPPOSED to do?

Here is a question I received on the concept "Application Security can be used to define and measure Quality" (slides here
Quality is a measure of how successful a product is in what it is SUPPOSED to do.

AppSec is a measure of how many and what things product does that it is NOT SUPPOSED to.

These two are not related. A startup may have a good quality product full of security holes, and a bank may have a highly secure product that is also of great quality.
I think that measuring Quality by only looking at the success rate of a product is a very narrow definition of Quality.

For example here is how Wikipedia defines Quality which shows that Quality is a very wide issue.
In business, engineering and manufacturing, quality has a pragmatic interpretation as the non-inferiority or superiority of something; it is also defined as fitness for purpose. Quality is a perceptual, conditional, and somewhat subjective attribute and may be understood differently by different people. Consumers may focus on the specification quality of a product/service, or how it compares to competitors in the marketplace. Producers might measure the conformance quality, or degree to which the product/service was produced correctly.

Support personnel may measure quality in the degree that a product is reliable, maintainable, or sustainable. A quality item (an item that has quality) has the ability to perform satisfactorily in service and is suitable for its intended purpose.

Sherif Mansour on the OWASP leaders list had a great comment which was:
My take on it is that not all software quality issues are security issues, but all software security issues are software quality issues. If thats the case then security is certainly a measurement of quality.
If we go back to the pollution analogy I make in the sides, in the past, there were many successful products (loved by customer) that also had a high number of external costs. These costs were not (initially) paid by the end customer. Change only occurred, when we (as a society and a large number of individual cases) started to really understand the costs of that pollution.

In fact, I would argue that IT IS A Quality issue, when a product does things that it is NOT supposed to do. Specially (and this is very important), when the makers of that product don't even understand or are aware of those 'extra features'

The reality is that because most companies can't really measure the lack of Quality of their products (i.e. the number of un-intended side-effects, the cost of making changes, the lack of understanding on how the product/app/web-service really works, the lack of up-to-date documentation, the lack of testability, the weak deployment process, the number of bugs discovered by end-users, the interdependencies between the multiple modules, etc...), they focus on the end result (pleasing the customer) and handle the side effects (i.e. incidents) as they come.

The problem that we have now in 2016, is that software development has been doing this for so long that the number of criminal attackers (which are the ones that cause really damage) are starting to reach critical mass.

Note: While trying to find the source of the 'running faster then you vs the bear' story, I found this page which has the perfect analogy:

Two strangers were out backpacking in the woods. They came upon each other and decided to walk the next bit together. Around a bend in the trail they came face to face with a bear. One stranger drops to his knee, fetches his running shoes from his backpack and begins the removing his hiking boots. The other stranger just stares and says, "There is no way you can run faster than that bear."

The kneeling stranger stands up and replies, "I don't have to be faster than the bear. I only have to be faster than you."

We discovered (in a mental exercise, not from experience!) that that little anecdote does not apply to sharks/barracuda and skin diving. The shark/barracuda is so much faster than the swimmers that it can circle several times and even take taste tests before it decides. This is surely a metaphor for something in our profession.