I have to admit that I really think this is a good idea. Shay Chen (@sectooladdict) has put together a project to evaluate Web Application Vulnerability scanners. He calls it WAVSEP.

The project is currently being hosted on Google code.

Taken from the project description:

Project WAVSEP currently includes the following test cases:

Vulnerabilities:

• Reflected XSS: 66 test cases, implemented in 64 jsp pages (GET & POST)
• Error Based SQL Injection: 80 test cases, implemented in 76 jsp pages (GET & POST )
• Blind SQL Injection: 46 test cases, implemented in 44 jsp pages (GET & POST )
• Time Based SQL Injection: 10 test cases, implemented in 10 jsp pages (GET & POST )

False Positives:

• 7 different categories of false positive Reflected XSS vulnerabilities (GET & POST )
• 10 different categories of false positive SQL Injection vulnerabilities (GET & POST)

Additional Features:

• A simple web interface for accessing the vulnerable pages
• Sample detection & exploitation payloads for each and every test case
• Database connection pool support, ensuring the consistency of scanning results

Also – on Shay’s blog he has posted a comparison of majority of Web Application Vulnerability Scanners (both Commercial & Open-Source).

An argument could be made that the project’s focus in terms of vulnerabilities is too narrow (not covering dom-based XSS as an example), and as much as I can see the argument I’m sure the argument is coming from people that are sitting on their ass not doing anything for the community like Shay is doing.

In my opinion – I think this is good for the community and I hope that the project gains some good momentum and really grows.

Good work Shay!