Testing, testing

OK, we’ve used that as a title before. However, it seems quite apposite as this is my first published blog here, and it’s related to anti-malware testing. (See what I did there? :-D)

This is actually a retread of my heavily re-edited blog at securiteam. But since it concerns (obliquely, for legal reasons) an issue that some of us discussed at VB 2009, I’m quite happy to repurpose some of it here.

Principle 3 of the AMTSO (Anti-Malware Testing Standards Organization) guidelines document (http://www.amtso.org/amtso—download—amtso-fundamental-principles-of-testing.html) states that “Testing should be reasonably open and transparent.”

The document goes on to explain what information on the test and the test methodology it’s reasonable to ask for.

So my first question is, is it open and transparent for an anti-malware tester who claims that his tests are compliant with AMTSO guidelines to decline to answer a vendor’s questions or give any information about the reported performance of their product unless they buy a copy of the report or pay a consultancy fee to the tester?

Secondly, there is, of course, nothing to stop an anti-malware tester soliciting payment from the vendors whose products have been tested both in advance of the test and in response to requests for further information. But is he then entitled to claim to be independent and working without vendor funding? In what respect is this substantially different to the way in which certification testing organizations work, for example?

AMTSO will be considering those questions at its next meeting (in Prague, next week).  But there are a lot of people inside and outside AVIEN who are seriously concerned with testing standards, as an aid to evaluating products for use in their own organizations, or because they have a vocational interest in making or supporting products that are impacted by fair/unfair or good/bad testing, and I’d be more than a little interested in hearing your views.

David Harley CISSP FBCS CITP
Chief Operations Officer, AVIEN

One thought on “Testing, testing

  1. Pingback: AVIEN and Testing | ThreatBlog

Leave a Reply