OK, we’ve used that as a title before. However, it seems quite apposite as this is my first published blog here, and it’s related to anti-malware testing. (See what I did there? 
)
This is actually a retread of my heavily re-edited blog at securiteam. But since it concerns (obliquely, for legal reasons) an issue that some of us discussed at VB 2009, I’m quite happy to repurpose some of it here.
Principle 3 of the AMTSO (Anti-Malware Testing Standards Organization) guidelines document (http://www.amtso.org/amtso—download—amtso-fundamental-principles-of-testing.html) states that “Testing should be reasonably open and transparent.”
The document goes on to explain what information on the test and the test methodology it’s reasonable to ask for.
So my first question is, is it open and transparent for an anti-malware tester who claims that his tests are compliant with AMTSO guidelines to decline to answer a vendor’s questions or give any information about the reported performance of their product unless they buy a copy of the report or pay a consultancy fee to the tester?
Secondly, there is, of course, nothing to stop an anti-malware tester soliciting payment from the vendors whose products have been tested both in advance of the test and in response to requests for further information. But is he then entitled to claim to be independent and working without vendor funding? In what respect is this substantially different to the way in which certification testing organizations work, for example?
AMTSO will be considering those questions at its next meeting (in Prague, next week). But there are a lot of people inside and outside AVIEN who are seriously concerned with testing standards, as an aid to evaluating products for use in their own organizations, or because they have a vocational interest in making or supporting products that are impacted by fair/unfair or good/bad testing, and I’d be more than a little interested in hearing your views.
David Harley CISSP FBCS CITP
Chief Operations Officer, AVIEN
Tags: AMTSO, AMTSO compliance, AMTSO Guidelines, AMTSO principles, Anti-malware Testing, Anti-Malware Testing Standards Organization, AVIEN, certification testing, Virus Bulletin 2009
[...] Testing, Testing (yes, Andrew and I did use that as the title of an ESET conference paper!) asks whether an anti-malware testing organization can claim that its testing is "open and transparent" (i.e. in accordance with principle three of the AMTSO fundamental principles of testing document) if that information is only made available for a fee to the company that makes a tested product, and whether making such a charge before the test really qualifies as "vendor independent". (These are issues that are likely to come up for heated discussion at the AMTSO Workshop in Prague next month.) [...]