Category Archives: AMTSO

Anti-malware testing resource

Testing security software has been part of my life for almost as long as I’ve been involved with computing: not only in terms of evaluating the efficiency of products and technologies for the organizations I worked for, but as an independent tester (especially of Mac AV) way back in the 90s. I stopped testing when I began to foresee a time when I simply wouldn’t have the time or resources to do justice to what even then was a difficult job. There was a time around 2006 when I was discussing roles on both sides of the vendor/tester divide, but for better or worse, I went over to the dark side and focused on supplying consultancy services to the AV industry (primarily ESET). However, I didn’t escape the testing controversy, being involved almost from the beginning in in the Anti-Malware Testing Standards Organization (AMTSO) and even serving for nearly three years on its Board of Directors.

While I’m still in sympathy with the ultimate aims of AMTSO, when the organization decided that the blog I set up on behalf of the Board no longer met its needs, I found myself needing a platform where I could continue to provide independent commentary on testing issues. Hence, the Anti-Malware Testing blog. While most of the material there right now consists of articles I originally posted to the AMTSO blog (as an independent commentator, not on behalf of AMTSO) that are no longer available elsewhere, it’s primarily intended for new articles. (I am, however, currently working on a resource page similar to the one on the extinct amtso.wordpress.com blogsite, with links to useful articles, papers and other testing-related resources.)

Right now there are three new articles there:

  • Explaining the Anti-Malware Testing Blog is what the title suggests it is.
  • Imperva-ious to Criticism looks at Imperva’s continued defence of its flawed quasi-test methodology, which inappropriately tried to use VirusTotal as a measure of the detection abilities of anti-virus/anti-malware products.
  • A Little Light Relief is a little lighter in tone. Literally. It points to an entertaining article by Robert Slade. After all, if I had to take testing seriously all the time, I’d get very depressed.

Compliments of the season to all our readers, and very best wishes for the New Year.

David Harley CITP FBCS CISSP
Small Blue-Green World/Mac Virus
ESET Senior Research Fellow

AMTSO members’ workshop

Don’t you hate it when people send you “reminders” meaning “here’s something I should have told you about before”?

Well, here’s something that would have been a reminder if I’d actually blogged it here before. 🙂

The next members meeting of AMTSO (the Anti-Malware Testing Standards Organization, a body whose intent to raise AV testing standards is very dear to the hearts of some of us here), is at San Mateo, California, on February 10th-11th.

More details, including a preliminary agenda, at http://www.amtso.org/meetings.html.

David Harley CITP FBCS CISSP

More AMTSO stuff

They say there’s no such thing as bad publicity, though quite who ‘they’ are, and why ‘they’ would make such a clearly daft statement is beyond me. It seems that AMTSO has had it’s fair share of bad publicity recently –  a further example is the piece by Ed Moyle over on his blog at http://www.securitycurve.com/wordpress/archives/1773. It’s a long article, but it does show that Ed clearly doesn’t understand (or doesn’t want to accept) what AMTSO is trying to do – maybe that does just mean that AMTSO needs a better PR representation. Anyway, once again Kurt Wismer (or perhaps I should adopt his anti capitalist rendering and use kurt wismer) has provided some excellent analysis of Ed’s piece over on his blog at http://anti-virus-rants.blogspot.com/2010/07/i-see-standards-organization.html

There’s little more that really needs to be said from my perspective. For the record, I personally agree with Kurt (just can’t seem to get my head around the ‘kurt’ thing), in his analysis of the NSS report done by AMTSO – which seems to be at the root of this whole anti AMTSO campaign. The central point is that NSS did a good job, and came very close to the ideal – (if you haven’t read the review, then it’s here). It’s unfortunate that that has been taken as a negative thing or a slight against them to say that they did not fully meet the ideal standard set by AMTSO – it was still far better than many other tests, and I have every hope that people are sensible enough to recognise that. It’s hard for me to see quite how Ed jumps from that report to an accusation that AMTSO is ‘Slapping the labs’ – an argument even harder to see when a lab like Dennis Technology Lab (who have very similar methodology to NSS) voluntarily submitted their own test for the AMTSO review process (see the report here).

If there’s one thing we can learn from this, it’s that it does seem that there’s a double standard here – testers can criticise AV vendors with impunity in their reviews and tests of AV products, but when someone tries to apply that same process and rigour to the tests done by those testers, that is somehow anathema. Personally, I think that’s shoddy thinking, and I have no doubt that AMTSO will continue to strive, as it has done from inception, to provide the public with an insight into tests, and to support good testing practice (and incidentally point out less than ideal practice where needed).

Andrew Lee
AVIEN CEO / CTO K7 Computing

Virus Researchers are community outcasts

Lately I’ve been reading a lot of blogs and articles attacking and defending AMTSO and their attempt at establishing standards for the testing of counter-malware products. Unfortunately I think BOTH sides are missing the larger picture here. AMTSO was formed to address some critical shortcomings in the testing of counter-malware products: some tests were arguably unethical, most unscientific and some just poor from the word go. So where does the dissent come from? It comes from the very people who done or supported those poor non-science based tests. Yet it goes beyond that. The people who are condemning AMTSO and their efforts are in some cases well respected in the general security arena, and are very knowledgeable, and this is the rub. These people, most people in academia, and in management as well do not recognize Malware research and prevention as a specialty niche. They attempt to apply the same rule-set to fighting a malware outbreak as they do a simple intrusion, and see nothing wrong with that solution.

A majority of people not engaged in the Malware field as a profession still feel that the average Security Professional has the same knowledge and skill sets as used by the Counter Malware Professionals. Unfortunately nothing can be further from the truth. It goes beyond the abilities and skills for reverse engineering, programming, and identifying abnormal network traffic. This argument goes back to at least the early 1990’s when in a panel discussion a firewalls specialist attempted to answer a question about a virus. On that panel was Wolfgang Stiller, creator of Integrity Master Anti-Virus, Wolfgang interrupted him saying along the lines of “look I’m here for the virus questions, I would never presume to speak with authority or experience on firewalls issues, but you presume to have the same experience and expertise with viruses that I do, and that is mistaken”. Similar exchanges have happened on other panels with people such as Robert Vibert and Rob Rosenberger, among others. These are also the same people who demand that anti-malware products protect against threats that are not viruses, nor are they specifically malware, but “Potentially unwanted programs”. So this is not a new phenomenon. The question in my mind is why does it still exist?

Anti-Virus ‘Experts’ helped establish the disaster recovery field, and were among the very first to teach classes in th at subject. It was the Anti-Virus Researchers who developed the field of Computer Forensics, in both cases it was the Anti-Virus field that had the necessary expertise and skill set needed to fill the holes and expand the career field. So now that Disaster Recovery, and Computer Forensics are recognized as specialty fields and given a high degree of respect from schools and management, what happened to the Anti-Virus researcher? Their mindset is not of an operational nature, they bore easily, some may even say they have attention deficit disorder (ADD), yet they are anal about doing things the same way every-time. They dwell on minutiae, arguing to the point of splitting hairs. I sometimes think some of my colleagues can SEE the traffic on the wire in their minds eye. Yet with all this contribution to the Computer Security Community they are still (almost purposely) maligned and misunderstood. At a Virus Bulletin Conference, I stated that we as a community must take action or go from the ranks of professional, to the ranks of the tradesmen. I still don’t know what action that is, or how to go about it, but AMTSO is a good step in that direction, and the naysayers need to start looking outside their comfort zone and realize they know enough to be dangerous and not enough to be helpful at this point.

Ken Bechtel
Team Anti-Virus
Virus Researcher and Security pontificator

AMTSO – The herding of the cats continues

I’ve spent the last couple of days in Prague (never a real hardship) at the AMTSO (Anti-Malware Testing Standards Organization) conference. The subject of Testing is one that I, and many others in the industry, have been interested in for a long time. Indeed, my main contribution to the AVIEN Malware Defense Guide was a chapter discussing testing. The whole reason for AMTSO forming was to try to create some clarity around the increasingly complex issues of testing. It may seem to some – particularly those who may never have attended an event involving large numbers of people with (slightly or wildly) differing opinions – that the wheels of AMTSO grind very slowly. However, this is not the case, these are complex issues, and the important thing is to ensure that if a document is published, that it should meet the aims and principles of the organisation. To that end all documents must be fully discussed and formally voted upon by the membership. The meetings are a productive time where final adjustments to the documents that have been put together over the past months can be made, and these documents voted upon.

There are already signs that AMTSO is having a positive effect, many testers have joined in the effort – as clearly, bad testing also has a negative effect on their reputations, and many mentions of the group have been seen in the press and in security circles. I hope that the increased awareness will encourage people to get involved, and that the progress will continue. The conference was interesting for all, with some good discussion on controversial topics. Keep an eye out for a press release over the next couple of weeks, and the appearance of some news on the AMTSO site.

Anti-malware testing is something that really does affect anyone who has a computer, so it’s great to know that there is a group dedicated to promoting ethical practice and laying out guidelines for good testing that can showcase the abilities of modern products.

As a member of AMTSO (but not an official representative of it), I’m happy to say that I fully support the efforts, and while it may seem slow, and often progress does involve a level of complexity akin to herding cats, it’s a worthwhile effort, and it is to be hoped that it will continue to go from strength to strength

Andrew Lee CISSP
AVIEN CEO

Testing, testing

OK, we’ve used that as a title before. However, it seems quite apposite as this is my first published blog here, and it’s related to anti-malware testing. (See what I did there? :-D)

This is actually a retread of my heavily re-edited blog at securiteam. But since it concerns (obliquely, for legal reasons) an issue that some of us discussed at VB 2009, I’m quite happy to repurpose some of it here.

Principle 3 of the AMTSO (Anti-Malware Testing Standards Organization) guidelines document (http://www.amtso.org/amtso—download—amtso-fundamental-principles-of-testing.html) states that “Testing should be reasonably open and transparent.”

The document goes on to explain what information on the test and the test methodology it’s reasonable to ask for.

So my first question is, is it open and transparent for an anti-malware tester who claims that his tests are compliant with AMTSO guidelines to decline to answer a vendor’s questions or give any information about the reported performance of their product unless they buy a copy of the report or pay a consultancy fee to the tester?

Secondly, there is, of course, nothing to stop an anti-malware tester soliciting payment from the vendors whose products have been tested both in advance of the test and in response to requests for further information. But is he then entitled to claim to be independent and working without vendor funding? In what respect is this substantially different to the way in which certification testing organizations work, for example?

AMTSO will be considering those questions at its next meeting (in Prague, next week).  But there are a lot of people inside and outside AVIEN who are seriously concerned with testing standards, as an aid to evaluating products for use in their own organizations, or because they have a vocational interest in making or supporting products that are impacted by fair/unfair or good/bad testing, and I’d be more than a little interested in hearing your views.

David Harley CISSP FBCS CITP
Chief Operations Officer, AVIEN