Monthly Archives: July 2010

More AMTSO stuff

They say there’s no such thing as bad publicity, though quite who ‘they’ are, and why ‘they’ would make such a clearly daft statement is beyond me. It seems that AMTSO has had it’s fair share of bad publicity recently –  a further example is the piece by Ed Moyle over on his blog at http://www.securitycurve.com/wordpress/archives/1773. It’s a long article, but it does show that Ed clearly doesn’t understand (or doesn’t want to accept) what AMTSO is trying to do – maybe that does just mean that AMTSO needs a better PR representation. Anyway, once again Kurt Wismer (or perhaps I should adopt his anti capitalist rendering and use kurt wismer) has provided some excellent analysis of Ed’s piece over on his blog at http://anti-virus-rants.blogspot.com/2010/07/i-see-standards-organization.html

There’s little more that really needs to be said from my perspective. For the record, I personally agree with Kurt (just can’t seem to get my head around the ‘kurt’ thing), in his analysis of the NSS report done by AMTSO – which seems to be at the root of this whole anti AMTSO campaign. The central point is that NSS did a good job, and came very close to the ideal – (if you haven’t read the review, then it’s here). It’s unfortunate that that has been taken as a negative thing or a slight against them to say that they did not fully meet the ideal standard set by AMTSO – it was still far better than many other tests, and I have every hope that people are sensible enough to recognise that. It’s hard for me to see quite how Ed jumps from that report to an accusation that AMTSO is ‘Slapping the labs’ – an argument even harder to see when a lab like Dennis Technology Lab (who have very similar methodology to NSS) voluntarily submitted their own test for the AMTSO review process (see the report here).

If there’s one thing we can learn from this, it’s that it does seem that there’s a double standard here – testers can criticise AV vendors with impunity in their reviews and tests of AV products, but when someone tries to apply that same process and rigour to the tests done by those testers, that is somehow anathema. Personally, I think that’s shoddy thinking, and I have no doubt that AMTSO will continue to strive, as it has done from inception, to provide the public with an insight into tests, and to support good testing practice (and incidentally point out less than ideal practice where needed).

Andrew Lee
AVIEN CEO / CTO K7 Computing

Virus Researchers are community outcasts

Lately I’ve been reading a lot of blogs and articles attacking and defending AMTSO and their attempt at establishing standards for the testing of counter-malware products. Unfortunately I think BOTH sides are missing the larger picture here. AMTSO was formed to address some critical shortcomings in the testing of counter-malware products: some tests were arguably unethical, most unscientific and some just poor from the word go. So where does the dissent come from? It comes from the very people who done or supported those poor non-science based tests. Yet it goes beyond that. The people who are condemning AMTSO and their efforts are in some cases well respected in the general security arena, and are very knowledgeable, and this is the rub. These people, most people in academia, and in management as well do not recognize Malware research and prevention as a specialty niche. They attempt to apply the same rule-set to fighting a malware outbreak as they do a simple intrusion, and see nothing wrong with that solution.

A majority of people not engaged in the Malware field as a profession still feel that the average Security Professional has the same knowledge and skill sets as used by the Counter Malware Professionals. Unfortunately nothing can be further from the truth. It goes beyond the abilities and skills for reverse engineering, programming, and identifying abnormal network traffic. This argument goes back to at least the early 1990’s when in a panel discussion a firewalls specialist attempted to answer a question about a virus. On that panel was Wolfgang Stiller, creator of Integrity Master Anti-Virus, Wolfgang interrupted him saying along the lines of “look I’m here for the virus questions, I would never presume to speak with authority or experience on firewalls issues, but you presume to have the same experience and expertise with viruses that I do, and that is mistaken”. Similar exchanges have happened on other panels with people such as Robert Vibert and Rob Rosenberger, among others. These are also the same people who demand that anti-malware products protect against threats that are not viruses, nor are they specifically malware, but “Potentially unwanted programs”. So this is not a new phenomenon. The question in my mind is why does it still exist?

Anti-Virus ‘Experts’ helped establish the disaster recovery field, and were among the very first to teach classes in th at subject. It was the Anti-Virus Researchers who developed the field of Computer Forensics, in both cases it was the Anti-Virus field that had the necessary expertise and skill set needed to fill the holes and expand the career field. So now that Disaster Recovery, and Computer Forensics are recognized as specialty fields and given a high degree of respect from schools and management, what happened to the Anti-Virus researcher? Their mindset is not of an operational nature, they bore easily, some may even say they have attention deficit disorder (ADD), yet they are anal about doing things the same way every-time. They dwell on minutiae, arguing to the point of splitting hairs. I sometimes think some of my colleagues can SEE the traffic on the wire in their minds eye. Yet with all this contribution to the Computer Security Community they are still (almost purposely) maligned and misunderstood. At a Virus Bulletin Conference, I stated that we as a community must take action or go from the ranks of professional, to the ranks of the tradesmen. I still don’t know what action that is, or how to go about it, but AMTSO is a good step in that direction, and the naysayers need to start looking outside their comfort zone and realize they know enough to be dangerous and not enough to be helpful at this point.

Ken Bechtel
Team Anti-Virus
Virus Researcher and Security pontificator

The edge of reason(ableness): AV Testing and the new creation scientists

First, let me start out by saying that I am in a bad mood. I probably shouldn’t write when I’m in this mood, because I’m in danger of just ranting, but I’m going to anyway. I’m in a bad mood because I am pretty fed up that some people are so deliberately trying to destroy something I’ve personally (along with many others) worked very hard to build in the last couple of years.

I’m in a bad mood because writing this is distracting me from the many other things that I need to do, and get paid to do.

I’m in a bad mood because I’m fed up with hearing that I, and others like me, have no right to comment on things that fall directly within my realm of expertise (and goodness knows, that’s a narrow enough realm) – and that if I do, it’s simply self-interested nonsense.

Secondly, let me also point out that although I’m now going to reveal that, yes, I’m talking about Anti-Malware Testing, and may mention AMTSO, I’m not speaking on behalf of AMTSO, nor my employer, nor anyone else, but me, myself and I (oh, that there were so many of us).

So, “What’s the rumpus?*” Well, in what has become an almost unbelievable farce, the last few weeks have seen mounting attacks on the AMTSO group and what it does.

For some background – those who are interested can read these articles.

http://kevtownsend.wordpress.com/2010/06/27/anti-malware-testing-standards-organization-a-dissenting-view/

http://krebsonsecurity.com/2010/06/anti-virus-is-a-poor-substitute-for-common-sense/

There are some very good points in the second (Krebs) article, although cantankerous is not something that I would say characterizes AMTSO all that well – as Lysa Myers has pointed out ‘AMTSO is made of people‘, and I think the generally negative tone employed is a shame. The first (Townsend) article is way more problematic; there’s just so much wrong with Mr Townsend’s thinking that I don’t really know where to start. Fortunately, Kurt Wismer has already done a great job of responding here, and David Harley an equally competent job here.

So why my response? Well, probably because I certainly am cantankerous.

I’m also, almost uniquely in this industry (David Harley is another), formerly one of those “users” that Mr Townsend is so adamant should be controlling the process of AMTSO’s output – indeed, the whole of AVIEN was set up in the year 2000 as an organisation of interested, non-vendor employed, users – albeit users who knew something about anti-malware issues. We were users responsible for protecting large enterprises, who wanted to be able to share breaking anti-virus information without the interference of Vendors or the noise of such cesspools as alt.comp.virus. We wanted good, reliable information.

I, like David Harley, later joined the industry as a Vendor, but I still understand what it is to be a user, and that was also a huge consideration in the setup of AMTSO – as so many have said before, and I want to reiterate here, bad testing of anti-virus products hurts everyone, the user most especially.

However, this debate is much more than just one on which we can ‘agree to differ’  – like whether Germany or Spain has the better football team might be – it’s much more fudamental than that.

Indeed, the only real analogy that comes close is that of the battle currently raging between the so called  faith based ‘science’ of creationists (let’s not prevaricate, Intelligent Design is just a euphemism for Creationism), and the research based science of evolutionary biologists and so on.

On the one hand, you have anti-malware researchers, professional testers and so on; people who study malware every day, who constantly deal with the realities of malware exploiting users, and who understand better than anyone the challenges that we face in tackling malware – if you like, the “Richard Dawkinses of anti-malware” (though I certainly would not claim to match his eloquence nor intelligence) –  and on the other hand, we have those outside the industry who say that we’re all wrong, that we’re just a “self-perpetuating cesspool populated by charlatans” (yet none the less, a cesspool at which the media feeds most voraciously), that nobody needs AV, and that everything the AV community does or says is bunk.

What I find so extraordinary (in both cases) is that those who are most in a position to provide trusted commentary on the subject are so ignored, in favour of those who have shrill, but ill-informed voices. Why is it that information from a tester; who may have just woken up one morning and decided to ‘test’ antivirus products; is taken on faith as being correct and true; and yet, when a group of professional people give up their time voluntarily, and work together to try to produce some documentation that sets out the ways in which anti-malware products can be tested effectively (and, no, that has nothing in particular to do with the WildList) and reliably, is it so violently decried as self-interested nonsense. It’s a terrible shame that science is so deliberately ignored in the face of popular opinion. Unfortunately, millions of people CAN be wrong, and often are.

AMTSO is not about dictating truth, but rather pointing out ways in which truth can be reliably found (and importantly, where it cannot).

I refuse to lie down and take it when someone tries to tell me that I’ve no right to point out the truth – and I’m not talking about truth based on some millenia old scripture, but real, hard, repeatable, scientifically verifiable, researched fact. If that makes me as unpopular as Richard Dawkins is to a creationist, then so be it.

If you’re interested in understanding why anti-virus testing is so important (and why so many professional testers participate in AMTSO) then, please, do have a read of the AMTSO scriptures er… documents, here.

Andrew Lee – AVIEN CEO, Cantankerous AV researcher.

* If you’ve not seen the excellent movie “Miller’s Crossing” you won’t know where that quote comes from.

(Thanks to Graham Cluley for pointing out that the first link didn’t go to the correct page.)

Brief hiatus

Our reader may note that it’s been quiet around here for a few weeks. Far from this being due to a lack of news, it’s rather that there have been a huge number of other things demanding time and attention. Not least of these is me trying to submit my master’s thesis on time, that and a few conferences, papers and other matters mean that we’re a little understaffed at AVIEN right now. Normal intermittent service should be resumed shortly.

Andrew Lee
AVIEN CEO / CTO K7 Computing