VTC's PC Viruskiller test 1997
------------------------------------------------ Executive Summary: VTC University of Hamburg PC Scanner Test 97-02: ------------------------------------------------ Virus Test Center (VTC) at Hamburg University`s Faculty for Informatics recently tested on-demand scanners for their ability to identify PC viruses. Tests were performed on VTCs virus databases, which were frozen on their November 30, 1996 status to give AV producers a fair chance to support updates within 8 weeks. The test goal was to determine detections rates, reliability (=consistency) of virus identification and reliability of detection of submitted or publicly available scanners. Moreover, development of detection quality was measured where more than one update was available during the test period (including updates up to February 9, 1997). Essential information about the virus databases: 10,704 File Viruses in 58,000 infected files, 827 System Viruses in 2,577 infected images, and 143 Macro Viruses in 472 infected documents. With threats from non-viral malware growing, the test also included 18 non-viral malicious programs (droppers, virus generators, trojan horses etc). The final test contained versions of the following scanners: Alert (Look), AVAST! (Alwil), AVG (Grisoft), AVP (KAMI Ltd), AVScan (H+B EDV), DSAV (Dr. Solomon), DrWeb (Dialogue Science), F-Prot, F-MacroW (Frisk Software), F/Win (Kurtzhals), IBM AV (IBM), Integrity Master (Stiller Research), InVircible (NetZ), Norman Virus Control (Norman DD), Norton AV (Symantec), PCVP (CSE), Scan (McAfee), Sweep (Sophos), TBAV (ThunderByte), TNT (Carmel), VDS (Aadvanced Research Group), Virex (Datawatch), Virus Buster (Leprechaun), Virus Hunter (Dialogue Sc.), Virus Track (On Technology), Xscan (Anyware). Sereval more scanners were tested but completely failed to meet VTCs test criteria (see test report). Moreover, our attempts to contact several reputed AV producers was answered with electronic silence. Overview of File/Boot/Makro Virus Detection Rates: ================================================================ | Scanner | Number of File | Number of Boot ! Number of Macro! | Codename: | Viruses (%): | Viruses (%): ! Viruses (%) | |===========+================+================+================+ | Total: | 10,704 (100 %) | 827 (100 %) ! 143 (100 %) | ============+================+================+================+ | ALERT/Look: | | al41013 | 10,487 (98,0%) | 804 (97,2%) | 100 (69,9%) | | al41014 | 10,507 (98,2%) | 807 (97,6%) | 126 (88,1%) | | al41015 | 10,580 (98,8%) | 774 (93,6%) | 138 (96,5%) | | AVAST!/Alwil: =============================================== | av75013 | 10,506 (98,2%) | 808 (97,7%) | 104 (72,7%) | | av75014 | 14,511 (98,2%) | 807 (97,6%) | 126 (88,1%) | | av77001 | 10,584 (98,9%) | 778 (94,1%) | 142 (99,3%) | | AVG/Grisoft: ================================================= | avg41 | 8,481 (79,2%) | 586 (70,9%) | 36 (25,2%) | | AVP/KAMIS: =================================================== | avp222 | 10,578 (98,8%) | 802 (97,0%) | 138 (96,5%) | | avpl30 | 10,589 (98,5%) | 536 (64,8%) | 142 (99,3%) | | AVScan/H+B EDV: ============================================== | avs293b | 6,850 (64,0%) | 464 (56,1%) | 25 (17,5%) | | avs320 | 7,852 (73,4%) | 504 (60,9%) | 83 (58,0%) | | DrWeb/Dialogue SC: =========================================== | drw318d | 9,984 (93,3%) | 357 (43,2%) | 129 (90,2%) | | drw318f | 9,979 (93,2%) | 366 (44,3%) | 129 (90,2%) | | DSAV/DrSolomon: ============================================== | dsav766 | 10,657 (99,6%) | | 111 (77,6%) | | dsav767 | 10,674 (99,7%) | 820 (99,2%) | 120 (83,9%) | | dsav768 | 10,675 (99,7%) | 825 (99,8%) | 140 (97,9%) | | F-PROT: ====================================================== | fmac102 | n/a n/a | n/a n/a | 141 (98,6%) | | fpr225 | 9,705 (90,7%) | 703 (85,0%) | 62 (43,4%) | | F/Win: ======================================================= | fwin402 | n/a n/a | n/a n/a | 139 (97,2%) | | fwin403 | n/a n/a | n/a n/a | 139 (97,2%) | | IBM AV: ====================================================== | ibm251 | 10,017 (93,6%) | See problems | 93 (65,0%) | | Integrity Master: ============================================ | itm311a | See problems | See problems | 41 (28,7%) | | itm311b | See problems | 107 (12,9%) | 117 (81,8%) | | InVircible: ================================================== | inv612a | 838 (7,8%) | See problems | See problems | | inv612d | 829 (7,7%) | See problems | See problems | | Norton AV/Symantec: ========================================== | nav30j | See problems | 543 (65,7%) | 110 (76,9%) | | nav30f | 8,638 (80,7%) | 553 (66,9%) | 121 (84,6%) | | Norman VC: =================================================== | nvc351 | 9,359 (87,4%) | 711 (86,0%) | 19 (13,3%) | | PCVP/CSE: ==================================================== | pcvp239 | 7,267 (67,9%) | 397 (48,0%) | See problems | | pcvp240 | 7,266 (67,9%) | 335 (40,5%) | See problems | | Scan/McAfee: ================================================= | scn253d | 8,701 (81,3%) | 670 (81,0%) | 114 (79,7%) | | scn253f | 8,981 (83,9%) | 682 (82,5%) | 136 (95,1%) | | Sweep/Sophos: ================================================ | swp293 | 10,111 (94,5%) | 780 (94,3%) | 106 (74,1%) | | swp294 | 10,262 (95,9%) | 784 (94,8%) | 125 (87,4%) | | TBAV/ThunderByte: ============================================ | tbav706 | 10,178 (95,1%) | 751 (90,8%) | 101 (70,6%) | | tbav707 | 10,223 (95,5%) | 650 (78,6%) | 103 (72,0%) | | TNT/Carmel: ================================================== | tnt964 | See problems | 379 (45,8%) | 32 (22,4%) | | tnt971 | 6,209 (58,0%) | 373 (45,1%) | 88 (61,5%) | | VDS/Adv.Res.Group: =========================================== | vds31 | See problems | See problems | 23 (16,1%) | | Virex: ======================================================= | vrx299 | See problems | See problems | 11 ( 7,7%) | | Virus Buster/Leprechaun: ===================================== | vbs482 | 4,457 (41,6%) | See problems | See problems | | vbs484 | 4,457 (41,6%) | See problems | See problems | | vbsl484 | 7,794 (72,8%) | See problems | See problems | | vb48415 | 4,614 (43,1%) | See problems | See problems | | Virus Hunter/Dialogue Sc: ==================================== | vht1663 | 2,018 (18,9%) | | See problems | | vht1678 | 2,066 (19,3%) | 301 (36,4%) | See problems | | Virus Track/On Technology: =================================== | vit9606 | 4,873 (45,5%) | See problems | 9 ( 6,3%) | | XScan/Anyware: =============================================== | xsc233 | 6,365 (59,5%) | See problems | See problems | |============+================+================+================+ For explanation of the different columns, abbreviations of scanner versions etc: see TEST-972.TXT. Eval #1: Evaluation for overall virus detection rates: ------------------------------------------------------ The following grid is applied to classify scanners: - detection rate above 95% : the scanner is graded "excellent" - detection rate above 90% : the scanner is graded "very good" - detection rate of 80-90% : the scanner is graded "good enough" - detection rate of 70-80% : the scanner is graded "not good enough" - detection rate of 60-70% : the scanner is graded "rather bad" - detection rate of 50-60% : the scanner is graded "very bad" - detection rate below 50% : the scanner is graded "useless" To assess an "overall grade" (including boot, file and macro virus detection), the lowest of the related results is used to classify the resp. scanner. If several scanners of the same producer has been tested, grading is applied to the most actual version (which is, on most cases, the version with highest detection rates). Only scanners where all tests were completed are considered; here, the most actual version with test completed was selected. The following list indicates those scanners graded into one of the upper three categories: "Excellent" scanners: DSAV 768 (99,7% 99,8% 97,0%) AVP 2.2 (98,8% 97,0% 96,5%) "Very Good" scanners: AVAST!77/1 (98,9% 94,1% 99,3%) Alert41/15 (98,8% 93,6% 96,5%) Sweep 294 (95,9% 94,8% 95,1%) "Good Enough" scanners: F-PROT2.25 (90,7% 85,0% 98,6%=F-MacroW 102) Scan 2.53 (83,9% 82,5% 95,1%) Remark: The following scanners fail a good classification by just one category: AVPlite, DrWeb, TBAV 706 and NAV. Concerning "In-The-Wild" viruses, a much more rigid grid must be applied to classify scanners, as the likelyhood is significant that a user may find such a virus on her/his machine. The following grid is applied: - detection rate is 100% : scanner is "excellent" - detection rate is >95% : scanner is "very good" - detection rate is >90% : scanner is "good" - detection rate is <90% : scanner is "risky" "Excellent" scanners: DSAV 768 (100% 100% 100%) AVP 2.22 (100% 100% 100%) "Very Good" scanners: FPROT 2.25 (99,2% 98,9% 100%) Scan 2.53 (100% 96,7% 100%) NAV 3.0 (99,2% 96,7% 100%) Sweep 2.94 (100% 100% 95,5%) "Good" scanners: AVAST! 77/1 (100% 94,5% 100%) Alert 41/15 (100% 93,4% 100%) TBAV 707 (100% 100% 90,9%) Eval #2: Evaluation for detection by virus classes: --------------------------------------------------- Some scanners are specialised on detecting some class of viruses (either in deliberately limiting themselves to one class, esp. macro viruses, or as that part is significantly better as other parts). It is therefore worth notifying which scanners perform best in detecting file, boot and macro viruses. The same grades are applied as in the "overall" grading (see 1). 2.1 Detection of file viruses: ------------------------------ "Excellent" scanners: DSAV 768 (99,7%) AVAST! 97/1 (98,9%) AVP 2.22 (98,8%) Alert 41/15 (98,8%) Sweep 294 (95,9%) TBAV 706 (95,5%) "Very Good" scanners: IBM AV 2.51 (93,6%) DrWeb 318 (93,2%) F-PROT 2.25 (90,7%) "Good" scanners: Norman VC 351 (87,4%) Scan 2.5.3 (83,9%) NAV 3.0 (80,7%) 2.2 Detection of boot viruses: ------------------------------ "Excellent" scanners: DSAV 768 (99,8%) AVP 2.2 (97,0%) "Very Good" scanners: Sweep 2.94 (94,8%) AVAST! 77/1 (94,1%) Alert 41/15 (93,6%) "Good" scanners: Norman VC 351 (86,0%) F-Prot 2.25 (85,0%) Scan 2.53 (82,5%) 2.3 Detection of macro viruses: ------------------------------- "Excellent" scanners: AVAST!77/1 (99,3%) AVP 3 lite (99,3%) F-MacroW 1.02 (98,6%) F/Win 4.03 (97,2%) DSAV 768 (97,0%) Alert 41/15 (96,5%) Scan 2.5.3 (95,1%) "Very Good" scanners: DrWeb 316 (90,2%) "Good" scanners: Sweep 2.94 (87,4%) NAV 3.0 (84,6%) ITM 3.11b (81,8%) Eval #3: Evaluation of Macro Malware detection: ----------------------------------------------- Several scanners are able to detect also non-viral malware. As existence of macro malware is published, the (yet small) macro malware database was used for an initial test for malware detection. The following grid is applied to classify detection of macro malware: - detection rate > 90% : the scanner is graded "excellent" - detection rate of 80-90% : the scanner is graded "very good" - detection rate of 60-80% : the scanner is graded "good enough" - detection rate of < 60% : the scanner is graded "not good enough" "Excellent" scanners: ---------- "Very Good" scanners: AVP 3 lite (86,7%) DSAV 768 (86,7%) "Good Enough" scanners: AVAST 77/1 (66,7%) NAV 3.0 (66,7%) Alert 41/15 (60,0%) DrWeb 318 (60,0%) Scan 2.5.3 (60,0%) More detiled information about the test, its methods and viral databases, as well as detailed test results are available for anonymous FTP downloading from: ftp.informatik.uni-hamburg.de /pub/virus/texts/tests/vtc/pc-av/1997-02/test-972.zip General information is also available from VTCs HomePage (VTC is part of working group "AGN"): agn-www.informatik.uni-hamburg.de/vtc or: www.informatik.uni-hamburg.de/AGN/ Any comment and critical remark which helps VTC learning to improve our teste methods will be warmly welcomed. The next comparative test is planned for May-June 1997, with viral databses frozen On April 30, 1997. Any AV producer wishing to participate in that test is invited to submit related products. On behalf of the VTC Test Crew: Dr. Klaus Brunnstein (February 20, 1997)