Recently for several different reasons we have taken a look at claims made about quite a few web security services and products and noticed that there are often bold claims that seem highly unlikely to be true. There also is a decided lack of evidence put forward to back those claims, much less independent third-party evidence to back them. So we were surprised today to see a press release from the web security company SiteLock, Independent Testing Shows SiteLock Web-Based Malware Protection Outperforms Traditional Endpoint Solutions. Considering their track record, we wouldn’t have expected them to be the ones actually having independent testing done. After taking a quick look over the details of the testing, we found that, not surprisingly, the testing was actually the opposite of independent.
The test compared their ability to detect malicious code on websites against McAfee Complete Endpoint Protection ability to do that. It appears the testing was intended to show that products of the type that McAfee have are not well suited to doing this, but since there is not a comparison of multiple products in each category, the value that could be gleamed from the test is limited.
One finding of the test did stick out to us though:
SiteLock SMART provided 100 percent PHP and JavaScript coverage versus less than 6 percent coverage by McAfee
Considering the poor quality we have seen with SiteLock’s detection in the past, whether it be missing malicious code or falsely identifying non-malicious code as malicious, we had a hard time believing that that they would possibly detect 100 percent of the tested malicious code in a reasonable test.
A key question to determine the quality of the testing was how the malicious code being used in the test was picked. When the items tested are mentioned in the press release there source isn’t mentioned either time, here:
Testing a group of nearly 3,000 web-based malware examples to determine effectiveness
and here:
Tests showed that the SiteLock solution detected and cleaned 100 percent of the samples provided.
Looking at the complete report (PDF) by the Tolly Group, there is the answer:
The test corpus was provided by SiteLock and consisted of 2,972 samples that included PHP, JavaScript and other executables that contained a variety of attacks.
So SiteLock provided the items that would be tested against. The test clearly isn’t independent at that point and the real story would have been if SiteLock didn’t detect them all.
The complete report also makes no claim to the testing being independent.
While SiteLock knew ahead of time what would be tested, since they provided the items being tested, the company behind the other product was not allowed to see them:
In accordance with Tolly’s Fair Testing Charter, Tolly personnel invited representatives from McAfee/Intel to participate in the testing. Because of internal guidelines, McAfee/Intel was unable to review the test corpus in detail and had no further comment.
Based on that you wonder what else might be in their internal guidelines that could skew the test results.
A Better Alternative to SiteLock For Cleaning Up a Hacked Website
If your web host is pushing you to hire SiteLock to clean up a hacked website, we provide a better alternative, where we actually properly clean up the website.