When it comes to web security information, getting good information is difficult in large part because there are so many people handing out advice that really are not qualified to do that (that includes many security companies), which can lead to falsehoods gaining widespread exposure. Even when you get past that it can be difficult because there are not always easily determined answers. Take for example the security of various web applications, trying to say how secure one is or how the security of one compares to another is difficult without doing a full security review of one or more of them.
Quicker methods don’t necessarily produce great results. For example you could look at how many vulnerabilities in them have led to websites being exploited in the past, but take a phrase from another industry, past performance does not guarantee future results.
Another method that is sometimes used is to look at how many vulnerabilities have been discovered in the software. There are a number of problems with that. One is that is you don’t know how many undiscovered vulnerabilities still exist in the software. It would also seem likely that more popular software would have more people looking over the code, so that they would have had more discovered than comparably secure software that is less popular. There is also the problem that not all vulnerabilities are equal importance, in fact most vulnerabilities discovered in web application have little chance of being exploited on the average website. You could try to mitigate by only counting more severe vulnerabilities, but as we have found severity ratings can be quite inaccurate.
Another problem we recently ran across when looking at something related to Plone, is that comparisons are not always done using comparable statistics.
The Plone developer’s described their security track record this way:
Unfortunately, there are no trustworthy statistics on vulnerability numbers available. This is mostly due to the fact that security databases are often outdated or incomplete. That said, Plone consistently has fewer vulnerabilities reported on databases than other content management systems. As our security team diligently reports any vulnerabilities found and works with vendors such as Red Hat to ensure our reports are complete and accurate this is strong evidence for Plone’s security
Here is how that is portrayed on the Wikipedia entry for the software:
Mitre is a not-for-profit corporation which hosts the Common Vulnerabilities and Exposures (CVE) Database. The CVE database provides a worldwide reporting mechanism for developers and the industry and is a source feed into the U.S. National Vulnerability Database (NVD). According to Mitre, as of 2013-05-29, Plone has the lowest number of reported lifetime and year to date vulnerabilities when compared to other popular Content Management Systems. This security record has led to widespread adoption of Plone by government and non-governmental organizations, including the FBI.
The following table compares the number of CVEs as reported by Mitre. It should be noted that logged CVEs take into account vulnerabilities exposed in the core product as well as the modules of the software, of which, the included modules may be provided by 3rd party vendors and not the primary software provider.
Comparison of Common Vulnerabilities and Exposures CMS First released CVEs Plone 2003 62 Joomla 2005 638 WordPress 2003 934 Drupal 2001 969
If you compare Plone to WordPress there is a great disparity, but looking at the vulnerabilities listed for both there is a glaring issue, hinted at by this portion of the Wikipedia entry:
It should be noted that logged CVEs take into account vulnerabilities exposed in the core product as well as the modules of the software, of which, the included modules may be provided by 3rd party vendors and not the primary software provider.
Currently there are 1024 entries for WordPress in the database, but if you start looking through them you will notice that many of them are for plugins and not for WordPress itself. There are 849 instances of the word “plugin” on that page, though some of those relate to WordPress and some are multiple instances of it in the same entry. There are another 60 entries that involve the word “theme”, many of which related vulnerabilities in themes. If subtract those two values from the overall entry total you get 115.
The current total for Plone is still 62 entries. None of those entries include the word “add-on”.
Considering the popularity of WordPress, the difference between the crude calculation of vulnerabilities in the core of WordPress and Plone total number doesn’t seem that significant. It also worth noting that we can’t recall a vulnerability in the core of WordPress that has led to lots of website being hacked in many years, maybe verging on a decade.
But what about the vulnerabilities in plugins versus that lack of mention of Plone addons. It is possible that Plone add-ons are quite secure, but one explanation is raw numbers. As of writing this there 47,665 reported to be on the WordPress’ Plugin Directory. The Plone number is much smaller, on the page that lists “all Plone add-ons” there are currently 2861 packages and some addons have multiple packages, so the number of addons is even smaller than that.