OpenX

Where Revive Adserver is Getting It Right and Wrong

It has been a little over six months since the software formerly known as OpenX changed hands and became Revive Adserver. We thought now would be a good chance to look at an important improvement that has occurred and pretty serious problem that has popped up.

Before we get to that we should note that anyone still running OpenX should upgrade as soon as possible as Revive Adserver has fixed several security vulnerabilities. Other than the bug we will get to later in the post, the upgrade should be rather seamless.

Improved Security

The story of the later years of OpenX was a series of security problems and a lack of concern for security that lead to at least some of those problems. In one instance their systems were breached and someone was able to modify the OpenX downloads so that malicious code was included. In another their systems were breached and used in conjunction with a vulnerability in OpenX, that the OpenX developers had been warned about, to gain access to individual OpenX installations. Another ongoing issue was that OpenX was not releasing the details of what changes were being made in releases. Doing this is important when security vulnerabilities are fixed as it allows others to double check that the issue has been resolved and it also helps people cleaning up hacked ad servers (like us) to know if the vulnerability that was exploited is an old vulnerability that has been fixed or a new vulnerability that would need to be reported to the developer to get fixed.

So far the people behind Revive Adserver have been handling things much better. For the last security vulnerability found in the software, which existed long before it became Revive Adserver, it was promptly fixed and a security advisory with details of it was released.

Our own experience with reporting a security issue to the OpenX and Revive Adserver teams showed the dramatic difference between the two. In June of 2012 a vulnerability was discovered in the XmlRpc component of the Zend Framework. Shortly afterward we sent an email to OpenX’s security address alerting them to the vulnerability in the component, which was included OpenX. We never heard anything back from them and the vulnerable component was never fixed. After the software became Revive Adserver we remembered the issue and decided to try reporting the issue again. Not only did we get a prompt response, but it was clear that they had actually looked into the scope of the vulnerability. As the components were no longer used in Revive Adserver the only way the vulnerability could be exploited is if a plugin used them. To resolve the issue they removed those components in the next release of Revive Adserver, 3.0.3.

A Serious Bug Unfixed

The latest release of Revive Adserver, 3.0.3, has a serious bug that causes new statistics data to stop showing up. Since statistics are important function of the ad server this is something that should have been promptly fixed, but about three weeks later it hasn’t. There were a couple of threads started on their forum (one of which is currently labeled as being HOT) shortly after the release raising the issue and identifying the problem. There were then a couple of bug reports filed, which were closed with a developer directing people to the forum. The latest bug report was filed a week ago and has yet to receive a response from the developers. This situation seems to indicate that improvements could be made in the handling of bug reports and that better pre-release testing might be needed, so that this type of bug can be spotted before it gets into a released version.

For those waiting on an official fix, the easiest way to resolve this for now is to go to the file /lib/RV.php and change the line:

require_once RV_PATH . '/lib/pear/PEAR.php';

require_once 'pear/PEAR.php';

Upgrading From OpenX to Revive Adserver Is a Minor Upgrade

We have recently gotten a number of questions about how much disruption upgrading from OpenX to the new Revive Adserver causes and as other undoubtedly have the same questions we wanted to address those for a wider audience. The good news is that the upgrade should be seamless in most instances. While the software has new name – due to ownership of the software being transferred – and a jump in version number from 2.8 to 3.0, the changes so far have been under the hood. This means that you won’t have to make changes to zones, campaigns, banners, ad positions, etc.

Two of the releases of Revive Adserver (3.0.0 and 3.0.2) have fixed security vulnerabilities that could lead to the ad server being hacked, so if you haven’t upgrading yet you should do that as soon as possible. There have also been bug fixes and modernization, including support for PHP 5.4 and 5.5, included in the new versions so far.

If you have previously done an upgrade between versions of OpenX 2.8 then you should find the process to be the same when upgrading to Revive Adserver.

So far the only issue we have run in to with the upgrade is that in one instance the upgrade failed to remove the OpenX Market plugin, which had been deprecated. The failure to remove that caused the admin interface to not work due to a Failed Opening Required error for the file /lib/ox/m2M/xmlrpcexecutor.php. If that occurs you can delete the /www/admin/plugins/oxMarket/ directory allowing access to the admin interface where you can fully remove the plugin and the openXWorkflow plugin, which should also have been removed.

If you are looking for someone to handle the upgrade for you, we can do a one-time upgrade for you or we can handle upgrades on an ongoing basis for you (insuring that you always get security fixes applied within a day of their release).

OpenX Doesn’t Take Security Seriously

Earlier this week it was discovered that the downloads of OpenX 2.8.10 had been modified at some point to include malicious code that allowed remote code execution. OpenX’s blog post about the incident starts with the claim that “OpenX takes security seriously.”. This isn’t the first time they have claimed that in a blog post (that previous blog post has the dubious distinction of being the third post named Security Matters on their blog). The claim that they take security seriously is hard to square with what happened in this instance, especially in light of previous events. Unlike the issues mentioned in those previous blog posts, which involved unintentional security vulnerabilities, in this case someone was able to gain access to OpenX’s website and modify files on the website to include malicious code without being detected by them. It only came to light that the files had been modified after the vulnerability added to the download was being actively exploited.

That isn’t something that should happen and it would be a big red flag that security isn’t taken seriously if it had only happened once. But this doesn’t seem to be the first time that OpenX’s website has been breached. It appears that their website was previously breached and used to exploit OpenX ad servers in April of last year. OpenX 2.8.10 wasn’t released until September of last year, so this most recent issue would have come either from a subsequent breach or from them not shutting off access after the first breach was detected.

Their post emphasizes that their other products were not impacted by the vulnerability in the downloads, but considering they were breached and didn’t detect it, it reasonable to be concerned that the breach may have reached other parts of their systems. Their post gives no indication that they made any check to insure that is the case.

The claim that they take security seriously is even harder to believe in light of the fact that they fail to take basic security measures with their website even after having their website breached at least twice. This can be seen by their use of an outdated version of WordPress on the very blog were they are claiming to take security seriously:

WordPress 3.4.1 is eleven months out of date and there have been three updates with security fixes released (3.4.2, 3.5.1, and 3.5.2). The announcement for 3.5.2, released on June 21, included this message, which OpenX has ignored:

This is a security release for all previous versions and we strongly encourage you to update your sites immediately.

WordPress is very easy to update, so if they can’t manage to do that it seems likely that they are failing to take other more complicated security measures that need to be taken when a website is being targeted, as theirs has been.

OpenX Ignores Security Issue

Back in July of last year we sent an email to OpenX’s security email address to inform that there was a vulnerability in the Zend Framework that ships with OpenX. We never heard anything back from them and the vulnerable file has not been updated in OpenX.

Does Security Really Matter to OpenX?

On December 1st OpenX finally made a public announcement on their blog about OpenX 2.8.8, which fixed a vulnerability that had already been exploited for some time before OpenX 2.8.8 was released. There post claims “If ever we find an issue, we address it quickly and communicate any updates as soon as possible.” Would anyone think a month is “as soon as possible”. What makes the length of time for the announcement even more troubling is that back on November 8 when we posted about the lack of a public announcement, and other issues, we had many visitors from OpenX visiting the blog so if they hadn’t yet thought it was important to make announcement before that they should by then.

Their post begins with the claim that “OpenX takes security seriously.” It hard to take that seriously considering that that this is third post on their blog titled Security Matters (1, 2) making the same claim and yet they have had to continually released fixes to vulnerabilities after those are already being exploited. It is understandable that software can have vulnerabilities, but when hackers are finding and exploiting them first instead of the developers finding and fixing them first it is an indication that their process for insuring the security of their code is lacking.

While there has been a fair amount of time between new vulnerabilities being exploited, and then fixed by OpenX, it is reasonable to consider that it might not be due a limited number of vulnerabilities but a lack of need to exploit more vulnerabilities. From what we have seen there seems to plenty of ad server running outdated versions of OpenX that hackers have been able to exploit well after new versions are released, so it doesn’t seem unreasonable to think that hackers might know of or could easily find more vulnerabilities in OpenX but as long there are enough ad servers running on outdated versions of OpenX to exploit there would be no need to make OpenX aware of a new vulnerability so that it can eventually be used when they run low on outdated ad servers to exploit.

It also is hard to take them seriously when there is such a public example of them not following their own advice. As part of their post they say “It’s critical to the safe maintenance and operation of any software that you not only maintain a current version of the software, but also take steps to regularly audit accounts that have access to your system.” They correctly state that it is critical to keep software up to date, but you don’t have look far to see that they don’t follow their own advice. The blog that they posted to is running WordPress 2.6.2 (if you want to see when websites are running out of date version versions of WordPress and other software check out our web browser extension for Firefox and Chrome). That version is now over three years out of date. They have failed to apply the last 16 releases that included security updates and 27 overall.

The CHANGELOG.txt file for www.openx.com indicates that it is running Drupal 6.19, which, if accurate, means the Drupal install is a year out of date and they missed a security update for that as well.

OpenX Continues Questionable Security Posture

Last Thursday OpenX released version 2.8.8 of their software. They have yet to make any sort of public announcement of the update on their blog or anywhere else that we could find. The only information given, found on the Product Updates page in the OpenX admin interface, says that:

It is highly recommended to install this update as soon as possible, because it contains a number of security fixes. The version of OpenX which you are currently using might be vulnerable to certain attacks and is probably not secure.

With a release that includes important security fixes, as this seems to be, you would expect that they would want to make sure people that use their software would be well aware of the update.

There was no information was given as to what the vulnerabilities were or what other changes were made in the new version. This is a continuing practice from OpenX as we have written about before. While it is understandable that developers would want to limit the amount of information to make it harder to for people to be able to exploit the vulnerabilities, hackers have shown that they are able to hack OpenX without this information and the information would be useful for people not looking to hack OpenX. To repeat what we said after the last OpenX release, “[w]ithout knowing what the issue or issues that were fixed makes it hard to determine the source of a hacking, potentially leading to new vulnerabilities that are exploited in OpenX going undiagnosed in the future if the OpenX installation hacked was running an out of date version.” It also makes it hard for anyone to independently verify the vulnerabilities were fully and properly fixed in the newer version.

The larger concern we have now is that OpenX seems to continue to be releasing security fixes in response to vulnerabilities being actively exploited, commonly referred to as zero-day exploits, instead them being found beforehand during development or during subsequent security reviews. We know that with past vulnerabilities they were being exploited before updates were released. We have seen some reporting that vulnerabilities in the last version were being exploited (with the most specific report we were not able to replicate the vulnerability, but that could be because of using a different server configuration) before this version was released. This at least means that users keeping the software up to date are not safe from being hacked, which they generally are with most web software that have a good track record of finding and fixing vulnerabilities in their software before they can be exploited. It also could be an indication that OpenX is not as concerned about the security of the software as they need to be for something that is so widely deployed.

What makes there apparent lack of concern towards the security of their software more troubling is the way they used the update message for 2.8.8 as a chance to promote their hosted solutions. This is the message that followed the warning about the need to update:

OpenX also provides both free and Enterprise hosted versions of the ad server, offering significant improvements in both infrastructure and functionality. Both of these products are managed and operated by the OpenX team, including upgrades, maintenance, and security scans, freeing you and your team from handling such issues. If ad serving is mission-critical to your business, we suggest contacting our team to learn more about OpenX Enterprise. As always, please let us know of any potential security problems by emailing security@openx.org.

All the hacks of OpenX we have dealt with so far have been due to security vulnerabilities in the OpenX software and not due directly to something related to self-hosting. In many of those cases OpenX had released a update before they were hacked, so automatic upgrades provided by their hosted solutions would have helped. But unless OpenX is providing their hosted customers with a more secure version of OpenX, then the hosted customers remain as vulnerable before the fixes for the security vulnerabilities are released. The quality of their security scans should be in question as well, if vulnerabilities keep getting found and exploited before they are fixed by OpenX.

Update (November 14, 2011):

Another thing that should be noted when considering how OpenX views the importance of security is the fact that their blog is still running WordPress 2.6.2. One of the most basic and important security measure anyone running a website should be doing is making sure they keep any software running on the website up to date. The version they are currently running is now over three years out of date. Since version 2.6.2 there have been 16 releases that include security fixes that they have missed (and 26 overall releases).

Deletion of ofc_upload_image.php Causes Failure of OpenX Upgrade

Last month it was disclosed that there was a vulnerability in the Video Ads plugin for OpenX. The vulnerability is contained in the ofc_upload_image.php file located in/www/admin/plugins/videoReport/lib/ofc2/ directory and is currently being exploited to cause ad servers to include malware on the banner pages they serve. The Video Ads plugin was first included with OpenX in version 2.8.4 and the version included with 2.8.5 and 2.8.6 also contained the vulnerability. The version including in OpenX 2.8.7 does not include the vulnerability, the ofc_upload_image.php file is empty.

In the Product Updates page listing for OpenX 2.8.7, in the OpenX admin interface, it states:

If you recently upgraded to version 2.8.6, you can simply install an upgraded video ad plug-in available [here] or remove the following file: admin/plugins/videoReport/lib/ofc2/ofc_upload_image.php from your installation.

Others have also made the suggestion that should delete the file. You should not delete the file as this will cause future upgrades of OpenX to fail. Instead, if you are running version 2.8.6 and are not upgrading to version 2.8.7 you should delete the content of the file but not the file itself. If you are currently running version 2.8.5 or below you should upgrade to 2.8.7 as those versions contain other security vulnerabilities.

If you have not done an upgrade since deleting the file adding an empty file named ofc_upload_image.php in the /www/admin/plugins/videoReport/lib/ofc2/ directory will prevent a future upgrade from failing.

If you are currently doing an upgrade and are receiving a red box that says “One or more plugin files couln’t be located, check the install.log file for more information” after you enter the path on the page that says “Provide the path to your previous OpenX installation.” you need to add an empty file named ofc_upload_image.php in the /www/admin/plugins/videoReport/lib/ofc2/ directory and then reenter the path. If you are not sure what the path is you can find it in the configuration file. The path is listed in the webDir parameter, make sure to remove the /www/images from the end of the path listed in the parameter.

If you previously attempted the upgrade and now receive a message that says “Your OpenX database and file structure are both using the most recent version and therefore no upgrade is required at this time. Please click Continue to proceed to the OpenX administration panel.” when you tried to try to perform the upgrade again you have two options. For the first, you will need to change the value of the oa_version record, in the _application_variable table of the database used by OpenX , to version number of OpenX you are currently running and then you need to start the upgrade process again (including deleting the new installation and then uploading a new copy of it). For the second, you will need replace the old OpenX installation with the new one and then you will then need to manually reinstall the plugins. The plugin installation files can be found in the /etc/plugins directory of the OpenX download.

OpenX Continues To Release Updates Without Details of Changes

OpenX has released a 2.8.7 which patches a vulnerability that could cause OpenX to be compromised. Previous vulnerabilities have led to numerous OpenX installations to be hacked and infected with malware. No detail has been given on what the vulnerability was or what, if any, other changes were made in this release. The new version does include an updated version of openXVideoAds plugin that patches a vulnerability in an earlier version. Without knowing what the issue or issues that were fixed makes it hard to determine the source of a hacking, potentially leading to new vulnerabilities that are exploited in OpenX going undiagnosed in the future if the OpenX installation hacked was running an out of date version.

OpenX lack of details of changes began with version 2.8.4, which was released in January of 2010. Beginning with that release the only information on changes that have been made is a link to https://developer.openx.org. The information about releases in this section of the website are not complete. The listing for Version 2.8.6 list only one item that was fixed, it does not indicate that a fix for a “potentially serious SQL injection vulnerability” and bug that caused advertisers to disappear were also patched in the update. The listing for 2.8.7 only lists 13 unresolved issues.