OVH: Database quota exceeded

OVH emailed me a few weeks back telling me that my shared database for the plan that powers was approaching its (huge!) quota of 25MB, and then again last week to tell me that this time, the quota was reached.

Once you reach the quota, the DB is placed in read-only mode, although SQL `DELETE` commands do go through correctly, as we’ll see later.

So my first instinct was to see what was wrong, by going into the PhpMyAdmin that OVH gives to each shared DB owner. It confirmed that the database was too big, mainly because of two tables: the main culprit at 9MB was wp_comments, the comments on this blog, and the second one at 5MB was its related sibling wp_commentmeta. The root cause being, of course, spam: all these comments were properly intercepted and classified as spam by Akismet, but as long as I didn’t purge them, they were still taking valuable disk space.

So I thought I could just delete the comments that Akismet marked as spam (as that info is available directly in the table) and go on with my day, but unfortunately no – the deletion went through, but the table was still marked as being 9MB, including about 7MB of “overhead“. How do we reclaim this overhead? By running OPTIMIZE TABLE… Which we cannot do as we’re in read-only mode.

At this point, I took a dump of the database, and deleted it through the OVH admin interface, recreated a new database and reimported that dump: solved! The new DB clocked at about 14MB, enough for the foreseeable future.

Lesson learned: clean your spam.

(PS: in the few days that passed between the db clean and me writing this article, I got another 355 spam comments. Yay.)

“They use some weird padding…”

A few days ago, a colleague was telling me about a project where she needs to implement a crypto scheme from an external vendor in order to talk to their API over HTTP. For complicated (and probably wrong) reasons, they decided to eschew TLS and develop their own system instead, relying on DES –not even triple DES! Basic DES, the one from the ’70s that is horribly insecure today– and RC4, which isn’t great either.

The whole scheme was bad, but my colleague added “and they also use that strange padding scheme – because the plaintext length needs to be a multiple of 8 bytes, at the end of every message, they put seven “Bell” characters!”.

The bell character? That’s odd. I mean, it’s in ASCII, and not usually part of any plaintext, so it’s probably safe to use as padding, but… Wait a second – padding with strange characters, all the same? That rings a bell!

And indeed it does – it’s PKCS#7!

PKCS#7 is meant to pad a message until it reaches the next block boundary, to use with block ciphers. It works by appending n characters of ASCII value 0xn, and of course the ASCII codepoint of the bell character is 0x07!

“Oh, that explains a lot. Now I won’t have to add blank spaces until it reaches (x mod 8) + 1 bytes and pad with bell characters”, my colleague said. I guess that’s the danger when you’re given a bad scheme to implement: it’s harder to realise when they actually do something right.

(Hat’s up to the Matasano crypto challenges: despite doing only level 1 and 2 if the memory serves –it was a while back–, they’re super useful for these sort of cryptography basics.)

Updating a tiny Rails app from Rails 3.1 to Rails 4.2

In 2011 I wrote a small Rails app in order to learn Ruby better and see what all the fuss was about – this was Antipodes, a website that shows you the antipodes of a given point or country/state using google maps.

I built it using the latest and greatest version of Rails available at the time, which was 3.2. It has since fell to various security issues and has been superseded by newest version, and is currently unsupported.

I’ve been aware of these limitations and decided not to carry on hosting on such an old version, so I just stopped the nginx instance that was powering it and left it aside.

Until now! I had some time to spare recently, so I decided to upgrade.

Updating to Rails 3.2

The railsguide documentation about upgrading suggests not to update straight to the latest version of Rails, but to do it by steps instead. The first one for me was to update from 3.1 to 3.2.

First up, let’s update our Gemfile to pick up on the new Rails version, and let’s dive in!

The first issue I ran into was that the :timestamps model field definition are now marked as NON NULL. I wasn’t actively using these, so I decided to remove them from the DB rather than fixing DB import code.

My next issue was that some gems would not install properly – I decided to use the newest version of Ruby available on the system, 2.2, and it was not happy at my Gemfile requiring ruby-debug19. Fair enough, let’s remove it.

My next problem didn’t come from Rails itself, but rather from the gem I used to generate Google Maps, Gmaps4Rails. It underwent a serious rewrite in the past 4 years and now needed very different code under the hood – no problem. It also allowed me to clean some of the .coffee files and make better use of the assets pipeline.

An lo, the website was running under Rails 3.2!

Upgrading to Rails 4.0

The next step was to upgrade to Rails 4.0. This was very straightforward, a quick change in the Gemfile and a change to route handling (match was retired in favour of using the actual HTTP verbs of the route being handled, e.g. get) made it work, and a couple of new config options silenced the deprecation warnings.

Upgrading to Rails 4.2

And finally, the upgrade from Rails 4.0 to Rails 4.2 was made through the Gemfile only, no update was needed on the code itself.


And here we are! Antipodes is now up to date with its dependencies, and waiting for a new nginx+passenger to run again (more on that soon!). is back!

So, had been in a pretty sorry state for a couple of months now, and I kept thinking I should go have a look into it and get to the bottom of the issue.

And the bottom of the issue was the 6000 spam emails sitting in the inbox, making the server crash at startup.

They’re now deleted, and everything is back up and happy. I’m currently thinking about different monitoring options, but given it’s all email-based, no solution that I know of seem overly practical to me. Any idea would be appreciated. :)

“So, the tests sometimes fail inexplicably” is a sentence you probably hear pretty often if you’re running any type of full-stack, browser-driven integration tests over a large-enough code base, especially when customising on top of an existing product.

Today’s instance was puzzling at first – the tests would sometimes fail to log in at all. That is, open the login page, fill in the username and password, wait until the URL change and assert that we’re now on the dashboard – nope, failure. It happens about 5% of the time, breaks straight away (no timeout), but happens to seemingly random users as different points of testing.

Well, time to output some debug information. First, let’s see whether that “wait until URL change” logic is working properly by outputting the URL after it changed – yes, the URL has indeed changed, and is now back into the login page with a little ?error=true tacked to the end.

An error at login? Let’s check the username and password… No, they’re definitely correct. And the problem is intermittent anyway. Could it be the backend rejecting valid credentials every so often? I’d never heard of that happening, but who knows. Let’s keep that in mind and possibly come back to it later.

As an added debugging, I made the test output the values of the username and password field sbefore they get submitted, and now we’ve got something interesting – the values filled in aren’t the values we told Cucumber to fill in! Instead of e.g. “username” / “password“, the username is filled in and sent as “usernameassword” and the password as “p“.

Uh? Ah!

And lo and behold, the culprit is unmasked – as a useful convenience, the login page executes a tiny bit of Javascript on page load that sets the focus to the “username” field so that a user doesn’t have to click or tab through it, but can start typing straight away. And Cucumber+Phantom faithfully emulating a browser and executing the javascript on that page will run into a race condition in which it starts entering keystrokes into a field, only to get its focus stolen by javascript and writing the rest of the keystrokes into the other field.

And the bug was of course intermittent because it would only happen when Cucumber was writing the password concurrently with page load. Any time before that and the fields would be correct, any time after that and the fields would be just as correct.


Our solution? Watir::Wait.until { browser.username_field.focused? } before actually filling in the fields and logging in. Works like a charm!