// TODO: come up with witty tagline

Perl::Critic is a nifty syntax analyzer able to parse your Perl code, warn you against common mistakes and hint you towards best practices. It’s available either as a Perl module or a standalone shell script (perlcritic). Unfortunately, there is no standard way to integrate it with Jenkins.

Jenkins, the continuous-integration-tool-formerly-known-as-Hudson, is the cornerstone of our continuous building process at work. It checks out the latest build from Git, runs a bunch of tests (mainly Selenium, as we develop a website) and keeps track of what goes wrong and what goes right. We wanted to integrate Perl::Critic to Jenkins’ diagnostics to keep an eye on some errors that could creep in our codebase.

So Jenkins doesn’t do Perl::Critic. However, Jenkins supports TAP. TAP, the Test Anything Protocol, is an awesomely simple format to express test results that goes as follow:

1..4
ok 1 my first test
ok 2 another successful test
not ok 3 oh, this one failed
ok 4 the last one's ok

The first line (the plan) announces how many tests there are, and each following line is a test result beginning by either “ok” or “not ok” depending on what gave.

Based on such a simple format, we can use a bit of shell scripting to mangle perlcritic’s output to be TAP-compatible:

# Perl::Critic                             \
# with line numbers (nl)...                \
# prepend 'ok' for passed files...         \
# and 'not ok' for each error...           \
# and put everything in a temp file

perlcritic $WORKSPACE                      \
  | nl -nln                                \
  | sed 's/\(.*source OK\)$/ok \1/'        \
  | sed '/source OK$/!s/^.*$/not ok &/'    \
  > $WORKSPACE/perlcritic_tap.results.tmp

# Formatting: add the TAP plan, and output the tap results file.
# TAP plan is a line "1..N", N being the number of tests
echo 1..`wc -l < $WORKSPACE/perlcritic_tap.results.tmp` \
 |cat - $WORKSPACE/perlcritic_tap.results.tmp > $WORKSPACE/perlcritic_tap.results

# Cleanup
rm -f $WORKSPACE/perlcritic_tap.results.tmp

(§WORKSPACE is a Jenkins-set variable referring to where the current build is being worked on.)

And voilà! Jenkins reads the TAP result file and everything runs smoothly.

The only downside of this approach is that the number of tests will vary. For example, a single .pm file containing 5 Perl::Critic violations will show up as 5 failed tests, but fixing these will turn into a single successful test.

You probably heard of GNU screen. It’s handy, ubiquitous, and dead simple. Here’s how to use it!

Open a terminal and type:

screen

You’re welcomed by an introduction message, press enter, and… You’re in a shell. Uh?

Screen is simple

screen is a terminal manager, so it’s logical that the first thing you see when you start it is a terminal.

This terminal is as vanilla as the terminal we started from. Just try it:

See? No black magic here, simply a terminal.

Screen is simple

The only difference is that Ctrl+a is now a special key combination that you can use to invoke screen‘s commands.

So let’s take a break and quit screen. Type Ctrl+a to let screen know you want its attention, then d, as in detach. There! You’re back in your first terminal.

Let’s go back in screen and learn some more! Just type:

screen -r

The -r stands for reattach: screen will re-open the last session, the one we detached from. You can see the results of the commands we entered earlier are still here.

We just saw a great feature of screen: the ability to log out and log back in without losing anything. Do you have something long to on a server? SSH into the server, launch screen, launch the task, detach from screen, log out from SSH, go back home, enjoy a good dinner and a well-deserved night of sleep, come back to work, SSH into the server, launch screen -r, and it’s just as if you never left.

Screen is simple

You can already use screen just like that, but let’s just see another nifty feature: multiple terminals!

In screen, type Ctrl-a, then ‘c’, short for create. You’re in a shell. Uh?

Screen is simple

You just created another terminal. screen can manage plenty of simultaneous terminals, not just one. To see a list of them, type Ctrl+a, then the quote symbol “, and you will see your two terminals. Just use the arrows to select which one you want to open.

There you go, you know screen! See, I told you it was simple.

Misc useful commands

Do you want to change the name of a terminal in screen‘s list? In that terminal, Ctrl+a and A.
Do you want to go directly to a specific terminal? Ctrl+a and its number.
Do you want to go to the previous/next terminal? Ctrl+a and p or n.
Do you want to switch to the previous terminal quickly? Ctrl+a and Ctrl+a.
Do you want to remap Ctrl+a to another key, say Ctrl+b? Just put escape ^b in your .screenrc.

As a follow-up on my previous post on the question, which advocated a simple (but bad) approach to keeping GimmeSomeTune running, here’s a better way!

The Good Thing ™ to do is to use OS X’s built-in mechanism to start and keep processes running, namely launchd.

What we have to do is simply to write a plist containing the info needed by launchd, namely:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
	<dict>
		<key>KeepAlive</key>
		<true/>
		<key>Label</key>
		<string>at.eternalstorms.gstlauncherdaemon</string>
		<key>ProgramArguments</key>
		<array>
			<string>/Applications/GimmeSomeTune.app/Contents/MacOS/GimmeSomeTune</string>
		</array>
		<key>RunAtLoad</key>
		<true/>
	</dict>
</plist>

Save this as a plist file in ~/Library/LaunchAgents. The name doesn’t really matter, but the best way to keep everything tidy and adhere to OS X’s standards is to call it at.eternalstorms.gstlauncherdaemon.plist.

Alright! Now GimmeSomeTune is going to start when you log in, and launchd will make sure it keeps running (i.e. relaunch it if it crashes). To tell launchd to use that plist file right now without having to log out and back in again, run:

launchctl load -w ~/Library/LaunchAgents/at.eternalstorms.gstlauncherdaemon.plist

And conversely, to stop it:

launchctl unload -w ~/Library/LaunchAgents/at.eternalstorms.gstlauncherdaemon.plist

During the past few weeks, I have been writing a C++ plugin to grade C41 digital intermediates in Cinelerra, an open-source Linux video editor. C41 is the most common chemical process for negatives, resulting in films that look like this — you probably know it if you’ve ever shot film cameras.

Of course, after scanning those negatives, you have to process (“grade”) them to turn them back to positive. And it’s not as simple as merely inverting the values of each channel for each pixel; C41 has a very pronounced orange shift that you have to take into account.

The algorithm

The core algorithm for this plugin was lifted from a script written by JaZ99wro for still photographs, based on ImageMagick, which does two things:
- Compute “magic” values for the image
- Apply a transformation to each channel (R, G, B) based on those magic values

The problem with film is that due to tiny changes between the images, the magic values were all over the place from one frame to the other. Merely applying JaZ’s script on a series of frames gave a sort of “flickering” effect, with colours varying from one frame to the other, which is unacceptable effect for video editing.

The plugin computes those magic values for each frame of the scene, but lets you pick and fix specific values for the duration of the scene. The values are therefore not “optimal” for each frame, but the end result is visually very good.

However, doing things this way is slow: less than 1 image/second for 1624*1234 frames.

Optimising: do less

The first idea was to make optional the computing of the magic values: after all, when you’re batch processing a scene with fixed magic values, you don’t need to compute them again for each frame.

It was a bit faster, but not by much. A tad more than an image/second maybe.

Optimising: measure

The next step —which should have been the first!— was to actually benchmark the plugin, and see where the time was spent. Using clock_gettime() for maximum precision, the results were:

~0.3 seconds to compute magic values (0.2s to apply a box blur to smooth out noise, and 0.1s to actually compute the values)

~0.9 seconds to apply the transformation

Optional computing of the magic values was indeed a step in the right direction, but the core of the algorithm was definitely the more computationally expensive. Here’s what’s to be computed for each pixel:

row[0] = (magic1 / row[0]) - magic4;
row[1] = pow((magic2 / row[1]),1/magic5) - magic4;
row[2] = pow((magic3 / row[2]),1/magic6) - magic4;

With row[0] being the red channel, row[1] the green channel, and row[2] the blue channel.

The most expensive call here is pow(), part of math.h. We don’t need to be extremely precise for each pixel value, so maybe we can trade some accuracy for raw speed?

Optimising: do better

Our faithful friend Google, tasked with searching for a fast float pow() implementation, gives back Ian Stephenson’s implementation, a short and clear (and more importantly, working) version of pow().

But we can’t just throw that in without analysing how it affects the resulting frame. The next thing to do was to add a button that would switch between the “exact” version and the approximation: the results were visually identical.

Just to be sure, I measured the difference between the two methods, and it shows an average 0.2% difference, going as high as 5% for the worst case, which are acceptable values.

And the good news is that the plugin now only takes 0.15 to 0.20 second to treat each image, i.e. between 5 and 6 images/second — an 8-fold gain since the first version. Mission accomplished!

Today was the last day of my Rails week. I added some database-backing to my app (with a fully scaffolded model and all!) for the countries’ data and refactored a fair bit, though I’m still unsure about a few decisions I made, such as if I should put the base data in seeds.rb or in a migration. Oh, well.

The website is available here:

http://antipodes.plui.es

And its source code is on GitHub.

So?

Writing this website taught me quite a lot about Rails in general, from its innards to automated deployment. I still have a lot to learn, and more importantly a lot of practice to do before I can say I’m competent with Rails, but that was a great start. *self-pat on the back*

The more important thing to me is that even though I didn’t have the time to learn each of the aspects of the Rails ecosystem inside and out, I have a better overview of “what does what” in Rails, and a lot of good pointers to learn more when needed.

Buzzword bingo!

All in all, I learned (“began learning” would be more appropriate):

• Vagrant, and a tiny bit of Chef
• Rails (eh!): MVC, migrations, the global directory structure, views/partials, the asset pipeline, etc.
• RVM
• Bundler
• Rake
• Spork
• Autotest
• RSpec
• Capistrano

Loose ends

Well, although I think I understood a fair amount of what I set out to learn, I still don’t grasp Chef at all, and didn’t really adhere to the philosophy of TDD through the week. My tests are really basic and not very satisfying; writing meaningful tests seems like quite an difficult art that I’ll have to learn more about.

Another thing that bugs me a bit is that it took more time than I originally thought to do a lot of the things I set out to do. This is probably due to inexperience, so in a way I’m curing it? I guess? I probably could have gained some time by asking a few questions on things like IRC, but it felt a bit stupid when there’s such a trove of information about Rails online. Indeed, googling and reading guides or StackOverflow threads / mail threads always ended up giving the answer; but maybe not as fast as IRC would’ve been.

Oh, and the design of the website is pretty terrible, but that wasn’t really the goal (and CSS has never been my forte).

What now?

Well, as far as Rails go, I’m ready to tackle bigger things. I hope to find a Rails job in Wellington (wink wink nudge nudge if you’re reading this from New Zealand ;) ) and put this freshly-acquired knowledge to good use!

I also hope these blog posts might help a newcomer to Rails, but they ended up being half ranting and half specific bug-finding, so I’m not sure of their value as a learning tool. Or as a read to anyone else than me actually. I’ll just post this on HN and let the crowd decide.