For me, I first learned this lesson when making use of twisted, writing some convenience classes to help with some of the somewhat odd class structure they have. (Sidebar: twisted is an extremely powerful framework, but their naming schemes are not what they could be.) Twisted makes heavy use of a deferred model where callbacks are executed in separate threads, while mission-critical operations run in the main thread. My convenience classes exposed further callbacks that could be overridden in subclasses, but I made the critical mistake of not executing that code inside of a try/except block.

Twisted has learned this lesson. In fact, their deferred model makes it very hard to throw a real exception. If your callbacks fail, execution takes a different path — calling errback functions. In fact, twisted is so pessimistic about callbacks (rightly so) that you just can’t make enough exceptions to break out of errback functions. However, wrapped in my convenience classes were pieces of code that were mission critical, and my not catching exceptions in the callbacks I provided was causing me a world of hurt.

That whole experience was enough to make me learn my lesson. Then, a few days ago I encountered it again in a different library, in a different language, in a different project, where I was exposing callbacks for user interface code in JavaScript. The logical / functional chunk of code exposed events that the UI would be interested in, but was too trusting, leading to errors in callbacks skipping over critical parts of the code.

All in all, when exposing callbacks, never trust a callback to not throw an exception. Even if you wrote the callbacks it’s executing (as was the case with both of these instances, at least in the beginning). Callbacks are a courtesy — a chance for code to be notified of an event, but like many courtesies, they can be abused.

def hello(first, last):
	print 'Hello %s %s' % (first, last)
 
hello(last='Lecocq', first='Dan')

In fact, you can programmatically gain insight into functions with the inspect module. But suppose you want to be able to accept an arbitrary number of parameters. For example, for a printf equivalent. Or where I encountered it in wanting to read a module name from a configuration file, as well as the arguments to instantiate it. In this case, you’d get the module and class as a string and then a dictionary of the arguments to make an instance of it. Of course, Python always has a way. In this case, **kwargs.

This is actually dictionary unpacking, taking all the keys in a dictionary and mapping them to argument names. For example, in the above example, I could say:

hello(**{'last':'Lecocq', 'first':'Dan'})

Of course, in that case it’s a little verbose, but if you’re getting a dictionary of arguments programmatically, then it’s invaluable. But wait, there’s more! Not only can you use the **dict operator to map a dictionary into parameters, but you can accept arbitrary parameters with it, too!

def kw(**kwargs):
	for key, value in kwargs.items():
		print '%s => %s' % (key, value)
 
kw(**{'hello':'Howdy!', 'first':'Dan'})
kw(hello='Howdy!', first='Dan')

Magic! No matter how you invoke the function, it has access to the parameters. You can even split the difference, making some parameters named and some parameters variable. For example, if you wanted to create an instance of a class that you passed a name in for, initialized with the arguments you give it:

def factory(module, cls, **kwargs):
	# The built-in __import__ does just what it sounds like
	m = __import__(module)
	# Now get the class in that module
	c = getattr(m, cls)
	# Now make an instance of it, given the args
	return c(**kwargs)
 
factory('datetime', 'datetime', year=2011, month=11, day=8)
factory('decimal', 'Decimal', value=7)

This is one place where Python’s flexibility is extremely useful.

Pipes, for those who don’t know, are the bees knees. They’re the cat’s meow. They allow you to (as the name implies) make a pipeline between one or more programs, with the output of one feeding into the input of others. Suppose, for example, that we want to find out how many files that contain ‘.a’ there are in a directory. There’s a tool you might know, ‘ls,’ that lists all the files in a directory. And ‘grep’ is a tool to search for lines of text that match a regular expression. And ‘wc’ is a tool that can count the number of bytes, words, lines, etc. in a file.

Typically, each of these operates in isolation, reading from a file (in the case of grep and wc), or… standard input. And they all write to standard output. A pipe is away to hook up one’s process’ standard output file descriptor to the standard input file descriptor of the another, making one the producer of information and the other the consumer:

ls -l /path/to/some/directory | grep '.a' | wc -l

This is typical of the design of many command line utilities. Most either come with an option to read from standard in (usually either the absence of a filename or a single ‘-’). And most do exactly one task well. Each has one very specific purpose, but is generally happy to play along with others.

File redirection is another handy tool that is related to named pipes. File redirection lets you either read the contents of a file as if it were standard input, or have a process write to it as if it were standard output. Going back to the earlier example, if we wanted to store a list of the all such files in our own file called ‘list’:

ls | grep '.a' > list

Easy as pie. Now… for named pipes. They’re also called ‘FIFO‘s for their first-in-first-out behavior. You can make one with ‘mkfifo <filename>’. And then, feel free to read from it and write to it. Perhaps in two different terminals:

# In one terminal:
mkfifo test
cat < test
 
# In another terminal:
echo 'hello' > test

The first terminal, cat plugs along doing the one thing it knows how to do: display what it reads in out to standard out. Take a minute for what has just happened to sink in. You were able to have one process wait around until it had something read… from a pipe. And in a completely different terminal, you had a different process communicate with the first one through opening a file. This is a mechanism that’s commonly used for inter-process communication (IPC) for obvious reasons — everyone knows how to read from and write to a file, so it makes use of a known paradigm. But wait — it gets even better.

Suppose you want to aggregate some statistics about how many different types of requests your application serves, but you don’t want to have to write that in. Or maybe it’s an application that you know already just writes to a log file. Of course, you could trawl the log file, but there are conceivably cases where you don’t want the overhead of keeping around huge files, so you’d rather avoid it if possible. You have to be careful when doing this (not all applications play nicely with named pipes — mostly surrounding blocking described below), but chances are you might be able to dupe the application into just logging to a named pipe! If you remove the log file and in that same path you make a pipe, then your work is done — just read from that pipe to aggregate your statistics periodically. This works particularly well with the python logging module.

Reading from and writing to a named pipe can be a little more nuanced, however. Some things to bear in mind:

• Opening a named pipe can block, so consider opening them non-blocking. Of course, it depends on your access pattern, but if you’re not sure if another process has written to the pipe and you don’t want that to trip up your reading, non-blocking is the way to go.
• Named pipes have ‘no size.’ If you write to a pipe, data gets queued up for the other end to read, but even before that gets read, stat(1) reports that the file has a size of 0 bytes. So, you can’t rely on a change in file size to know it’s ready for reading.
• Instead, use select, poll, epoll, etc. to detect read/write-ability on the pipe. If you’re only interested in one file descriptor, then you can go ahead and use select, but if you’re starting to listen to too many, perhaps one of the others is a better idea

import psutil
 
psutil.phymem_usage().percent
# 31.2
psutil.virtmem_usage().percent
# 0.0

Pretty handy tool if you’re doing any sort of monitoring!

But rebuilding your development environment comes up more than just upgrading your OS. If you need to migrate to that new computer you got, or that work gave you, or help someone else get up and running with a project you’re thinking about releasing. Admittedly, it took me a little while to learn this lesson, but finally it’s drilled into my head: keep build notes!

A couple weeks ago I was trying to install an internal package whose docs hadn’t been updated in a very long time. After struggling and hitting countless snags, I finally got it up and running when I got an email that was along the lines of, “Oh, if you could write down what problems you ran into, that would be great.” Fortunately, I just made notes of what I had done in order to get it built, and I was able to whip off a reply with speed that surprised the recipient.

Even at a system-wide level, I try to make it a habit to record every package I install/build associated with development. It makes it extremely easy to get set up on the next system, even if the instructions have to be updated for a new environment. I call it a manifest and I manage it as a flat file, though I know there are package managers that can do a lot of heavy lifting for me. However, I find that no package manager is perfect and so even if I make use of one for certain libraries, it’s important to me to have everything documented in one place. At a minimum (and you probably don’t need more than this) keep the following:

1. Package name and version – Maybe you needed readline 6.1 to get your project running, or you know that such-and-such version is buggy for your purposes.
2. Why you installed it – I find that many libraries I install are used for a particular project, and so it’s useful to have the motivation for getting it.
3. How you installed it – Whether it was macports or a typical configure, make and install, how did you build it? Did you need special flags to make it go? You will absolutely forget these, so why not write them down? Even just copy and paste from your history!

I can’t stress enough how much easier this has made my development life in a lot of ways, and how little a time investment it is.

Apparently, there are a couple conventions for doing this. First, you can inject a configuration step (à la autoconf, for example) which would detect what platform you’re building on in a robust way and build a Makefile for you. Second, if you’re lazy or autoconf would be like hitting a fly with a hammer, you can use make’s conditionals:

# Ensure that this gets declared in time,
# and fill it with the result of `uname`
UNAME := $(shell uname)

# If the environment is Darwin...
ifeq ($(UNAME), Darwin)
CXXFLAGS = # Something Apple-y
else
CXXFLAGS = # Something Linux-y
endif


Simple enough!

While this is fine and dandy (and admittedly obviously attractive in ways), there is a big problem SEOmoz co-worker Brandon pointed out: There’s no way to determine which code path will be taken.

For a little bit of context, I came across a set of type-conversion operators that seemed reasonable enough. They tried to cover the whole gamut of possible primitive types:

operator unsigned long long() const;
operator long long() const;
operator unsigned long() const { return operator unsigned long long(); }
operator long() const { return operator long long(); }
...


The compiler has absolutely no problem with the above declaration. The class you put that in will happily compile, but the problem arises when you try to use it:

Widget w(...);
// Suddenly, the compiler complains, not knowing which operator to use
unsigned long int foo = w;


At this point, the compiler puts its foot down. What to me seems unintuitive is that even though there is an conversion operator to this exact type, the compiler won’t use it. What’s even more bizarre to me is that typedefs and in-header definitions can further muddle things up:

operator long long() const;
operator long() const;
operator int() const;
operator short() const;
// For whatever reason, let's say you do this:
operator int32_t() const {
return operator long long();
}


Even though int32_t will be the same as one of those other types, this will still compile. It makes a certain amount of sense when viewed in the context of the compiler because after all, it only does so much processing on headers because they’re going to be directly included wherever you use them. You actually don’t get duplicate symbols in this case, and thus no “previously-defined” error. In reality, their function definitions are the same, and they actually get mangled to the same name (on my system the operators for int32_t and int both mangle to ‘_ZNK6WidgetcviEv’):

# See what mangled symbols actually appear
nm -j widget.o
# See what demangled symbols are actually there
nm -j widget.o | sed s/__/_/ | grep -v .eh | c++filt -n


The above (with in-header definitions) is exactly what we encountered in the code. We (well, a co-worker) suspected that the reason that the sort of multiple definition was allowed was that the names were getting mangled based on their typedef name string (mangled on int32_t instead of the actual type it maps to), but this is not the case. If you move the in-header definition for the int32_t operator into the .cpp file, the compiler will complain to you earlier.

My first inclination when dealing with the “conversion to type long long is ambiguous” error was to ask for an explicit conversion: static_cast<long long int>(myWidget). However, this doesn’t work either. So even in this scenario, you can’t even ask for a specific type conversion operator. From what I can gather, type-conversion operators are a double-edged lightsaber: few things in C++ were added without a purpose, but it’s extremely important to understand that exact purpose and its risks. To require that type conversions are explicit you should generally use something like:

template
const T convert() const {
...
}

template <>
const bool convert() const {
// Your conversion to bool
...
}


For instance, I was extremely frustrated this week to find that on OS X Lion, gcc-4.4.5 just would… not… compile. Frustrating stuff. I was tasked with porting this enormous existing in-house code base (of about 60-100k lines) to Mac, but was dismayed to find that they required C++0x features, which are unsupported in gcc 4.2.

Giving up, I turned to MacPorts as a broken, empty shell of a man. MacPorts was able to compile it, though relegated to /opt/, and though I could add that to my path, this new version of gcc wasn’t ready to consider the libraries I had installed in /usr/local/ by hand. Of course, I could edit all the makefiles, or do some other magics, but it turns out MacPorts can be bent to your will.

Like most, I had installed the binary release of MacPorts, configured to live in /opt/, but if you instead build from source, you can:

./configure --prefix=/usr/local --with-unsupported-prefix


This has not only MacPorts reside in /usr/local/, but then it will also in turn install its packages there as well! I don’t think I’m the only one who appreciates that kind of consistency — all my libraries in the right place. I still feel slightly dirty whenever I have to rely on port, but at least when I do, I can save a little face.

So, whenever I open up one such block, I add a little macro:

#ifndef MAC
// ... The Linux-only code
// ... takes up
// ... a lot
// ... of space
#else
#error "Don't forget to implement it as Mac!"
#endif


At least I’ll always catch it at compile time, and when I fix/add the Mac-only code, then I can go ahead and remove that macro. It’s not that I’m forgetful, but I’ve shot myself in the foot so much at this point.

G++ kept giving me errors for each of those lines: “error: expected unqualified-id before ‘signed’,” referring to the line “using ::int8_t.” I’m a little embarrassed that I couldn’t figure it out right away, but eventually I figured out that it was caused by int8_t being #define’d somewhere else. For those of you that don’t know, #define really just takes one term and substitutes every subsequent occurrence of that term with one you provide.

// If you define it as a macro
#define int8_t signed char
// Then this line will be interpreted very differently from how you expect
using ::int8_t;
// Gets interpreted as "using ::signed char"!!!


And this is what g++ had been complaining about. That is not legal C++ syntax; I’m sorry I doubted you, g++! But there still remained a larger question: where were these types getting defined? I didn’t want to be in the business of “patching” a library, and especially the largely impeccable Boost library. Typically these types (int8_t, uint8_t, etc.) are defined in cstdint or stdint.h, but looking at the system’s version, I found to my surprise that they were not macro-defined, but typedef’d, which is the right way to do it.

Sidebar: In general, you should be using typedef instead of #define for this reason, and another very good reason. Because #define macros just go through code and blindly replace references, it can be difficult to trace the origin of a type, but typedefs are carried through, and so even after preprocessing, you can still see what the semantic meaning was (that you wanted int8_t specifically, not just something that happens to be the same type). And when debugging, this extra type information can be helpful. Similarly, you should generally also use const to define constants in your code, instead of #define macros, because while you might remember what a magic number is when you write it into your library, the meaning of that particular constant becomes unclear when you encounter it’s value when debugging. (If you haven’t, read Scott Meyer’s Effective C++.)

Getting back to the morality tale, the library I was porting wasn’t macro-defining int8_t, and stdint.h wasn’t, so where was the culprit? Clearly there are potentially hundreds of places it could be, and I was running out of good guesses. Luckily, SEOmoz C++ shaman Martin taught me a little ninja magic: use the -E flag with g++ to only run the preprocessor stage, and redirect the output into a file! When compiling with ‘make,’ it typically spits out the offending g++ command, so rerun it to just pass that one through the preprocessor, which reaches out and fetches the header files and gloms them in order into one giant input file. Then, step through this file to see where “define” and “int8_t” occur on the same line! In two minutes we were able to find the header that was causing all this trouble, where I had spent two hours learning and reading about where the problem might be.

In the end we found it in a very small library that we happened to use, and on Mac we had just been using a slightly old version and this problem had been fixed in subsequent releases. Still, I’m glad to have added this preprocessor trick to my toolkit.

To me, it’s a pretty clear sign that you’ve done something wrong if people are writing plugins for you browser to restart it, because it consumes too much RAM with some regularity.

One of my favorite questions, though, was one about pouring bottles of champagne: Suppose you are in charge of pouring the champagne for the midnight toast at a prestigious party with more VIPs than you can count. You have 10 waiters, who are wheeling in 1000 bottles of champagne and some bad news; exactly one of these bottles is poisoned, and if you drink even a single drop, you will become violently ill in one hour. If you serve tainted champagne to any guest, you and your employees will be fired on the spot.

Your waiters are sympathetic to this fact, and have thus volunteered to be taste-testers — a night home on the couch being miserable is better than being unemployed. Though, because of time constraints, you only have time for one testing before people get sick and you have to serve champagne.

The naive approach is to split the bottles up into groups of 100, and have each waiter try a drop from each. You would know from whichever waiter gets sick which group of 100 bottles has the bad bottle, but there would be 99 good bottles wasted, and of course, the wasted good bottles get deducted from your pay.

So, you want everyone to keep their jobs (and not serve tainted booze) while wasting as few good bottles as possible. How many bottles do you have to waste? Click below to reveal solution.

Show ▼

My uninformed search led me to QUnit, from the same people that bring us jQuery, which we already use heavily. We found it immediately simple to use, expressive and powerful. Much of this particular site involves AJAX, and as such, we wanted to be able to test and make sure our interface to these resources is working, as well. Not to mention that most operations rely on information retrieved from remote resources.

To that end, the vast majority of our tests use their asyncTest function, which lets you perform any kind of asynchronous request, and then in your callback, you indicate to the system that your test has all its necessary information and can continue. For example,


asyncTest("Our Site's API", function() {
$.ajax({...
success: function(response) {
ok('data' in response, "We expect data in the response.");
}
}
}


One big ‘get’ in my mind is that it’s from the same group that produces a library we already use heavily, so they tend to be thought-out in similar ways. Plus, it runs in the browser, and has nice styling for the interface that make your unit tests look extra classy!

The QUnit site has a lot more examples and demos, but this concludes my shameless plug for a unit testing quite I’ve come to appreciate very much.

The first major point of git is that everyone has their own copy of the repository. When you commit changes, you commit them to your local copy of the repository. If you are working on a group project, there is a shared resource that can be pushed to and pulled from, but I actually like that it takes an extra command to do that — it forces me to make sure that it’s what I actually want to do. Now, to the bare-bones commands:

1. git init — Wherever you’re writing your code, type “git init”. It creates an empty repository in that directory (there’s a magical hidden folder “.git” that gets created and knows things).
2. git status — Git knows which files have changed since you last saved changes, and it will happily tell you which files are new and changed with this command.
3. git add — When you change files and are at a point where it make sense to save changes to your code (a bug fix, a new feature, etc.), then tell git which files you want to put in this commit with “git add”. If committing a set of changes with git is like shooting a gun, then adding files to be committed is like loading that gun. Git knows which files have changed, but it can make sense to group changed files into different logical commits. For example, if you fix two bugs between commits, you might want to add the changes for each bug fix separately.
4. git commit — When you have added a bunch of changed files to be committed, now you’re ready to actually commit those changes. Type “git commit -m ‘A short, meaningful summary of the changes that happened.’”

There is a lot more to git, and a million tutorials, that will explain things in more detail, but these are the commands I spend 95% of my time using, and enough to at least get you starting tracking changes for anything and everything. It doesn’t matter very much what version control you use, just use something.

There are other issue trackers with an API, but this one seemed reasonable enough so we used it. The big get for us in building a web-based app, is that it lowers the barrier for reporting bugs. We set up our development server to include a bug-reporting button that collected a little report that then sent it to our server, which uses the Unfuddle API to add a ticket. It also records and submits a little information about the user’s browser, current session, etc. that might help in debugging their problem.

Not the most complicated or profound system in the world, but it does put user-reported (and tester-reported) bugs in the same place we check our to-dos and other issues. No checking a support email account we remember once a month. Just puts bugs right were we’re looking anyway. I hope to share a little bit of the backend code soon!

1. Version control – svn, git, something! Anything, really! Just use something!
2. Unit testing – sure we talked about it, but its utility isn’t clear until you use it in earnest
3. Documentation – always felt more like a chore than a tool
4. Issue tracking – Joel on Software said it best: “I was born with only two bug-storing-slots in my brain.”
5. More best practices – Effective C++ (it’s worth it — get it and read it) alone was worth more than the course was.

I concede that these are things whose utility might be hard to convey to college students who may or may not be working on projects they really believe in. Still, very important.

One of the things I like about documentation is that you can refer to the documentation when uncertain about side-effects, input, etc. of functions and you don’t really feel like reading through code and trying to keep it in your brain-RAM. Not only that, but I usually refer to my own documentation to get an idea of the guarantee that a function was offering, to try to make sure that it’s making those same guarantees after I make changes. Not only this, but articulating functionality crystalizes it’s implications in your mind. When describing how something works to a friend, it often reminds you of issues you hadn’t considered. Writing documentation is the same in this respect.

When working with other people, while it’s nice to be able to talk face-to-face, the issues we think of that need changing go in one ear and out the other for me. Not that it wasn’t a good idea, but my brain-RAM is not infinite. Bugs, features, ideas, issues all get paged out of my head. Write them down instead, which is where issue tracking comes in. Design questions and decisions come out of issues all the time, and the trail of comments and discussion make a nice reference for documentation.

I find that when I’m fixing a bug, the functionality of that chunk of code gets reduced to that particular feature in my mind. So, if when I try using it, if that bug is fixed, then my job is done. But we’ve all experienced unintended side effects, where in fixing one thing, we break another. Having a list of features to check is a step up from trying to remember the things that need to work, but it’s much less than the power of unit testing. I’ve recently been using UnitTest++ for C++ applications, and QUnit from the jQuery people, and have been pleased to thrilled. It allows you to check functionality exhaustively and quickly on a whim. Even the task of writing the tests forces you to define the functional requirements of your code, and I find it helps to clarify a lot.

Version control lets you try crazy new approaches and track changes, isn’t that great? I was helping a friend last semester with some of his code, and he was trying to make code changes with copy, paste, and the undo buffer. In the end, he ended up introducing more bugs into the code than he was fixing. It’s intractable to keep these kinds of code changes in your brain-RAM. Use something, use anything. I have a qualm with svn because it was something I used so rarely, I could never remember how to set it up. Even a barrier as small as that was enough to prevent me from using version control at all. I love git because it’s so easy to set up, I use it for almost anything: from a report (if using LaTeX), to a weekend project, to my thesis. There are tons of tools out there, but just use something.