A Party to the World

Taskmaster from DISQUS

On 05.20.12, In computer science, By dan.lecocq

I have been waiting for an occasion to use dcramer’s taskmaster, which is a queueing system meant for large, infrequently-invoked (even one-off) tasks. In his original blog post brings up one of the features that was particularly striking to me — you don’t put jobs into the queue per se, but you describe a generator that yields all the jobs that should be put in the queue.

Occasionally at SEOmoz, we want to perform sanity checks on customer accounts, or transitioning from one backend to another, etc. In particular, we’ve been transitioning to a new queueing system, and we wanted to go through every customer and ensure that they had a recent crawl, and further, were definitely in the new system. Unfortunately, much of the data we have to check involves a lookup into Cassandra (that can’t be turned into a bulk operation very easily). Cassandra’s not necessarily the problem, but just the latency between requests. So, spawn off 20 or so workers with taskmaster, each given the details about the customer that we needed to verify.

The serial version takes 4-5 hours. It took 15 minutes to get taskmaster installed and grokked, and then the task itself took an hour. Already a win!

SEOmoz Blog Post — Introducing qless

On 05.18.12, In computer science, By dan.lecocq

I recently wrote a new dev blog post at SEOmoz about qless, a project I’ve been working on. It’s a queueing system that takes advantage of Redis 2.6′s server-side Lua scripting.

Happy queueing!

Tagged with: lua • qless • queueing • redis • seomoz

Redis and Lua for Robust, Portable Libraries

On 03.11.12, In computer science, By dan.lecocq

Redis 2.6 has support for server-side Lua scripting. Off hand, this may not seem like a big deal, but it offers some surprisingly powerful features. I’ll give a little bit of background on why I’m interested in this in the first place, and then I’ll show why this unassuming feature is so extremely useful for otherwise impossible atomic operations, as well as for easy language portability and performance.

For example, I’ve recently been working on a Redis-based queueing system (heavily inspired by Resque, but with some added twists) and a lot of functionality that I wanted to support would have been prohibitively difficult without Redis’ support for Lua. For example, I want to make sure that jobs submitted to this queueing system do not simply get dropped on the floor. A worker is given a temporary exclusive lock on a job, and must either complete it or heartbeat it within a certain amount of time. If that worker does not, it’s presumed that the worker dropped the job and it can be given to a new worker.

Now let’s imagine what this locking mechanism would have to look like in order to be correct. First, we’d probably maintain a list of jobs in a queue that have been popped off, but not yet completed, sorted by when their lock expires. When a client tries to get a job, it should first check for expired locks, and if it finds any, it should assume responsibility for those jobs. So this client sees an expired lock, and attempts to update the metadata associated with the job to reflect that it now has that job. In the mean time, the original client has swooped in and tried to complete the job despite the expired lock, removes the entry for the lock, and updates the job data to reflect its completion. It’s possible that the second client updates the job data after this, and inserts a new lock for itself, putting the system into an inconsistent state.

Yes, Redis has a mechanism for this, but it’s only so strong. There’s the `MULTI`, `WATCH` and `EXEC` combo, which allows you to detect the situation when another client has tried to modify a key for which you’re trying to perform an atomic operation and allows you to try the operation again. But for highly contentious keys, you can spend a lot of time backing off and failing. That’s frustrating.

Redis’ Lua support has an interesting guarantee: Lua scripts in Redis are guaranteed to be executed atomically. No other commands can be run on the Redis instance while the Lua script is running. With that in place, you are free to no worry in the slightest about these sorts of race conditions, because they just won’t happen. You can access as many keys as you’d like, without having to worry about `WATCH`-ing them for changes, and implement as simple or complex a locking mechanism as you’d like.

Another interesting feature that comes out of this is that if you implement your next Redis-based library as a collection of Lua scripts, then you can write bindings in other languages in a flash. The only requirement is that those new bindings must be able to read in a file, load the script, and then have Redis bindings to invoke those scripts. Clients no longer have to worry about mimicking any arbitrarily complex logic in their own language — they just rely on these Lua scripts that can be shared across all the bindings! This may go without saying, but maintaining bindings is something that can be a bit of a nuisance. One example that jumps to mind immediately is working with Redis from Node.js if a lot of successive commands have to be chained together. It can get extremely messy.

Not only this laundry list of wonderful features spring out of this Lua support, but it’s surprisingly performant. Without giving too much away, at SEOmoz, I recently implemented the queueing system I mentioned to support scheduled work items, heartbeating, priority and statistics collection in a collection of about 10-12 Lua scripts. Initial benchmarks have hit 4500 job pop/complete transactions per second on a 2011-ish MacBook Pro. At least for our purposes, this is plenty of room to roam. And let me assure you, these scripts are not always simple, and so the fact that Redis can still maintain good performance in the face of arbitrary scripts speaks volumes about the quality of Redis.

Tagged with: lua • portable • queueing • redis

The Cost of Except in Python

On 01.08.12, In miscellany, By dan.lecocq

I was curious recently about how much of a performance penalty try/except blocks incur in python. Specifically, 1) does it incur much of a cost if no exception is thrown (accepting only a penalty when something exceptional happens) and 2) how does it compare to if/else statements where possible? A snippet to answer the first question:

import timeit
 
withTryNoThrow = '''
	try:
		a = int('5')
	except ValueError:
		pass
'''
 
withTryThrow = '''
	try:
		a = int('z')
	except ValueError:
		pass
'''
 
withoutTry = '''
	a = int('5')
'''
 
results = {
	'withoutTry'    : timeit.Timer(withoutTry    ).timeit(100000),
	'withTryNoThrow': timeit.Timer(withTryNoThrow).timeit(100000),
	'withTryThrow'  : timeit.Timer(withTryThrow  ).timeit(100000)
}
 
for k, v in results.items():
	print '%20s => %fs' % (k, v)

For me, the results looked something like this:

      withTryNoThrow => 0.082781s
          withoutTry => 0.082880s
        withTryThrow => 0.261147s

It would appear that while catching exceptions is expensive, catching non-exceptions is very cheap. I imagine that the reason is mostly because when you throw an exception, you actually instantiate an exception object of some kind, which necessarily introduces some overhead. In the absence of that object creation, things can be relatively fast.

Now, for the second question. This particular question came up when deciding whether or not I should try fetching a key from a dictionary and catching an exception when it’s absent, or if I should use the get method and then check if the result is None.

import timeit
 
setup = '''d = dict({
	'some'   : 1,
	'keys'   : 2,
	'are'    : 3,
	'present': 4,
	'others' : 5,
	'arent'  : 6
})'''
 
tryExcept = '''
	try:
		a = d['doesntexist']
	except KeyError:
		pass
'''
 
getIfElse = '''
	a = d.get('doesntexist', None)
	if a == None:
		pass
	else:
		pass
'''
 
getNoIf = '''
	a = d.get('doesntexist', None)
'''
 
results = {
	'tryExcept' : timeit.Timer(tryExcept, setup).timeit(100000),
	'getIfElse' : timeit.Timer(getIfElse, setup).timeit(100000),
	'getNoIf'   : timeit.Timer(getNoIf  , setup).timeit(100000)
}
 
for k, v in results.items():
	print '%20s => %fs' % (k, v)

For this second test, the results looked something like this for me:

             getNoIf => 0.019980s
           tryExcept => 0.083638s
           getIfElse => 0.027422s

Obviously, if your program is amenable to just using a default value, then happiness ensues. Failing that, using the get method and then if/else is much faster than the try/except alternative.

Fine Print: I am running Python 2.7.1 on a 2011-ish MacBookPro.

Chef-Solo and God — A Divine Duo

On 01.06.12, In computer science, SEOmoz, By dan.lecocq

I recently wrote a new post on the SEOmoz dev blog about our deployment with chef-solo and god on EC2.

Tagged with: chef • god • seomoz

Python’s Logging Module – Exceptions

On 10.19.11, In miscellany, By dan.lecocq

I’m a big fan of python’s logging module. It supports configuration files, multiple handlers (for both writing to the screen while writing to a file, for example), output formatting like crazy, and many other delicious features. One that I’ve only recently encountered is its exception method.

The basic idea of the logging module is that you can get a logger from a factory (that allows multiple pieces of code to easily access the same logical logging entity). From there, you add handlers that output messages to various places (files, screen, sockets, HTTP endpoints, etc.). Every message you log is done at a specific level, and then the configuration of the logger determines whether or not to record messages of a certain severity:

import logging
 
# Get a logger instance
logger = logging.getLogger('testing')
 
# Some initialization of handlers here, 
# unimportant in this context
 
# Print out at various levels
logger.warn('Oops! Something happened')
logger.info('Did you know that X?')
logger.debug('Index is : %i' % ...)

What’s great about the module is that it separates your messages from how they’re displayed and where. For debugging, it’s nice to be able to flip a switch and turn on a more verbose mode. Or for production to tell it to shut up and only log messages that are really critical. What the ‘exception’ method does is to not only log a message as an error, but to also print a nice backtrace of where the error took place!

try:
	# So something here
	raise Exception('oops!')
except:
	logger.exception('Such-and-such failed! Stack trace to follow:')
	# Stack trace appears in the log

Tagged with: exceptions • logging • python

Never Trust Callbacks

On 10.16.11, In computer science, By dan.lecocq

It’s a lesson that has now been hammered home repeatedly in my head: never trust callbacks. Just don’t. Go ahead and execute them, but if you trust them to not throw exceptions or errors, you are in for a world of unhappiness.

For me, I first learned this lesson when making use of twisted, writing some convenience classes to help with some of the somewhat odd class structure they have. (Sidebar: twisted is an extremely powerful framework, but their naming schemes are not what they could be.) Twisted makes heavy use of a deferred model where callbacks are executed in separate threads, while mission-critical operations run in the main thread. My convenience classes exposed further callbacks that could be overridden in subclasses, but I made the critical mistake of not executing that code inside of a try/except block.

Twisted has learned this lesson. In fact, their deferred model makes it very hard to throw a real exception. If your callbacks fail, execution takes a different path — calling errback functions. In fact, twisted is so pessimistic about callbacks (rightly so) that you just can’t make enough exceptions to break out of errback functions. However, wrapped in my convenience classes were pieces of code that were mission critical, and my not catching exceptions in the callbacks I provided was causing me a world of hurt.

That whole experience was enough to make me learn my lesson. Then, a few days ago I encountered it again in a different library, in a different language, in a different project, where I was exposing callbacks for user interface code in JavaScript. The logical / functional chunk of code exposed events that the UI would be interested in, but was too trusting, leading to errors in callbacks skipping over critical parts of the code.

All in all, when exposing callbacks, never trust a callback to not throw an exception. Even if you wrote the callbacks it’s executing (as was the case with both of these instances, at least in the beginning). Callbacks are a courtesy — a chance for code to be notified of an event, but like many courtesies, they can be abused.

Tagged with: callbacks • exceptions • javascript • lessons learned • python

Python and Arbitrary Function Arguments – **kwargs

On 09.14.11, In computer science, Python, By dan.lecocq

Python has a pretty useful policy: named arguments. When you call a function, you can explicitly say that such-and-such value is what you’re providing for a particular argument, and can even include them in any order:

def hello(first, last):
	print 'Hello %s %s' % (first, last)
 
hello(last='Lecocq', first='Dan')

In fact, you can programmatically gain insight into functions with the inspect module. But suppose you want to be able to accept an arbitrary number of parameters. For example, for a printf equivalent. Or where I encountered it in wanting to read a module name from a configuration file, as well as the arguments to instantiate it. In this case, you’d get the module and class as a string and then a dictionary of the arguments to make an instance of it. Of course, Python always has a way. In this case, **kwargs.

This is actually dictionary unpacking, taking all the keys in a dictionary and mapping them to argument names. For example, in the above example, I could say:

hello(**{'last':'Lecocq', 'first':'Dan'})

Of course, in that case it’s a little verbose, but if you’re getting a dictionary of arguments programmatically, then it’s invaluable. But wait, there’s more! Not only can you use the **dict operator to map a dictionary into parameters, but you can accept arbitrary parameters with it, too!

def kw(**kwargs):
	for key, value in kwargs.items():
		print '%s => %s' % (key, value)
 
kw(**{'hello':'Howdy!', 'first':'Dan'})
kw(hello='Howdy!', first='Dan')

Magic! No matter how you invoke the function, it has access to the parameters. You can even split the difference, making some parameters named and some parameters variable. For example, if you wanted to create an instance of a class that you passed a name in for, initialized with the arguments you give it:

def factory(module, cls, **kwargs):
	# The built-in __import__ does just what it sounds like
	m = __import__(module)
	# Now get the class in that module
	c = getattr(m, cls)
	# Now make an instance of it, given the args
	return c(**kwargs)
 
factory('datetime', 'datetime', year=2011, month=11, day=8)
factory('decimal', 'Decimal', value=7)

This is one place where Python’s flexibility is extremely useful.

Tagged with: keyword arguments • kwargs • python

Kevin Mitnick’s Ghost in the Wires

On 09.12.11, In books, By dan.lecocq

I recently finished reading one of Kevin Mitnick‘s books, Ghost in the Wires. Fantastic. I constantly found it amazing that someone had lived that life, hacking, evading capture, changing identities. Reads like an action movie at many points, and in fact, several movies have been made loosely (and one very loosely) based on his life. Mitnick often talks about how much the “myth of Mitnick” is inflated or distorted, especially in the media and particularly with the movies.

As it turns out, Mitnick lived briefly in Seattle, and with my interest piqued, I figured I might be able to track down his old apartment. He describes going home one day before realizing his was being followed, and in the course of the description he mentions a few street names and the part of town he lived in. And at the end of the book, there’s a photo of the apartment, slightly too grainy to read the name of the building. But clear enough to read the number. A little time with Google Maps and found it! Being so close, I figured I’d drop by to take a picture:

Tagged with: kevin mitnick • seattle

Named Pipes

On 09.11.11, In computer science, By dan.lecocq

Yesterday I encountered a concept I hadn’t known about: named pipes. They’re essentially a path that acts as a pipe for reading from / writing to. In that sense, you work with them like with file redirection and traditional files. But that data doesn’t get stored anywhere really permanent. All data that goes through it is meat to be written once, and read once, and it comes with a performance boost of not having to write large chunks to disk.

Pipes, for those who don’t know, are the bees knees. They’re the cat’s meow. They allow you to (as the name implies) make a pipeline between one or more programs, with the output of one feeding into the input of others. Suppose, for example, that we want to find out how many files that contain ‘.a’ there are in a directory. There’s a tool you might know, ‘ls,’ that lists all the files in a directory. And ‘grep’ is a tool to search for lines of text that match a regular expression. And ‘wc’ is a tool that can count the number of bytes, words, lines, etc. in a file.

Typically, each of these operates in isolation, reading from a file (in the case of grep and wc), or… standard input. And they all write to standard output. A pipe is away to hook up one’s process’ standard output file descriptor to the standard input file descriptor of the another, making one the producer of information and the other the consumer:

ls -l /path/to/some/directory | grep '.a' | wc -l

This is typical of the design of many command line utilities. Most either come with an option to read from standard in (usually either the absence of a filename or a single ‘-’). And most do exactly one task well. Each has one very specific purpose, but is generally happy to play along with others.

File redirection is another handy tool that is related to named pipes. File redirection lets you either read the contents of a file as if it were standard input, or have a process write to it as if it were standard output. Going back to the earlier example, if we wanted to store a list of the all such files in our own file called ‘list’:

ls | grep '.a' > list

Easy as pie. Now… for named pipes. They’re also called ‘FIFO‘s for their first-in-first-out behavior. You can make one with ‘mkfifo <filename>’. And then, feel free to read from it and write to it. Perhaps in two different terminals:

# In one terminal:
mkfifo test
cat < test
 
# In another terminal:
echo 'hello' > test

The first terminal, cat plugs along doing the one thing it knows how to do: display what it reads in out to standard out. Take a minute for what has just happened to sink in. You were able to have one process wait around until it had something read… from a pipe. And in a completely different terminal, you had a different process communicate with the first one through opening a file. This is a mechanism that’s commonly used for inter-process communication (IPC) for obvious reasons — everyone knows how to read from and write to a file, so it makes use of a known paradigm. But wait — it gets even better.

Suppose you want to aggregate some statistics about how many different types of requests your application serves, but you don’t want to have to write that in. Or maybe it’s an application that you know already just writes to a log file. Of course, you could trawl the log file, but there are conceivably cases where you don’t want the overhead of keeping around huge files, so you’d rather avoid it if possible. You have to be careful when doing this (not all applications play nicely with named pipes — mostly surrounding blocking described below), but chances are you might be able to dupe the application into just logging to a named pipe! If you remove the log file and in that same path you make a pipe, then your work is done — just read from that pipe to aggregate your statistics periodically. This works particularly well with the python logging module.

Reading from and writing to a named pipe can be a little more nuanced, however. Some things to bear in mind:

Opening a named pipe can block, so consider opening them non-blocking. Of course, it depends on your access pattern, but if you’re not sure if another process has written to the pipe and you don’t want that to trip up your reading, non-blocking is the way to go.
Named pipes have ‘no size.’ If you write to a pipe, data gets queued up for the other end to read, but even before that gets read, stat(1) reports that the file has a size of 0 bytes. So, you can’t rely on a change in file size to know it’s ready for reading.
Instead, use select, poll, epoll, etc. to detect read/write-ability on the pipe. If you’re only interested in one file descriptor, then you can go ahead and use select, but if you’re starting to listen to too many, perhaps one of the others is a better idea

Tagged with: fifo • named pipes • operating system • os • pipes