Static binaries relying on iconv

Symptoms

I often build static binaries (you know, ghc --optl-static --optl-pthread) when I need to run some program on a system that is different from my build system.  Somehow, this seems easier than installing a full build setup.  When i recently tried to run such an executable on a different system, the system responded with mkTextEncoding: invalid argument.  Googling didn’t exactly help, except pointing me to something iconv-related as the culprit.

Diagnosis

It turns out that Haskell programs built with GHC now (version 6.12?) rely on iconv to …well, do some Unicode stuff.  Static linking bundles iconv in the binary, but apparently iconv goes off and reads some dynamic libraries anyway, embedding their paths (as they are on the build system) inside the executable.

Needless to say, this breaks when the target system decides to put these library bits elsewhere.  Specifically, Ubuntu puts these files in
/usr/lib/gconv/, while Red Hat puts 32-bit versions in that directory, and 64-bit versions in /usr/lib64/gconv/.  A 64-bit binary built on a Ubuntu systems thus tries to load 32-bit library code, and I guess we’re lucky we even get an error message.

Remedy

Far be it from me to question the wisdom of embedding paths to local library code in static executables; instead, let me just commend the developers for providing the GCONV_PATH environment variable, which, when set to point to the lib64 directory, made it possible to run my executable.

Update: Apparently, the exact error produced can vary, and I recently got openFile: invalid argument (Invalid argument) instead.  Using strace it was clear that the executable was loading the wrong library again, and setting GCONV_PATH solved the problem.

Laziness change in binary

Symptoms

I have written a small program to analyze and extract information from 454 sequencing file.  In order to efficiently process these files, which can be fairly large, I use the excellent binary library to decode SFF files at disk speeds.  On one occasion, this program started to use a lot of memory, but with a little help from the community, this got sorted out by some inlining.  Recently, the same thing appeared to happen again, but in spite of all my tampering with the previously offending code, I was unable to build any version without this behavior.

Diagnosis

After some quick testing that showed that Data.Binary.decode was, contrary to earlier behavior, strictly evaluating its input before returning anything.  A quick mail to the binary developers confirmed that this behavior had been changed, as it sped up GHC itself.

Edit: I didn’t find it previously, but dons has a nice writeup of the change.

Remedy

Flower depends on being able to lazily process the list of sequences in its input, and the new version of binary not only caused it use excessive memory, but also slowed it down.  I don’t really have any better solution to this (and my use case is probably not important enough to sacrifice GHC performance for than to require binary version less than 0.5.

The original version

I’ve taken the liberty of cleaning things up a bit, and removing some context that I sure hope isn’t necessery.  Bascially, the task is to take a list of ByteString input lines consisting of whitespace separated decimal integers, and build a ByteString consisting of single byte quality values corresponding to those integers.

Below is how the naive quality parsing function might look, unpacking each word to String in order to use read.  Note that B is lazy ByteString.Char8, BB is lazy ByteString, that is, the Word8 version.

BB.pack $ map (read . B.unpack) $ B.words $ B.unlines ls

We don’t expect this to do terribly well, and I guess it’s no surprise when this parses my test file of about 10MB in 24 seconds.

Improved versions

The key to improved performance is first and foremost to avoid the unpacking and parsing of strings.  Thankfully, the ByteString library provides a readInt function.

BB.pack [lookup x | x <- concatMap B.words ls]
    where
    lookup x = case B.readInt x of Just (v,_) -> fromIntegral v
                                   Nothing -> error "Unparsable qual value"

This isn’t a lot more complicated than our initial attempt, but the speed increas is considerable: slightly less than 2 seconds for the test file, more than a tenfold improvement.  The ByteString implementation will share the storage of the separate words with the original strings, but since readInt gives us back the rest of the string in addition to the parsed integer, we might as well make use of it:

BB.pack $ readInts $ B.unlines ls
    where readInts xs = case B.readInt xs of 

                          Just (i,rest) -> fromIntegral i : readInts (B.dropWhile isSpace rest)
                          Nothing -> []

This turns out to be a bit faster, time is now 1.6 seconds.  Another 20% shaved off.

Final version

We’re really only interested in Word8 values, since quality values always are small, and since that’s what gets encoded in the result anyway.  The previous versions takes a detour by reading Ints and using fromIntegral to convert them to the desired size.  It bears noting that there is no error checking involved, fromIntegral will happily and silently truncate any number beyond its target range.  So lets do things explicitly, using Word8s throughout the computation:

BB.pack $ go 0 ls
    where
    isDigit x = x <= 58 && x >= 48
    go i (s:ss) = case BB.uncons s of 

                    Just (c,rs) -> if isDigit c then go (c - 48 + 10*i) (rs:ss)
                                   else let rs' = BB.dropWhile (not . isDigit) rs
                                        in if BB.null rs' then i : go 0 ss else i : go 0 (rs':ss)
                    Nothing -> i : go 0 ss
    go _ [] = []

This is the fastest one so far, clocking in at 0.94 seconds, over 40% faster than the best readInt version, and about 25 times faster than the naive version.  Still, 10MB/s is well below the average hard disk.

So is there more room for improvement?  The most obvious wart to me is the rather artificial splitting into lines.  This is mostly an artifact of some early design desicions, and it should be possible to eliminate the splitting earlier on and saving even more time by simplify this function quite a bit.

If you spot anything else, or have suggestions, I (and my darcs repo) am all ears.

Edit: Since some people have asked, I’ve wrapped up a simple test program along with some test files at http://malde.org/~ketil/biohaskell/qualparsetest.  This is a simplified version, if you want to be really helpful, you could always look at Bio.Sequence.Fasta in the Bioinformatics library and see if you can speed up e.g. dephd -i input.fasta input.qual -F /dev/null.

The Haskell bioinformatics library has for some time now supported reading and writing the SFF format, which is used by the oldest (previousest?) of the next generation technologies, namely Roche’s 454 sequencing.  Once the library functionality is in place, it is easy to develop small tools for doing the various chores.   After spending some time in anticipation of the hordes of programmers no doubt rushing to exploit the monumental effort put I down in  the library, I’ve instead written a few programs myself, including tools information/statistics extraction (flower),  extracting sequences by various criteria (fselect), simulating sequencing (pyrosim), and repairing broken SFF files (frepair).  This is their story.

The bioinformatics library includes functions and data structures to read and write and extract data from SFF files.  It provides reasonable performance (meaning that in most cases, my disk is the limiting factor).  It is also a clean room implementation, built from official documentation, but not based on Roche’s (or anybody else, for that matter) code.  It comes with an LGPL license, which I hope will make it useful while encouraging back contributions.

Flower

This program can extract  various information from SFF files.  To some extent, it overlaps with ’sffinfo’ and ’sfffile’ from 454, but in addition to generating Fasta sequences (optionally with quality), it can also extract directly to the more compact FastQ format.  It also can output a (huge) table of flow values, the histogram of flow values (useful for estimating the flow distributions, and thus probabilities for the base calls), or a one-line summary of each read that includes some statistics on lenghts and quality. Here’s the usage info:

flower: Usage: flower -[f|q|r|R] <file.sff> [<file2.sff> ..]
  -r  output reads in Fasta format
  -R  output reads in Fasta format with associated .qual
      (generates files instead of writing to <stdout>)
  -q  output in FastQ format
  -f  output the flowgram in tabular format
  -h  output a histogram table of flow values
  -s  output a summary of each read

FSelect

FSelect takes an SFF file and produces a new SFF file containing a subset of the sequences, using the same statistics as Flower can output.  It has a small expression language built in, so that you can build more complex logical queries.  For instance, if you want to extract the sequences with lenght between 300 and 400, and with a K² score greater than 0.7, you could do

fselect "And (Func LT k2 0.7) (And (Func GT len 200) (Func LT len 400))" FL61AHU01.sff

Okay, it’s a bit clunky, but the syntax should be reasonably straightforward: Logical operators are And, Not, and Or, while Func op f v defines a function using op (either LT or GT) to compare f (one of k2, len, tlen, ncount)  to each read.  Output is generated in a file named “selected.sff”.
FSelect can also select random reads, using the select expression "Rand p", where p is the probability for selection.  (I.e. fselect "Rand 0.2" FL61AHU01.sff will select each read with a probability of 0.20, giving you approximately 20% of the reads.  Random selection can not be combined with other criteria at this point, if you want this, you’ll have to run fselect multiple times.

FRecover

We had some issues with broken SFF files, specifically there were block of zero bytes at random places. Both flower and sffinfo just terminate on encountering a broken read, so I implemented a simple utility that attempts to skip the broken block and continue extracting good reads beyond the trouble.

PyroSim

This attempts to simulate pyrosequencing, for now it only does the GS20  generation of 454 sequencing, but the other generations should be easy to add.  The main problem is that the algorithm for quality calling is insufficiently documented.  GS20 has the advantage that quality for a homopolymer is uniquely derived from the flow value (modulo rounding), so reverse engineering it is fairly straightforward.

Anyway, pyrosim takes a ‘generation’  (at the moment, this is only GS20) and a Fasta file as input parameters, picks random points in the Fasta file, and produces the correspondig flowgram, including a suitable perturbation of the values to introduce the expected measure of noise, calls the bases and quality, and outputs an SFF file.

Availability

Flower, FSelect and FRecover are part of the “flower” package, available from the darcs archive at http://malde.org/~ketil/biohaskell/flower

PyroSim is available separately (for now, at least) from the darcs archive at  http://malde.org/~ketil/biohaskell/pyrosim

I try to upload what I consider stable versions to HackageDB, please check the Bioinformatics category there.  Currently, these programs are in a bit of flux, so going with the darcs repo is probably a good idea at this point.

I’d really like to have packages for the most common Linux distributions as well (i.e. .debs and .rpms), but I don’t know the details of how to produce them, and while I’ve made half-hearted attempts in the past, I guess I just don’t really desire it enough.  I’d be happy to see somebody package it up, so if you know how, please go ahead.

Filtering out empty sequences.

Phred often produces zero-length sequences, and this confuses other programs.  While BLAST will just output a warning, SeqClean — or to be precise, cln2qual — will break down.   (My own code using the Bioinformatics library treats all sequences the same regardless of length, so zero-length sequences are perfectly okay). Anyway, you can now use dephd -z to eliminate them from the output.

Sequence Clipping

Sequence trimming or clipping is often necessary to remove contamination like vector sequence, or simply low quality sequence parts.  Typically, both of these occur at the ends of the sequences.  Many programs (including dephd, but also phred, lucy, seqclean and others) add trimming information to the sequence header.  Dephd is now able to act on this information and clip the sequences.  The trimming information is now obsolete as the coordinates have changed, so they are replaced with the clipping coordinates.   Dephd also provides its own quality assessment, and with the -q option, sequence ends where the sliding windown average quality is below 15 will be clipped.  This is pretty heavy-handed, but it seems I get better EST clustering with this enabled.

Old features

Of course we retain the old features: reads PHD and Fasta/Qual files, mask (to lower case/N but don’t clip) by quality, generate quality plots, outputs Fasta/Qual, and ranking sequences by quality.

Edit: In the latest release, there’s now also a fix for a problem with drawing quality graphs with gnuplot.  It turns out that my shiny new Ubuntu ships with gnuplot 4.2, but your crappy old distribution ships with an older version, and that there are some incompatibilities in the input formats.  I’ve now reverted this to use old-style format only, so hopefully it should work with gnuplots back to 3.7 or so.   And for those SLS or MCC interrim die-hards out there, I’ll even add an option to dump the gnuplot file itself, so that you can copy it to a floppy and generate the plots on a computer with a modern color display. How’s that for user friendly?

The safety that we seek can be exemplified by sequence alignment or similarity scores, as calculated by e.g. the BLAST suite of programs.  Here we have e.g. blastn for comparing nucleotide sequences, blastp for comparing amino acid sequences, and blastx for comparing nucleotides to proteins by first translating the nucleotides to the possible corresponding amino acid sequences. We’ll aim for an align function that does the right thing for the various combination of sequence types.

The old approach

Previously, there was only one Sequence data type, defined as:

data Sequence = Seq SeqLabel SeqData (Maybe QualData)

(where SeqLabel, SeqData and QualData are represented as various ByteStrings, but you can think of them as synonyms for String)

As mentioned, this isn’t really leveraging the type system much, and at the very least, it’d be a good thing to be able to differentiate between nucleotide and  amino acid sequence (a.k.a. peptides or proteins).

The algebraic data type approach

The default functional programming approach would be to use an algebraic (sum) type, what the imperative programmers woudl call a union.

data Sequence = Nucleotide ... | Protein ...

This is a good solution when the two branches have different structure, but here you’d essentially just repeat the structure.  Moreover, all functions will need to do run-time checks for each case, imposing a cost both in function complexity, running time, and type safety.  Note that performing the alignment usually requires a score matrix describing the penalty for replacing any given character with any other.

align :: Matrix -> Sequence -> Sequence -> Alignment

align mx (Nucleotide ...) (Nucleotide ...) =
align mx (Nucleotide ...) (Protein ...) =
align mx (Protein ...) (Nucleotide ...) =
align mx (Protein ...) (Protein ...) =

This is not even complete, since a similar restriction applies to the score matrices – sometimes it will contain replacement penalties for the nucleotide alphabet, and sometimes it will contain penalties for the amino alphabet, and the appropriate matrix must be used in each brach of the align function.

Another problem with this is that this requires you to over-specify the type.  Sometimes you don’t know or care what kind of sequence you have.  Say you are selecting sequences by name from a Fasta file.  Since the file format is the same you don’t really care what kind of sequence it is, and forcing it to be one or the other is….immoral.

The third way: phantom types

The chosen approach is instead to tag the Sequence type with a phantom type parameter, phantom meaning it does not affect the actual representation of the data.  It looks like this:

data Sequence t = Seq ....  -- but no data member of type 't'!

Now, we can write our alignment functions, and make them safer to use:

align :: Matrix t -> Sequence t -> Sequence t -> Alignment
alignX :: Matrix Amino -> Sequence Nuc -> Sequence Amino -> Alignment

Note that we also phantom-typed the Matrix type.  With this approach,  incorrect usage like comparing sequences of different type with the generic align will be flagged by the compiler, so run time checks are no longer necessary.

On the other hand, readFasta can now be given the type:

readFasta :: Filepath -> IO (Sequence a)

or:

readFasta :: FilePath -> IO (Sequence Unknown)

depening on how much you trust the programmer.  Since I’m writing most of the programs using this library, I know how much you can trust application programmers, so the second and safer method is chosen.

 In practice

The old biolib repository has now been replaced by two new ones: biolib-stable, currently containing version 0.3.5, and biolib-unstable at 0.4.0. Perhaps unsurprisingly, the latter version contains the phantomly typed Sequence definition.

Currently, three types are used for tags:  Amino, which is the type for the amino acid alphabet,and Nuc and Unknown, which are dummy types without any data constructors, and used solely for this purpose.

So what are the experiences so far?  Well, it does complicate things.   For some file types the sequence type is known, for instance the output of nucleotide sequencing machines in the form of ABI, SCF, or SFF files.  But often, file formats are agnostic with respect to sequence types, the most ubiquitous offender being the Fasta format. Currently, this is handled by the Unknown type tag, but I’m not entirely convinced this is the optimal solution.

Code needs to be updated, but mostly this is relatively easy.  The type system will spot the difficulties, and often you can get by by just replacing Sequence in the type signatures with Sequence t.  Just make sure that t isn’t already used in the signature – I stumbled into this one.

Of course, this is one step towards increased leveraging of the type system.  One could go further, and  tag different type of nucleotide sequences based on sequencing technology: Sanger sequencing has quite different error characteristics from 454 sequencing, and Solexa has a completely different interpretation of the quality values than either of those.  Also, one might want to include the presence or absence of quality data in the type as well.  There are also different amino acid alphabets – should they have different types?  This is a difficult design space – how much information should be encoded in the type?

While I think this is an improvement, I’m very curious how this works out in practice, or if there are other options I should consider.  Please comment!

Also, since short reads are all the rage, and my flower program appears to be used a bit, I’ve done a quick writeup of its features and use as a static page.  I’ll try to keep it updated as things progress.  Popularity – Yay!

Finally, I got some help compiling everything on some less mainstream operating systems (“Windows”, I think it is called).  Mostly, things appear to work, and some improvements – albeit portability-neutral ones – were made.  Portability  – Yay!.

First things first, here are the files:

Bloom filters for bioinformatics – paper

Bloom filters for bioinformatics – slides

The new addition to the slides consist of the counting bloom filters, and a brief overview on how to locate matches in a Bloom filter.  The standard counting Bloom filter consists of replacing the array of bits with an array of bit buckets (of size b, say).  This lets the filter count up to 2^b occurrences of each element, typically using saturating counts.  If you want to retain the false positive rate, this means you’ll expand the size of the Bloom filter by a factor of b, which can be quite significant. In the example below, b is set to 3, and the filter saturates at a count of 7.

Therefore, I propose a different kind of counting bloom filter, using a distributed count.  Keep in mind that we have an infinite sequence of hash functions.  When inserting a new value, we can check the k hash functions to see if it’s already there.  If so, we calculate some additional hash functions to represent the count of two for this value.  And so on, until we find a count (and corresponding set of hashes) that have at least one zero bit, set those bits to one, and move on to the next value.  It may make sense here to decrease the number of hash values as the count goes up.  In the example below, we see that x is already inserted, and two new hash values are calculated to increase the count.

There’s a trade off, of course.  Insertion and lookup is no longer (as) constant, but  proportional to value count. However, the Bloom filter is likely to be more compact – given n values in total, where u are unique, the standard counting filter needs a size proportional to bu, while the distributed counting filter needs size proportional to n.  If you want to count accurately, b needs to be set to the logarithm of the max count of any value.

Finally, it’s also possible to use Bloom filters for searching – for instance, locating unique words in a genome. A lookup in a Bloom filter basically gives one bit of information – present or not present.  We use this with a series of m Bloom filters, each indexing the regions of the genome corresponding to one bit of location information.  Looking up a unique word in the set of filters reveals the location with m bits of precision.

In other news, inspired by an article on programmer productivity at LWN, I ran darcs-graph on my code to see how I do.  I guess I consider myself an about average programmer, and it looks like I can average about five commits a day when I’m working on a project.  I’m occasionally touching 20 commits, but that’s probably a built-up backlog of patches.  Let’s see where this puts me:

If you accept the proposition that the best programmers outperform the poor ones by a factor of 50, an average programmer should outperform a poor one by a factor of about seven, and also be outperformed by a top-notch one by the same amount.  So if I’m about average, the best ones should be able to beat me by a factor of about seven, which means 35 commits a day, or about ten minutes per commit throughout the day.  Scaling up my peak commit at 20/day would give 140 commits, about 20 per hour, or on average 3 minutes between each.  Conversely, a lousy programmer should be struggling hard to make a commit at all during his working day.  Both ends of the spectrum seem a bit incredible, but I guess the lousy end is slightly more so, and I have to accept the fact that I’m officially a below-average programmer.  But before I get all depressed about it, I must point out that on the Haskell community overview, no project gets anywhere near 35 commits on average.  The busiest is GHC, which touches 15 commits.  The most likely explanation is that the 50:1 ratio is way off the mark, and that the old 10:1 ratio is closer to the mark.

As an aside: Productivity is often measured in source lines of code (SLOC), which is duly criticized for being imprecise.  For instance, some of the most important and beneficial changes remove lines, what’s that, negative productivity?  In contrast, I find that I really like the number of commits approach.  A commit — at least for me — is a small piece of modification, usually one to maybe ten lines of effective changes.  As such, it represents sort of a minimal, atomic modification to the code, and encapsulates the smallest coherent unit of brain sweat.  In the Wikipedia page linked to above, Bill Gates is quoted as saying “Measuring programming progress by lines of code is like measuring aircraft building progress by weight.“  In contrast, measuring programming progress by commits is like measuring aircraft building progress by parts — which I think is much more sensible.  Of course, other people may have different notions on the granularity of patches.  Most of what I’ve seen seems to agree with mine, though.

(Even more) random notes

Here’s a rather entertaining read on productivity, although it uses time and quality as metrics, and the standard error for the listed projects is about 0.5 (i.e. time use is about 20 hours ± 10 hours), with a factor of 10 between the worst and best programmer average.  And, disregarding the span from worst to best, the shape of the distribution has certain ramifications, too.

One of the things I remember from Brooks is that he ascribed one order of magnitude productivity improvement from moving from assembly to a high level language.  Further, he thought that no such improvement was possible again from moving to yet higher level languages.  I tend to agree with Yanniss’ law that this is too pessimistic, although I also think a doubling every six years is too optimistic.

Hackage submissions updated

There seems to have been problems with some of the bioinformatics applications on Hackage, thanks to Don S. for pointing it out. That should be fixed now by new uploads, but I’m still waiting for the automatic builds to register results. An, since you ask, it was all my fault for being sloppy with version dependencies. It’d all work with a recent biolib. Speaking of which,

A home page for the bioinformatics library

I’ve finally updated the static home page for the library, it can be admired (especially if you remember what Oscar Wilde had to say about that) here.

I’ve discovered HPC

No, not high-performance computing (which sucks anyway), but GHC’s new ability to do coverage profiling.  Adding it to the default testing procedure was just a couple of extra options, the results can be admired by browsing the HTML files in the darcs repository.  This is very cool, almost too easy, and looks pretty good. The only downside is that it exposes my sloppy attitude to testing.

The last few years, I’ve been working in bioinformatics, the science of taking the results of hard working computer scientists and applying it to the results of equally hard working biologists. You probably think this looks like an easy route to fame and fortune, and you’d be right. In fact, most bioinformaticists just string together some crusty old command line tools with Perl scripts, or even a bit of Java code, and call it a day.

To make things a bit more interesting – interesting to me, that is, the biologists won’t know the difference, and most comuter scientist are too busy teaching Java to web programmers to notice – I’m using Haskell instead. I don’t know why you are using Haskell, but I‘m using it because it put the fun back in programming for me. (I used to program C++, so it’s not like I need to do this for a living.)

Anyway – the intent is that I will document code here – especially when it is particularly neat or cool in some way. I’ll also discuss experiences with the competition, when I say crusty command line tools, I do have particular tools and particular crustiness in mind.