The original version

I’ve taken the liberty of cleaning things up a bit, and removing some context that I sure hope isn’t necessery.  Bascially, the task is to take a list of ByteString input lines consisting of whitespace separated decimal integers, and build a ByteString consisting of single byte quality values corresponding to those integers.

Below is how the naive quality parsing function might look, unpacking each word to String in order to use read.  Note that B is lazy ByteString.Char8, BB is lazy ByteString, that is, the Word8 version.

BB.pack $ map (read . B.unpack) $ B.words $ B.unlines ls

We don’t expect this to do terribly well, and I guess it’s no surprise when this parses my test file of about 10MB in 24 seconds.

Improved versions

The key to improved performance is first and foremost to avoid the unpacking and parsing of strings.  Thankfully, the ByteString library provides a readInt function.

BB.pack [lookup x | x <- concatMap B.words ls]
    where
    lookup x = case B.readInt x of Just (v,_) -> fromIntegral v
                                   Nothing -> error "Unparsable qual value"

This isn’t a lot more complicated than our initial attempt, but the speed increas is considerable: slightly less than 2 seconds for the test file, more than a tenfold improvement.  The ByteString implementation will share the storage of the separate words with the original strings, but since readInt gives us back the rest of the string in addition to the parsed integer, we might as well make use of it:

BB.pack $ readInts $ B.unlines ls
    where readInts xs = case B.readInt xs of 

                          Just (i,rest) -> fromIntegral i : readInts (B.dropWhile isSpace rest)
                          Nothing -> []

This turns out to be a bit faster, time is now 1.6 seconds.  Another 20% shaved off.

Final version

We’re really only interested in Word8 values, since quality values always are small, and since that’s what gets encoded in the result anyway.  The previous versions takes a detour by reading Ints and using fromIntegral to convert them to the desired size.  It bears noting that there is no error checking involved, fromIntegral will happily and silently truncate any number beyond its target range.  So lets do things explicitly, using Word8s throughout the computation:

BB.pack $ go 0 ls
    where
    isDigit x = x <= 58 && x >= 48
    go i (s:ss) = case BB.uncons s of 

                    Just (c,rs) -> if isDigit c then go (c - 48 + 10*i) (rs:ss)
                                   else let rs' = BB.dropWhile (not . isDigit) rs
                                        in if BB.null rs' then i : go 0 ss else i : go 0 (rs':ss)
                    Nothing -> i : go 0 ss
    go _ [] = []

This is the fastest one so far, clocking in at 0.94 seconds, over 40% faster than the best readInt version, and about 25 times faster than the naive version.  Still, 10MB/s is well below the average hard disk.

So is there more room for improvement?  The most obvious wart to me is the rather artificial splitting into lines.  This is mostly an artifact of some early design desicions, and it should be possible to eliminate the splitting earlier on and saving even more time by simplify this function quite a bit.

If you spot anything else, or have suggestions, I (and my darcs repo) am all ears.

Edit: Since some people have asked, I’ve wrapped up a simple test program along with some test files at http://malde.org/~ketil/biohaskell/qualparsetest.  This is a simplified version, if you want to be really helpful, you could always look at Bio.Sequence.Fasta in the Bioinformatics library and see if you can speed up e.g. dephd -i input.fasta input.qual -F /dev/null.

The SFF file format is documented in the GS FLX documentation, page 445-448.  There also exist an open C implementation in io_lib from the  Staden package, in addition to the proprietary implementation provided to Roche’s customers bundled with the sequencing machine.

structure and implementation

The SFF format is a relatively straightforward one, consisting of a header (the common header), followed by a number of read blocks corresponding to the individual sequences.  Each read block contains a header block and a data block. There is also an optional index, whose format is not defined as part of the SFF format.

Here’s the direct translation to Haskell:

data SFF = SFF CommonHeader [ReadBlock]

data CommonHeader = CommonHeader {
          magic                                   :: Word32
        , version                                 :: Word32
        , index_offset                            :: Int64
        , index_length, num_reads                 :: Int32
        , cheader_length, key_length, flow_length :: Int16
        , flowgram_fmt                            :: Word8
        , flow, key                               :: ByteString
     }

data ReadHeader = ReadHeader {
      rheader_length, name_length           :: Int16
    , num_bases                             :: Int32
    , clip_qual_left, clip_qual_right
    , clip_adapter_left, clip_adapter_right :: Int16
    , read_name                             :: ByteString
}

data ReadBlock = ReadBlock {
      read_header                :: ReadHeader
    -- The data block
    , flowgram                   :: [Int16]
    , flow_index, bases, quality :: ByteString
    }

We can almost get by with these data structures and the straightforward Binary instances.  One slightly complicating feature is that each block must be aligned to an 8-byte boundary.  Another one is that we will in some places need some information (lengths and counts) from previously read data, as none of the arrays are size-prefixed.

Note that the data structures above represent the data on disk very closely, later on, static items like the magic number and version will be removed and hardwired into the code instead.

Testing and optimization

While I love to do automated unit testing with QuickCheck, it’s not so straightforward to generate random instances for complex structures like flowgrams.  So in order to test the program, I added two functions to the library: ‘test’ and ‘convert’. The former just prints the information from the CommonHeader and the two first ReadBlocks.  I can then use sffinfo or a similar tool to check that the two programs are in agreement.   The latter reads an SFF file building the SFF data structure, and then serializes it back to a file (modulo the index).

I also wrote a small application, ‘flower’, that reads SFF files and provides various outputs.  The first one is just a table with all the flow values, which will be useful for statistical analysis of these data.  This immediately revealed a performance problem.    (The file contains about 120 000 sequences, each with 168 flow values):

./flower ../biolib/DZX0XNV01.sff > /dev/null  265.68s user 1.08s system 99% cpu 4:28.76 total

The good news is that reading the SFF file is quite fast, the bad news is that formatting the information takes a relatively large amount of time.  Here’s the offending code:

showread :: CommonHeader -> ReadBlock -> [String]
showread h rd = let rn = unpack (read_name $ read_header rd)
in map ((p,c,v) -> printf "%st%dt%st%1.2f" rn p [c] (fi v))
$ zip3 [(1::Int)..] (unpack $ flow h) (flowgram rd)

fi :: Int16 -> Double
fi = (/100) . fromIntegral

The main culprit turned out to be ‘printf’, which I suspect needs to interpret the format string every time.  Also, we’ll need a way to format floating point values.

showread :: CommonHeader -> ReadBlock -> [String]
showread h rd = let rn = unpack (read_name $ read_header rd)
in map ((p,c,v) -> concat [rn,"t",show p,"t",[c],"t",fi v]) -- printf "%st%dt%st%1.2f" rn p [c] (fi v))
$ zip3 [(1::Int)..] (unpack $ flow h) (flowgram rd)

fi :: Int16 -> String
fi = (x -> showFFloat (Just 2) x "") . (/100) . fromIntegral

Running this gives us:

./flower ../biolib/DZX0XNV01.sff > /dev/null  192.65s user 0.69s system 99% cpu 3:15.01 total

Of course,  we’re still building up Strings — that is, linked lists of unicode characters, which is very often a performance problem.  Let’s try the universal string fastifier: Data.Bytestring.  And while we could add a ‘pack’ to the floating point formatting, we’ll try to build it directly:

showread :: CommonHeader -> ReadBlock -> [ByteString]
showread h rd = let rn = read_name $ read_header rd
in map ((p,c,v) -> B.concat [rn,t,B.pack (show p),t,B.pack [c],t,fi v])
$ zip3 [(1::Int)..] (unpack $ flow h) (flowgram rd)

fi :: Int16 -> ByteString
fi i = let (a,x) = i `divMod` 1000
(b,y) = x `divMod` 100
(c,d) = y `divMod` 10
mkdigit = chr . (+48) . fromIntegral
in B.pack [mkdigit a,mkdigit b,'.',mkdigit c,mkdigit d]

The variable t is just a bytestring tab.  Let’s benchmark it again:

./flower ../biolib/DZX0XNV01.sff > /dev/null  55.70s user 0.35s system 99% cpu 56.217 total

It turns out that floating point formatting is still a problem.   Let’s replace it with a table lookup:

fi = (!) farray

farray :: Array Int16 ByteString
farray = listArray (0,10000) [B.pack (showFFloat (Just 2) i "") | i <- [0,0.01..99.99::Double]]

This is about as fast as it gets:

./flower -f ../biolib/DZX0XNV01.sff > /dev/null  39.08s user 0.26s system 99% cpu 39.573 total

Note that it generates about half a gigabyte of output, or a rate of more than 10MB/s — faster than wc can count it.  Flower can also extract the reads (Fasta format, takes 0.7 seconds, approximately 200 000 seqeunces per second), reads with quality (Fastq format, about same speed), or a histogram of flow value frequencies (somewhat slower at 2.2 seconds).

Availability

The bioinformatics library and the flower source code are available here , and there’s also a linux executable for flower, which I may be updating as development progresses.

swords s = take (fromIntegral (seqlength s)+1-k) . map (B.take k) . B.tails . B.map toUpper $ seqdata s

Here, seqdata returns a lazy bytestring, which is also what’s hiding behind the B. qualifier.  Basically, this just builds the list of all lenght-k substrings.  This should, in my opinion, stream nicely, and run in constant space and linear time.  What on earth makes this quadratic?  On only some systems?   Time to dissect the systems in question:

Comparing our environments, one obvious difference was that the fast system was built using GHC 6.8.1, while the slow ones were 6.8.2.  So we checked with a 6.8.1 snapshot – no difference, still slow.  We checked the various source repositories involved for some stray patch that somehow had avoided distribution – nothing turned up. Comparing Cabal setup files didn’t reveal anything obvious, except that the newer Cabal emitted a lot more information.

Somewhere, something had to be different. Could it be bloomfilter not being a good consumer for the generated words?  The bio library messing up FASTA parsing?  Something else entirely?

Looking at the swords function, it looks like a good candidate for deforestation and/or fusion.  Could there be some optimization not being performed in some cases, for some strange reasons?  Since I remember fusion being mentioned occasionally in the context of bytestrings, I checked the libraries again, and found something I’d missed previously:  Bytestring 0.9.1 on the fast system, 0.9.0.1 on the slow ones.  While the announcement didn’t promise more than a few percent better performance, no stone could be left unturned.  And this proved to be it: after upgrading bytestring, and recompiling binary, biolib, and bloomfilter to use it, my laptop was as fast as the server.

I’m not sure exactly what caused this, but apparently there were some issues with bytestring fusion, and fusion is disabled in current bytestring versions. At any rate, it seems the newer version is safer, this is now the minimum requirement in bio.cabal.

Comparing sequences to find similarity is a common occurrence in bioinformatics.  For instance, one might want to know where a certain gene is located in the chromosome, or which sequence fragments are similar enough to originate from the same gene. To speed up searches, it is common to index sequences in questions as overlapping, substrings (k-tuples, q-grams).  This index seems like an obvious target for Bloom filters — large data, time critical, some false positives anyway — but for some reason, there is almost no such applications that use them. Until now.

Sequence clustering

Sequence clustering is a commonly used technique which is usually based on sequence similarity.  I’ve written one sequence clusterer, xsact, which is based on blocks of exact matches. There are many others, and another example is d2cluster, which is based on occurence of fixed length words — which is right up Bloom alley, right?

So a straightforward way to build a Bloom filter based sequence clusterer is to represent each cluster as a set of words — stored as a Bloom filter.  Now, adding a new sequence to the clusters is a simple matter of extracting the words from the sequence, identifying the cluster(s) containing a sufficient number of these words, and adding the remaining words to that cluster (or the union of the clusters, in the case multiple clusters match).

The interesting thing about this approach is that the whole thing becomes O(kn), for k clusters and data size n.  I think all other clustering algorithms are based on sequence pairs, which makes them O(n²) — in the worst case, you need to check all pairs. (However, a straightforward similarity-based clustering will have worst-case behavior when no sequence math each other, while suffix-based methods like xsact will have worst case when all sequences match — so perhaps there is room for a better middle ground?)

Anyway — while the above looks promising, there is one snag: EST sequences can occur as a copy of the gene, or due various properties of the manufacturing process, the gene’s reverse complement.  Thus, we need to be able to compare sequences in both directions simultaneously. This could be achieved with a slightly creative hashing function, but to keep things simple, we’ll stick with the ossified mental sweat in the provided implementation.

Index and search

A somewhat related area is indexing and search.  Let’s say you have a bunch of DNA sequences (of for each chromosome, perhaps), and a set of ESTs, which you’ll remember are gene fragments in an unpredictable mix of forward and reverse-complement directions.  Here’s the plan:

1. index by building a Bloom filter containing the q-tuples for each chromosome
2. for each EST, look up each q-tuple (first forward, then rev.comp.) against the filters, and assign it to the chromosome containing the most q-tuples
3. align each EST against the designated chromosome using traditional methods

As far as I can tell at this point, for word size q, c chromosomes of length m and e ESTs of lenght n, this should run in something like  O(qn) + O(qenc) + O(emn), compared to just aligning directly at O(ecmn). Note that mn is the big factor here, so on a large scale, we’re reducing the total work by a factor of c.  (Of course, in real life, nm would be too large, and you’d use a heuristic, subquadratic alignment step, but this is for illustration purposes.  Try to be a bit generous, will you?)

Further plans

That concludes the current plan, but there are certainly improvements that can be made, two obvious ones are

• hash function that hashes a word and its reverse complement to the same value
• break long sequences into partially (1/3) overlapping regions to speed things up even more, as well as give more accurate placements

Another thing that struck me is that in my annotation tool, I could probably use this as a faster way to store the set of matching proteins before extracting GO terms.  Currently, the performance is limited by XML parsing, so it’s probably not worth the bother at the moment.

More is likely to come up, but now it’s time to implement something!

Profiling with Haskell

First we need to identify the actual bottlenecks, using GHC’s profiling options. This means compiling the program with -prof -auto-all, and the Makefile shipped with xml2x already has a target for this. (I could probably do it with Cabal as well, but I never bothered to find out. Lazy me.) A “feature” of the bioinformatics library is that its Cabal file contains this line:

ghc-options: -Wall -O2 -fexcess-precision -funbox-strict-fields -auto-all


The final “auto-all” ensures that when the library gets compiled with profiling support, cost centres will be assigned to its internal functions too, thus the profiling run will reveal where time is being spent in more detail. This practice is probably frowned upon by people who are into that kind of thing, but here it is crucial, as the profile reveals this:

Fri May 16 18:28 2008 Time and Allocation Profiling Report  (Final)

xml2x.p +RTS -p -sxml2x.p2.stats -RTS -v -C --reg --go-def=GO.terms_and_ids
  --anno=gene_association.goa_uniprot small.xml -o small3.csv

total time  =     4073.12 secs   (203656 ticks @ 20 ms)
total alloc = 1082,832,292,360 bytes  (excludes profiling overheads)

COST CENTRE                    MODULE               %time %alloc

mkAnn                          Bio.Sequence.GOA        60.8   56.2
readXML                        Bio.Alignment.BlastXML  27.4   38.4
hsp2match                      Bio.Alignment.BlastXML   3.1    1.7
readGOA                        Bio.Sequence.GOA         1.9    0.4
hit2hit                        Bio.Alignment.BlastXML   1.6    1.8
protTerms                      Main                     1.6    0.0
sequence'                      Bio.Util                 1.1    0.2

In short, the function named mkAnn consumes almost two thirds of the time here, and is an obvious target for closer inspection. The function looks like this:

data Annotation = Ann !UniProtAcc !GoTerm !EvidenceCode deriving (Show)
newtype GoTerm = GO Int deriving (Eq,Ord)
type UniProtAcc = ByteString

mkAnn :: ByteString -> Annotation
mkAnn = pick . B.words
    where pick (_db:up:rest) = pick' up $ findGo rest
          pick' up' (go:_:ev:_) = Ann (copy up') (read $ unpack go) (read $ unpack ev)
          findGo = dropWhile (not . B.isPrefixOf (pack "GO:"))

This is a slightly clunky way of extracting a bit of information from a line of text; the text in question is GOA, a mapping between proteins and standard Gene Onthology terms. It’s a text file with about 20 million lines, so we use lazy bytestrings to deal with it. Each line is broken into words, the second word (“up”) is the protein name. Then a variable number of words later, the GO term itself appears, and the final item we want is the third word after the GO term. Note the copy, ensuring that we don’t keep around a large part of the input file.

So what’s wrong here? Asking in the café, Don suggested that unpacking bytestrings is a less than recommendable practice. Still, I find it is a useful idiom, and the read $ unpack s should garbage-collect the characters as soon as they are consumed. Ideally, the string should be deforested or fused away. I happily agree that this is a lazy programmer’s approach and not optimal – but on the other hand, it shouldn’t be that costly?

GHC core to the rescue – not.

This is the stage where I looked at GHC core. I guess it’s one of those things that looks easy when professionals do it, but when you get home and try it on your own, you just have no idea. The three-line function above resulted in four pages of cryptic core output. Call me lazy and ignorant, but I decided to take just one more little peek at the source first.

Let’s check out those read-unpack constructs. The read instance for GoTerm checks that the string starts with “GO:”, and then reads an Int. The second reads an EvidenceCode, and since these are defined as short abbreviations in all upper case, I just wrote a data type with corresponding nullary constructors, and derived Read for it.

data EvidenceCode = IC  -- Inferred by Curator
	          | IDA -- Inferred from Direct Assay
	          | IEA -- Inferred from Electronic Annotation
	          | IEP -- Inferred from Expression Pattern
	          | IGC -- Inferred from Genomic Context
	          | IGI -- Inferred from Genetic Interaction
	          | IMP -- Inferred from Mutant Phenotype
	          | IPI -- Inferred from Physical Interaction
	          | ISS -- Inferred from Sequence or Structural Similarity
	          | NAS -- Non-traceable Author Statement
	          | ND  -- No biological Data available
	          | RCA -- inferred from Reviewed Computational Analysis
	          | TAS -- Traceable Author Statement
	          | NR  -- Not Recorded
     deriving (Read,Show,Eq)

Maybe this wasn’t such a good idea after all? Let’s try with custom, bytestring-based parsers for EvidenceCode and GoTerm:

getEC :: ByteString -> EvidenceCode
getEC s = case B.uncons s of
            Just ('I',s') -> case B.uncons s' of
                               Just ('C',_) -> IC
                               Just ('D',_) -> IDA
                               Just ('E',s'') -> case B.head s'' of 'A' -> IEA
                                                                    'P' -> IEP
                                                                    _ -> e 1
                               Just ('G',s'') -> case B.head s'' of 'C' -> IGC
                                                                    'I' -> IGI
                                                                    _ -> e 2
                               Just ('M',_) -> IMP
                               Just ('P',_) -> IPI
                               Just ('S',_) -> ISS
                               _ -> e 3
            Just ('N',s') -> case B.head s' of 'A' -> NAS
                                               'D' -> ND
                                               'R' -> NR
                                               _ -> e 4
            Just ('R',_) -> RCA
            Just ('T',_) -> TAS
            _ -> e 5
    where e :: Int -> a
          e n = error ("Illegal GO evidence code ("++show n++"): "++unpack s)

getGo :: ByteString -> GoTerm
getGo bs = GO $ fst $ maybe e id (B.readInt $ B.drop 3 bs)
    where e = error ("Unable to parse GO term"++unpack bs)

Time to re-run with profiling:

Sat May 17 19:25 2008 Time and Allocation Profiling Report  (Final)

xml2x.p +RTS -sxml2x.2pp.stats -p -RTS -v -C --reg --go-def=GO.terms_and_ids
    --anno=gene_association.goa_uniprot small.xml -o small5p.csv

        total time  =     2022.36 secs   (101118 ticks @ 20 ms)
        total alloc = 622,807,651,828 bytes  (excludes profiling overheads)

COST CENTRE                    MODULE               %time %alloc

readXML                        Bio.Alignment.BlastXML  48.6   66.7
mkAnn                          Bio.Sequence.GOA        29.8   23.6
hsp2match                      Bio.Alignment.BlastXML   5.2    3.0
readGOA                        Bio.Sequence.GOA         3.6    0.7
hit2hit                        Bio.Alignment.BlastXML   3.2    3.0
protTerms                      Main                     3.1    0.0
sequence'                      Bio.Util                 2.1    0.4
countIO                        Bio.Util                 1.4    0.6
getFrom                        Bio.Alignment.BlastXML   1.1    0.7

We see that mkAnn has fallen way behind the XML parsing in time consumption, and until Neil gets around to do a bytestring version of his otherwise excellent tagsoup library, there isn’t all that much left to do. Total time has been cut in half, from over an hour to 35 minutes or so. There’s still 30% of the run time to be shaved even more closely, perhaps you can suggest the remaining culprit? My guess would be the findGo function. How should it be rewritten?

Lessons learned

Profiling is nice to pinpoint the hotspots. Bytestrings is the way to go for fast code. GHC generates crappy derived parsers for data types, perhaps especially so for data types with many constructors. Writing manual parsers is tedious, but a bit of tedium can go a long way.

Unfortunatly, I am still not entirely happy with xml2x. The profiling numbers lie, as they don’t — I believe — include GC time. This particular program uses an inordinate amount of time collecting garbage. I suspect it is because I store GO terms as 16 bit integers in unboxed arrays in an insufficiently strict Data.Map, but I will have to investigate this closer.

But not today.