flowers

Here’s a better method that identifies poly-A tails by finding an optimal, quality adjusted alignment in linear time.

A quick introduction

Although the definitions of what constitutes a gene vary considerably, we’ll use the term to refer to a region of DNA that get transcribed, that is, copied from DNA into an mRNA molecule, which in turn will be used as a blueprint for assembling a protein. After transcription, the mRNA molecule then undergoes polyadenylation, a process where a string of adenines (the ‘A’ of the nucleotide alphabet) gets appended to an mRNA molecule before it is exported from the nucleus.

Identifying poly-A tails are important for several reasons.

1. It positively identifies the end of the transcript. If you don’t have a poly-A in your sequence, you have no way to know how far the molecule extends beyond the end of the sequence. You can also find alternatively terminated transcripts this way.
2. It positively identifies the end of the transcript. Anything after the poly-A tail is linker or vector sequence, and can safely be trimmed off. Even if, as is often the case, it is too low quality to be recognized by your average vector masking software.
3. It provides useful information about the transcript, as the poly-A tail is important for things like protecting the mRNA from degradation.

Unfortunately utilities often trim poly-A tails by default (e.g. SeqClean), or just ignore it (e.g. BLAST’s low-complexity
filter).

Quality based alignment

When a molecule is sequenced, the analog output from the sequencing machine is stored as a chromatogram. In order to be useful, the sequence is called, that is, translated to a string of letters from the familiar {A,C,G,T} nucleotides alphabet. In addition, the base caller will associate each letter with a quality value. This is derived from an estimate of the probability of the call being incorrect, and for quality value Q, the error probability estimate is

ϵ = 10-Q/10

Traditionally sequence alignment simply aligns the string of characters using a fixed positive score (reward) for aligning similar characters, and fixed negative scores (penalties) for either substituting a different character, opening a gap, or extending a previous gap.

However, taking into account the quality value, we can do better, and instead of fixed scores, we can adjust the scores dynamically according to quality.

Using this method, the penalty for e.g. aligning two different characters will depend on the quality of the characters: high quality means a high penalty, low quality — lower penalty (since there’s a greater chance one of them was incorrectly called).

Scoring of alignments

When calculating the score of an alignment, we really want to answer the question: how likely is this sequence to be a real poly-A sequence, as opposed to just a random string? In other words, we are comparing our sequence against two models: the poly-A model, and the background model. Our score will use the ratio of probabilities of the string being produced by the two models.

For the poly-A model, only As are allowed, so the probability of a character occurring is 1 for As and 0 for the others. For the background model, we’ll just take a uniform distribution of nucleotides, each getting a probability of 0.25.

Using this scheme, the score for a string s is simply 1/0.25 = 4 for each A, and 0/0.25 = 0 for all others. We usually work with the logarithm of these numbers to make them more manageable.

The optimal alignment is then simply the longest run of As, since as soon as you multiply with a zero (or add -infinity, if you use log-scores), you lose the whole score.

Adding quality to the mix

Of course, the actual sequence isn’t perfect, and even the poly-A tail is likely to contain the odd G, C, or T. To determine exactly how likely is where the quality value enters the picture. Using the formula above, we can calculate the error estimate and decide what the penalty for a mismatch and reward for a match should be.

For the poly-A model, the probability for a match (that is, an actual ‘A’ in the sequence) is 1-ϵ, the probability of a mismatch (a non-A) is ϵ/3 (since only one of the three possible substititutions is an A, and for simplicity, we give them equal probability). Using the formula for  ϵ as a function of Q (and hopefully not introducing any errors), I get the scores to be:

match q = log (4*(1-1/10**(q/10)))
mismatch q = log 4 - log 3 - q/10*log 10

Now, we can use this to do a standard Smith-Waterman alignment, calculating a dynamic programming matrix, and searching for an optimal local alignment.

However, since we’re aligning against a repeated nucleotide, there’s no real need for a second dimension, and we can use the following recurrence to calculate the “polyA-score” M for each position i:

M_i = max (0, S_i + M_i-1)

To implement this, we first define the list of scores by applying match and mismatch to the list of (nucleotide,quality) pairs. We also define a scanl-based function to calculate a list of cumulative scores:

scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
cumulative = scanl (\a b -> let r = a + b in max 0 r)

The only remaining thing is to identify the maximal value which marks the end of the poly-A tail, and the corresponding 0 value that indicates the start. I wrote a recursive function called ”findmax” for this, but a better programmer will probably be able to do this with a fold.

Including the parts discussed briefly above, the whole thing looks like this:

findPolyA :: Sequence Nuc -> Maybe (Int,Int)
findPolyA (Seq _ d mq) =
let qd = zip (B.unpack d) (maybe (repeat 15) BB.unpack mq)
scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
match x' = let x = fromIntegral x' in log (4*(1-1/10**(x/10)))
mismatch x' = let x = fromIntegral x' in log 4 - log 3 - x/10*log 10
cumulative = scanl (\a b -> let r = a + b in max 0 r) 0
(zi,mi,maxscore) = findmax $ cumulative scores
in if maxscore > 12 then Just (zi+1,mi) else Nothing  -- arbitrary constant alert!

findmax :: [Double] -> (Int,Int,Double)
findmax = go 0 (0,0,0) . zip [0..]
where go _ cm [] = cm
go _ cm ((i,0):rest) = go i cm rest
go last_z (cmz,cmi,cmx) ((i,x):rest) = if x > cmx then go last_z (last_z,i,x) rest
else go last_z
(cmz,cmi,cmx) rest

Availability

This method is implemented in a simple tool called “trimpolya” (darcs repo), and also in the more general “dephd” (darcs, hackage) sequence analysis package.

Installing biolib

First I need to install the bioinformatics library.  I’m about to release 0.4.1, but this is also a good opportunity to check that everything works with 0.4 (which is what you’ll find on Hackage), so let’s do that first.  Using darcs, I pull the repo up to the 0.4 tag (but you can of course get the tarball from Hackage):

% ./Setup.hs configure
Configuring bio-0.4...
Setup.hs: At least the following dependencies are missing:
QuickCheck <2, binary -any


(Side note: you may notice that I run Setup.hs directly, as opposed to using runhaskell. I prefer it this way, but you may have to do a chmod +x Setup.hs if you downloaded this from the darcs repository or similar.)

Since we want to use the system libraries as far as possible, these libraries are just an apt-get away:

sudo apt-get install libghc6-quickcheck1\*
sudo apt-get install libghc6-binary-\*


Now, let’s try again:

% ./Setup.hs configure
Configuring bio-0.4...
% ./Setup.hs build
Preprocessing library bio-0.4...
Building bio-0.4...
Binary: Int64 truncated to fit in 32 bit Int
ghc: panic! (the 'impossible' happened)
(GHC version 6.10.4 for i386-unknown-linux):
Prelude.chr: bad argument

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Okay, this was not what was supposed to happen.  As always, dropping to #haskell on IRC is the first thing to do, and sure enough:

<sereven> ketil: that's also shown up for xmonad users when .hi and .o files weren't cleaned
between rebuilds mixing different versions. usually between ghc updates IIRC.


Let’s try to get rid of old cruft lying about, polluting directories:

./Setup.hs clean && ./Setup.hs configure && ./Setup.hs build

Sure enough, this time it worked. For good measure, we’ll run the unit tests:

make test

After a zillion tests, we notice that everything is go, great!

Applications

Next, it is time to go through the list of bioinformatics applications.  Since my working directory is a mess of branches and versions, we’ll just go over the published applications and versions on Hackage.

xsact is an application to do sequence (in particular EST sequence) clustering.  It predates and thues doesn’t actually use the bioinformatics library, but we’ll check it anway.  So we try the familiar command line:

./Setup.hs clean && ./Setup.hs configure && ./Setup.hs build

And things compile.  However, the version on Hackage is outdated, so we’ll upload a new version, 1.7.1.  One test case still fails, but I can’t imagine anybody is using it to generate Newick-formatted trees — I am certainly not — and since there are many equally correct outputs (including tree rotations and rounding modes), output is likely correct anyway.  Holler if you need it!

rbr is an application to mask repeats in sequence data.  Normally, this is done using a library of known repeats, but this application tries to do it using statistics, making the — I think justifiable — assumption that repeats are going to be more common than non-repeats. The version on Hackage is old, and only works with the library prior to 0.4, so again this is a good time to push the latest changes out in the limelight.  Compiling this works great, by the way.

cluster_tools is a package that contains a bunch of binaries, useful for working with the results of sequence clustering, including extracting various information from ACE files.  This uses another library, called simpleargs, that simplifies command line argument parsing for simple cases.  Again, the Hackage version is for bio<0.4, so a new version will be pushed.  At the same time, we make a mental note to push version 0.2 of simpleargs to Hackage as well, instead of keeping age-old modifications buried forever.

dephd is my Swiss-army-knife of sequence analysis, and lets you do various things like converting between formats, plotting and trimming by quality.  This is a more live project than most of the others (I’m currently working on improved quality trimming and automatic generation of files for submission to GenBank), but the currently available version also compiles without incident.

estreps is a couple of programs I needed for repeat analysis, perhaps not tremendously interesting, but at least rselect, which lets you select randomized subsets from Fasta files, might be of interest to some? We try the usual invocation to compile, and get:

src/Unigene.hs:24:23:
Couldn't match expected type `a' against inferred type `Unknown'
`a' is a rigid type variable bound by
the type signature for `clusters' at src/Unigene.hs:22:41


This error arises due to the introduction of phantom types for identifying sequences introduced in bio 0.4. Unfortunately, this version of estreps contains some adaption to this model, so it won’t compile against older versions either. So it looks like yet another sdist for Hackage. Look for version 0.3.1.

flower is a utility for extracting information from SFF files (containing sequences from Roche’s 454 machines).  Although a new version is around the corner, the old 0.2 just works.

xml2x is a utility for converting BLAST results in XML format into CSVs, that somehow is more compatible with biologists.  Trying to compile it fails, with the following error:

src/Xml2X.hs:152:49:
Couldn't match expected type `[b]'
against inferred type `Maybe Bio.Sequence.KEGG.KO'
In the first argument of `concatMap', namely `(flip M.lookup ks)'
In the first argument of `($)', namely
`concatMap (flip M.lookup ks)'
In the second argument of `($)', namely
`concatMap (flip M.lookup ks) $ map chop $ map subject fs'


It turns out that somewhere along the way, the lookup function from Data.Map was de-generalized from working on arbitrary monads to just returning a Maybe. I was using this to return a list, using the empty list to signal an unsuccessful lookup. This is easily remedied, but that means yet another sdist for Hackage.

korfu is a utility for identifying open reading frames in sequence data.  It hasn’t yet been ported to version 0.4 of the library, but works if you install 0.3.5.  I updated this too, since it didn’t have a category.  Now it too resides in the bioinformatics section.

Summing up

In retrospect, it seems like giving old code a thorough spring cleaning once in a while. Although nothing really critical or difficult happened, a good number of small annoyances were discovered, and a bunch of new sdists are now ready to be uploaded to Hackage. Next will be converting all this into debian packages.

The important question is of course, how do we avoid this in the future? During development, it is important to be able to modify libraries and appliations, but installing a new version of the biolib, say, overwrites the old one, and suddenly I’m compiling and testing everything against a different library than Joe Random Hackage User is going to find. I have some thoughts on how to avoid this, but if you have a method that works nicely, I’m all ears.

•  My biohaskell slides
• slides’ LaTeX source, using beamer.cls
• Lazy PSSM JFP-paper

The original version

I’ve taken the liberty of cleaning things up a bit, and removing some context that I sure hope isn’t necessery.  Bascially, the task is to take a list of ByteString input lines consisting of whitespace separated decimal integers, and build a ByteString consisting of single byte quality values corresponding to those integers.

Below is how the naive quality parsing function might look, unpacking each word to String in order to use read.  Note that B is lazy ByteString.Char8, BB is lazy ByteString, that is, the Word8 version.

BB.pack $ map (read . B.unpack) $ B.words $ B.unlines ls

We don’t expect this to do terribly well, and I guess it’s no surprise when this parses my test file of about 10MB in 24 seconds.

Improved versions

The key to improved performance is first and foremost to avoid the unpacking and parsing of strings.  Thankfully, the ByteString library provides a readInt function.

BB.pack [lookup x | x <- concatMap B.words ls]
    where
    lookup x = case B.readInt x of Just (v,_) -> fromIntegral v
                                   Nothing -> error "Unparsable qual value"

This isn’t a lot more complicated than our initial attempt, but the speed increas is considerable: slightly less than 2 seconds for the test file, more than a tenfold improvement.  The ByteString implementation will share the storage of the separate words with the original strings, but since readInt gives us back the rest of the string in addition to the parsed integer, we might as well make use of it:

BB.pack $ readInts $ B.unlines ls
    where readInts xs = case B.readInt xs of 

                          Just (i,rest) -> fromIntegral i : readInts (B.dropWhile isSpace rest)
                          Nothing -> []

This turns out to be a bit faster, time is now 1.6 seconds.  Another 20% shaved off.

Final version

We’re really only interested in Word8 values, since quality values always are small, and since that’s what gets encoded in the result anyway.  The previous versions takes a detour by reading Ints and using fromIntegral to convert them to the desired size.  It bears noting that there is no error checking involved, fromIntegral will happily and silently truncate any number beyond its target range.  So lets do things explicitly, using Word8s throughout the computation:

BB.pack $ go 0 ls
    where
    isDigit x = x <= 58 && x >= 48
    go i (s:ss) = case BB.uncons s of 

                    Just (c,rs) -> if isDigit c then go (c - 48 + 10*i) (rs:ss)
                                   else let rs' = BB.dropWhile (not . isDigit) rs
                                        in if BB.null rs' then i : go 0 ss else i : go 0 (rs':ss)
                    Nothing -> i : go 0 ss
    go _ [] = []

This is the fastest one so far, clocking in at 0.94 seconds, over 40% faster than the best readInt version, and about 25 times faster than the naive version.  Still, 10MB/s is well below the average hard disk.

So is there more room for improvement?  The most obvious wart to me is the rather artificial splitting into lines.  This is mostly an artifact of some early design desicions, and it should be possible to eliminate the splitting earlier on and saving even more time by simplify this function quite a bit.

If you spot anything else, or have suggestions, I (and my darcs repo) am all ears.

Edit: Since some people have asked, I’ve wrapped up a simple test program along with some test files at http://malde.org/~ketil/biohaskell/qualparsetest.  This is a simplified version, if you want to be really helpful, you could always look at Bio.Sequence.Fasta in the Bioinformatics library and see if you can speed up e.g. dephd -i input.fasta input.qual -F /dev/null.

The SFF file format is documented in the GS FLX documentation, page 445-448.  There also exist an open C implementation in io_lib from the  Staden package, in addition to the proprietary implementation provided to Roche’s customers bundled with the sequencing machine.

structure and implementation

The SFF format is a relatively straightforward one, consisting of a header (the common header), followed by a number of read blocks corresponding to the individual sequences.  Each read block contains a header block and a data block. There is also an optional index, whose format is not defined as part of the SFF format.

Here’s the direct translation to Haskell:

data SFF = SFF CommonHeader [ReadBlock]

data CommonHeader = CommonHeader {
          magic                                   :: Word32
        , version                                 :: Word32
        , index_offset                            :: Int64
        , index_length, num_reads                 :: Int32
        , cheader_length, key_length, flow_length :: Int16
        , flowgram_fmt                            :: Word8
        , flow, key                               :: ByteString
     }

data ReadHeader = ReadHeader {
      rheader_length, name_length           :: Int16
    , num_bases                             :: Int32
    , clip_qual_left, clip_qual_right
    , clip_adapter_left, clip_adapter_right :: Int16
    , read_name                             :: ByteString
}

data ReadBlock = ReadBlock {
      read_header                :: ReadHeader
    -- The data block
    , flowgram                   :: [Int16]
    , flow_index, bases, quality :: ByteString
    }

We can almost get by with these data structures and the straightforward Binary instances.  One slightly complicating feature is that each block must be aligned to an 8-byte boundary.  Another one is that we will in some places need some information (lengths and counts) from previously read data, as none of the arrays are size-prefixed.

Note that the data structures above represent the data on disk very closely, later on, static items like the magic number and version will be removed and hardwired into the code instead.

Testing and optimization

While I love to do automated unit testing with QuickCheck, it’s not so straightforward to generate random instances for complex structures like flowgrams.  So in order to test the program, I added two functions to the library: ‘test’ and ‘convert’. The former just prints the information from the CommonHeader and the two first ReadBlocks.  I can then use sffinfo or a similar tool to check that the two programs are in agreement.   The latter reads an SFF file building the SFF data structure, and then serializes it back to a file (modulo the index).

I also wrote a small application, ‘flower’, that reads SFF files and provides various outputs.  The first one is just a table with all the flow values, which will be useful for statistical analysis of these data.  This immediately revealed a performance problem.    (The file contains about 120 000 sequences, each with 168 flow values):

./flower ../biolib/DZX0XNV01.sff > /dev/null  265.68s user 1.08s system 99% cpu 4:28.76 total

The good news is that reading the SFF file is quite fast, the bad news is that formatting the information takes a relatively large amount of time.  Here’s the offending code:

showread :: CommonHeader -> ReadBlock -> [String]
showread h rd = let rn = unpack (read_name $ read_header rd)
in map ((p,c,v) -> printf "%st%dt%st%1.2f" rn p [c] (fi v))
$ zip3 [(1::Int)..] (unpack $ flow h) (flowgram rd)

fi :: Int16 -> Double
fi = (/100) . fromIntegral

The main culprit turned out to be ‘printf’, which I suspect needs to interpret the format string every time.  Also, we’ll need a way to format floating point values.

showread :: CommonHeader -> ReadBlock -> [String]
showread h rd = let rn = unpack (read_name $ read_header rd)
in map ((p,c,v) -> concat [rn,"t",show p,"t",[c],"t",fi v]) -- printf "%st%dt%st%1.2f" rn p [c] (fi v))
$ zip3 [(1::Int)..] (unpack $ flow h) (flowgram rd)

fi :: Int16 -> String
fi = (x -> showFFloat (Just 2) x "") . (/100) . fromIntegral

Running this gives us:

./flower ../biolib/DZX0XNV01.sff > /dev/null  192.65s user 0.69s system 99% cpu 3:15.01 total

Of course,  we’re still building up Strings — that is, linked lists of unicode characters, which is very often a performance problem.  Let’s try the universal string fastifier: Data.Bytestring.  And while we could add a ‘pack’ to the floating point formatting, we’ll try to build it directly:

showread :: CommonHeader -> ReadBlock -> [ByteString]
showread h rd = let rn = read_name $ read_header rd
in map ((p,c,v) -> B.concat [rn,t,B.pack (show p),t,B.pack [c],t,fi v])
$ zip3 [(1::Int)..] (unpack $ flow h) (flowgram rd)

fi :: Int16 -> ByteString
fi i = let (a,x) = i `divMod` 1000
(b,y) = x `divMod` 100
(c,d) = y `divMod` 10
mkdigit = chr . (+48) . fromIntegral
in B.pack [mkdigit a,mkdigit b,'.',mkdigit c,mkdigit d]

The variable t is just a bytestring tab.  Let’s benchmark it again:

./flower ../biolib/DZX0XNV01.sff > /dev/null  55.70s user 0.35s system 99% cpu 56.217 total

It turns out that floating point formatting is still a problem.   Let’s replace it with a table lookup:

fi = (!) farray

farray :: Array Int16 ByteString
farray = listArray (0,10000) [B.pack (showFFloat (Just 2) i "") | i <- [0,0.01..99.99::Double]]

This is about as fast as it gets:

./flower -f ../biolib/DZX0XNV01.sff > /dev/null  39.08s user 0.26s system 99% cpu 39.573 total

Note that it generates about half a gigabyte of output, or a rate of more than 10MB/s — faster than wc can count it.  Flower can also extract the reads (Fasta format, takes 0.7 seconds, approximately 200 000 seqeunces per second), reads with quality (Fastq format, about same speed), or a histogram of flow value frequencies (somewhat slower at 2.2 seconds).

Availability

The bioinformatics library and the flower source code are available here , and there’s also a linux executable for flower, which I may be updating as development progresses.

swords s = take (fromIntegral (seqlength s)+1-k) . map (B.take k) . B.tails . B.map toUpper $ seqdata s

Here, seqdata returns a lazy bytestring, which is also what’s hiding behind the B. qualifier.  Basically, this just builds the list of all lenght-k substrings.  This should, in my opinion, stream nicely, and run in constant space and linear time.  What on earth makes this quadratic?  On only some systems?   Time to dissect the systems in question:

Comparing our environments, one obvious difference was that the fast system was built using GHC 6.8.1, while the slow ones were 6.8.2.  So we checked with a 6.8.1 snapshot – no difference, still slow.  We checked the various source repositories involved for some stray patch that somehow had avoided distribution – nothing turned up. Comparing Cabal setup files didn’t reveal anything obvious, except that the newer Cabal emitted a lot more information.

Somewhere, something had to be different. Could it be bloomfilter not being a good consumer for the generated words?  The bio library messing up FASTA parsing?  Something else entirely?

Looking at the swords function, it looks like a good candidate for deforestation and/or fusion.  Could there be some optimization not being performed in some cases, for some strange reasons?  Since I remember fusion being mentioned occasionally in the context of bytestrings, I checked the libraries again, and found something I’d missed previously:  Bytestring 0.9.1 on the fast system, 0.9.0.1 on the slow ones.  While the announcement didn’t promise more than a few percent better performance, no stone could be left unturned.  And this proved to be it: after upgrading bytestring, and recompiling binary, biolib, and bloomfilter to use it, my laptop was as fast as the server.

I’m not sure exactly what caused this, but apparently there were some issues with bytestring fusion, and fusion is disabled in current bytestring versions. At any rate, it seems the newer version is safer, this is now the minimum requirement in bio.cabal.

@{sequence header}
{sequence data}
+{sequence header}
{quality data}

Note that the sequence header is repeated in there, apparently somebody thought that would be a good idea. The {sequence data} part looks like it does in a Fasta file, except that here it has to be on a single line. The {quality data} is ASCII, each letter representing the quality value 33 lower than it’s ASCII value.  This opens up another possibility of getting it wrong, since the line of quality data can (and will!) start out with ‘+’ or ‘@’ occasionally.

Anyway, the implementation seems to be pretty efficient, I wrote a simple program to count the number of sequences in a file:

module Main where

import Bio.Sequence
import System

main = do
  [f] <- getArgs
  print . length =<< readFastQ f

Testing it on this 440MB file ran in (first cold, then warm cache):

./countFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  1.77s user 0.41s system 53% cpu 4.091 total
./countFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  1.26s user 0.19s system 89% cpu 1.631 total

For comparison, I also tried it with ‘grep’:

grep '^@' ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  5.61s user 0.46s system 98% cpu 6.181 total

..and in addition to being 50% slower, it gives the wrong answer, since the ‘@’ delimiter may occur in the quality data. Another nice thing is that the Haskell program is IO bound, so it would be even faster if I had a better disk. (Note to self: talk to boss about getting an SSD for my laptop).

Update: I did a quick test, comparing it to the old Fasta format.  First, to convert, I replaced the last line in the program above with

readFastQ f >>= writeFasta (f++".fasta")

and compiled it as convertFQ.  Running this:

./convertFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  9.51s user 2.76s system 40% cpu 30.037 total

Then, I made a countFa by changing the last line to:

print . length =<< readFasta f

Running this on the Fasta file generated just now (282MB), I get:

./countFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  1.26s user 0.19s system 89% cpu 1.631 total

Here, grep takes 2.25 seconds user time (and gives the correct answer), we’re still faster.

The only cloud on the horizon is that there is some disagreement about the format.  For example, somebody thinks quality always should start with a ‘!’ (viz. zero).  Maq developers think it’s okay to drop the repeated sequence name.  Rockefeller thinks the quality data should be text digits, like the Qual format. And the good people at Solexa had to go and slightly alter…not the format itself, but it’s interpretation, using a different formula to calculate the error probabilities from the quality values. So basically, given a file, there’s no way to know whether it uses Solexa-style quality information, or regular, Phred-style quality.  If you get quality values above 60, then maybe you’re interpreting it wrong.  Then again, maybe not. Sigh.

Comparing sequences to find similarity is a common occurrence in bioinformatics.  For instance, one might want to know where a certain gene is located in the chromosome, or which sequence fragments are similar enough to originate from the same gene. To speed up searches, it is common to index sequences in questions as overlapping, substrings (k-tuples, q-grams).  This index seems like an obvious target for Bloom filters — large data, time critical, some false positives anyway — but for some reason, there is almost no such applications that use them. Until now.

Sequence clustering

Sequence clustering is a commonly used technique which is usually based on sequence similarity.  I’ve written one sequence clusterer, xsact, which is based on blocks of exact matches. There are many others, and another example is d2cluster, which is based on occurence of fixed length words — which is right up Bloom alley, right?

So a straightforward way to build a Bloom filter based sequence clusterer is to represent each cluster as a set of words — stored as a Bloom filter.  Now, adding a new sequence to the clusters is a simple matter of extracting the words from the sequence, identifying the cluster(s) containing a sufficient number of these words, and adding the remaining words to that cluster (or the union of the clusters, in the case multiple clusters match).

The interesting thing about this approach is that the whole thing becomes O(kn), for k clusters and data size n.  I think all other clustering algorithms are based on sequence pairs, which makes them O(n²) — in the worst case, you need to check all pairs. (However, a straightforward similarity-based clustering will have worst-case behavior when no sequence math each other, while suffix-based methods like xsact will have worst case when all sequences match — so perhaps there is room for a better middle ground?)

Anyway — while the above looks promising, there is one snag: EST sequences can occur as a copy of the gene, or due various properties of the manufacturing process, the gene’s reverse complement.  Thus, we need to be able to compare sequences in both directions simultaneously. This could be achieved with a slightly creative hashing function, but to keep things simple, we’ll stick with the ossified mental sweat in the provided implementation.

Index and search

A somewhat related area is indexing and search.  Let’s say you have a bunch of DNA sequences (of for each chromosome, perhaps), and a set of ESTs, which you’ll remember are gene fragments in an unpredictable mix of forward and reverse-complement directions.  Here’s the plan:

1. index by building a Bloom filter containing the q-tuples for each chromosome
2. for each EST, look up each q-tuple (first forward, then rev.comp.) against the filters, and assign it to the chromosome containing the most q-tuples
3. align each EST against the designated chromosome using traditional methods

As far as I can tell at this point, for word size q, c chromosomes of length m and e ESTs of lenght n, this should run in something like  O(qn) + O(qenc) + O(emn), compared to just aligning directly at O(ecmn). Note that mn is the big factor here, so on a large scale, we’re reducing the total work by a factor of c.  (Of course, in real life, nm would be too large, and you’d use a heuristic, subquadratic alignment step, but this is for illustration purposes.  Try to be a bit generous, will you?)

Further plans

That concludes the current plan, but there are certainly improvements that can be made, two obvious ones are

• hash function that hashes a word and its reverse complement to the same value
• break long sequences into partially (1/3) overlapping regions to speed things up even more, as well as give more accurate placements

Another thing that struck me is that in my annotation tool, I could probably use this as a faster way to store the set of matching proteins before extracting GO terms.  Currently, the performance is limited by XML parsing, so it’s probably not worth the bother at the moment.

More is likely to come up, but now it’s time to implement something!

As in any program, there are many occasions where you want to effect some particular change during some part of the program.  The archetypical example is allocation of local variables. After allocation, the variables are then available to the program until they run out of scope, they then get deallocated automatically.  The technique can be generalized beyond this.  For instance, you (or rather I) may want to set a $STAGE variable that indicates the current processing stage, and which should be unset when the stage has finished executing.  Or you may want to run some processing in a different directory, in which case you really want to remember to return to the previous directory when you finish.  The purpose of bracketing is to wrap a section of code with an initial part to be run in advance, and a final part to be run afterwards.

Some examples

When I toyed with PHP ages ago, I’d often find myself building a section of a page by a) generating a header with some opening tags, b) generating some content, and c) generating a footer with some closing tags.  And when the exact contents of these pieces depend on various factors, and such sections would nest in complicated ways, it should not be a surprise that getting a) and c) to correspond exactly could be a challenge.  With absolutely no enforcement by the language (which may have improved in later years, I wouldn’t know), this was very fragile.

For HTML the solution is simple: instead of generating open and close tags separately, have a function take the tag and its contents, and output the contents appropriately surrounded by open and close tags.  If you build the whole document this way, you guarantee that each open tag will have a matching close tag, and that tags will be properly nested.  Another example is Common Lisps with-open-file macro.

In Haskell, there’s bracket (in Control.Exception), which in addition to being vastly more general also is a regular function, thus once again proving Haskell’s vast technical and moral superiority over the more pedestrian languages….but I digress.  I suspect the rather original name is supposed to allude to how brackets (as in those banana-shaped glyphs surrounding this text) consist of an opening bracket, some contents, and a closing bracket.  Anyway, we like Haskell, so we use Haskell terminology.

As a final note, observe that brackets are similar to stack allocation, manual resource management is similar to manual memory management, while using finalizers/destructors is similar to garbage collection.  (It’s tempting to add “pick any two”.)

Generalized bracket

While implementing the EST pipeline, I found myself needing, and implementing, several bracket-like functions (including the two previously mentioned: setting and unsetting a variable, and running a subcomputation in a separate directory). Thus, the question that poses itself is: Is it possible to do this in a more general way, akin to Haskell’s bracket?  Here’s my currently best attempt:

bracket(){
    CLOSE=$2
    eval $1
    shift; shift
    eval $*
    eval $CLOSE
}

First we store the second parameter, which will be the “close” action in a variable.  We then execute the first parameter (the “open” action), using eval so that variables can be set etc.  We then skip the two first arguments, using shift twice, then eval the main action, and finally, eval the “close” action.

This allows stuff like:

bracket "mkdir mytmpdir && pushd mytmpdir" "popd" mkfiles

where the mkfiles function is run inside a temporary directory, and where execution resumes in the original directory after completion. Another example is

bracket "echo Entering first stage; STAGE=first" "STAGE=none" echo Current stage is '$STAGE'

Note the careful quoting of the variable with single quotes, we don’t want $STAGE to evaluated before it is set in the bracket function. In other words, the single quotes lets us pass the literal string $STAGE, sort of pass by name semantics instead of the default pass by value.

Perfection is the enemy…

If you are an experienced shell programmer, you may at this point have formed an opinion that I am not. And you’d be right, of course, but even I can see that there’s (at least) one obvious bug: we define a global variable named CLOSE.  Not only does this have the potential to clash with an existing variables, it also prevents recursive calls to bracket. Possibly, we should generate variable names, or have $CLOSE be a stack, or something…But hey, Mr. Know-it-all, if you’re so damn good, why not post a comment explaining how it’s really done?

In other words: feedback and comments are most welcome (although for spam-prevention you may have to register first)!