flowers

Here’s a better method that identifies poly-A tails by finding an optimal, quality adjusted alignment in linear time.

A quick introduction

Although the definitions of what constitutes a gene vary considerably, we’ll use the term to refer to a region of DNA that get transcribed, that is, copied from DNA into an mRNA molecule, which in turn will be used as a blueprint for assembling a protein. After transcription, the mRNA molecule then undergoes polyadenylation, a process where a string of adenines (the ‘A’ of the nucleotide alphabet) gets appended to an mRNA molecule before it is exported from the nucleus.

Identifying poly-A tails are important for several reasons.

1. It positively identifies the end of the transcript. If you don’t have a poly-A in your sequence, you have no way to know how far the molecule extends beyond the end of the sequence. You can also find alternatively terminated transcripts this way.
2. It positively identifies the end of the transcript. Anything after the poly-A tail is linker or vector sequence, and can safely be trimmed off. Even if, as is often the case, it is too low quality to be recognized by your average vector masking software.
3. It provides useful information about the transcript, as the poly-A tail is important for things like protecting the mRNA from degradation.

Unfortunately utilities often trim poly-A tails by default (e.g. SeqClean), or just ignore it (e.g. BLAST’s low-complexity
filter).

Quality based alignment

When a molecule is sequenced, the analog output from the sequencing machine is stored as a chromatogram. In order to be useful, the sequence is called, that is, translated to a string of letters from the familiar {A,C,G,T} nucleotides alphabet. In addition, the base caller will associate each letter with a quality value. This is derived from an estimate of the probability of the call being incorrect, and for quality value Q, the error probability estimate is

ϵ = 10-Q/10

Traditionally sequence alignment simply aligns the string of characters using a fixed positive score (reward) for aligning similar characters, and fixed negative scores (penalties) for either substituting a different character, opening a gap, or extending a previous gap.

However, taking into account the quality value, we can do better, and instead of fixed scores, we can adjust the scores dynamically according to quality.

Using this method, the penalty for e.g. aligning two different characters will depend on the quality of the characters: high quality means a high penalty, low quality — lower penalty (since there’s a greater chance one of them was incorrectly called).

Scoring of alignments

When calculating the score of an alignment, we really want to answer the question: how likely is this sequence to be a real poly-A sequence, as opposed to just a random string? In other words, we are comparing our sequence against two models: the poly-A model, and the background model. Our score will use the ratio of probabilities of the string being produced by the two models.

For the poly-A model, only As are allowed, so the probability of a character occurring is 1 for As and 0 for the others. For the background model, we’ll just take a uniform distribution of nucleotides, each getting a probability of 0.25.

Using this scheme, the score for a string s is simply 1/0.25 = 4 for each A, and 0/0.25 = 0 for all others. We usually work with the logarithm of these numbers to make them more manageable.

The optimal alignment is then simply the longest run of As, since as soon as you multiply with a zero (or add -infinity, if you use log-scores), you lose the whole score.

Adding quality to the mix

Of course, the actual sequence isn’t perfect, and even the poly-A tail is likely to contain the odd G, C, or T. To determine exactly how likely is where the quality value enters the picture. Using the formula above, we can calculate the error estimate and decide what the penalty for a mismatch and reward for a match should be.

For the poly-A model, the probability for a match (that is, an actual ‘A’ in the sequence) is 1-ϵ, the probability of a mismatch (a non-A) is ϵ/3 (since only one of the three possible substititutions is an A, and for simplicity, we give them equal probability). Using the formula for  ϵ as a function of Q (and hopefully not introducing any errors), I get the scores to be:

match q = log (4*(1-1/10**(q/10)))
mismatch q = log 4 - log 3 - q/10*log 10

Now, we can use this to do a standard Smith-Waterman alignment, calculating a dynamic programming matrix, and searching for an optimal local alignment.

However, since we’re aligning against a repeated nucleotide, there’s no real need for a second dimension, and we can use the following recurrence to calculate the “polyA-score” M for each position i:

M_i = max (0, S_i + M_i-1)

To implement this, we first define the list of scores by applying match and mismatch to the list of (nucleotide,quality) pairs. We also define a scanl-based function to calculate a list of cumulative scores:

scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
cumulative = scanl (\a b -> let r = a + b in max 0 r)

The only remaining thing is to identify the maximal value which marks the end of the poly-A tail, and the corresponding 0 value that indicates the start. I wrote a recursive function called ”findmax” for this, but a better programmer will probably be able to do this with a fold.

Including the parts discussed briefly above, the whole thing looks like this:

findPolyA :: Sequence Nuc -> Maybe (Int,Int)
findPolyA (Seq _ d mq) =
let qd = zip (B.unpack d) (maybe (repeat 15) BB.unpack mq)
scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
match x' = let x = fromIntegral x' in log (4*(1-1/10**(x/10)))
mismatch x' = let x = fromIntegral x' in log 4 - log 3 - x/10*log 10
cumulative = scanl (\a b -> let r = a + b in max 0 r) 0
(zi,mi,maxscore) = findmax $ cumulative scores
in if maxscore > 12 then Just (zi+1,mi) else Nothing  -- arbitrary constant alert!

findmax :: [Double] -> (Int,Int,Double)
findmax = go 0 (0,0,0) . zip [0..]
where go _ cm [] = cm
go _ cm ((i,0):rest) = go i cm rest
go last_z (cmz,cmi,cmx) ((i,x):rest) = if x > cmx then go last_z (last_z,i,x) rest
else go last_z
(cmz,cmi,cmx) rest

Availability

This method is implemented in a simple tool called “trimpolya” (darcs repo), and also in the more general “dephd” (darcs, hackage) sequence analysis package.

Installing biolib

First I need to install the bioinformatics library.  I’m about to release 0.4.1, but this is also a good opportunity to check that everything works with 0.4 (which is what you’ll find on Hackage), so let’s do that first.  Using darcs, I pull the repo up to the 0.4 tag (but you can of course get the tarball from Hackage):

% ./Setup.hs configure
Configuring bio-0.4...
Setup.hs: At least the following dependencies are missing:
QuickCheck <2, binary -any


(Side note: you may notice that I run Setup.hs directly, as opposed to using runhaskell. I prefer it this way, but you may have to do a chmod +x Setup.hs if you downloaded this from the darcs repository or similar.)

Since we want to use the system libraries as far as possible, these libraries are just an apt-get away:

sudo apt-get install libghc6-quickcheck1\*
sudo apt-get install libghc6-binary-\*


Now, let’s try again:

% ./Setup.hs configure
Configuring bio-0.4...
% ./Setup.hs build
Preprocessing library bio-0.4...
Building bio-0.4...
Binary: Int64 truncated to fit in 32 bit Int
ghc: panic! (the 'impossible' happened)
(GHC version 6.10.4 for i386-unknown-linux):
Prelude.chr: bad argument

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Okay, this was not what was supposed to happen.  As always, dropping to #haskell on IRC is the first thing to do, and sure enough:

<sereven> ketil: that's also shown up for xmonad users when .hi and .o files weren't cleaned
between rebuilds mixing different versions. usually between ghc updates IIRC.


Let’s try to get rid of old cruft lying about, polluting directories:

./Setup.hs clean && ./Setup.hs configure && ./Setup.hs build

Sure enough, this time it worked. For good measure, we’ll run the unit tests:

make test

After a zillion tests, we notice that everything is go, great!

Applications

Next, it is time to go through the list of bioinformatics applications.  Since my working directory is a mess of branches and versions, we’ll just go over the published applications and versions on Hackage.

xsact is an application to do sequence (in particular EST sequence) clustering.  It predates and thues doesn’t actually use the bioinformatics library, but we’ll check it anway.  So we try the familiar command line:

./Setup.hs clean && ./Setup.hs configure && ./Setup.hs build

And things compile.  However, the version on Hackage is outdated, so we’ll upload a new version, 1.7.1.  One test case still fails, but I can’t imagine anybody is using it to generate Newick-formatted trees — I am certainly not — and since there are many equally correct outputs (including tree rotations and rounding modes), output is likely correct anyway.  Holler if you need it!

rbr is an application to mask repeats in sequence data.  Normally, this is done using a library of known repeats, but this application tries to do it using statistics, making the — I think justifiable — assumption that repeats are going to be more common than non-repeats. The version on Hackage is old, and only works with the library prior to 0.4, so again this is a good time to push the latest changes out in the limelight.  Compiling this works great, by the way.

cluster_tools is a package that contains a bunch of binaries, useful for working with the results of sequence clustering, including extracting various information from ACE files.  This uses another library, called simpleargs, that simplifies command line argument parsing for simple cases.  Again, the Hackage version is for bio<0.4, so a new version will be pushed.  At the same time, we make a mental note to push version 0.2 of simpleargs to Hackage as well, instead of keeping age-old modifications buried forever.

dephd is my Swiss-army-knife of sequence analysis, and lets you do various things like converting between formats, plotting and trimming by quality.  This is a more live project than most of the others (I’m currently working on improved quality trimming and automatic generation of files for submission to GenBank), but the currently available version also compiles without incident.

estreps is a couple of programs I needed for repeat analysis, perhaps not tremendously interesting, but at least rselect, which lets you select randomized subsets from Fasta files, might be of interest to some? We try the usual invocation to compile, and get:

src/Unigene.hs:24:23:
Couldn't match expected type `a' against inferred type `Unknown'
`a' is a rigid type variable bound by
the type signature for `clusters' at src/Unigene.hs:22:41


This error arises due to the introduction of phantom types for identifying sequences introduced in bio 0.4. Unfortunately, this version of estreps contains some adaption to this model, so it won’t compile against older versions either. So it looks like yet another sdist for Hackage. Look for version 0.3.1.

flower is a utility for extracting information from SFF files (containing sequences from Roche’s 454 machines).  Although a new version is around the corner, the old 0.2 just works.

xml2x is a utility for converting BLAST results in XML format into CSVs, that somehow is more compatible with biologists.  Trying to compile it fails, with the following error:

src/Xml2X.hs:152:49:
Couldn't match expected type `[b]'
against inferred type `Maybe Bio.Sequence.KEGG.KO'
In the first argument of `concatMap', namely `(flip M.lookup ks)'
In the first argument of `($)', namely
`concatMap (flip M.lookup ks)'
In the second argument of `($)', namely
`concatMap (flip M.lookup ks) $ map chop $ map subject fs'


It turns out that somewhere along the way, the lookup function from Data.Map was de-generalized from working on arbitrary monads to just returning a Maybe. I was using this to return a list, using the empty list to signal an unsuccessful lookup. This is easily remedied, but that means yet another sdist for Hackage.

korfu is a utility for identifying open reading frames in sequence data.  It hasn’t yet been ported to version 0.4 of the library, but works if you install 0.3.5.  I updated this too, since it didn’t have a category.  Now it too resides in the bioinformatics section.

Summing up

In retrospect, it seems like giving old code a thorough spring cleaning once in a while. Although nothing really critical or difficult happened, a good number of small annoyances were discovered, and a bunch of new sdists are now ready to be uploaded to Hackage. Next will be converting all this into debian packages.

The important question is of course, how do we avoid this in the future? During development, it is important to be able to modify libraries and appliations, but installing a new version of the biolib, say, overwrites the old one, and suddenly I’m compiling and testing everything against a different library than Joe Random Hackage User is going to find. I have some thoughts on how to avoid this, but if you have a method that works nicely, I’m all ears.

•  My biohaskell slides
• slides’ LaTeX source, using beamer.cls
• Lazy PSSM JFP-paper

The original version

I’ve taken the liberty of cleaning things up a bit, and removing some context that I sure hope isn’t necessery.  Bascially, the task is to take a list of ByteString input lines consisting of whitespace separated decimal integers, and build a ByteString consisting of single byte quality values corresponding to those integers.

Below is how the naive quality parsing function might look, unpacking each word to String in order to use read.  Note that B is lazy ByteString.Char8, BB is lazy ByteString, that is, the Word8 version.

BB.pack $ map (read . B.unpack) $ B.words $ B.unlines ls

We don’t expect this to do terribly well, and I guess it’s no surprise when this parses my test file of about 10MB in 24 seconds.

Improved versions

The key to improved performance is first and foremost to avoid the unpacking and parsing of strings.  Thankfully, the ByteString library provides a readInt function.

BB.pack [lookup x | x <- concatMap B.words ls]
    where
    lookup x = case B.readInt x of Just (v,_) -> fromIntegral v
                                   Nothing -> error "Unparsable qual value"

This isn’t a lot more complicated than our initial attempt, but the speed increas is considerable: slightly less than 2 seconds for the test file, more than a tenfold improvement.  The ByteString implementation will share the storage of the separate words with the original strings, but since readInt gives us back the rest of the string in addition to the parsed integer, we might as well make use of it:

BB.pack $ readInts $ B.unlines ls
    where readInts xs = case B.readInt xs of 

                          Just (i,rest) -> fromIntegral i : readInts (B.dropWhile isSpace rest)
                          Nothing -> []

This turns out to be a bit faster, time is now 1.6 seconds.  Another 20% shaved off.

Final version

We’re really only interested in Word8 values, since quality values always are small, and since that’s what gets encoded in the result anyway.  The previous versions takes a detour by reading Ints and using fromIntegral to convert them to the desired size.  It bears noting that there is no error checking involved, fromIntegral will happily and silently truncate any number beyond its target range.  So lets do things explicitly, using Word8s throughout the computation:

BB.pack $ go 0 ls
    where
    isDigit x = x <= 58 && x >= 48
    go i (s:ss) = case BB.uncons s of 

                    Just (c,rs) -> if isDigit c then go (c - 48 + 10*i) (rs:ss)
                                   else let rs' = BB.dropWhile (not . isDigit) rs
                                        in if BB.null rs' then i : go 0 ss else i : go 0 (rs':ss)
                    Nothing -> i : go 0 ss
    go _ [] = []

This is the fastest one so far, clocking in at 0.94 seconds, over 40% faster than the best readInt version, and about 25 times faster than the naive version.  Still, 10MB/s is well below the average hard disk.

So is there more room for improvement?  The most obvious wart to me is the rather artificial splitting into lines.  This is mostly an artifact of some early design desicions, and it should be possible to eliminate the splitting earlier on and saving even more time by simplify this function quite a bit.

If you spot anything else, or have suggestions, I (and my darcs repo) am all ears.

Edit: Since some people have asked, I’ve wrapped up a simple test program along with some test files at http://malde.org/~ketil/biohaskell/qualparsetest.  This is a simplified version, if you want to be really helpful, you could always look at Bio.Sequence.Fasta in the Bioinformatics library and see if you can speed up e.g. dephd -i input.fasta input.qual -F /dev/null.

The Haskell bioinformatics library has for some time now supported reading and writing the SFF format, which is used by the oldest (previousest?) of the next generation technologies, namely Roche’s 454 sequencing.  Once the library functionality is in place, it is easy to develop small tools for doing the various chores.   After spending some time in anticipation of the hordes of programmers no doubt rushing to exploit the monumental effort put I down in  the library, I’ve instead written a few programs myself, including tools information/statistics extraction (flower),  extracting sequences by various criteria (fselect), simulating sequencing (pyrosim), and repairing broken SFF files (frepair).  This is their story.

The bioinformatics library includes functions and data structures to read and write and extract data from SFF files.  It provides reasonable performance (meaning that in most cases, my disk is the limiting factor).  It is also a clean room implementation, built from official documentation, but not based on Roche’s (or anybody else, for that matter) code.  It comes with an LGPL license, which I hope will make it useful while encouraging back contributions.

Flower

This program can extract  various information from SFF files.  To some extent, it overlaps with ’sffinfo’ and ’sfffile’ from 454, but in addition to generating Fasta sequences (optionally with quality), it can also extract directly to the more compact FastQ format.  It also can output a (huge) table of flow values, the histogram of flow values (useful for estimating the flow distributions, and thus probabilities for the base calls), or a one-line summary of each read that includes some statistics on lenghts and quality. Here’s the usage info:

flower: Usage: flower -[f|q|r|R] <file.sff> [<file2.sff> ..]
  -r  output reads in Fasta format
  -R  output reads in Fasta format with associated .qual
      (generates files instead of writing to <stdout>)
  -q  output in FastQ format
  -f  output the flowgram in tabular format
  -h  output a histogram table of flow values
  -s  output a summary of each read

FSelect

FSelect takes an SFF file and produces a new SFF file containing a subset of the sequences, using the same statistics as Flower can output.  It has a small expression language built in, so that you can build more complex logical queries.  For instance, if you want to extract the sequences with lenght between 300 and 400, and with a K² score greater than 0.7, you could do

fselect "And (Func LT k2 0.7) (And (Func GT len 200) (Func LT len 400))" FL61AHU01.sff

Okay, it’s a bit clunky, but the syntax should be reasonably straightforward: Logical operators are And, Not, and Or, while Func op f v defines a function using op (either LT or GT) to compare f (one of k2, len, tlen, ncount)  to each read.  Output is generated in a file named “selected.sff”.
FSelect can also select random reads, using the select expression "Rand p", where p is the probability for selection.  (I.e. fselect "Rand 0.2" FL61AHU01.sff will select each read with a probability of 0.20, giving you approximately 20% of the reads.  Random selection can not be combined with other criteria at this point, if you want this, you’ll have to run fselect multiple times.

FRecover

We had some issues with broken SFF files, specifically there were block of zero bytes at random places. Both flower and sffinfo just terminate on encountering a broken read, so I implemented a simple utility that attempts to skip the broken block and continue extracting good reads beyond the trouble.

PyroSim

This attempts to simulate pyrosequencing, for now it only does the GS20  generation of 454 sequencing, but the other generations should be easy to add.  The main problem is that the algorithm for quality calling is insufficiently documented.  GS20 has the advantage that quality for a homopolymer is uniquely derived from the flow value (modulo rounding), so reverse engineering it is fairly straightforward.

Anyway, pyrosim takes a ‘generation’  (at the moment, this is only GS20) and a Fasta file as input parameters, picks random points in the Fasta file, and produces the correspondig flowgram, including a suitable perturbation of the values to introduce the expected measure of noise, calls the bases and quality, and outputs an SFF file.

Availability

Flower, FSelect and FRecover are part of the “flower” package, available from the darcs archive at http://malde.org/~ketil/biohaskell/flower

PyroSim is available separately (for now, at least) from the darcs archive at  http://malde.org/~ketil/biohaskell/pyrosim

I try to upload what I consider stable versions to HackageDB, please check the Bioinformatics category there.  Currently, these programs are in a bit of flux, so going with the darcs repo is probably a good idea at this point.

I’d really like to have packages for the most common Linux distributions as well (i.e. .debs and .rpms), but I don’t know the details of how to produce them, and while I’ve made half-hearted attempts in the past, I guess I just don’t really desire it enough.  I’d be happy to see somebody package it up, so if you know how, please go ahead.

Filtering out empty sequences.

Phred often produces zero-length sequences, and this confuses other programs.  While BLAST will just output a warning, SeqClean — or to be precise, cln2qual — will break down.   (My own code using the Bioinformatics library treats all sequences the same regardless of length, so zero-length sequences are perfectly okay). Anyway, you can now use dephd -z to eliminate them from the output.

Sequence Clipping

Sequence trimming or clipping is often necessary to remove contamination like vector sequence, or simply low quality sequence parts.  Typically, both of these occur at the ends of the sequences.  Many programs (including dephd, but also phred, lucy, seqclean and others) add trimming information to the sequence header.  Dephd is now able to act on this information and clip the sequences.  The trimming information is now obsolete as the coordinates have changed, so they are replaced with the clipping coordinates.   Dephd also provides its own quality assessment, and with the -q option, sequence ends where the sliding windown average quality is below 15 will be clipped.  This is pretty heavy-handed, but it seems I get better EST clustering with this enabled.

Old features

Of course we retain the old features: reads PHD and Fasta/Qual files, mask (to lower case/N but don’t clip) by quality, generate quality plots, outputs Fasta/Qual, and ranking sequences by quality.

Edit: In the latest release, there’s now also a fix for a problem with drawing quality graphs with gnuplot.  It turns out that my shiny new Ubuntu ships with gnuplot 4.2, but your crappy old distribution ships with an older version, and that there are some incompatibilities in the input formats.  I’ve now reverted this to use old-style format only, so hopefully it should work with gnuplots back to 3.7 or so.   And for those SLS or MCC interrim die-hards out there, I’ll even add an option to dump the gnuplot file itself, so that you can copy it to a floppy and generate the plots on a computer with a modern color display. How’s that for user friendly?

The safety that we seek can be exemplified by sequence alignment or similarity scores, as calculated by e.g. the BLAST suite of programs.  Here we have e.g. blastn for comparing nucleotide sequences, blastp for comparing amino acid sequences, and blastx for comparing nucleotides to proteins by first translating the nucleotides to the possible corresponding amino acid sequences. We’ll aim for an align function that does the right thing for the various combination of sequence types.

The old approach

Previously, there was only one Sequence data type, defined as:

data Sequence = Seq SeqLabel SeqData (Maybe QualData)

(where SeqLabel, SeqData and QualData are represented as various ByteStrings, but you can think of them as synonyms for String)

As mentioned, this isn’t really leveraging the type system much, and at the very least, it’d be a good thing to be able to differentiate between nucleotide and  amino acid sequence (a.k.a. peptides or proteins).

The algebraic data type approach

The default functional programming approach would be to use an algebraic (sum) type, what the imperative programmers woudl call a union.

data Sequence = Nucleotide ... | Protein ...

This is a good solution when the two branches have different structure, but here you’d essentially just repeat the structure.  Moreover, all functions will need to do run-time checks for each case, imposing a cost both in function complexity, running time, and type safety.  Note that performing the alignment usually requires a score matrix describing the penalty for replacing any given character with any other.

align :: Matrix -> Sequence -> Sequence -> Alignment

align mx (Nucleotide ...) (Nucleotide ...) =
align mx (Nucleotide ...) (Protein ...) =
align mx (Protein ...) (Nucleotide ...) =
align mx (Protein ...) (Protein ...) =

This is not even complete, since a similar restriction applies to the score matrices – sometimes it will contain replacement penalties for the nucleotide alphabet, and sometimes it will contain penalties for the amino alphabet, and the appropriate matrix must be used in each brach of the align function.

Another problem with this is that this requires you to over-specify the type.  Sometimes you don’t know or care what kind of sequence you have.  Say you are selecting sequences by name from a Fasta file.  Since the file format is the same you don’t really care what kind of sequence it is, and forcing it to be one or the other is….immoral.

The third way: phantom types

The chosen approach is instead to tag the Sequence type with a phantom type parameter, phantom meaning it does not affect the actual representation of the data.  It looks like this:

data Sequence t = Seq ....  -- but no data member of type 't'!

Now, we can write our alignment functions, and make them safer to use:

align :: Matrix t -> Sequence t -> Sequence t -> Alignment
alignX :: Matrix Amino -> Sequence Nuc -> Sequence Amino -> Alignment

Note that we also phantom-typed the Matrix type.  With this approach,  incorrect usage like comparing sequences of different type with the generic align will be flagged by the compiler, so run time checks are no longer necessary.

On the other hand, readFasta can now be given the type:

readFasta :: Filepath -> IO (Sequence a)

or:

readFasta :: FilePath -> IO (Sequence Unknown)

depening on how much you trust the programmer.  Since I’m writing most of the programs using this library, I know how much you can trust application programmers, so the second and safer method is chosen.

 In practice

The old biolib repository has now been replaced by two new ones: biolib-stable, currently containing version 0.3.5, and biolib-unstable at 0.4.0. Perhaps unsurprisingly, the latter version contains the phantomly typed Sequence definition.

Currently, three types are used for tags:  Amino, which is the type for the amino acid alphabet,and Nuc and Unknown, which are dummy types without any data constructors, and used solely for this purpose.

So what are the experiences so far?  Well, it does complicate things.   For some file types the sequence type is known, for instance the output of nucleotide sequencing machines in the form of ABI, SCF, or SFF files.  But often, file formats are agnostic with respect to sequence types, the most ubiquitous offender being the Fasta format. Currently, this is handled by the Unknown type tag, but I’m not entirely convinced this is the optimal solution.

Code needs to be updated, but mostly this is relatively easy.  The type system will spot the difficulties, and often you can get by by just replacing Sequence in the type signatures with Sequence t.  Just make sure that t isn’t already used in the signature – I stumbled into this one.

Of course, this is one step towards increased leveraging of the type system.  One could go further, and  tag different type of nucleotide sequences based on sequencing technology: Sanger sequencing has quite different error characteristics from 454 sequencing, and Solexa has a completely different interpretation of the quality values than either of those.  Also, one might want to include the presence or absence of quality data in the type as well.  There are also different amino acid alphabets – should they have different types?  This is a difficult design space – how much information should be encoded in the type?

While I think this is an improvement, I’m very curious how this works out in practice, or if there are other options I should consider.  Please comment!

Also, since short reads are all the rage, and my flower program appears to be used a bit, I’ve done a quick writeup of its features and use as a static page.  I’ll try to keep it updated as things progress.  Popularity – Yay!

Finally, I got some help compiling everything on some less mainstream operating systems (”Windows”, I think it is called).  Mostly, things appear to work, and some improvements – albeit portability-neutral ones – were made.  Portability  – Yay!.

First things first, here are the files:

Bloom filters for bioinformatics – paper

Bloom filters for bioinformatics – slides

The new addition to the slides consist of the counting bloom filters, and a brief overview on how to locate matches in a Bloom filter.  The standard counting Bloom filter consists of replacing the array of bits with an array of bit buckets (of size b, say).  This lets the filter count up to 2^b occurrences of each element, typically using saturating counts.  If you want to retain the false positive rate, this means you’ll expand the size of the Bloom filter by a factor of b, which can be quite significant. In the example below, b is set to 3, and the filter saturates at a count of 7.

Therefore, I propose a different kind of counting bloom filter, using a distributed count.  Keep in mind that we have an infinite sequence of hash functions.  When inserting a new value, we can check the k hash functions to see if it’s already there.  If so, we calculate some additional hash functions to represent the count of two for this value.  And so on, until we find a count (and corresponding set of hashes) that have at least one zero bit, set those bits to one, and move on to the next value.  It may make sense here to decrease the number of hash values as the count goes up.  In the example below, we see that x is already inserted, and two new hash values are calculated to increase the count.

There’s a trade off, of course.  Insertion and lookup is no longer (as) constant, but  proportional to value count. However, the Bloom filter is likely to be more compact – given n values in total, where u are unique, the standard counting filter needs a size proportional to bu, while the distributed counting filter needs size proportional to n.  If you want to count accurately, b needs to be set to the logarithm of the max count of any value.

Finally, it’s also possible to use Bloom filters for searching – for instance, locating unique words in a genome. A lookup in a Bloom filter basically gives one bit of information – present or not present.  We use this with a series of m Bloom filters, each indexing the regions of the genome corresponding to one bit of location information.  Looking up a unique word in the set of filters reveals the location with m bits of precision.