Part of my day job has recently been working on a de novo sequencing project. As we are are exploring many methods for genome assembly, it is imporant to be able to evaluate the resulting assembly in order to eliminate poor methods, improve on the more promising ones, and of course in the end select the best assembly. I have implemented a pipeline that performs this evaluation, and this is an attempt to briefly describe its goals and implementation.

Measures and input data

In order to assess quality, we rely on both internal and external measures. For instance, we can calculate the N50 contig size from the genome assembly directly, making it an internal measure. Alternatively, we can count the number of ESTs we are able to align to the assembly, making it an external measure.

For external measures, we will make use of many different types of data:

• High-throughput DNA-sequencing reads
• Paired-end and mate-pair reads
• EST reads
• Known DNA fragments, including the mitochondrion genome

Defining qualities

The pipeline addresses four aspects of assembly quality that ideally are independent of each other:

1. Completeness: does the genome assembly contain every bit of the genome, or is a part of it missing?

2. Fragmentation: does the genome come as a few, long, sequences, or as a zillion tiny fragments? And if the former, are the sequences correctly put together?

3. Accuracy: does the assembled genome sequence contain small-scale errors, from e.g. base calling errors?

4. Redundancy: is there a one-to-one correspondence between locations in the assembly and the actual genome, or does the assembly contain multiple copies of parts of the genome?

Note that it isn’t necessary that there is a fixed, optimal value for each measure that indicates a perfect assembly, it is only important that we can compare the scores for different assembly in order to determine which assembly is superior for that particular measure.

Completeness

Genomic (DNA) reads are aligned to the genome, and the proportion having matches is reported. We use BWA to align Illumina and 454 reads, and rely on ‘samtools flagstats’ to report the matches.

Also, various other sequences are mapped using BLAT, below we have a set of transcripts assembled from ESTs and RNASeq data, the mitochondrion genome, and fosmid ends. Here we report the average size of each match (as a percentage of the lenght of the query sequence), as well as the total number of sequences matching.

COMPLETENESS:
  Illumina mapped:      77.0411%
  PSL hits:
    all_transcripts_165.rna.psl.uniq:   92.0361%, 28398 matching sequences
    l.salmonis-mtDNA.dna.psl.uniq:      90.1952%, 3 matching sequences
    fosmids_fwd.dna.psl.uniq:   66.002%, 9411 matching sequences
    fosmids_rev.dna.psl.uniq:   63.7143%, 5886 matching sequences

Fragmentation

Fragmentation is measured by the standard N50 contig size, as well as N25 and N75. In order for it to be comparable, we use a fixed estimate for the genome size, not the size of the assembly itself.

Also, we count the proportion of Illumina reads that have the ‘proper pair’ flag set by BWA. Note that in order to make fragmentation independent of completeness, this is measured as a fraction of the matching reads, not all reads.

FRAGMENTATION:
  N25/50/75:    76308   46116   24010   using 600000000 as total size.
  Illumina pairs proportion:    67.7091%

Accuracy

We measure accuracy by looking at individual scores of alignments, in this case, genomic reads are aligned with BWA, and the alignments score is averaged.

ACCURACY:
  SFF average match:    148.565
  Illum average match:  46.7718

Redundancy

Redundancy looks at the total size of the assembly, comparing it to the estimated size. Then, we count the number of alignments of 454 reads and compare it to the number of query sequences.

Finally, we make use of the BLAT alignments, summing up the alignment scores as a proportion of the best scores for each sequence.

REDUNDANCY:
  Total size: 675758373, 112.626% of estimated size.
  SFF reads mapped:     117.235%
  PSL score, total vs unique:
    all_transcripts_165.rna.psl:            233.564%    67439222/28873929
    l.salmonis-mtDNA.dna.psl:       179.406%    76115/42426
    fosmids_fwd.dna.psl:            1046.08%    67459110/6448777
    fosmids_rev.dna.psl:            963.377%    37459791/3888385

Implementation

The pipeline is built as a makefile, which uses a mix of shell, AWK, and the occasional Haskell executable to do most of the heavy lifting.

Make is the widely used build system traditionally used in the Unix world. Essentially, the build process is described using a declarative language, and the make tool works out the process needed to constuct what’s desired.

Some features that makes it especially suitable is that:

• it avoids redoing work when it’s not necessary
• it automatically parallelizes the build process
• it is ubiquitously available

Make

The basic syntax of make is pretty simple (but never fear, it quickly gets complicated enough!). Here’s an example rule:

foo: bar
	process bar > foo

This tells make that, in order to build the file foo, it first needs the file bar, and when that is in order – either because it exists already, or because make can use some other rule to construct it – foo will be built by running the command process bar.

Make supports pattern-based rules, the archetypal example is probably something like:

%.o: %.c
	$(CC) $<

Which means that we can build any file ending in .o from a prerequisite having the same name, except ending with .c, by running the command line from the variable CC (presumably the C compiler) on the prerequisite (from the built-in variable intuitively named $<).

While this makes it easy to set up rules for building stuff from other stuff, the caveat here is that it’s all pattern based, typically using a wildcard (%) and a suffix. This means that, for instance, paired sequence data in FASTQ must be named something.1.txt and something.2.txt.¹

In addition, make has a set of built-in string manipulation functions, they typically look somewhat like:

STATS    := $(patsubst %.bam,%.stats,$(BAMS))

This populates the variable \$(STATS) with the contents of the variable \$(BAMS), except that it substitutes .stats for .bam for each element.

Shell

The commands in make files are executed by the shell, so it’s not too difficult to do things like loops. For instance, you can do:

all-fstats: $(FSTATS)
	for a in $(FSTATS); do echo "$a	`grep % $a | tr '\n' '\t'`"; done > $@

I.e., to build the file all-fstats, we first make sure we have everything in the \$(FSTATS) variable, then we loop over them, doing some grep and tr stuff, and directing output into \$@, yet another intuitively named variable, this time referring to the target of the current rule, or in this case all-fstats. Note that to refer to shell variables, they must be prefixed with an extra \$, a single dollar refers to makefile variables.

AWK

AWK is useful to collect data and perform arithmetic. For instance, we construct a table summarizing the interesting stats from all-fstats like so:

all-fstats.tab: all-fstats
	sed -e 's/\t[^\t]*(\([0-9][0-9]*\.[0-9]*%\))/\t\1/g' < $< | \
		awk ' BEGIN { OFS="\t" } { print; mapped += $2; paired += $3; single += $4 } \
		END { print "totals:    ",mapped/NR "%",paired/NR "%",single/NR "%" }' > $@

AWK is used here to accumulate sums of the various variables, and to calculate and print the averages in the end.

Haskell

There are also some tools written in Haskell that performs various processing. This includes psluniq, which extracts the best hit from PSL files, atcounts, which calculates GC/AT-ratio for each contig.

The whole thing is distributed as a Cabal library, so the required Haskell tools will be compiled or pulled in as dependencies on installation. Note that you’ll probably have to add $HOME/.cabal/bin to your $PATH.

Some thoughts

Make is Unix’s answer to democracy: it’s clearly the worst build tool out there, except for the others. It is often criticised for not being a full programming language, but I think this is a Good Thing: a more restrictive language is easier to get right. In fact, most of the problems of make stems from it having all kinds of extra functionality bolted on top.

One difficulty is the need to juggle between different paradigms, for instance, the various string processing functions in make go a long way in spite of all their messy syntax. But you still can’t escape some munging with AWK or Perl or similar tools. Wouldn’t it be nice if this were unified? Similarly, you have different layers with different sets of variables. Perhaps the ideal tool is make rules with Perl implementations?

An interesting alternative is Neil Mitchell’s shake, which does all of this in Haskell. In addition, it traces dependencies indirectly, so no more problems with forgotten dependencies. If or when I get around to trying out the alternatives, that’s clearly at the top of my list.

In practice

Getting it

The pipeline is available as a darcs repository, you should be able to get it simply by doing

darcs get http://malde.org/~ketil/asmeval

Of course, you’ll also need a bunch of tools, including the GHC compiler, the biopsl library, aligners BWA and BLAT, samtools, etc. Hopefully, you already have standard Unix tools like sed and awk.

On debian-derived Linux distributions, you should be able to get at least some of the way doing:

sudo apt-get install cabal-install darcs bwa
cabal update
darcs get http://malde.org/~ketil/asmeval
cd asmeval
cabal install

It looks like you need to install BLAT separately, and I may be missing some stuff - do let me know and I’ll update this post.

Configuring it

The configuration essentially consists of setting a bunch of make variables. This can be done on the command line, e.g.:

asmeval GENOME=mygenome.fasta ESTS=transcripts.fasta

However, asmeval will also look for a file called CONFIG in its current directory, which also can set these variables. It needs to be in makefile format, and is just included by the main makefile. The above translates to a CONFIG file containing:

GENOME := mygenome.fasta
ESTS   := transcripts.fasta

Note that this lets you use make’s string processing facilities in all their byzantine glory. There is an example (CONFIG.example) provided in the repository that can be used as a template.

Running it

After configuration, it should be possible to run everything by executing asmeval, which invokes make with some default options. These include -r to avoid the built-in rules (which only slow things down), and turning on warnings for uninitialized variables, in case you forgot something in the CONFIG file. You can add further options, for instance you may feel inclined to add -j to get parallel processing.

In order to run in the background and collect various information, I often type:

nice nohup time asmeval -j8 &

Note that j allows make to spawn multiple processes in parallel, but does nothing to take advantage of multi-threaded executables like BWA. There is a configuration variable, THREADS, you can set for this, so if an alternative to the above could be

nice nohup time asmeval THREADS=4 -j2 &

This will speed up some of the sub-processes, but run fewer of them in parallel.

The report

The default summary is output in a file called report, which contains the main statistics collected about the assembly. Hopefully it is self-explanatory enough to be useful.

So I’ll try to illustrate with an example I ran across.

In bioinformatics, sequence alignment (or edit distance) is part of our staple diet. For instance, an unknown protein can be aligned to known proteins to determine its function and structure. Sets of proteins from different species can be aligned against each other to determine phylogenetic lineage. And so on. So let’s look at alignments.

Definitions: an alignment of a pair of sequences is a mapping (f, say) from positions in one sequence to positions in the other. In addition, we require two invariants, first that the mapping is one-to-one:

f x == f y <=> x == y

This makes the alignment invertible, so there exists a mapping f' back from the second sequence to the first one. Makes sense, no? Second, the alignment is monotonic, satisfying:¹

f x > f y <=> x > y

In other words, the positions make up subsequences of equal lengths. So you can skip bits in either sequence, but you’re not allowed to flip things around.

In order to model this in Haskell, we can define a position to simply be an Int, and use a list of pairs of positions. So we can define a type alias for this:

type Alignment = [(Int,Int)]

Although this is very specific, it does feel a bit unsatisfactory from a typing perspective: the type doesn’t enforce either of the two invariants. E.g. it allows definitions like:

wrong1 = [(1,1),(2,1)]  -- maps both a[1] and a[2] to b[1]
wrong2 = [(1,2),(2,1)]  -- maps the wrong way around

It’s hard to come up with a good data structure that enforces the invariants (suggestions welcome!), but at least, it’s very concrete, and the compiler willl now protect you from accidentally substitute an alignments with a String, or any other type. Right?

Again, if you’re a dynamic typing enthusiast, you might argue that this enforcement is not such a big deal, you’re not very likely to pass strings by accident here anyway – and I agree. In fact, I think it is much worse that all the Ints are interchangeable; it is all too easy to, say, compare positions from different sequences, or performing arithmetic on them (what do you get when you multiply two positions?).

For example, we can align a sequence against another via a third sequence. Basically, given sequences A, B, and C, we have alignments A to B and B to C, and would like to derive the alignment of A to C. Now this isn’t too complicated:

trans_align :: Alignment -> Alignment -> Alignment  -- type signature
trans_align ((p1,p2):ps) ((q1,q2):qs)
	| p2 == q1 = (p1,q2) : trans_align ps qs        -- return match and continue
	| p2 < q1 = trans_align ps ((q1,q2):qs)         -- skip first 'p'
	| q1 < p2 = trans_align ((p1,p2):ps) qs         -- skip first 'q'
trans_align _ _ = []                                -- one input is empty

The first line declares the type of the function; it takes two alignments and returns a third. Then, the input alignments are pattern matched to extract the first pair of each alignment, and their position in sequence B are compared. If they are the same, we have found a position in A that maps to a position in C, if not, we skip the one which is smallest, and in any case we continue. The final line is for when the pattern match fails, i.e. one of the alignments is empty. In that case, no more mappings can be constructed, and we return the empty list.

But what if we make our Alignment type less specific:

type Alignment a b = [(a,b)]

Here, a and b are type variables, they represent any type. So e.g.

[("foo",42),("bar",1337)]

is a valid alignment of type Alignment String Int, for instance. In other words, we no longer require indices to different sequences to have the same type. We can of course give trans_align the type:

trans_align :: Alignemnt Int Int -> Alignemnt Int Int -> Alignemnt Int Int
-- index into:            A   B                B   C                A   C

But we’re not acutally using any Int operations here, and clearly, if the returned alignment is from A to C, we can enforce this by instead giving trans_align a type of:

trans_align :: Alignment x y -> Alignment y z -> Alignment x z

(In reality, we also need an Ord constraint on y, without this, we couldn’t compare positions in B in our implementation.²)

See what have we done? In the end we’ll probably just use Int for all the type variables (x, y, and z), but the implementation of trans_align doesn’t know this. Thus the type ensures that indices in the different sequences are kept separately, and things like comparing indices from different sequences, or returning indices from the wrong sequence are no longer not just bad ideas, they’re illegal, and enforced by the compiler. How cool is that? This function is even prohibited from comparing indices in A or C, all it can do with them is return them. And: unlike passing strings off as ints, mixing up indexes is a mistake that is easy to make.

In retrospect, this is perhaps obvious: The more general a type is - that is, the less you know aobut its specifics - the less you are allowed to do with it. A general type constrains its implementation. And the less you can do with it, the fewer wrong things are possible. Perfection is achieved not when there is nothing left to add, but when there is nothing left to take away. And perfect use of the type system is when all the wrong programs are eliminated, and only correct implementations are legal.

Hakyll is a templating system, so it consists of a set of HTML templates and a Haskell program that expands them and builds a set of static pages constiuting the site. There are an assortment of auxiliary files, like images and css that are just copied. Post content is written in markdown¹, and metadata specified at the top is available to the template processing.

This setup has some advantages: I can develop code or textual content on my laptop, build, and check the site by accessing a file:// URL. I use darcs version control, and when I’m happy with the result, I simply darcs push the changes to the server, and run hakyll.hs to rebuild the site. Also, the site being static, there are few holes to exploit (which were my main reason for moving off Wordpress).

However, although there’s a blog example included with Hakyll, it’s not entirely ready to use. There are plenty of batteries are included, but you need to fit them correctly, so to speak.

Fixing the RSS feed

In a fit of hubris, I emailed the maintainer of Planet Haskell to ask to be included. The quick and brief reply stated that my site, and in particular, its RSS feed, was not deemed adequate. So I started to implement the list of required fixes:

Dates were missing.

By default, Hakyll assumes that posts are named according to date, and sets data variables from the file name. I don’t care much for that, so to get a reasonable date field in the RSS, I added a published field to each post’s metadata:

published: 2012-01-06T15:00:00Z

The default RSS rendering function looked like this:

-- Render RSS feed
match  "rss.xml" $ route idRoute
create "rss.xml" $
     requireAll_ "posts/*" >>> renderRss feedConfiguration

In order to sort on the published field, I added a custom sort:

match  "rss.xml" $ route idRoute
create "rss.xml" $ requireAll_ "posts/*"
   >>> arr (reverse . chronological)
   >>> renderRss feedConfiguration

chronological = sortBy $ comparing $ getField "published"

The RSS file was missing post contents.

It’s a bit unclear to me what the description field is supposed to contain in RSS; Wordpress has this “more” thing, and includes stuff above it, typically an introduction, but most people seem to just include the post contents.

The easy way out is to include the post body, and the easy way to do that, is to just copy it into a variable. Here’s the diff:

      match "posts/*" $ do
          route   $ setExtension ".html"
          compile $ pageCompiler
 +            >>> arr (copyBodyToField "description")
              >>> applyTemplateCompiler "templates/post.html"
              >>> applyTemplateCompiler "templates/default.html"
              >>> relativizeUrlsCompiler

Hopefully, my RSS feed will pop up in Planet H. real soon now, in the meantime, why, you’ll just have to subscribe directly, won’t you?

Other tweaks

Now, since the description field holds the whole post contents, I changed the metadata field to summary (since that’s what it contained). To actually make use of this, I added this variable to templates/postitem.html, the template responsible for rendering postitems, that is, items when posts are listed.

 <li>
     <a href="$url$">$title$</a>
     - <em>$date$</em> - by <em>$author$</em>
+    <p class="noindent">$summary$</p>
 </li>

One important problem is that I’m sometimes sloppy, and I want site compilation to fail if there’s something wrong - if, say, I forgot to specify an important field. For this, I can use trySetField, which only evaluates its argument if the field isn’t already present. Having the value be error thus will halt compilation:

match  "rss.xml" $ route idRoute
create "rss.xml" $ requireAll_ "posts/*"
   >>> arr (reverse . chronological)
   >>> arr (map (trySetField "summary" $ error "Missing field: description"))
   >>> arr (map (trySetField "published" $ error "Missing field: published"))
   >>> renderRss feedConfiguration

In addition, I’ve made minor edits to template and CSS, but although I use cooler software than Twan, I think it’s evident that he has greater CSS-fu, so I’ll spare you the details…

Shoulders of giants, or forever voyaging through dark seas?

As you may have noticed, there are no forum included. My WP forum drowned in spam, so this is at least 50% intentional. I have included my email address, so if you have something interesting to say, I’ll update the post, taking it into account. Otherwise, discuss away on G+ or Reddit - that’s what they’re for.

All in all, I’m pretty happy with this. There is a certain amount of yak-shaving involved, but on the other hand: infinite flexibility and reuse of awesome code like pandoc, as well as excellent support (hint: Jasper spends all his spare time hanging around on #hakyll, just itching for you to bring forth your complaints.)

GACT - The General Alignment Clustering Tool

Yes, I too can haz a nucleotide acronym! GACT reads a set of sequence alignments (currently only BLAT’s PSL format is supported, but I intend to add BLAST tabular and possibly others), and clusters the sequences that are matched to each other.

Currently, it supports brief output (where each line represents a cluster, and consists of the labels of the sequences in the cluster) and log output, (where the cluster lines are followed by the set of alignments forming the cluster).

Supporting tools

Most aligners are rather “blind”, and in many cases only a subset of the produced alignments are useful. PSL support is implemented in the biopsl library, and it comes with a couple of tools to help extract what you want.

For instance, for EST clustering, you want the alignments to represent overlaps: if the end of sequence A matches the beginning of sequence B, that’s good, but if they have a short match in the middle, you’re probably not that interested. The pslfilter command lets you select reads based on percent identity of match, percent of query or target sequence covered, and on overhang (i.e. the shortest distance from the match to the end of a sequence).

In other cases, you just want to cluster by best match, there’s an app…uh, a tool for that, too, namely psluniq.

Availability

Available on Hackage, or from the darcs repositories.

If you do Illumina sequencing, you’re probably getting paired reads, i.e. you get two reads from each molecule, one from each end. This is usually pretty reliable, but for more complicated mate pair protocols, there’s often chimeric sequences where the two members of a pair actually come from different molecules, and sometimes the protocol fails for other reasons. In any case, it might be useful to look closer at the result when you map the reads to their reference.

To simplify this, I’ve uploaded a small tool, bamstats to Hackage. It uses Nick Ingolia’s samtools wrapper to extract relevant information from BAM files, and then summarizes it. Here are the examples:

Insert size statistics:

% bam stats -n 1000000 test.bam
Alignment                count     prop    mean   stdev    skew    kurt
innies                  484874  96.97%    365.3    24.1    -3.9    34.9
outies                     141   0.03%    112.8   217.3     8.0    62.8
lefties                      0   0.00%      NaN     NaN     NaN     NaN
righties                     0   0.00%      NaN     NaN     NaN     NaN
Total reads:    1000000

Histogram of insert sizes:

% bam hist -n 1000000 -b 6 -m 600 test.bam
Alignment                count    prop      100     200     300     400     500     >
innies                  484874  96.97%        0    2204    5136  471596    5847      91
outies                     141   0.03%      138       1       0       0       0       2
lefties                      0   0.00%        0       0       0       0       0       0
righties                     0   0.00%        0       0       0       0       0       0
Total reads:    1000000

As usual, the --help option will tell you about the available tunables.

Flower update

For 454 sequences (and, I guess, for Ion torrent?), it is often informative to look at the raw data (or flow values) in the SFF files. Flower, which now ships as a part of the biosff library, is one option for doing this. The output option -h produces a table of flow values by nucleotide. The latest release adds -H, which produces a table of flow values by flow position. This can e.g. be used to build empirical distributions for use with flowsim.

One way to plot the output is to run

flower -h flows.dat input.sff

And then, with gnuplot do:

gnuplot> set view map
gnuplot> set logscale cb
gnuplot> splot "flows.dat" matrix with image

I hope somebody, somewhere, noticed that my blog has been offline for some weeks now. After upgrading the server it runs on, I decided not to continue running the WordPress instance. There are several reasons for that, among others:

• I’m not terribly happy or comfortable running PHP at all,
• I’m not happy about a system that is based on downloading large code dumps and dropping them in specific directories.
• I was never very comfortable with the web-editing widgets.
• My installation was hacked.

I think the last points illustrates the problems with the first two.

Anyway, having made the decision to bid farewell, I looked for alternatives. Somebody suggested that I might not need a full content manangement system, but should look into static site generating systems, like Hakyll. This is it.

Using Hakyll

Hakyll is really rather simple to use: there’s a blog example included, and I just snarfed that with some modifications. There’s a main file (hakyll.hs) which directs how everything is to be constructed, a template directory, a css directory, and a posts directory. Adding posts means writing a markdown file, and stuffing it in “posts”. All under version control, of course.

Plans for the future

I can’t, of course, simply discard all my old posts, irreplaceable treasures of our cultural heritage as they are. Currently, they’re all 503’ed, but you can find some of them in the Wayback Machine. When time allows, I’ll have to bring the old MySQL instance up, and somehow extract the text in a useful format.

Since this site is now static, there isn’t, and won’t be, a comment field. With WP, it was all spam anyway, and we can all discuss stuff on Google+, Reddit, or wherever. What I do want to do, is to provide a “comment section” with backlinks to any mentions of my posts. Probably, I’ll mine the Apache logs for referring sites, and possibly also use Google search. We’ll see what I can come up with.

Comparing sequences to find similarity is a common occurrence in bioinformatics. For instance, one might want to know where a certain gene is located in the chromosome, or which sequence fragments are similar enough to originate from the same gene. To speed up searches, it is common to index sequences in questions as overlapping, substrings (k-tuples, q-grams). This index seems like an obvious target for Bloom filters – large data, time critical, some false positives anyway – but for some reason, there are almost no such applications that use them. Until now.

Sequence clustering

Sequence clustering is a commonly used technique which is usually based on sequence similarity. I’ve written one sequence clusterer, xsact, which is based on blocks of exact matches. There are many others, and another example is d2cluster, which is based on occurence of fixed length words – which is right up Bloom alley, right?

So a straightforward way to build a Bloom filter based sequence clusterer is to represent each cluster as a set of words – stored as a Bloom filter. Now, adding a new sequence to the clusters is a simple matter of extracting the words from the sequence, identifying the cluster(s) containing a sufficient number of these words, and adding the remaining words to that cluster (or the union of the clusters, in the case multiple clusters match).

The interesting thing about this approach is that the whole thing becomes O(kn), for k clusters and data size n. I think all other clustering algorithms are based on sequence pairs, which makes them O(n²) – in the worst case, you need to check all pairs. (However, a straightforward similarity-based clustering will have worst-case behavior when no sequence math each other, while suffix-based methods like xsact will have worst case when all sequences match – so perhaps there is room for a better middle ground?)

Anyway – while the above looks promising, there is one snag: EST sequences can occur as a copy of the gene, or due various properties of the manufacturing process, the gene’s reverse complement. Thus, we need to be able to compare sequences in both directions simultaneously. This could be achieved with a slightly creative hashing function, but to keep things simple, we’ll stick with the ossified mental sweat in the provided implementation.

Index and search

A somewhat related area is indexing and search. Let’s say you have a bunch of DNA sequences (of for each chromosome, perhaps), and a set of ESTs, which you’ll remember are gene fragments in an unpredictable mix of forward and reverse-complement directions. Here’s the plan:

1. index by building a Bloom filter containing the q-tuples for each chromosome

2. for each EST, look up each q-tuple (first forward, then rev.comp.) against the filters, and assign it to the chromosome containing the most q-tuples

3. align each EST against the designated chromosome using traditional methods

As far as I can tell at this point, for word size q, c chromosomes of length m and e ESTs of lenght n, this should run in something like O(qn) + O(qenc) + O(emn), compared to just aligning directly at O(ecmn). Note that mn is the big factor here, so on a large scale, we’re reducing the total work by a factor of c. (Of course, in real life, nm would be too large, and you’d use a heuristic, subquadratic alignment step, but this is for illustration purposes. Try to be a bit generous, will you?)

Further plans

That concludes the current plan, but there are certainly improvements that can be made, two obvious ones are

• hash function that hashes a word and its reverse complement to the same value

• break long sequences into partially (1/3) overlapping regions to speed things up even more, as well as give more accurate placements

• Another thing that struck me is that in my annotation tool, I could probably use this as a faster way to store the set of matching proteins before extracting GO terms. Currently, the performance is limited by XML parsing, so it’s probably not worth the bother at the moment.

More is likely to come up, but now it’s time to implement something!

Hackage submissions updated

There seems to have been problems with some of the bioinformatics applications on Hackage, thanks to Don S. for pointing it out. That should be fixed now by new uploads, but I’m still waiting for the automatic builds to register results. An, since you ask, it was all my fault for being sloppy with version dependencies. It’d all work with a recent biolib. Speaking of which,

A home page for the bioinformatics library

I’ve finally updated the static home page for the library, it can be admired (especially if you remember what Oscar Wilde had to say about that) here.

I’ve discovered HPC

No, not high-performance computing which sucks anyway, but GHC’s new ability to do coverage profiling. Adding it to the default testing procedure was just a couple of extra options. This is very cool, almost too easy, and looks pretty good. The only downside is that it exposes my sloppy attitude to testing.