Here’s a better method that identifies poly-A tails by finding an optimal, quality adjusted alignment in linear time.

A quick introduction

Although the definitions of what constitutes a gene vary considerably, we’ll use the term to refer to a region of DNA that get transcribed, that is, copied from DNA into an mRNA molecule, which in turn will be used as a blueprint for assembling a protein. After transcription, the mRNA molecule then undergoes polyadenylation, a process where a string of adenines (the ‘A’ of the nucleotide alphabet) gets appended to an mRNA molecule before it is exported from the nucleus.

Identifying poly-A tails are important for several reasons.

1. It positively identifies the end of the transcript. If you don’t have a poly-A in your sequence, you have no way to know how far the molecule extends beyond the end of the sequence. You can also find alternatively terminated transcripts this way.
2. It positively identifies the end of the transcript. Anything after the poly-A tail is linker or vector sequence, and can safely be trimmed off. Even if, as is often the case, it is too low quality to be recognized by your average vector masking software.
3. It provides useful information about the transcript, as the poly-A tail is important for things like protecting the mRNA from degradation.

Unfortunately utilities often trim poly-A tails by default (e.g. SeqClean), or just ignore it (e.g. BLAST’s low-complexity
filter).

Quality based alignment

When a molecule is sequenced, the analog output from the sequencing machine is stored as a chromatogram. In order to be useful, the sequence is called, that is, translated to a string of letters from the familiar {A,C,G,T} nucleotides alphabet. In addition, the base caller will associate each letter with a quality value. This is derived from an estimate of the probability of the call being incorrect, and for quality value Q, the error probability estimate is

ϵ = 10-Q/10

Traditionally sequence alignment simply aligns the string of characters using a fixed positive score (reward) for aligning similar characters, and fixed negative scores (penalties) for either substituting a different character, opening a gap, or extending a previous gap.

However, taking into account the quality value, we can do better, and instead of fixed scores, we can adjust the scores dynamically according to quality.

Using this method, the penalty for e.g. aligning two different characters will depend on the quality of the characters: high quality means a high penalty, low quality — lower penalty (since there’s a greater chance one of them was incorrectly called).

Scoring of alignments

When calculating the score of an alignment, we really want to answer the question: how likely is this sequence to be a real poly-A sequence, as opposed to just a random string? In other words, we are comparing our sequence against two models: the poly-A model, and the background model. Our score will use the ratio of probabilities of the string being produced by the two models.

For the poly-A model, only As are allowed, so the probability of a character occurring is 1 for As and 0 for the others. For the background model, we’ll just take a uniform distribution of nucleotides, each getting a probability of 0.25.

Using this scheme, the score for a string s is simply 1/0.25 = 4 for each A, and 0/0.25 = 0 for all others. We usually work with the logarithm of these numbers to make them more manageable.

The optimal alignment is then simply the longest run of As, since as soon as you multiply with a zero (or add -infinity, if you use log-scores), you lose the whole score.

Adding quality to the mix

Of course, the actual sequence isn’t perfect, and even the poly-A tail is likely to contain the odd G, C, or T. To determine exactly how likely is where the quality value enters the picture. Using the formula above, we can calculate the error estimate and decide what the penalty for a mismatch and reward for a match should be.

For the poly-A model, the probability for a match (that is, an actual ‘A’ in the sequence) is 1-ϵ, the probability of a mismatch (a non-A) is ϵ/3 (since only one of the three possible substititutions is an A, and for simplicity, we give them equal probability). Using the formula for  ϵ as a function of Q (and hopefully not introducing any errors), I get the scores to be:

match q = log (4*(1-1/10**(q/10)))
mismatch q = log 4 - log 3 - q/10*log 10

Now, we can use this to do a standard Smith-Waterman alignment, calculating a dynamic programming matrix, and searching for an optimal local alignment.

However, since we’re aligning against a repeated nucleotide, there’s no real need for a second dimension, and we can use the following recurrence to calculate the “polyA-score” M for each position i:

M_i = max (0, S_i + M_i-1)

To implement this, we first define the list of scores by applying match and mismatch to the list of (nucleotide,quality) pairs. We also define a scanl-based function to calculate a list of cumulative scores:

scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
cumulative = scanl (\a b -> let r = a + b in max 0 r)

The only remaining thing is to identify the maximal value which marks the end of the poly-A tail, and the corresponding 0 value that indicates the start. I wrote a recursive function called ”findmax” for this, but a better programmer will probably be able to do this with a fold.

Including the parts discussed briefly above, the whole thing looks like this:

findPolyA :: Sequence Nuc -> Maybe (Int,Int)
findPolyA (Seq _ d mq) =
let qd = zip (B.unpack d) (maybe (repeat 15) BB.unpack mq)
scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
match x' = let x = fromIntegral x' in log (4*(1-1/10**(x/10)))
mismatch x' = let x = fromIntegral x' in log 4 - log 3 - x/10*log 10
cumulative = scanl (\a b -> let r = a + b in max 0 r) 0
(zi,mi,maxscore) = findmax $ cumulative scores
in if maxscore > 12 then Just (zi+1,mi) else Nothing  -- arbitrary constant alert!

findmax :: [Double] -> (Int,Int,Double)
findmax = go 0 (0,0,0) . zip [0..]
where go _ cm [] = cm
go _ cm ((i,0):rest) = go i cm rest
go last_z (cmz,cmi,cmx) ((i,x):rest) = if x > cmx then go last_z (last_z,i,x) rest
else go last_z
(cmz,cmi,cmx) rest

Availability

This method is implemented in a simple tool called “trimpolya” (darcs repo), and also in the more general “dephd” (darcs, hackage) sequence analysis package.

Filtering out empty sequences.

Phred often produces zero-length sequences, and this confuses other programs.  While BLAST will just output a warning, SeqClean — or to be precise, cln2qual — will break down.   (My own code using the Bioinformatics library treats all sequences the same regardless of length, so zero-length sequences are perfectly okay). Anyway, you can now use dephd -z to eliminate them from the output.

Sequence Clipping

Sequence trimming or clipping is often necessary to remove contamination like vector sequence, or simply low quality sequence parts.  Typically, both of these occur at the ends of the sequences.  Many programs (including dephd, but also phred, lucy, seqclean and others) add trimming information to the sequence header.  Dephd is now able to act on this information and clip the sequences.  The trimming information is now obsolete as the coordinates have changed, so they are replaced with the clipping coordinates.   Dephd also provides its own quality assessment, and with the -q option, sequence ends where the sliding windown average quality is below 15 will be clipped.  This is pretty heavy-handed, but it seems I get better EST clustering with this enabled.

Old features

Of course we retain the old features: reads PHD and Fasta/Qual files, mask (to lower case/N but don’t clip) by quality, generate quality plots, outputs Fasta/Qual, and ranking sequences by quality.

Edit: In the latest release, there’s now also a fix for a problem with drawing quality graphs with gnuplot.  It turns out that my shiny new Ubuntu ships with gnuplot 4.2, but your crappy old distribution ships with an older version, and that there are some incompatibilities in the input formats.  I’ve now reverted this to use old-style format only, so hopefully it should work with gnuplots back to 3.7 or so.   And for those SLS or MCC interrim die-hards out there, I’ll even add an option to dump the gnuplot file itself, so that you can copy it to a floppy and generate the plots on a computer with a modern color display. How’s that for user friendly?

swords s = take (fromIntegral (seqlength s)+1-k) . map (B.take k) . B.tails . B.map toUpper $ seqdata s

Here, seqdata returns a lazy bytestring, which is also what’s hiding behind the B. qualifier.  Basically, this just builds the list of all lenght-k substrings.  This should, in my opinion, stream nicely, and run in constant space and linear time.  What on earth makes this quadratic?  On only some systems?   Time to dissect the systems in question:

Comparing our environments, one obvious difference was that the fast system was built using GHC 6.8.1, while the slow ones were 6.8.2.  So we checked with a 6.8.1 snapshot – no difference, still slow.  We checked the various source repositories involved for some stray patch that somehow had avoided distribution – nothing turned up. Comparing Cabal setup files didn’t reveal anything obvious, except that the newer Cabal emitted a lot more information.

Somewhere, something had to be different. Could it be bloomfilter not being a good consumer for the generated words?  The bio library messing up FASTA parsing?  Something else entirely?

Looking at the swords function, it looks like a good candidate for deforestation and/or fusion.  Could there be some optimization not being performed in some cases, for some strange reasons?  Since I remember fusion being mentioned occasionally in the context of bytestrings, I checked the libraries again, and found something I’d missed previously:  Bytestring 0.9.1 on the fast system, 0.9.0.1 on the slow ones.  While the announcement didn’t promise more than a few percent better performance, no stone could be left unturned.  And this proved to be it: after upgrading bytestring, and recompiling binary, biolib, and bloomfilter to use it, my laptop was as fast as the server.

I’m not sure exactly what caused this, but apparently there were some issues with bytestring fusion, and fusion is disabled in current bytestring versions. At any rate, it seems the newer version is safer, this is now the minimum requirement in bio.cabal.

@{sequence header}
{sequence data}
+{sequence header}
{quality data}

Note that the sequence header is repeated in there, apparently somebody thought that would be a good idea. The {sequence data} part looks like it does in a Fasta file, except that here it has to be on a single line. The {quality data} is ASCII, each letter representing the quality value 33 lower than it’s ASCII value.  This opens up another possibility of getting it wrong, since the line of quality data can (and will!) start out with ‘+’ or ‘@’ occasionally.

Anyway, the implementation seems to be pretty efficient, I wrote a simple program to count the number of sequences in a file:

module Main where

import Bio.Sequence
import System

main = do
  [f] <- getArgs
  print . length =<< readFastQ f

Testing it on this 440MB file ran in (first cold, then warm cache):

./countFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  1.77s user 0.41s system 53% cpu 4.091 total
./countFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  1.26s user 0.19s system 89% cpu 1.631 total

For comparison, I also tried it with ‘grep’:

grep '^@' ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  5.61s user 0.46s system 98% cpu 6.181 total

..and in addition to being 50% slower, it gives the wrong answer, since the ‘@’ delimiter may occur in the quality data. Another nice thing is that the Haskell program is IO bound, so it would be even faster if I had a better disk. (Note to self: talk to boss about getting an SSD for my laptop).

Update: I did a quick test, comparing it to the old Fasta format.  First, to convert, I replaced the last line in the program above with

readFastQ f >>= writeFasta (f++".fasta")

and compiled it as convertFQ.  Running this:

./convertFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  9.51s user 2.76s system 40% cpu 30.037 total

Then, I made a countFa by changing the last line to:

print . length =<< readFasta f

Running this on the Fasta file generated just now (282MB), I get:

./countFQ ../download/200x36x36-071113_EAS56_0053-s_1_1.fastq  1.26s user 0.19s system 89% cpu 1.631 total

Here, grep takes 2.25 seconds user time (and gives the correct answer), we’re still faster.

The only cloud on the horizon is that there is some disagreement about the format.  For example, somebody thinks quality always should start with a ‘!’ (viz. zero).  Maq developers think it’s okay to drop the repeated sequence name.  Rockefeller thinks the quality data should be text digits, like the Qual format. And the good people at Solexa had to go and slightly alter…not the format itself, but it’s interpretation, using a different formula to calculate the error probabilities from the quality values. So basically, given a file, there’s no way to know whether it uses Solexa-style quality information, or regular, Phred-style quality.  If you get quality values above 60, then maybe you’re interpreting it wrong.  Then again, maybe not. Sigh.

Comparing sequences to find similarity is a common occurrence in bioinformatics.  For instance, one might want to know where a certain gene is located in the chromosome, or which sequence fragments are similar enough to originate from the same gene. To speed up searches, it is common to index sequences in questions as overlapping, substrings (k-tuples, q-grams).  This index seems like an obvious target for Bloom filters — large data, time critical, some false positives anyway — but for some reason, there is almost no such applications that use them. Until now.

Sequence clustering

Sequence clustering is a commonly used technique which is usually based on sequence similarity.  I’ve written one sequence clusterer, xsact, which is based on blocks of exact matches. There are many others, and another example is d2cluster, which is based on occurence of fixed length words — which is right up Bloom alley, right?

So a straightforward way to build a Bloom filter based sequence clusterer is to represent each cluster as a set of words — stored as a Bloom filter.  Now, adding a new sequence to the clusters is a simple matter of extracting the words from the sequence, identifying the cluster(s) containing a sufficient number of these words, and adding the remaining words to that cluster (or the union of the clusters, in the case multiple clusters match).

The interesting thing about this approach is that the whole thing becomes O(kn), for k clusters and data size n.  I think all other clustering algorithms are based on sequence pairs, which makes them O(n²) — in the worst case, you need to check all pairs. (However, a straightforward similarity-based clustering will have worst-case behavior when no sequence math each other, while suffix-based methods like xsact will have worst case when all sequences match — so perhaps there is room for a better middle ground?)

Anyway — while the above looks promising, there is one snag: EST sequences can occur as a copy of the gene, or due various properties of the manufacturing process, the gene’s reverse complement.  Thus, we need to be able to compare sequences in both directions simultaneously. This could be achieved with a slightly creative hashing function, but to keep things simple, we’ll stick with the ossified mental sweat in the provided implementation.

Index and search

A somewhat related area is indexing and search.  Let’s say you have a bunch of DNA sequences (of for each chromosome, perhaps), and a set of ESTs, which you’ll remember are gene fragments in an unpredictable mix of forward and reverse-complement directions.  Here’s the plan:

1. index by building a Bloom filter containing the q-tuples for each chromosome
2. for each EST, look up each q-tuple (first forward, then rev.comp.) against the filters, and assign it to the chromosome containing the most q-tuples
3. align each EST against the designated chromosome using traditional methods

As far as I can tell at this point, for word size q, c chromosomes of length m and e ESTs of lenght n, this should run in something like  O(qn) + O(qenc) + O(emn), compared to just aligning directly at O(ecmn). Note that mn is the big factor here, so on a large scale, we’re reducing the total work by a factor of c.  (Of course, in real life, nm would be too large, and you’d use a heuristic, subquadratic alignment step, but this is for illustration purposes.  Try to be a bit generous, will you?)

Further plans

That concludes the current plan, but there are certainly improvements that can be made, two obvious ones are

• hash function that hashes a word and its reverse complement to the same value
• break long sequences into partially (1/3) overlapping regions to speed things up even more, as well as give more accurate placements

Another thing that struck me is that in my annotation tool, I could probably use this as a faster way to store the set of matching proteins before extracting GO terms.  Currently, the performance is limited by XML parsing, so it’s probably not worth the bother at the moment.

More is likely to come up, but now it’s time to implement something!

As in any program, there are many occasions where you want to effect some particular change during some part of the program.  The archetypical example is allocation of local variables. After allocation, the variables are then available to the program until they run out of scope, they then get deallocated automatically.  The technique can be generalized beyond this.  For instance, you (or rather I) may want to set a $STAGE variable that indicates the current processing stage, and which should be unset when the stage has finished executing.  Or you may want to run some processing in a different directory, in which case you really want to remember to return to the previous directory when you finish.  The purpose of bracketing is to wrap a section of code with an initial part to be run in advance, and a final part to be run afterwards.

Some examples

When I toyed with PHP ages ago, I’d often find myself building a section of a page by a) generating a header with some opening tags, b) generating some content, and c) generating a footer with some closing tags.  And when the exact contents of these pieces depend on various factors, and such sections would nest in complicated ways, it should not be a surprise that getting a) and c) to correspond exactly could be a challenge.  With absolutely no enforcement by the language (which may have improved in later years, I wouldn’t know), this was very fragile.

For HTML the solution is simple: instead of generating open and close tags separately, have a function take the tag and its contents, and output the contents appropriately surrounded by open and close tags.  If you build the whole document this way, you guarantee that each open tag will have a matching close tag, and that tags will be properly nested.  Another example is Common Lisps with-open-file macro.

In Haskell, there’s bracket (in Control.Exception), which in addition to being vastly more general also is a regular function, thus once again proving Haskell’s vast technical and moral superiority over the more pedestrian languages….but I digress.  I suspect the rather original name is supposed to allude to how brackets (as in those banana-shaped glyphs surrounding this text) consist of an opening bracket, some contents, and a closing bracket.  Anyway, we like Haskell, so we use Haskell terminology.

As a final note, observe that brackets are similar to stack allocation, manual resource management is similar to manual memory management, while using finalizers/destructors is similar to garbage collection.  (It’s tempting to add “pick any two”.)

Generalized bracket

While implementing the EST pipeline, I found myself needing, and implementing, several bracket-like functions (including the two previously mentioned: setting and unsetting a variable, and running a subcomputation in a separate directory). Thus, the question that poses itself is: Is it possible to do this in a more general way, akin to Haskell’s bracket?  Here’s my currently best attempt:

bracket(){
    CLOSE=$2
    eval $1
    shift; shift
    eval $*
    eval $CLOSE
}

First we store the second parameter, which will be the “close” action in a variable.  We then execute the first parameter (the “open” action), using eval so that variables can be set etc.  We then skip the two first arguments, using shift twice, then eval the main action, and finally, eval the “close” action.

This allows stuff like:

bracket "mkdir mytmpdir && pushd mytmpdir" "popd" mkfiles

where the mkfiles function is run inside a temporary directory, and where execution resumes in the original directory after completion. Another example is

bracket "echo Entering first stage; STAGE=first" "STAGE=none" echo Current stage is '$STAGE'

Note the careful quoting of the variable with single quotes, we don’t want $STAGE to evaluated before it is set in the bracket function. In other words, the single quotes lets us pass the literal string $STAGE, sort of pass by name semantics instead of the default pass by value.

Perfection is the enemy…

If you are an experienced shell programmer, you may at this point have formed an opinion that I am not. And you’d be right, of course, but even I can see that there’s (at least) one obvious bug: we define a global variable named CLOSE.  Not only does this have the potential to clash with an existing variables, it also prevents recursive calls to bracket. Possibly, we should generate variable names, or have $CLOSE be a stack, or something…But hey, Mr. Know-it-all, if you’re so damn good, why not post a comment explaining how it’s really done?

In other words: feedback and comments are most welcome (although for spam-prevention you may have to register first)!

The EST sequencing process

1. mRNA is extracted from a sample, and a primer-linker is attached. Since mRNAs contain a poly-A tail (a string of As at the end), the primer contains a string of Ts that will hybridize to the poly-A tail, and the short double-stranded end is ready to initiate the duplication to cDNA.
2. Reverse transcriptase makes a reverse-complemented cDNA copy of the mRNA. This process will go on for some length, but not necessarily to the end of the mRNA transcript – the resulting cDNA strand will thus often start some distance from the beginning of the mRNA.
3. the RNA is removed, and the cDNA methylated to make it more stable.
4. polymerase duplicates the cDNA strand, making it double stranded.
5. adapters are attached to each end of the double-stranded DNA segment. These are short sequences that are designed to match one of the vector’s (below) cloning sites, fixing the DNA segment as an insert in the vector.
6. the primer-linker contains a target for a restriction enzyme. This enzyme now chops off part of the primer-linker, discarding the adapter at that end, and revealing the sequence that will ligate (bind) to the other cloning site in the vector.
7. The vector is a short, circular DNA sequence (plasmid or phagemid) that will work like a small chromosome when inserted into a suitable host, typically a bacteria like E. coli. Our DNA segment is now ligated to the vector at both ends, and the whole thing is duplicated by growing a bacteria colony.

At this point, there has been a number of opportunities for mishaps. The primer-linker can ligate to something else than a poly-A tail (1), reverse transcription can be – and often is – cut short (2), cDNA can be incompletely methylated, making it a possible target for restriction enzymes that will chop it up (3), adapters can ligate with each other, forming chimeric sequences (5), the primer-linker can avoid being cut, leading to it being retained in the sequence (6), and the bacteria can assimilate two vectors, or none, the insert can end up in the wrong direction, possibly a vector can acquire an insert from bacterial DNA (7), and there’s probably more.

Sequences and consequences

What this means, is that in addition to actual mRNA sequence, we will usually get sequences containing vector, and quite often primer, linker, adapter, or E.coli sequence as well. This needs to be identified, and discarded (masked or trimmed off) before further analysis.

I’ve been experimenting with the following programs, SeqClean by TIGR and Lucy, by Chou and Holmes (2001).

SeqClean

Pros: Masks against a database of vectors and a database of contaminants. Masks low complexity.

Cons: Does not take into account sequence quality. Inherent model expects vector in the first and second third of the sequence, sometimes failing to trim vector, and sometimes trimming valid sequence. Trims sequences instead of marking them, making it harder to compare results.

Lucy

Pros: Uses knowledge of vector splice sites, giving it the opportunity for more accurate masking and differentiation between vector and spurious matches. Takes quality into account.

Cons: Very picky about vector input, and in particular splice site sequences must be in the correct orientation and sequence. A bug makes it fail on long Fasta headers (although I wrote a fix for that). Can only deal with a single vector at a time, meaning you need to know for each sequence the vector used.

I’ve noticed that even after masking with either program, some vector remains. And, it turns out, I’m not the only one, my buddies at CBU has the had the same problem, and there’s also a paper (Chen et al, 2006). Unfortunately, the remedy suggested there didn’t work for our sequences, and in fact, only seems to help for one particular vector type – that we don’t use.

Currently, I use a rather ugly hack, using RBR to mask, first against E.coli contamination with high stringency, and then against vector, using a more lenient settings. Finally, I use SeqClean to chop off the masked-out bits, as RBR only Xes out the offending parts, but doesn’t trim, and TGICL doesn’t deal gracefully with masked but untrimmed sequences…but that’s another bug.

The plan?

It should be relatively easy to write a tool that combines the strengths of Lucy and SeqClean, while avoiding the weaknesses. A useful tool could use BLAST to find all vectors, know about vector-adapter-linker-primer combinations, and identify them. It could pinpoint sequences that do not follow the typical vector-linker-sequence-poly-A-linker-vector pattern, and perhaps identify chimeric sequences and other artifacts.

It could do a lot of things. Unfortunately, at the moment the margin of my calendar is too small to fit it in…