bio-0.4.6: A bioinformatics library

The library provides functions and data structures to work with various kinds of bioinformatics data.  It has its own page here.  New features include support for BLAT’s PSL format, fixes to SFF and limiting the binary dependency to <0.5 to maintain necessary laziness.

flowsim-0.2.6: A simulator for 454 pyrosequencing data

Flowsim is new on Hackage, but also has its own page. The current version is split into two parts, clonesim which simulates fragmenting of a genome, and flowsim proper, which simulates flowgrams from the sequences, and does base- and quality calling in order to output SFF files, similar to those generated from the real thing.  Flowsim will be presented at ECCB’10 in Ghent at the end of September.

flower-0.3: A set of tools to work with pyrosequencing data

This is a – dare I say bouquet?  of various tools for working on and with 454 data.  Flower itself extracts various information from SFF files, and is documented here, but the package includes other tools as well, namely: the quite renamable flowt which attempts to remove duplicate clones, an artifact that appears to occur frequently in these data; the more appropriately named frename which relabels reads uniquely (useful if downstream software requires it), and frecover which recovers corrupted SFF files, which happened to us once, but so far hasn’t happened again.

They should all now be a cabal install away, so if you use these, please let me know how you fare, and whether you find them useful!  I also try to provide Linux binaries for recent versions, and hope to provide proper Linux distribution packages (you know, .debs and .rpms) in the future. If you want to help out, there’s also the darcs repositories for biolib, flower, and flowsim.

% grep libghc6 by_inst.ubuntu.txt | grep -- '-dev ' | head -20
9063  libghc6-mtl-dev                 8399   175  7991   228     5 (Ian Lynagh)
11976 libghc6-glib-dev                4305    25  4184    94     2 (Liyang Hu)
13542 libghc6-x11-dev                 3153    73  2980    99     1 (Ian Lynagh)
14725 libghc6-cairo-dev               2600    17  2535    47     1 (Liyang Hu)
14959 libghc6-xmonad-dev              2502    68  2350    83     1 (Joachim Breitner)
15290 libghc6-gtk-dev                 2367    17  2307    42     1 (Liyang Hu)
15486 libghc6-xmonad-contrib-dev      2288    68  2143    77     0 (Joachim Breitner)
16528 libghc6-quickcheck-dev          1947     2  1938     6     1 (Ian Lynagh)
16829 libghc6-utf8-string-dev         1859    13  1829    16     1 (Unknown)
16953 libghc6-opengl-dev              1825    21  1765    38     1 (Ian Lynagh)
17782 libghc6-network-dev             1622    25  1553    43     1 (Ian Lynagh)
18393 libghc6-gstreamer-dev           1505     9  1447    48     1 (Unknown)
19606 libghc6-xhtml-dev               1276    25  1194    56     1 (Ian Lynagh)
20109 libghc6-x11-xft-dev             1191    69  1042    80     0 (Unknown)
20159 libghc6-parsec-dev              1183     3  1177     1     2 (Ian Lynagh)
20900 libghc6-gtkglext-dev            1079     5  1055    19     0 (Liyang Hu)
21727 libghc6-parsec2-dev              967    41   858    68     0 (Unknown)
22701 libghc6-regex-base-dev           847    21   799    27     0 (Arjan Oosting)
22955 libghc6-pcre-light-dev           819     8   774    37     0 (Unknown)
23107 libghc6-hunit-dev                803    15   767    20     1 (Ian Lynagh)

% grep libghc6 by_inst.debian.txt | grep -- '-dev ' | head -20
6207  libghc6-mtl-dev                 1058   223   649   185     1 (Debian Haskell Group)
7643  libghc6-x11-dev                  677   144   435    98     0 (Debian Haskell Group)
7949  libghc6-xmonad-dev               620   134   392    94     0 (Joachim Breitner)
8414  libghc6-xmonad-contrib-dev       548   125   329    94     0 (Joachim Breitner)
10664 libghc6-x11-xft-dev              349   122   197    30     0 (Debian Haskell Group)
11352 libghc6-network-dev              310    89   181    39     1 (Debian Haskell Group)
11353 libghc6-quickcheck-dev           310     9   287    13     1 (Ian Lynagh)
12947 libghc6-regex-base-dev           238    72   131    34     1 (Debian Haskell Group)
13182 libghc6-parsec2-dev              229    82   101    46     0 (Debian Haskell Group)
13376 libghc6-opengl-dev               222    70   125    27     0 (Debian Haskell Group)
13420 libghc6-regex-posix-dev          220    66   119    34     1 (Debian Haskell Group)
13731 libghc6-hunit-dev                208    74   106    27     1 (Debian Haskell Group)
13788 libghc6-xhtml-dev                206    64   110    32     0 (Debian Haskell Group)
13885 libghc6-regex-compat-dev         203    61   109    32     1 (Debian Haskell Group)
14128 libghc6-stm-dev                  195    71   104    19     1 (Debian Haskell Group)
14665 libghc6-http-dev                 178    60    91    26     1 (Debian Haskell Group)
14707 libghc6-parallel-dev             177    63    81    32     1 (Debian Haskell Group)
14864 libghc6-glut-dev                 173    62    89    22     0 (Debian Haskell Group)
15111 libghc6-html-dev                 166    58    91    16     1 (Debian Haskell Group)
15143 libghc6-zlib-dev                 165    57    80    27     1 (Debian Haskell Group)

Some observations:

• MTL is by far the most popular library, but it’s only number nine on Don’s list
• GHC (not shown) has about twice the registrations of MTL, in both distributions
• but its 17K registered users on Ubuntu is only 1% of the reports, and 2K registered on Debian is about 2%
• the most popular libraries after that are graphics-related (X11, xmonad, and Gtk, Cairo and OpenGL on Ubuntu)

Static binaries relying on iconv

Symptoms

I often build static binaries (you know, ghc --optl-static --optl-pthread) when I need to run some program on a system that is different from my build system.  Somehow, this seems easier than installing a full build setup.  When i recently tried to run such an executable on a different system, the system responded with mkTextEncoding: invalid argument.  Googling didn’t exactly help, except pointing me to something iconv-related as the culprit.

Diagnosis

It turns out that Haskell programs built with GHC now (version 6.12?) rely on iconv to …well, do some Unicode stuff.  Static linking bundles iconv in the binary, but apparently iconv goes off and reads some dynamic libraries anyway, embedding their paths (as they are on the build system) inside the executable.

Needless to say, this breaks when the target system decides to put these library bits elsewhere.  Specifically, Ubuntu puts these files in
/usr/lib/gconv/, while Red Hat puts 32-bit versions in that directory, and 64-bit versions in /usr/lib64/gconv/.  A 64-bit binary built on a Ubuntu systems thus tries to load 32-bit library code, and I guess we’re lucky we even get an error message.

Remedy

Far be it from me to question the wisdom of embedding paths to local library code in static executables; instead, let me just commend the developers for providing the GCONV_PATH environment variable, which, when set to point to the lib64 directory, made it possible to run my executable.

Update: Apparently, the exact error produced can vary, and I recently got openFile: invalid argument (Invalid argument) instead.  Using strace it was clear that the executable was loading the wrong library again, and setting GCONV_PATH solved the problem.

Laziness change in binary

Symptoms

I have written a small program to analyze and extract information from 454 sequencing file.  In order to efficiently process these files, which can be fairly large, I use the excellent binary library to decode SFF files at disk speeds.  On one occasion, this program started to use a lot of memory, but with a little help from the community, this got sorted out by some inlining.  Recently, the same thing appeared to happen again, but in spite of all my tampering with the previously offending code, I was unable to build any version without this behavior.

Diagnosis

After some quick testing that showed that Data.Binary.decode was, contrary to earlier behavior, strictly evaluating its input before returning anything.  A quick mail to the binary developers confirmed that this behavior had been changed, as it sped up GHC itself.

Edit: I didn’t find it previously, but dons has a nice writeup of the change.

Remedy

Flower depends on being able to lazily process the list of sequences in its input, and the new version of binary not only caused it use excessive memory, but also slowed it down.  I don’t really have any better solution to this (and my use case is probably not important enough to sacrifice GHC performance for than to require binary version less than 0.5.

flowers

Here’s a better method that identifies poly-A tails by finding an optimal, quality adjusted alignment in linear time.

A quick introduction

Although the definitions of what constitutes a gene vary considerably, we’ll use the term to refer to a region of DNA that get transcribed, that is, copied from DNA into an mRNA molecule, which in turn will be used as a blueprint for assembling a protein. After transcription, the mRNA molecule then undergoes polyadenylation, a process where a string of adenines (the ‘A’ of the nucleotide alphabet) gets appended to an mRNA molecule before it is exported from the nucleus.

Identifying poly-A tails are important for several reasons.

1. It positively identifies the end of the transcript. If you don’t have a poly-A in your sequence, you have no way to know how far the molecule extends beyond the end of the sequence. You can also find alternatively terminated transcripts this way.
2. It positively identifies the end of the transcript. Anything after the poly-A tail is linker or vector sequence, and can safely be trimmed off. Even if, as is often the case, it is too low quality to be recognized by your average vector masking software.
3. It provides useful information about the transcript, as the poly-A tail is important for things like protecting the mRNA from degradation.

Unfortunately utilities often trim poly-A tails by default (e.g. SeqClean), or just ignore it (e.g. BLAST’s low-complexity
filter).

Quality based alignment

When a molecule is sequenced, the analog output from the sequencing machine is stored as a chromatogram. In order to be useful, the sequence is called, that is, translated to a string of letters from the familiar {A,C,G,T} nucleotides alphabet. In addition, the base caller will associate each letter with a quality value. This is derived from an estimate of the probability of the call being incorrect, and for quality value Q, the error probability estimate is

ϵ = 10-Q/10

Traditionally sequence alignment simply aligns the string of characters using a fixed positive score (reward) for aligning similar characters, and fixed negative scores (penalties) for either substituting a different character, opening a gap, or extending a previous gap.

However, taking into account the quality value, we can do better, and instead of fixed scores, we can adjust the scores dynamically according to quality.

Using this method, the penalty for e.g. aligning two different characters will depend on the quality of the characters: high quality means a high penalty, low quality — lower penalty (since there’s a greater chance one of them was incorrectly called).

Scoring of alignments

When calculating the score of an alignment, we really want to answer the question: how likely is this sequence to be a real poly-A sequence, as opposed to just a random string? In other words, we are comparing our sequence against two models: the poly-A model, and the background model. Our score will use the ratio of probabilities of the string being produced by the two models.

For the poly-A model, only As are allowed, so the probability of a character occurring is 1 for As and 0 for the others. For the background model, we’ll just take a uniform distribution of nucleotides, each getting a probability of 0.25.

Using this scheme, the score for a string s is simply 1/0.25 = 4 for each A, and 0/0.25 = 0 for all others. We usually work with the logarithm of these numbers to make them more manageable.

The optimal alignment is then simply the longest run of As, since as soon as you multiply with a zero (or add -infinity, if you use log-scores), you lose the whole score.

Adding quality to the mix

Of course, the actual sequence isn’t perfect, and even the poly-A tail is likely to contain the odd G, C, or T. To determine exactly how likely is where the quality value enters the picture. Using the formula above, we can calculate the error estimate and decide what the penalty for a mismatch and reward for a match should be.

For the poly-A model, the probability for a match (that is, an actual ‘A’ in the sequence) is 1-ϵ, the probability of a mismatch (a non-A) is ϵ/3 (since only one of the three possible substititutions is an A, and for simplicity, we give them equal probability). Using the formula for  ϵ as a function of Q (and hopefully not introducing any errors), I get the scores to be:

match q = log (4*(1-1/10**(q/10)))
mismatch q = log 4 - log 3 - q/10*log 10

Now, we can use this to do a standard Smith-Waterman alignment, calculating a dynamic programming matrix, and searching for an optimal local alignment.

However, since we’re aligning against a repeated nucleotide, there’s no real need for a second dimension, and we can use the following recurrence to calculate the “polyA-score” M for each position i:

M_i = max (0, S_i + M_i-1)

To implement this, we first define the list of scores by applying match and mismatch to the list of (nucleotide,quality) pairs. We also define a scanl-based function to calculate a list of cumulative scores:

scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
cumulative = scanl (\a b -> let r = a + b in max 0 r)

The only remaining thing is to identify the maximal value which marks the end of the poly-A tail, and the corresponding 0 value that indicates the start. I wrote a recursive function called ”findmax” for this, but a better programmer will probably be able to do this with a fold.

Including the parts discussed briefly above, the whole thing looks like this:

findPolyA :: Sequence Nuc -> Maybe (Int,Int)
findPolyA (Seq _ d mq) =
let qd = zip (B.unpack d) (maybe (repeat 15) BB.unpack mq)
scores = map (\(c,q) -> if toUpper c=='A' then match q else mismatch q) qd
match x' = let x = fromIntegral x' in log (4*(1-1/10**(x/10)))
mismatch x' = let x = fromIntegral x' in log 4 - log 3 - x/10*log 10
cumulative = scanl (\a b -> let r = a + b in max 0 r) 0
(zi,mi,maxscore) = findmax $ cumulative scores
in if maxscore > 12 then Just (zi+1,mi) else Nothing  -- arbitrary constant alert!

findmax :: [Double] -> (Int,Int,Double)
findmax = go 0 (0,0,0) . zip [0..]
where go _ cm [] = cm
go _ cm ((i,0):rest) = go i cm rest
go last_z (cmz,cmi,cmx) ((i,x):rest) = if x > cmx then go last_z (last_z,i,x) rest
else go last_z
(cmz,cmi,cmx) rest

Availability

This method is implemented in a simple tool called “trimpolya” (darcs repo), and also in the more general “dephd” (darcs, hackage) sequence analysis package.

Installing biolib

First I need to install the bioinformatics library.  I’m about to release 0.4.1, but this is also a good opportunity to check that everything works with 0.4 (which is what you’ll find on Hackage), so let’s do that first.  Using darcs, I pull the repo up to the 0.4 tag (but you can of course get the tarball from Hackage):

% ./Setup.hs configure
Configuring bio-0.4...
Setup.hs: At least the following dependencies are missing:
QuickCheck <2, binary -any


(Side note: you may notice that I run Setup.hs directly, as opposed to using runhaskell. I prefer it this way, but you may have to do a chmod +x Setup.hs if you downloaded this from the darcs repository or similar.)

Since we want to use the system libraries as far as possible, these libraries are just an apt-get away:

sudo apt-get install libghc6-quickcheck1\*
sudo apt-get install libghc6-binary-\*


Now, let’s try again:

% ./Setup.hs configure
Configuring bio-0.4...
% ./Setup.hs build
Preprocessing library bio-0.4...
Building bio-0.4...
Binary: Int64 truncated to fit in 32 bit Int
ghc: panic! (the 'impossible' happened)
(GHC version 6.10.4 for i386-unknown-linux):
Prelude.chr: bad argument

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

Okay, this was not what was supposed to happen.  As always, dropping to #haskell on IRC is the first thing to do, and sure enough:

<sereven> ketil: that's also shown up for xmonad users when .hi and .o files weren't cleaned
between rebuilds mixing different versions. usually between ghc updates IIRC.


Let’s try to get rid of old cruft lying about, polluting directories:

./Setup.hs clean && ./Setup.hs configure && ./Setup.hs build

Sure enough, this time it worked. For good measure, we’ll run the unit tests:

make test

After a zillion tests, we notice that everything is go, great!

Applications

Next, it is time to go through the list of bioinformatics applications.  Since my working directory is a mess of branches and versions, we’ll just go over the published applications and versions on Hackage.

xsact is an application to do sequence (in particular EST sequence) clustering.  It predates and thues doesn’t actually use the bioinformatics library, but we’ll check it anway.  So we try the familiar command line:

./Setup.hs clean && ./Setup.hs configure && ./Setup.hs build

And things compile.  However, the version on Hackage is outdated, so we’ll upload a new version, 1.7.1.  One test case still fails, but I can’t imagine anybody is using it to generate Newick-formatted trees — I am certainly not — and since there are many equally correct outputs (including tree rotations and rounding modes), output is likely correct anyway.  Holler if you need it!

rbr is an application to mask repeats in sequence data.  Normally, this is done using a library of known repeats, but this application tries to do it using statistics, making the — I think justifiable — assumption that repeats are going to be more common than non-repeats. The version on Hackage is old, and only works with the library prior to 0.4, so again this is a good time to push the latest changes out in the limelight.  Compiling this works great, by the way.

cluster_tools is a package that contains a bunch of binaries, useful for working with the results of sequence clustering, including extracting various information from ACE files.  This uses another library, called simpleargs, that simplifies command line argument parsing for simple cases.  Again, the Hackage version is for bio<0.4, so a new version will be pushed.  At the same time, we make a mental note to push version 0.2 of simpleargs to Hackage as well, instead of keeping age-old modifications buried forever.

dephd is my Swiss-army-knife of sequence analysis, and lets you do various things like converting between formats, plotting and trimming by quality.  This is a more live project than most of the others (I’m currently working on improved quality trimming and automatic generation of files for submission to GenBank), but the currently available version also compiles without incident.

estreps is a couple of programs I needed for repeat analysis, perhaps not tremendously interesting, but at least rselect, which lets you select randomized subsets from Fasta files, might be of interest to some? We try the usual invocation to compile, and get:

src/Unigene.hs:24:23:
Couldn't match expected type `a' against inferred type `Unknown'
`a' is a rigid type variable bound by
the type signature for `clusters' at src/Unigene.hs:22:41


This error arises due to the introduction of phantom types for identifying sequences introduced in bio 0.4. Unfortunately, this version of estreps contains some adaption to this model, so it won’t compile against older versions either. So it looks like yet another sdist for Hackage. Look for version 0.3.1.

flower is a utility for extracting information from SFF files (containing sequences from Roche’s 454 machines).  Although a new version is around the corner, the old 0.2 just works.

xml2x is a utility for converting BLAST results in XML format into CSVs, that somehow is more compatible with biologists.  Trying to compile it fails, with the following error:

src/Xml2X.hs:152:49:
Couldn't match expected type `[b]'
against inferred type `Maybe Bio.Sequence.KEGG.KO'
In the first argument of `concatMap', namely `(flip M.lookup ks)'
In the first argument of `($)', namely
`concatMap (flip M.lookup ks)'
In the second argument of `($)', namely
`concatMap (flip M.lookup ks) $ map chop $ map subject fs'


It turns out that somewhere along the way, the lookup function from Data.Map was de-generalized from working on arbitrary monads to just returning a Maybe. I was using this to return a list, using the empty list to signal an unsuccessful lookup. This is easily remedied, but that means yet another sdist for Hackage.

korfu is a utility for identifying open reading frames in sequence data.  It hasn’t yet been ported to version 0.4 of the library, but works if you install 0.3.5.  I updated this too, since it didn’t have a category.  Now it too resides in the bioinformatics section.

Summing up

In retrospect, it seems like giving old code a thorough spring cleaning once in a while. Although nothing really critical or difficult happened, a good number of small annoyances were discovered, and a bunch of new sdists are now ready to be uploaded to Hackage. Next will be converting all this into debian packages.

The important question is of course, how do we avoid this in the future? During development, it is important to be able to modify libraries and appliations, but installing a new version of the biolib, say, overwrites the old one, and suddenly I’m compiling and testing everything against a different library than Joe Random Hackage User is going to find. I have some thoughts on how to avoid this, but if you have a method that works nicely, I’m all ears.

•  My biohaskell slides
• slides’ LaTeX source, using beamer.cls
• Lazy PSSM JFP-paper

The original version

I’ve taken the liberty of cleaning things up a bit, and removing some context that I sure hope isn’t necessery.  Bascially, the task is to take a list of ByteString input lines consisting of whitespace separated decimal integers, and build a ByteString consisting of single byte quality values corresponding to those integers.

Below is how the naive quality parsing function might look, unpacking each word to String in order to use read.  Note that B is lazy ByteString.Char8, BB is lazy ByteString, that is, the Word8 version.

BB.pack $ map (read . B.unpack) $ B.words $ B.unlines ls

We don’t expect this to do terribly well, and I guess it’s no surprise when this parses my test file of about 10MB in 24 seconds.

Improved versions

The key to improved performance is first and foremost to avoid the unpacking and parsing of strings.  Thankfully, the ByteString library provides a readInt function.

BB.pack [lookup x | x <- concatMap B.words ls]
    where
    lookup x = case B.readInt x of Just (v,_) -> fromIntegral v
                                   Nothing -> error "Unparsable qual value"

This isn’t a lot more complicated than our initial attempt, but the speed increas is considerable: slightly less than 2 seconds for the test file, more than a tenfold improvement.  The ByteString implementation will share the storage of the separate words with the original strings, but since readInt gives us back the rest of the string in addition to the parsed integer, we might as well make use of it:

BB.pack $ readInts $ B.unlines ls
    where readInts xs = case B.readInt xs of 

                          Just (i,rest) -> fromIntegral i : readInts (B.dropWhile isSpace rest)
                          Nothing -> []

This turns out to be a bit faster, time is now 1.6 seconds.  Another 20% shaved off.

Final version

We’re really only interested in Word8 values, since quality values always are small, and since that’s what gets encoded in the result anyway.  The previous versions takes a detour by reading Ints and using fromIntegral to convert them to the desired size.  It bears noting that there is no error checking involved, fromIntegral will happily and silently truncate any number beyond its target range.  So lets do things explicitly, using Word8s throughout the computation:

BB.pack $ go 0 ls
    where
    isDigit x = x <= 58 && x >= 48
    go i (s:ss) = case BB.uncons s of 

                    Just (c,rs) -> if isDigit c then go (c - 48 + 10*i) (rs:ss)
                                   else let rs' = BB.dropWhile (not . isDigit) rs
                                        in if BB.null rs' then i : go 0 ss else i : go 0 (rs':ss)
                    Nothing -> i : go 0 ss
    go _ [] = []

This is the fastest one so far, clocking in at 0.94 seconds, over 40% faster than the best readInt version, and about 25 times faster than the naive version.  Still, 10MB/s is well below the average hard disk.

So is there more room for improvement?  The most obvious wart to me is the rather artificial splitting into lines.  This is mostly an artifact of some early design desicions, and it should be possible to eliminate the splitting earlier on and saving even more time by simplify this function quite a bit.

If you spot anything else, or have suggestions, I (and my darcs repo) am all ears.

Edit: Since some people have asked, I’ve wrapped up a simple test program along with some test files at http://malde.org/~ketil/biohaskell/qualparsetest.  This is a simplified version, if you want to be really helpful, you could always look at Bio.Sequence.Fasta in the Bioinformatics library and see if you can speed up e.g. dephd -i input.fasta input.qual -F /dev/null.

The Haskell bioinformatics library has for some time now supported reading and writing the SFF format, which is used by the oldest (previousest?) of the next generation technologies, namely Roche’s 454 sequencing.  Once the library functionality is in place, it is easy to develop small tools for doing the various chores.   After spending some time in anticipation of the hordes of programmers no doubt rushing to exploit the monumental effort put I down in  the library, I’ve instead written a few programs myself, including tools information/statistics extraction (flower),  extracting sequences by various criteria (fselect), simulating sequencing (pyrosim), and repairing broken SFF files (frepair).  This is their story.

The bioinformatics library includes functions and data structures to read and write and extract data from SFF files.  It provides reasonable performance (meaning that in most cases, my disk is the limiting factor).  It is also a clean room implementation, built from official documentation, but not based on Roche’s (or anybody else, for that matter) code.  It comes with an LGPL license, which I hope will make it useful while encouraging back contributions.

Flower

This program can extract  various information from SFF files.  To some extent, it overlaps with ’sffinfo’ and ’sfffile’ from 454, but in addition to generating Fasta sequences (optionally with quality), it can also extract directly to the more compact FastQ format.  It also can output a (huge) table of flow values, the histogram of flow values (useful for estimating the flow distributions, and thus probabilities for the base calls), or a one-line summary of each read that includes some statistics on lenghts and quality. Here’s the usage info:

flower: Usage: flower -[f|q|r|R] <file.sff> [<file2.sff> ..]
  -r  output reads in Fasta format
  -R  output reads in Fasta format with associated .qual
      (generates files instead of writing to <stdout>)
  -q  output in FastQ format
  -f  output the flowgram in tabular format
  -h  output a histogram table of flow values
  -s  output a summary of each read

FSelect

FSelect takes an SFF file and produces a new SFF file containing a subset of the sequences, using the same statistics as Flower can output.  It has a small expression language built in, so that you can build more complex logical queries.  For instance, if you want to extract the sequences with lenght between 300 and 400, and with a K² score greater than 0.7, you could do

fselect "And (Func LT k2 0.7) (And (Func GT len 200) (Func LT len 400))" FL61AHU01.sff

Okay, it’s a bit clunky, but the syntax should be reasonably straightforward: Logical operators are And, Not, and Or, while Func op f v defines a function using op (either LT or GT) to compare f (one of k2, len, tlen, ncount)  to each read.  Output is generated in a file named “selected.sff”.
FSelect can also select random reads, using the select expression "Rand p", where p is the probability for selection.  (I.e. fselect "Rand 0.2" FL61AHU01.sff will select each read with a probability of 0.20, giving you approximately 20% of the reads.  Random selection can not be combined with other criteria at this point, if you want this, you’ll have to run fselect multiple times.

FRecover

We had some issues with broken SFF files, specifically there were block of zero bytes at random places. Both flower and sffinfo just terminate on encountering a broken read, so I implemented a simple utility that attempts to skip the broken block and continue extracting good reads beyond the trouble.

PyroSim

This attempts to simulate pyrosequencing, for now it only does the GS20  generation of 454 sequencing, but the other generations should be easy to add.  The main problem is that the algorithm for quality calling is insufficiently documented.  GS20 has the advantage that quality for a homopolymer is uniquely derived from the flow value (modulo rounding), so reverse engineering it is fairly straightforward.

Anyway, pyrosim takes a ‘generation’  (at the moment, this is only GS20) and a Fasta file as input parameters, picks random points in the Fasta file, and produces the correspondig flowgram, including a suitable perturbation of the values to introduce the expected measure of noise, calls the bases and quality, and outputs an SFF file.

Availability

Flower, FSelect and FRecover are part of the “flower” package, available from the darcs archive at http://malde.org/~ketil/biohaskell/flower

PyroSim is available separately (for now, at least) from the darcs archive at  http://malde.org/~ketil/biohaskell/pyrosim

I try to upload what I consider stable versions to HackageDB, please check the Bioinformatics category there.  Currently, these programs are in a bit of flux, so going with the darcs repo is probably a good idea at this point.

I’d really like to have packages for the most common Linux distributions as well (i.e. .debs and .rpms), but I don’t know the details of how to produce them, and while I’ve made half-hearted attempts in the past, I guess I just don’t really desire it enough.  I’d be happy to see somebody package it up, so if you know how, please go ahead.

Filtering out empty sequences.

Phred often produces zero-length sequences, and this confuses other programs.  While BLAST will just output a warning, SeqClean — or to be precise, cln2qual — will break down.   (My own code using the Bioinformatics library treats all sequences the same regardless of length, so zero-length sequences are perfectly okay). Anyway, you can now use dephd -z to eliminate them from the output.

Sequence Clipping

Sequence trimming or clipping is often necessary to remove contamination like vector sequence, or simply low quality sequence parts.  Typically, both of these occur at the ends of the sequences.  Many programs (including dephd, but also phred, lucy, seqclean and others) add trimming information to the sequence header.  Dephd is now able to act on this information and clip the sequences.  The trimming information is now obsolete as the coordinates have changed, so they are replaced with the clipping coordinates.   Dephd also provides its own quality assessment, and with the -q option, sequence ends where the sliding windown average quality is below 15 will be clipped.  This is pretty heavy-handed, but it seems I get better EST clustering with this enabled.

Old features

Of course we retain the old features: reads PHD and Fasta/Qual files, mask (to lower case/N but don’t clip) by quality, generate quality plots, outputs Fasta/Qual, and ranking sequences by quality.

Edit: In the latest release, there’s now also a fix for a problem with drawing quality graphs with gnuplot.  It turns out that my shiny new Ubuntu ships with gnuplot 4.2, but your crappy old distribution ships with an older version, and that there are some incompatibilities in the input formats.  I’ve now reverted this to use old-style format only, so hopefully it should work with gnuplots back to 3.7 or so.   And for those SLS or MCC interrim die-hards out there, I’ll even add an option to dump the gnuplot file itself, so that you can copy it to a floppy and generate the plots on a computer with a modern color display. How’s that for user friendly?