Skip to main content

literally who hurt genomics to make you all encode one specific kind of number as the ASCII characters from ! to ~ as an integer input to some logarithm function, but then others of you changed the function but kept encoding it as a single ASCII character ranging from -5 to 62 (???), and then later they decided that -5 to 62 was silly and so they changed that to 0-62 and throwing away half the original range for no reason, except actually it's 0-40 by convention.

did anyone consider "encoding it as a number"

https://doi.org/10.1093/nar/gkp1137

#DataStandards #FileFormats #Genomics

PHRED SCORES AND THE QUAL FORMAT The PHRED software reads DNA sequencing trace files, calls bases and assigns a quality value to each base called (9,10). This introduced the PHRED quality score of a base call, defined in terms of the estimated probability of error: Q_{PHRED} = -10 * log10(P_e)
ANGER FASTQ FORMAT The FASTQ format was invented at the turn of the century at the Wellcome Trust Sanger Institute by Jim Mullikin, gradually disseminated, but never formally documented (Antony V. Cox, Sanger Institute, personal communication 2009). The closest thing to an official description from Sanger can be found on the MAQ/BWA website (13,14), but even this is incomplete. Full details of the file format, describing the read title, sequence and quality scores are given later. Here, we concentrate on how the quality scores were encoded into a simple string. Early FASTQ files were used for Sanger capillary sequencing, and it was natural to use PHRED quality scores (described above). Storing PHRED scores as single characters (or bytes) gave a simple but reasonably space efficient encoding. In order that the file be human readable and easily edited, this restricted the choices to the ASCII printable characters 32–126 (decimal), and since ASCII 32 is the space character, Sanger FASTQ files use ASCII 33–126 to encode PHRED qualities from 0 to 93 (i.e. PHRED scores with an ASCII offset of 33).
An important consequence of these equations is for high values the two scores are asymptotically equal, and after rounding to the nearest integer scores of ⋛10 are interchangeable (Figure 1). However, Solexa scores go down to −5 (approximating a random read error probability of 0.75). The Sanger offset of 33 can, therefore, no longer be used. Rather, an offset of 64 was chosen, meaning ASCII 59 to 126 can be used, allowing Solexa scores from −5 to 62 inclusive.
ILLUMINA 1.3+ FASTQ FORMAT Although Illumina initially continued to use the Solexa FASTQ variant, from Genome Analyzer Pipeline version 1.3 onwards (16), PHRED quality scores rather than Solexa scores were used. However, rather than adopt the original Sanger format, Illumina introduced a third incompatible FASTQ variant designed to be interchangeable with their earlier ‘Solexa FASTQ’ files for good quality reads. The Illumina 1.3+ FASTQ variant encodes PHRED scores with an ASCII offset of 64, and so can hold PHRED scores from 0 to 62 (ASCII 64–126), although currently raw Illumina data quality scores are only expected in the range 0–40. The OBF projects refer to this variant as the Illumina 1.3+ FASTQ format, under the format name ‘fastq-illumina’ (Table 1).
#genomics #datastandards #fileformats
in reply to jonny (good kind)

Dr. Angus Andrea Grieve-Smith
Dr. Angus Andrea Grieve-Smith
• •
There's no memory advantage to encoding it as a character vs a number, is there?
in reply to Dr. Angus Andrea Grieve-Smith

Eli Roberson (he/him)
Eli Roberson (he/him)
• •

@grvsmth filesize advantage. Quality as a number would require minimum two characters (single digit plus delimiter) and maximum of 3 characters (two numbers plus delimiter). Fixed width 0 led would still require two characters. ASCII coding makes it one character per quality score. Minimizes cold file sizes.

Now the multiple FASTQ quality ASCII coding specs are an artifact of Illumina choices. Sanger phred scale should have been the standard all along.

@Dr. Angus Andrea Grieve-Smith

Lo, thar be cookies on this site to keep track of your login. By clicking 'okay', you are CONSENTING to this.

⇧