fastp

A tool designed to provide fast all-in-one preprocessing for FastQ files. This tool is developed in C++ with multithreading supported to afford high performance.

Citation: Shifu Chen. 2023. Ultrafast one-pass FASTQ data preprocessing, quality control, and deduplication using fastp. iMeta 2: e107. https://doi.org/10.1002/imt2.107

• fastp
• features
• simple usage
• examples of report
• get fastp
  • install with Bioconda
  • or download the latest prebuilt binary for Linux users
  • or compile from source
    • Step 1: download and build libisal
    • step 2: download and build libdeflate
    • Step 3: download and build fastp
• input and output
  • output to STDOUT
  • input from STDIN
  • store the unpaired reads for PE data
  • store the reads that fail the filters
  • process only part of the data
  • do not overwrite exiting files
  • split the output to multiple files for parallel processing
  • merge PE reads
• filtering
  • quality filter
  • length filter
  • low complexity filter
  • Other filter
• adapters
• per read cutting by quality score
• base correction for PE data
• global trimming
• polyG tail trimming
• polyX tail trimming
• unique molecular identifier (UMI) processing
  • UMI example
• output splitting
  • splitting by limiting file number
  • splitting by limiting the lines of each file
• overrepresented sequence analysis
• merge paired-end reads
• duplication rate and deduplication
  • duplication rate evaluation
  • deduplication
• all options
• citations

features

0. comprehensive quality profiling for both before and after filtering data (quality curves, base contents, KMER, Q20/Q30, GC Ratio, duplication, adapter contents...)
1. filter out bad reads (too low quality, too short, or too many N...)
2. cut low quality bases for per read in its 5' and 3' by evaluating the mean quality from a sliding window (like Trimmomatic but faster).
3. trim all reads in front and tail
4. cut adapters. Adapter sequences can be automatically detected, which means you don't have to input the adapter sequences to trim them.
5. correct mismatched base pairs in overlapped regions of paired end reads, if one base is with high quality while the other is with ultra low quality
6. trim polyG in 3' ends, which is commonly seen in NovaSeq/NextSeq data. Trim polyX in 3' ends to remove unwanted polyX tailing (i.e. polyA tailing for mRNA-Seq data)
7. preprocess unique molecular identifier (UMI) enabled data, shift UMI to sequence name.
8. report JSON format result for further interpreting.
9. visualize quality control and filtering results on a single HTML page (like FASTQC but faster and more informative).
10. split the output to multiple files (0001.R1.gz, 0002.R1.gz...) to support parallel processing. Two modes can be used, limiting the total split file number, or limitting the lines of each split file.
11. support long reads (data from PacBio / Nanopore devices).
12. support reading from STDIN and writing to STDOUT
13. support interleaved input
14. support ultra-fast FASTQ-level deduplication
15. ...

If you find a bug or have additional requirement for fastp, please file an issue:https://github.com/OpenGene/fastp/issues/new

simple usage

• for single end data (not compressed)

fastp -i in.fq -o out.fq

• for paired end data (gzip compressed)

fastp -i in.R1.fq.gz -I in.R2.fq.gz -o out.R1.fq.gz -O out.R2.fq.gz

By default, the HTML report is saved to fastp.html (can be specified with -h option), and the JSON report is saved to fastp.json (can be specified with -j option).

examples of report

fastp creates reports in both HTML and JSON format.

• HTML report: http://opengene.org/fastp/fastp.html
• JSON report: http://opengene.org/fastp/fastp.json

get fastp

install with Bioconda

# note: the fastp version in bioconda may be not the latest
conda install -c bioconda fastp

or download the latest prebuilt binary for Linux users

This binary was compiled on CentOS, and tested on CentOS/Ubuntu

# download the latest build
wget http://opengene.org/fastp/fastp
chmod a+x ./fastp

# or download specified version, i.e. fastp v0.23.4
wget http://opengene.org/fastp/fastp.0.23.4
mv fastp.0.23.4 fastp
chmod a+x ./fastp

or compile from source

fastp depends on libdeflate and libisal, while libisal is not compatible with gcc 4.8. If you use gcc 4.8, your fastp will fail to run. Please upgrade your gcc before you build the libraries and fastp.

Step 1: download and build libisal

See https://github.com/intel/isa-l autoconf, automake, libtools, nasm (>=v2.11.01) and yasm (>=1.2.0) are required to build this isal

git clone https://github.com/intel/isa-l.git
cd isa-l
./autogen.sh
./configure --prefix=/usr --libdir=/usr/lib64
make
sudo make install

step 2: download and build libdeflate

See https://github.com/ebiggers/libdeflate

git clone https://github.com/ebiggers/libdeflate.git
cd libdeflate
cmake -B build
cmake --build build
cmake --install build

Step 3: download and build fastp

# get source (you can also use browser to download from master or releases)
git clone https://github.com/OpenGene/fastp.git

# build
cd fastp
make

# Install
sudo make install

You can add -j8 option to make/cmake to use 8 threads for the compilation.

input and output

fastp supports both single-end (SE) and paired-end (PE) input/output.

• for SE data, you only have to specify read1 input by -i or --in1, and specify read1 output by -o or --out1.
• for PE data, you should also specify read2 input by -I or --in2, and specify read2 output by -O or --out2.
• if you don't specify the output file names, no output files will be written, but the QC will still be done for both data before and after filtering.
• the output will be gzip-compressed if its file name ends with .gz

output to STDOUT

fastp supports streaming the passing-filter reads to STDOUT, so that it can be passed to other compressors like bzip2, or be passed to aligners like bwa and bowtie2.

• specify --stdout to enable this mode to stream output to STDOUT
• for PE data, the output will be interleaved FASTQ, which means the output will contain records like record1-R1 -> record1-R2 -> record2-R1 -> record2-R2 -> record3-R1 -> record3-R2 ...

input from STDIN

• specify --stdin if you want to read the STDIN for processing.
• if the STDIN is an interleaved paired-end stream, specify --interleaved_in to indicate that.

store the unpaired reads for PE data

• you can specify --unpaired1 to store the reads that read1 passes filters but its paired read2 doesn't, as well as --unpaired2 for unpaired read2.
• --unpaired1 and --unpaired2 can be the same, so the unpaired read1/read2 will be written to the same single file.

store the reads that fail the filters

• give --failed_out to specify the file name to store the failed reads.
• if one read failed and is written to --failed_out, its failure reason will be appended to its read name. For example, failed_quality_filter, failed_too_short etc.
• for PE data, if unpaired reads are not stored (by giving --unpaired1 or --unpaired2), the failed pair of reads will be put together. If one read passes the filters but its pair doesn't, the failure reason will be paired_read_is_failing.

process only part of the data

If you don't want to process all the data, you can specify --reads_to_process to limit the reads to be processed. This is useful if you want to have a fast preview of the data quality, or you want to create a subset of the filtered data.

do not overwrite exiting files

You can enable the option --dont_overwrite to protect the existing files not to be overwritten by fastp. In this case, fastp will report an error and quit if it finds any of the output files (read1, read2, json report, html report) already exists before.

split the output to multiple files for parallel processing

See output splitting

merge PE reads

See merge paired-end reads

filtering

Multiple filters have been implemented.

quality filter

Quality filtering is enabled by default, but you can disable it by -Q or disable_quality_filtering. Currently it supports filtering by limiting the N base number (-n, --n_base_limit), and the percentage of unqualified bases.  

To filter reads by its percentage of unqualified bases, two options should be provided:

• -q, --qualified_quality_phred       the quality value that a base is qualified. Default 15 means phred quality >=Q15 is qualified.
• -u, --unqualified_percent_limit   how many percents of bases are allowed to be unqualified (0~100). Default 40 means 40%

You can also filter reads by its average quality score

• -e, --average_qual if one read's average quality score <avg_qual, then this read/pair is discarded. Default 0 means no requirement (int [=0])

length filter

Length filtering is enabled by default, but you can disable it by -L or --disable_length_filtering. The minimum length requirement is specified with -l or --length_required.

For some applications like small RNA sequencing, you may want to discard the long reads. You can specify --length_limit to discard the reads longer than length_limit. The default value 0 means no limitation.

low complexity filter

Low complexity filter is disabled by default, and you can enable it by -y or --low_complexity_filter. The complexity is defined as the percentage of base that is different from its next base (base[i] != base[i+1]). For example:

# a 51-bp sequence, with 3 bases that is different from its next base
seq = 'AAAATTTTTTTTTTTTTTTTTTTTTGGGGGGGGGGGGGGGGGGGGGGCCCC'
complexity = 3/(51-1) = 6%

The threshold for low complexity filter can be specified by -Y or --complexity_threshold. It's range should be 0~100, and its default value is 30, which means 30% complexity is required.

Other filter

New filters are being implemented. If you have a new idea or new request, please file an issue.

adapters

Adapter trimming is enabled by default, but you can disable it by -A or --disable_adapter_trimming. Adapter sequences can be automatically detected for both PE/SE data.

• For SE data, the adapters are evaluated by analyzing the tails of first ~1M reads. This evaluation may be inacurrate, and you can specify the adapter sequence by -a or --adapter_sequence option. If adapter sequence is specified, the auto detection for SE data will be disabled.
• For PE data, the adapters can be detected by per-read overlap analysis, which seeks for the overlap of each pair of reads. This method is robust and fast, so normally you don't have to input the adapter sequence even you know it. But you can still specify the adapter sequences for read1 by --adapter_sequence, and for read2 by --adapter_sequence_r2. If fastp fails to find an overlap (i.e. due to low quality bases), it will use these sequences to trim adapters for read1 and read2 respectively.
• For PE data, the adapter sequence auto-detection is disabled by default since the adapters can be trimmed by overlap analysis. However, you can specify --detect_adapter_for_pe to enable it.
• For PE data, fastp will run a little slower if you specify the sequence adapters or enable adapter auto-detection, but usually result in a slightly cleaner output, since the overlap analysis may fail due to sequencing errors or adapter dimers.
• The most widely used adapter is the Illumina TruSeq adapters. If your data is from the TruSeq library, you can add --adapter_sequence=AGATCGGAAGAGCACACGTCTGAACTCCAGTCA --adapter_sequence_r2=AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT to your command lines, or enable auto detection for PE data by specifing detect_adapter_for_pe.
• fastp contains some built-in known adapter sequences for better auto-detection. If you want to make some adapters to be a part of the built-in adapters, please file an issue.

You can also specify --adapter_fasta to give a FASTA file to tell fastp to trim multiple adapters in this FASTA file. Here is a sample of such adapter FASTA file:

>Illumina TruSeq Adapter Read 1
AGATCGGAAGAGCACACGTCTGAACTCCAGTCA
>Illumina TruSeq Adapter Read 2
AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT
>polyA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

The adapter sequence in this file should be at least 6bp long, otherwise it will be skipped. And you can give whatever you want to trim, rather than regular sequencing adapters (i.e. polyA).

fastp first trims the auto-detected adapter or the adapter sequences given by --adapter_sequence | --adapter_sequence_r2, then trims the adapters given by --adapter_fasta one by one.

The sequence distribution of trimmed adapters can be found at the HTML/JSON reports.

per read cutting by quality score

fastp supports per read sliding window cutting by evaluating the mean quality scores in the sliding window. From v0.19.6, fastp supports 3 different operations, and you enable one or all of them:

• -5, --cut_front move a sliding window from front (5') to tail, drop the bases in the window if its mean quality is below cut_mean_quality, stop otherwise. Default is disabled. The leading N bases are also trimmed. Use cut_front_window_size to set the widnow size, and cut_front_mean_quality to set the mean quality threshold. If the window size is