Skip to main content

cleanyerreads

Tags: reads quality-control trimming filtering host-removal preprocessing named-workflow

Quality control and optional host read removal from raw sequencing reads.

This workflow performs comprehensive read quality control including trimming, adapter removal, quality filtering, and optionally removes host contamination using deacon, nohuman, or SRA Human Scrubber. It processes raw sequencing reads to produce high-quality clean reads ready for downstream analysis.

Usage​

clean-yer-reads CLI:

clean-yer-reads \
  --input samples.csv \
  --outdir results/

Nextflow:

nextflow run bactopia/bactopia/workflows/cleanyerreads/main.nf \
  --input samples.csv \
  --outdir results/

Outputs​

Expected Output Files​

<BACTOPIA_DIR>
├── <SAMPLE_NAME>
│   ├── main
│   │   ├── gather
│   │   │   ├── <SAMPLE_NAME>-meta.tsv
│   │   │   └── logs
│   │   │       ├── nf.command.{begin,err,log,out,run,sh,trace}
│   │   │       └── versions.yml
│   │   └── qc
│   │       ├── <SAMPLE_NAME>_SE.fastq.gz
│   │       ├── logs
│   │       │   ├── <SAMPLE_NAME>-fastp.log
│   │       │   ├── nf.command.{begin,err,log,out,run,sh,trace}
│   │       │   └── versions.yml
│   │       └── supplemental
│   │           ├── <SAMPLE_NAME>.fastp.html
│   │           ├── <SAMPLE_NAME>.fastp.json
│   │           ├── <SAMPLE_NAME>_SE-final.json
│   │           ├── <SAMPLE_NAME>_SE-final_fastqc.html
│   │           ├── <SAMPLE_NAME>_SE-final_fastqc.zip
│   │           ├── <SAMPLE_NAME>_SE-original.json
│   │           ├── <SAMPLE_NAME>_SE-original_fastqc.html
│   │           └── <SAMPLE_NAME>_SE-original_fastqc.zip
│   └── tools
│       └── srahumanscrubber
│           ├── <SAMPLE_NAME>.scrub.report.tsv
│           ├── <SAMPLE_NAME>.scrubbed.fastq.gz
│           └── logs
│               ├── nf.command.{begin,err,log,out,run,sh,trace}
│               └── versions.yml
└── bactopia-runs
    └── cleanyerreads-<TIMESTAMP>
        ├── merged-results
        │   ├── logs
        │   │   ├── meta-concat
        │   │   │   ├── nf.command.{begin,err,log,out,run,sh,trace}
        │   │   │   └── versions.yml
        │   │   └── scrubber-concat
        │   │       ├── nf.command.{begin,err,log,out,run,sh,trace}
        │   │       └── versions.yml
        │   ├── meta.tsv
        │   └── scrubber.tsv
        └── nf-reports
            ├── cleanyerreads-dag.dot
            ├── cleanyerreads-report.html
            └── cleanyerreads-timeline.html

Quality Control Reports​

FileDescription
supplemental/*_fastqc.*FastQC reports for raw and cleaned reads
supplemental/*-NanoPlot.*NanoPlot reports for Nanopore reads
supplemental/*.fastp.*Fastp quality reports (when applicable)
supplemental/*_original.jsonQuality metrics for original reads
supplemental/*_final.jsonQuality metrics for final reads

Cleaned Reads​

FileDescription
*.fastq.gzQuality controlled and trimmed reads
*.unclassified.fastq.gzHost-decontaminated reads (if scrubber enabled)

Host Decontamination​

note

Only created when host removal is enabled

FileDescription
*.kraken2.report.txtKraken2 classification report
*.scrub.report.tsvHuman contamination report

Audit Trail​

Below are files that can assist you in understanding which parameters and program versions were used.

Logs​

Each process that is executed will have a folder named logs. In this folder are helpful files for you to review if the need ever arises.

ExtensionDescription
.beginAn empty file used to designate the process started
.errContains STDERR outputs from the process
.logContains both STDERR and STDOUT outputs from the process
.outContains STDOUT outputs from the process
.runThe script Nextflow uses to stage/unstage files and queue processes based on given profile
.shThe script executed by bash for the process
.traceThe Nextflow trace report for the process
versions.ymlA YAML formatted file with program versions

Nextflow Reports​

These Nextflow reports provide great a great summary of your run. These can be used to optimize resource usage and estimate expected costs if using cloud platforms.

FilenameDescription
cleanyerreads-dag.dotThe Nextflow DAG visualization
cleanyerreads-report.htmlThe Nextflow Execution Report
cleanyerreads-timeline.htmlThe Nextflow Timeline Report
cleanyerreads-trace.txtThe Nextflow Trace report

Parameters​

Required Parameters​

The following parameters are how you will provide either local or remote samples to be processed by Bactopia.

ParameterTypeDefaultDescription
--samplesstringA FOFN (via bactopia prepare) with sample names and paths to FASTQ/FASTAs to process
--r1stringFirst set of compressed (gzip) Illumina paired-end FASTQ reads (requires --r2 and --sample)
--r2stringSecond set of compressed (gzip) Illumina paired-end FASTQ reads (requires --r1 and --sample)
--sestringCompressed (gzip) Illumina single-end FASTQ reads (requires --sample)
--ontstringCompressed (gzip) Oxford Nanopore FASTQ reads (requires --sample)
--hybridbooleanfalseCreate hybrid assembly using Unicycler. (requires --r1, --r2, --ont and --sample)
--short_polishbooleanfalseCreate hybrid assembly from long-read assembly and short read polishing. (requires --r1, --r2, --ont and --sample)
--samplestringSample name to use for the input sequences
--accessionsstringA file containing ENA/SRA Experiment accessions or NCBI Assembly accessions to processed
--accessionstringSample name to use for the input sequences
--assemblystringA assembled genome in compressed FASTA format. (requires --sample)
--check_samplesbooleanfalseValidate the input FOFN provided by --samples

Gather Parameters​

ParameterTypeDefaultDescription
--skip_fastq_checkbooleanSkip minimum requirement checks for input FASTQs
--min_basepairsinteger2241820The minimum amount of basepairs required to continue downstream analyses.
--min_readsinteger7472The minimum amount of reads required to continue downstream analyses.
--min_coverageinteger10The minimum amount of coverage required to continue downstream analyses.
--min_proportionnumber0.5The minimum proportion of basepairs for paired-end reads to continue downstream analyses.
--min_genome_sizeinteger100000The minimum estimated genome size allowed for the input sequence to continue downstream analyses.
--max_genome_sizeinteger18040666The maximum estimated genome size allowed for the input sequence to continue downstream analyses.
--attemptsinteger3Maximum times to attempt downloads
--use_enabooleanDownload FASTQs from ENA
--no_cachebooleanSkip caching the assembly summary file from ncbi-genome-download

csvtk concat Parameters​

ParameterTypeDefaultDescription
--csvtk_concat_optsstringExtra csvtk concat options in quotes

QC Parameters​

ParameterTypeDefaultDescription
--use_bbmapbooleanIllumina reads will be QC'd using BBMap
--use_porechopbooleanfalseUse Porechop to remove adapters from ONT reads
--skip_qcbooleanThe QC step will be skipped and it will be assumed the inputs sequences have already been QCed.
--skip_qc_plotsbooleanQC Plot creation by FastQC or Nanoplot will be skipped
--skip_error_correctionbooleanFLASH error correction of reads will be skipped.
--adaptersstringA FASTA file containing adapters to remove
--adapter_kinteger23Kmer length used for finding adapters.
--phixstringphiX174 reference genome to remove
--phix_kinteger31Kmer length used for finding phiX174.
--ktrimstringrTrim reads to remove bases matching reference kmers (choices: f, r, l)
--minkinteger11Look for shorter kmers at read tips down to this length, when k-trimming or masking.
--hdistinteger1Maximum Hamming distance for ref kmers (subs only)
--tpestringtWhen kmer right-trimming, trim both reads to the minimum length of either (choices: f, t)
--tbostringtTrim adapters based on where paired reads overlap (choices: f, t)
--qtrimstringrlTrim read ends to remove bases with quality below trimq. (choices: rl, f, r, l, w)
--trimqinteger6Regions with average quality BELOW this will be trimmed if qtrim is set to something other than f
--maqinteger10Reads with average quality (after trimming) below this will be discarded
--minlengthinteger35Reads shorter than this after trimming will be discarded
--ftminteger5If positive, right-trim length to be equal to zero, modulo this number
--tossjunkstringtDiscard reads with invalid characters as bases (choices: f, t)
--ainstringfWhen detecting pair names, allow identical names (choices: f, t)
--qoutstring33PHRED offset to use for output FASTQs (choices: 33, 64)
--maxcorinteger1Max number of corrections within a 20bp window
--sampleseedinteger42Set to a positive number to use as the random number generator seed for sampling
--ont_minlengthinteger1000ONT Reads shorter than this will be discarded
--ont_minqualinteger0Minimum average read quality filter of ONT reads
--porechop_optsstringExtra Porechop options in quotes
--nanoplot_optsstringExtra NanoPlot options in quotes
--bbduk_optsstringExtra BBDuk options in quotes
--fastp_optsstringExtra fastp options in quotes

SRA Human Scrubber Parameters​

ParameterTypeDefaultDescription
--use_srascrubberbooleanfalseUse SRAHumanScrubber for scrubbing human reads

Nohuman Download Parameters​

ParameterTypeDefaultDescription
--nohuman_dbstringPath to the nohuman database or directory to download it to
--nohuman_db_versionstringDatabase version to download (default: latest HPRC release)
--nohuman_save_as_tarballbooleanfalseSave the nohuman database as a tarball
--download_nohumanbooleanfalseDownload the nohuman database to the path given by --nohuman_db

Nohuman Run Parameters​

ParameterTypeDefaultDescription
--nohuman_dbstringPath to the nohuman database directory or tarball
--nohuman_confidencenumber0.0Kraken2 minimum confidence score for classification (0.0-1.0)
--nohuman_humanbooleanfalseInvert output to keep only human reads instead of removing them
--nohuman_save_reportbooleanfalseSave the Kraken2 classification report

Deacon Fetch Parameters​

ParameterTypeDefaultDescription
--deacon_index_namestringpanhuman-1Name of the pre-built deacon index to fetch
--download_deaconbooleanfalseDownload the deacon index to the datasets cache
--use_deaconbooleanfalseUse deacon for host read filtering

Deacon Filter Parameters​

ParameterTypeDefaultDescription
--deacon_abs_thresholdinteger2Minimum absolute number of minimizer hits for a match
--deacon_dbstringPath to a pre-existing deacon index (.idx) for host read filtering
--deacon_depletebooleantrueDiscard matching sequences instead of keeping them
--deacon_optsstringAdditional deacon filter options not covered by other parameters
--deacon_prefix_lengthinteger0Search only the first N nucleotides per sequence (0 for all)
--deacon_rel_thresholdnumber0.01Minimum relative proportion (0.0-1.0) of minimizer hits for a match
Dataset Parameters

Define where the pipeline should find input data and save output data.

ParameterTypeDefaultDescription
--speciesstringName of species for species-specific dataset to use
--ask_merlinbooleanAsk Merlin to execute species specific Bactopia tools based on Mash distances
--coverageinteger100Reduce samples to a given coverage, requires a genome size
--genome_sizeinteger0Expected genome size (bp) for all samples, required for read error correction and read subsampling
--use_baktabooleanUse Bakta for annotation, instead of Prokka
Optional Parameters

These optional parameters can be useful in certain settings.

ParameterTypeDefaultDescription
--outdirstringbactopiaBase directory to write results to
--skip_compressionbooleanfalseOutput files will not be compressed
--datasetsstringThe path to cache datasets to
--keep_all_filesbooleanfalseKeeps all analysis files created
Max Job Request Parameters

Set the top limit for requested resources for any single job.

ParameterTypeDefaultDescription
--max_retryinteger3Maximum times to retry a process before allowing it to fail.
--max_cpusinteger4Maximum number of CPUs that can be requested for any single job.
--max_memorystring128.GBMaximum amount of memory that can be requested for any single job.
--max_timestring240.hMaximum amount of time that can be requested for any single job.
--max_downloadsinteger3Maximum number of samples to download at a time
Nextflow Configuration Parameters

Parameters to fine-tune your Nextflow setup.

ParameterTypeDefaultDescription
--nfconfigstringA Nextflow compatible config file for custom profiles, loaded last and will overwrite existing variables if set.
--publish_dir_modestringcopyMethod used to save pipeline results to output directory. (choices: symlink, rellink, link, copy, copyNoFollow, move)
--infodirstring${params.outdir}/pipeline_infoDirectory to keep pipeline Nextflow logs and reports.
--forcebooleanfalseNextflow will overwrite existing output files.
--cleanup_workdirbooleanfalseAfter Bactopia is successfully executed, the work directory will be deleted.
Institutional config options

Parameters used to describe centralized config profiles. These should not be edited.

ParameterTypeDefaultDescription
--custom_config_versionstringmasterGit commit id for Institutional configs.
--custom_config_basestringhttps://raw.githubusercontent.com/nf-core/configs/masterBase directory for Institutional configs.
--config_profile_namestringInstitutional config name.
--config_profile_descriptionstringInstitutional config description.
--config_profile_contactstringInstitutional config contact information.
--config_profile_urlstringInstitutional config URL link.
Nextflow Profile Parameters

Parameters to fine-tune your Nextflow setup.

ParameterTypeDefaultDescription
--condadirstringDirectory to Nextflow should use for Conda environments
--registrystringquay.ioRegistry to pull Docker containers from.
--datasets_cachestring<HOME>/.bactopia/datasetsDirectory where downloaded datasets should be stored.
--singularity_cachestringDirectory where remote Singularity images are stored.
--singularity_pull_docker_containerbooleanInstead of directly downloading Singularity images for use with Singularity, force the workflow to pull and convert Docker containers instead.
--force_rebuildbooleanfalseForce overwrite of existing pre-built environments.
--queuestringgeneral,high-memoryComma-separated name of the queue(s) to be used by a job scheduler (e.g. AWS Batch or SLURM)
--cluster_optsstringAdditional options to pass to the executor. (e.g. SLURM: '--account=my_acct_name'
--container_optsstringAdditional options to pass to Apptainer, Docker, or Singularity. (e.g. Singularity: '-D pwd'
--disable_scratchbooleanfalseAll intermediate files created on worker nodes of will be transferred to the head node.
Helpful Parameters

Uncommonly used parameters that might be useful.

ParameterTypeDefaultDescription
--monochrome_logsbooleanDo not use coloured log outputs.
--nfdirbooleanPrint directory Nextflow has pulled Bactopia to
--sleep_timeinteger5The amount of time (seconds) Nextflow will wait after setting up datasets before execution.
--validate_paramsbooleantrueBoolean whether to validate parameters against the schema at runtime
--helpbooleanDisplay help text.
--wfstringbactopiaSpecify which workflow or Bactopia Tool to execute
--list_wfsbooleanList the available workflows and Bactopia Tools to use with '--wf'
--show_hidden_paramsbooleanShow all params when using --help
--help_allbooleanAn alias for --help --show_hidden_params
--versionbooleanDisplay version text.

Composition​

This workflow uses the following subworkflows:

  • bactopia_gather - Search, validate, gather, and standardize input samples.
  • bactopia_qc - Perform comprehensive quality control on sequencing reads.
  • scrubber - Remove contaminant sequences from metagenomic data.

Citations​

If you use this in your analysis, please cite the following.

Source​

View source on GitHub