pastForward - A pipeline for ancient and historical DNA based on Snakemake

pastForward analyzes raw ancient and historical DNA from a sequencing facility. It checks read quality and screens for contamination, with methods suited to the short, damaged reads such samples usually give. It maps reads to a reference genome and corrects them for DNA damage, so they are ready for downstream analysis. Optionally, it can also compare genomic features across time points or between specimens, such as transposon insertions, gene copy number changes, or endosymbiont strain replacements.

pastForward is built on Snakemake.

Note: pastForward uses two purpose-built tools: REVEAL for transposable element and genomic feature dynamics, and ECMSD for contamination screening against a curated mitochondrial database. See their READMEs for details on each tool.

Workflow Overview

For detailed information about the processing steps, see the Process Overview page. For common questions and troubleshooting, see the FAQ.

Quick Start

New to pastForward? These four steps take you from nothing to a finished run. Each one links to more detail.

1. Install Conda and Snakemake, and download pastForward. The Setup Guide walks through this step by step.
2. Add your species and sequencing data. Also covered in the Setup Guide, including how your read files need to be named.
3. Create a config.yaml file. Every pipeline stage is turned on by default, but can be adjusted if required. To run with default settings, a config with a project name and species list is enough. If you want to adjust the config but would rather not edit a text file by hand, open the Config Designer in your browser to create your config.
4. Run the pipeline. See Running the Pipeline below for the exact command.

Already running pastForward and want a newer version? See the Update Guide.

Running the Pipeline

Run ./pastForward from your project folder, the folder that directly contains workflow/, config/, and your <species>/ folders. See Project Structure for what that folder should look like.

For a new project, run these in order:

1. ./pastForward check: see what pastForward finds on disk for your config (species/individuals/references/...). Fix your data or config first if anything looks wrong here.
2. ./pastForward preview: see the output files a run would produce, including ones it'll skip.
3. ./pastForward dryrun (optional): a full Snakemake dry run, to double-check the exact rules that would execute.
4. ./pastForward run --cores 40: runs the pipeline, in the background by default.

./pastForward check
./pastForward preview
./pastForward dryrun            # optional

./pastForward run --cores 40             # runs the pipeline, in the background, with the suggested flags
./pastForward run --cores 40 --fg        # same, but in the foreground
./pastForward resume --cores 40          # like run, but also picks back up rules left incomplete by a crash/kill

./pastForward status           # project, config, PID, progress bar, and the last few pipeline steps of the tracked background run
./pastForward status --watch   # (or -w) same, plus a log tail, reprinted every 5s until the run ends (Ctrl-C to stop early)
./pastForward abort            # stop it gracefully (SIGTERM; snakemake shuts down its own subprocesses)
./pastForward abort --force    # (or -f) or kill it and everything it started, immediately
./pastForward unlock           # clear a stale lock left by a crashed run
./pastForward touch            # mark existing output files as up to date, so the next run skips the steps that made them
./pastForward doctor                        # list conda envs and whether each is built
./pastForward doctor --rebuild-envs ecmsd   # force one (or, with no names, all) to be recreated
./pastForward print-log        # print the most recently written log from logs/
./pastForward print-log --tail 50  # (or -t) only the last 50 lines (default 20)
./pastForward print-log --live     # (or -l) tail -f the log (Ctrl-C to stop)

./pastForward benchmark                 # how long each rule took and how much memory it used
./pastForward benchmark --emit-profile  # same, plus a Snakemake `set-resources:` block built from those numbers

./pastForward version          # print the pipeline version

./pastForward tools                         # list the setup helper tools
./pastForward tools create-species Dmel     # make Dmel's input folders and print a config snippet for it
./pastForward tools link-reads --source /data/run1 --species Dmel  # symlink every read file in /data/run1 into Dmel's read folder

run requires --cores <N> (or -j/--jobs). pastForward will not guess a thread count for you. --software-deployment-method conda, --keep-going, and --rerun-trigger mtime are added automatically, but if you pass one of them yourself, your value is used instead. (Any spelling of the deployment flag counts, including --sdm and the deprecated --use-conda.) Any other extra arguments (e.g. --forceall) go straight through to Snakemake. run also refuses to start if a tracked run is still alive in the same project folder. Stop that one with abort first.

touch runs snakemake --touch: it only updates the timestamps of output files that already exist, so Snakemake treats them as up to date and skips the steps that produced them. Nothing is recomputed and no file content changes. It is meant for cases where the results are fine but their timestamps are not, e.g. after copying results in from another machine. Output files that do not exist yet are skipped with a warning, and by default only files Snakemake already considers out of date are touched. Add --forcerun <rule> or --forceall to touch the rest as well.

benchmark summarizes what a run actually cost. Every step writes a small *.benchmark.jsonl file next to its log file, recording wall time, peak memory, threads, and input size, and benchmark collects them into one row per rule, ordered by total core-hours. Use it to see where a run spends its time, or, with --emit-profile, to turn those measurements into resource requests for an HPC cluster. See Running with Snakemake.

tools create-species <species> [<species> ...] sets up a new species. It creates the species folder with all its input/ subfolders in the project root and prints a species: block you can paste into config/config.yaml. Folders that already exist are left alone, so it is safe to run again. processed/ and results/ are not created, since the pipeline makes those itself. See Add a Species.

tools link-reads --source <folder> --species <species> brings in reads that already live somewhere else. The short forms are -d and -s, so -d /data/run1 -s Dmel works too, in either order. Every *.fastq.gz and *.fq.gz file directly inside the source folder gets a symlink in <species>/input/read_module/. Each link points to the file's absolute path and keeps its original name, so the names must already follow the raw read naming convention. Files whose names don't follow it are still linked, but listed in a warning, with the reason the pipeline would ignore or reject each one. Subfolders and hidden files are not searched. A file that is already in the read folder under the same name is left alone, so it is safe to run again. The species folder must exist, so run create-species first.

Each run/dryrun/touch writes a timestamped log to logs/ in your project folder. status reads the most recent run back out of there.

Snakemake keeps track of what's already been done and only re-runs steps that are missing or out of date. To start completely over, delete the relevant results and processed folders and run the pipeline again.

Want to call snakemake directly, tune its flags, run without the CLI wrapper, or run on an HPC cluster? See Running with Snakemake.

Reports

pastForward generates a MultiQC report for:

• Each species (all samples from that species together, so you can compare results across them)
  • Location: {species}/results/summary/species_level/{species}_multiqc.overall.html
• Each individual sample
  • Location: {species}/results/summary/individual_level/{individual}_multiqc.html

These reports summarize reads before and after trimming, taxonomic screening, coverage, deduplication, and damage rescaling. Use them to judge the quality of your sequenced reads and decide whether a sample needs additional library preparation.

Citation

If you use pastForward in your research, please cite:

Saadain, S., Kapun, M., & Kofler, R. (2026). pastForward: a Snakemake pipeline for ancient and historical DNA with eukaryote-wide taxonomic screening and tracking of copy-number variation. bioRxiv. https://doi.org/10.64898/2026.08.07.743613