KO
|
EN
gitlite — search
Search
#javascript
#python
#hacktoberfest
#react
#ai
#typescript
#llm
#go
#golang
#android
#machine-learning
#rust
#deep-learning
#linux
pipelines
★ 26
Open GitHub ↗
RPD Pipelines
Download README (.md)
Explore Similar Repositories
behat-step-time-logger
:
behat-step-time-logger
WordCloud-CN
:
程序员的浪漫！
XLog
:
Minimalistic logging PCL library for (.NET, WP8, UWP, Xamarin iOS, Xamarin Android)
jsongood
:
简单可扩展的轻量级的微服务网关(A Lightweight Mricro Service Gateway Framework)
fred
:
FRED simulator and associated paper
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
pipelines
?
Download (.md)
# NGS Pipeline Framework for [GIS](https://www.a-star.edu.sg/gis/) This folder contains workflow developed by the [Research Pipeline Development Team (RPD)](https://www.a-star.edu.sg/gis/our-science/technology-platforms/scientific-and-research-computing-platform.aspx) ## NOTE - This repo and its deployments are in maintenance mode - All newer pipelines are written in Nextflow. - Please contact us for more info - The NSCC setup has been affected multiple times by a forced change in setup. Let us know if you have trouble using it. ## Features - Cluster specifics are handled internally, i.e. users don't have to worry about scheduler usage. - Built-in check-pointing: Easy restart and skipping of already completed steps - Pipelines work out of the box on GIS's aquila (UGE) or the [National Super Computing Center (NSCC)](http://help.nscc.sg/) (PBS Pro) without any changes required by the user - Pipelines are divided into steps that are automatically run in parallel where possible and each step tries to make optimal use of resources ## Overview - Pipelines are organised into directories of specific category, e.g. `variant-calling` - Each pipeline has its own subfolder there and the corresponding wrapper script has the same name (e.g. `variant-calling/gatk/gatk.py`) - Each pipeline folder contains a README file describing the pipeline (e.g. [`variant-calling/gatk/README.md`](variant-calling/gatk/README.md)) - Furthermore, each pipeline folder contains an example flowchart of the workflow, called `example-dag.pdf` (see e.g. [`variant-calling/gatk/example-dag.pdf`](variant-calling/gatk/example-dag.pdf)) ## Existing Installations The following installations are available at different sites (referred to as `RPD_PIPELINES` below): - GIS: `/mnt/projects/rpd/pipelines/` - NSCC: `/data/users/astar/gis/rpd/pipelines/` Each of these contains one subfolder per pipeline version, e.g. `$RPD_PIPELINES/pipelines.2017-10` (referred to as `PIPELINE_ROOTDIR` below). Much of this framework assumes a certain setup and services to be present, as is the case in GIS / the NSCC. This repository is therefore of limited use to the general public. See `INSTALL.md` for simplistic installation instructions. Some pipelines only work at a certain site (due to system or software incompatibilities etc.). Refer to the table of pipelines below for details regarding availability. ## How to Run There are two ways to invoke a pipeline: either call the convenience wrapper, plainly called `run` or invoke the pipeline specific scripts directly: 1. Using the convenience wrapper (recommended) - The basic usage is `$PIPELINE_ROOTDIR/run name options`, where `name` is a pipeline name and `options` are valid options for this pipeline. - An example (GATK) would be `$PIPELINE_ROOTDIR/run gatk --help` - Just calling `$PIPELINE_ROOTDIR/run` will print a list of available pipelines and simple usage information 2. Direct invocation - Directly call the wrapper of the particular pipeline that you want to run, e.g. for GATK: `$PIPELINE_ROOTDIR/variant-calling/gatk/gatk.py` - Note, in this case you need to have a Python3 interpreter in your PATH, which is not needed if you use the convenience wrapper (see above) In either case, you must not prefix the script with `python`. - Note, there is no need to submit the script itself, as long as you run it from a cluster node - If called correctly, jobs will be run on the cluster automatically - Use `-h` or `--help` to display usage information - Use the `-v` option, so that more information is printed - All scripts create an output directory (option `-o`) containing the run environment - Your results will be saved to a corresponding subdirectory called `./out/` - Upon completion (success or error) an email will be send to the user (unless `--no-mail` was specified) pointing to the results. In addition a file called `report.html` will be generated containing some basic information about the analysis. - Should a pipeline fail for technical reasons (crash of a node, connectivity issues etc.) they can be easily restarted: cd into the output directory and `qsub run.sh >> logs/submission.log`. Upon restart, partially created files will be automatically deleted and the pipeline will skip already completed steps - Note, that the output directory has to be on a shared file-system, i.e. directories local to the cluster node like `/tmp` won't work, unless run in local mode) ### Example #### Variant calling with GATK for an Exome sample with two FastQ pairs fq1_x=x_R1.fastq.gz fq2_x=x_R2.fastq.gz fq1_y=y_R1.fastq.gz fq2_y=y_R2.fastq.gz bed=/path/to/SeqCap_EZ_Exome_v3_primary.bed outdir=/path/to/output-folder-for-this-analysis/ /path/to/pipelines/run gatk -o $outdir -1 $fq1_x $fq1_y -2 $fq2_x $fq2_y -s sample-name -t WES -l $bed # or # /path/to/pipelines/variant-calling/gatk/gatk.py -o $outdir -1 $fq1_x $fq1_y -2 $fq2_x $fq2_y -s sample-name -t WES -l $bed ## List of Pipelines | Name | Category | Notes | @GIS | @NSCC | | --- | --- | --- | --- | --- | | [bcl2fastq](bcl2fastq/README.md) | Production | Not for end-users | Y | Y | | [ATAC-seq](chromatin-profiling/atacseq/README.md) | Chromatin Profiling | | Y | Y | | [ChIP-seq](chromatin-profiling/chipseq/README.md) | Chromatin Profiling | | Y | Y | | [SG10K](custom/SG10K/README.md) | Custom | Not for end-users | Y | Y | | [ViPR](germs/vipr/README.md) | GERMS | | Y | Y | | [BWA-MEM](mapping/BWA-MEM/README.md) | Mapping | | Y | Y | | [Shotgun Metagenomics](metagenomics/shotgun-metagenomics/README.md) | Metagenomics | | Y | Y | | [Essential-Genes](metagenomics/essential-genes/README.md) | Metagenomics | Requires ref download | Y | Y | | [STAR-RSEM](rnaseq/star-rsem/README.md) | RNA-Seq | | Y | Y | | [Fluidigm-HT-C1-RNASeq](rnaseq/fluidigm-ht-c1-rnaseq/README.md)| RNA-Seq | | Y | N | | [Wafergen-scRNA](rnaseq/wafergen-scrna/README.md) | RNA-Seq | Requires cellular barcodes | Y | Y | | [LoFreq-Somatic](somatic/lofreq-somatic/README.md) | Somatic | | Y | N | | [Mutect](somatic/mutect/README.md) | Somatic | | Y | Y | | [GATK](variant-calling/gatk/README.md) | Variant-calling | | Y | Y | | [Lacer-LoFreq](variant-calling/lacer-lofreq/README.md) | Variant-calling | | Y | N | See `example-dag.pdf` in each pipeline's folder for a visual overview of the workflow. Note, most pipelines start with FastQ files as input, a few allow injection of BAM files. ## How it Works - All pipelines are based on [](http://snakemake.bitbucket.org) - Input will be a single FastQ file or a pair of FastQ files. Multiple of these can be given. Each pair is treated as one readunit (see also resulting `conf.yaml` file) and gets its own read-group assigned where appropriate. - Software versions are defined in each pipelines' `cfg/modules.yaml` and loaded via [Lmod](http://lmod.readthedocs.io/en/latest/) - Pipeline wrappers create an output directory containing all necessary configuration files, run scripts etc. - After creation of this folder, the analysis run is automatically submitted to the cluster (unless `--no-run` was used which gives you a chance to change the config file `conf.yaml`) - The actual run script is called `run.sh` - The main log file is `./logs/snakemake.log` (use `tail -f` to follow live progress) - After a successful run, the last line in the mail log file will read: `(100%) done` - Cluster log files can be found in the respective `./logs/` sub-directory ## Debugging Techniques First call the wrapper in question with `--no-run`. cd into the given outdir and then - Check the created `conf.yaml` - Execute a dryrun: `rm -f logs/snakemake.log; EXTRA_SNAKEMAKE_ARGS="--dryrun" bash run.sh; cat logs/snakemake.log` - Run locally: `nohup bash run.sh; tail -f logs/snakemake.log` ## (Multi) Sample Configuration If you have just one sample to analyse (no matter if multiple FastQ pairs or not), you will use options `-s`, `-1` and `-2` most of the time. To provide the pipeline with more information about your FastQ files (e.g. run-id etc.) you can create a sample configuration file (see below) and provide it to the wrapper script with `--sample-cfg` (thus replacing `-s`, `-1` and `-2`). You also need a sample configuration file if you want to analyse many samples identically with just one wrapper call. The easiest way to create such a file is to first create an Excel/CSV sheet listing all samples and FastQ files and convert it into a sample config file as described in the following: - Create an Excel sheet with the following columns: 1. sample name (mandatory; can be used repeatedly, e.g. if you have multiple FastQs per sample) 2. run id (allowed to be empty) 3. flowcell id (allowed to be empty) 4. library id (allowed to be empty) 5. lane id (allowed to be empty) 6. read-group id (allowed to be empty) 7. fastq1 (mandatory) 8. fastq2 (allowed to be empty) - Save the Excel sheet as CSV and run the following to convert it to a yaml config file: `tools/sample_conf.py -i <your>.csv -i <your>.yaml` Depending on how you created the CSV file you might want to set the CSV delimiter with `-d`, e.g. `-d ,` - Use the created yaml file as input for the pipeline wrapper (option `--sample-cfg your.yaml`) Please note, not all pipelines support this feature, for example the Chipseq and all somatic pipelines. In some cases multisample processing can lead to very high memory consumption by the snakemake master process itself, a side-effect which is hard to predict (the master process will be killed). The above configuration can be used for single sample processing as well, however, for single samples the corresponding use of options `-s`, `-1` and `-2` is usually easier. ## FAQ #### Where are my results? In the output directory that you specified with `-o`, under a subdirectory called `out`. Depending on the pipeline, the sample-name is added as well. #### How do I know the pipeline run is completed? You should have received an email. To double check run `tail logs/snakemake.log` in the output directory. It should either say `Nothing to be done` or `(100%) done` #### How do I submit the wrapper to the cluster? You don't. It's taken care of automatically. #### Which Python version should I use? None! Call scripts directly, i.e. without `python`. #### Pipeline execution failed. What now? First, simply try to restart the pipeline. In your output directory execute `qsub run.sh >> logs/submission.log`. If this still fails, you need to troubleshoot by examining the log files. You can ask us for help (see below). #### Can you write a pipeline for me? Yes. Please email us. A committee will decide on implementation priority. #### Can these pipelines be selected in / run from ELM? No. For now you will have to use the commandline. The Datahub team is working on a separate web-interface for running pipelines. ## Comments, For questions, feedback, bug reports etc. contact us: [Research Pipeline Development Team (RPD)](mailto:rpd@gis.a-star.edu.sg)
