PAT-2300 Add mixed input

[Lfulcrum]

This adds a number of input options to the pipeline as detailed in the README.md changes. Offers convenient flexibility for running with read data directly from iRODS.

This required several config imports and addition of a git submodule into the pipeline.

Validation is skipped for all parameters pertaining to MIXED_INPUT subworkflow, as it is assumed the validation is performed by that subworkflow.

[Lfulcrum]

Original MR reviewed by Harry (GlobalPneumoSeq#135)

[Lfulcrum]

Testing on farm:

bsub.py -q oversubscribed 4 test_with_irods_params_6 ./run_pipeline --reads test_input --manifest_of_reads manifest_of_reads.csv --runid 9870 --laneid 5 --plexid 53 --type bam --output test_with_irods_params_6 -profile singularity,lsf,sanger

With output:

executor >  local (5), lsf (142)
[56/c4ac1c] PIPELINE:GET_REF_GENOME_BWA_DB | 1 of 1 ✔
[68/f903a0] PIPELINE:GET_KRAKEN2_DB        | 1 of 1 ✔
[cf/b7b58f] PIPELINE:GET_SEROBA_DB         | 1 of 1 ✔
[00/c1176a] PIPELINE:GET_POPPUNK_DB        | 1 of 1 ✔
[d3/cfa05d] PIP…E:GET_POPPUNK_EXT_CLUSTERS | 1 of 1 ✔
[01/fbea67] PIPELINE:GET_ARIBA_DB          | 1 of 1 ✔
[4b/479fff] PIP…:IRODS_QUERY:JSON_PREP (1) | 1 of 1 ✔
[d3/a14e41] PIP…CTOR:IRODS_QUERY:BATON (1) | 1 of 1 ✔
[7f/348ae6] PIP…IRODS_QUERY:JSON_PARSE (1) | 1 of 1 ✔
[d3/96cb76] PIP…QUERY:METADATA_QUERIED (1) | 1 of 1 ✔
[b0/5f28d4] PIP…_EXTRACT:RETRIEVE_CRAM (1) | 1 of 1 ✔
[db/8a4b7f] PIP…_EXTRACT:COLLATE_FASTQ (1) | 1 of 1 ✔
[f4/b05c3d] PIP…ILE_VALIDATION (9870_5#53) | 6 of 6 ✔
[54/730055] PIP…INE:PREPROCESS (9870_5#53) | 6 of 6 ✔
[a0/15fec7] PIPELINE:READ_QC (9870_5#53)   | 6 of 6 ✔
[09/80ce24] PIP…SEMBLY_SHOVILL (9870_5#53) | 6 of 6 ✔
[be/7d0464] PIP…SSEMBLY_ASSESS (9870_5#53) | 6 of 6 ✔
[60/574c18] PIP…NE:ASSEMBLY_QC (9870_5#53) | 6 of 6 ✔
[69/251ea4] PIPELINE:MAPPING (9870_5#53)   | 6 of 6 ✔
[1a/e836c4] PIP…TO_SORTED_BAM (17175_7#59) | 6 of 6 ✔
[6a/2ea205] PIPELINE:SNP_CALL (17175_7#59) | 6 of 6 ✔
[b3/866e2a] PIP…HET_SNP_COUNT (17175_7#59) | 6 of 6 ✔
[6b/368eb2] PIP…NE:MAPPING_QC (17175_7#59) | 6 of 6 ✔
[b7/affcaf] PIPELINE:TAXONOMY (9870_5#53)  | 6 of 6 ✔
[b4/2039df] PIP…NE:TAXONOMY_QC (9870_5#53) | 6 of 6 ✔
[32/606bb7] PIP…NE:OVERALL_QC (17175_7#59) | 6 of 6 ✔
[4f/d08b7b] PIPELINE:LINEAGE (All samples) | 1 of 1 ✔
[71/f7124f] PIPELINE:SEROTYPE (17175_7#59) | 4 of 4 ✔
[90/dd325c] PIPELINE:MLST (17175_7#59)     | 4 of 4 ✔
[c4/dcc838] PIP…BP_RESISTANCE (17175_7#59) | 4 of 4 ✔
[48/467cfd] PIP…BP_RESISTANCE (17175_7#59) | 4 of 4 ✔
[bf/886056] PIP…ER_RESISTANCE (17175_7#59) | 4 of 4 ✔
[43/4384f9] PIP…ER_RESISTANCE (17175_7#59) | 4 of 4 ✔
[93/1575be] PIP…SAMPLE_REPORT (17175_7#59) | 6 of 6 ✔
[c6/b260fa] PIP…NE:GENERATE_OVERALL_REPORT | 1 of 1 ✔
[f1/71f1c7] SAV…NFO:GET_VERSION:IMAGES (1) | 1 of 1 ✔
[f1/8f7151] SAV…INFO:GET_VERSION:DATABASES | 1 of 1 ✔
[cd/bc2441] SAV…GET_VERSION:PYTHON_VERSION | 1 of 1 ✔
[47/a81533] SAV…:GET_VERSION:FASTP_VERSION | 1 of 1 ✔
[71/35c04f] SAV…_VERSION:UNICYCLER_VERSION | 1 of 1 ✔
[6d/9b7ef0] SAV…ET_VERSION:SHOVILL_VERSION | 1 of 1 ✔
[65/298283] SAV…:GET_VERSION:QUAST_VERSION | 1 of 1 ✔
[fc/d0998e] SAV…FO:GET_VERSION:BWA_VERSION | 1 of 1 ✔
[5b/fa7516] SAV…T_VERSION:SAMTOOLS_VERSION | 1 of 1 ✔
[48/b695a4] SAV…T_VERSION:BCFTOOLS_VERSION | 1 of 1 ✔
[20/49fff8] SAV…ET_VERSION:POPPUNK_VERSION | 1 of 1 ✔
[c2/78c1bd] SAV…O:GET_VERSION:MLST_VERSION | 1 of 1 ✔
[54/c09422] SAV…ET_VERSION:KRAKEN2_VERSION | 1 of 1 ✔
[95/1f5eff] SAV…GET_VERSION:SEROBA_VERSION | 1 of 1 ✔
[36/0b6868] SAV…:GET_VERSION:ARIBA_VERSION | 1 of 1 ✔
[5f/20fff9] SAVE_INFO:GET_VERSION:TOOLS    | 1 of 1 ✔
[62/ca8cc8] SAV…T_VERSION:COMBINE_INFO (1) | 1 of 1 ✔
[8d/71ccdb] SAVE_INFO:PARSE (1)            | 1 of 1 ✔
[6f/edc598] SAVE_INFO:SAVE (1)             | 1 of 1 ✔

The workflow ended at 2025-01-28 11:19:33 GMT. The duration was 25m

The pipeline has been completed successfully.
The output is located at /lustre/scratch126/pam/teams/team230/wr7/repo/gps-pipeline/test_with_irods_params_6.


Completed at: 28-Jan-2025 11:19:34
Duration    : 25m 1s
CPU hours   : 15.8
Succeeded   : 147



------------------------------------------------------------
Sender: LSF System <lsfadmin@farm22-oversubscribed03>
Subject: Job 535003: <test_with_irods_params_6> in cluster <farm22> Done

Job <test_with_irods_params_6> was submitted from host <farm22-head1> by user <wr7> in cluster <farm22> at Tue Jan 28 10:54:21 2025
Job was executed on host(s) <farm22-oversubscribed03>, in queue <oversubscribed>, as user <wr7> in cluster <farm22> at Tue Jan 28 10:54:22 2025
</nfs/users/nfs_w/wr7> was used as the home directory.
</data/pam/team230/wr7/scratch/repo/gps-pipeline> was used as the working directory.
Started at Tue Jan 28 10:54:22 2025
Terminated at Tue Jan 28 11:19:36 2025
Results reported at Tue Jan 28 11:19:36 2025

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
./run_pipeline --reads test_input --manifest_of_reads manifest_of_reads.csv --runid 9870 --laneid 5 --plexid 53 --type bam --output test_with_irods_params_6 -profile singularity,lsf,sanger
------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   136.84 sec.
    Max Memory :                                 1278 MB
    Average Memory :                             1198.51 MB
    Total Requested Memory :                     4000.00 MB
    Delta Memory :                               2722.00 MB
    Max Swap :                                   -
    Max Processes :                              18
    Max Threads :                                145
    Run time :                                   1509 sec.
    Turnaround time :                            1515 sec.