Pfthb

I am trying to take the GIAB data index files (which are CSVs), and download each file in Nextflow. I think I have the general structure right, but when I run nextflow run file.nf nothing happens.

Channel.fromPath(file('https://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/NA12878/sequence.index.NA12878_Illumina_HiSeq_Exome_Garvan_trimmed_fastq_09252015'))
 .splitCsv(header: true)
 .map it.FASTQ 
 .set giab_urls 


process download_giab 
 storeDir 'giab'

 input:
 file giab_url from giab_urls

 output:
 file '*.fastq' into giab_fastqs

 script:
 """
 lftp -c 'get $giab_url'
 """

The log file produced is as follows:

Nov-13 18:18:43.537 [main] DEBUG nextflow.cli.Launcher - $> /opt/miniconda3/bin/nextflow run main.nf
Nov-13 18:18:43.653 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 18.10.1
Nov-13 18:18:43.661 [main] INFO nextflow.cli.CmdRun - Launching `main.nf` [agitated_cori] - revision: 5cf3310536
Nov-13 18:18:43.757 [main] DEBUG nextflow.Session - Session uuid: c19f86b4-0eff-43de-8ad4-cb7936701490
Nov-13 18:18:43.758 [main] DEBUG nextflow.Session - Run name: agitated_cori
Nov-13 18:18:43.759 [main] DEBUG nextflow.Session - Executor pool size: 4
Nov-13 18:18:43.769 [main] DEBUG nextflow.cli.CmdRun - 
 Version: 18.10.1 build 5003
 Modified: 24-10-2018 14:03 UTC (25-10-2018 01:03 AEDT)
 System: Linux 4.15.0-38-generic
 Runtime: Groovy 2.5.3 on OpenJDK 64-Bit Server VM 1.8.0_181-8u181-b13-1ubuntu0.18.04.1-b13
 Encoding: UTF-8 (UTF-8)
 Process: 8747@michael-Latitude-7480 [127.0.1.1]
 CPUs: 4 - Mem: 23.4 GB (1.9 GB) - Swap: 2 GB (2 GB)
Nov-13 18:18:43.832 [main] DEBUG nextflow.Session - Work-dir: /home/michael/Programming/CromwellValidation/work [ext2/ext3]
Nov-13 18:18:43.832 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /home/michael/Programming/CromwellValidation/bin
Nov-13 18:18:43.904 [main] DEBUG nextflow.Session - Session start invoked
Nov-13 18:18:43.911 [main] DEBUG nextflow.processor.TaskDispatcher - Dispatcher > start
Nov-13 18:18:43.911 [main] DEBUG nextflow.script.ScriptRunner - > Script parsing
Nov-13 18:18:44.244 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Nov-13 18:18:44.586 [main] DEBUG nextflow.processor.ProcessFactory - << taskConfig executor: null
Nov-13 18:18:44.586 [main] DEBUG nextflow.processor.ProcessFactory - >> processorType: 'local'
Nov-13 18:18:44.593 [main] DEBUG nextflow.executor.Executor - Initializing executor: local
Nov-13 18:18:44.596 [main] INFO nextflow.executor.Executor - [warm up] executor > local
Nov-13 18:18:44.600 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=4; memory=23.4 GB; capacity=4; pollInterval=100ms; dumpInterval=5m
Nov-13 18:18:44.604 [main] DEBUG nextflow.processor.TaskDispatcher - Starting monitor: LocalPollingMonitor
Nov-13 18:18:44.605 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Nov-13 18:18:44.616 [main] DEBUG nextflow.executor.Executor - Invoke register for executor: local
Nov-13 18:18:44.672 [main] DEBUG nextflow.Session - >>> barrier register (process: download_giab)
Nov-13 18:18:44.676 [main] DEBUG nextflow.processor.TaskProcessor - Creating operator > download_giab -- maxForks: 4
Nov-13 18:18:44.736 [main] DEBUG nextflow.script.ScriptRunner - > Await termination 
Nov-13 18:18:44.736 [main] DEBUG nextflow.Session - Session await
Nov-13 18:18:44.758 [Actor Thread 3] DEBUG nextflow.Session - <<< barrier arrive (process: download_giab)
Nov-13 18:18:44.759 [main] DEBUG nextflow.Session - Session await > all process finished
Nov-13 18:18:44.813 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local)
Nov-13 18:18:44.813 [main] DEBUG nextflow.Session - Session await > all barriers passed
Nov-13 18:18:44.818 [main] DEBUG nextflow.trace.StatsObserver - Workflow completed > WorkflowStats[succeedCount=0; failedCount=0; ignoredCount=0; cachedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms]
Nov-13 18:18:44.826 [main] DEBUG nextflow.CacheDB - Closing CacheDB done
Nov-13 18:18:44.842 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Any ideas what I'm doing wrong here? None of the nextflow output is very enlightening.

It's need to map the fastq path string to a file object using the file function e.g.:

Channel.fromPath('https://raw.githubusercontent.com/genome-in-a-bottle/giab_data_indexes/master/NA12878/sequence.index.NA12878_Illumina_HiSeq_Exome_Garvan_trimmed_fastq_09252015')
 .splitCsv(header: true, sep:'t')
 .map file(it.FASTQ) 
 .set giab_urls 

Note also, you need to specify the sep option to handle TAB separated files and the file function is not needed when passing the url to the fromPath method.

You can find a description of this use case here.