Skip to content

Current Limitations ⚠️

  • Latest Funnel release (v0.11.11) requires specific bucket prefixing in the inputs and outputs
  • e.g. /mnt/disks/<BUCKET>/<FILE> instead of /<FILE>
  • Nextflow workflows are currently not supported (as Nextflow expects root-level working directories → /work)

Overview

The following steps illustrate how to run a TES tasks via GCP Batch utilizing Google Storage Buckets.

Quick Start

curl -fsSL https://calypr.github.io/funnel/install.sh | bash

funnel server run --Compute "gcp-batch"                \
                  --GCPBatch.Project "example-project" \
                  --GCPBatch.Location "us-central1"

3. Submit Task

gcp-example.json 
{
  "name": "Input/Output Test",
  "inputs": [
    {
      "url": "gs://tes-batch-integration/README.md",
      "path": "/mnt/disks/tes-batch-integration/README.md"
    }
  ],
  "outputs": [
    {
      "url": "gs://tes-batch-integration/README.md.sha256",
      "path": "/mnt/disks/tes-batch-integration/README.md.sha256"
    }
  ],
  "executors": [
    {
      "image": "alpine",
      "command": [
        "sha256sum",
        "/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"
      ]
    }
  ]
}

funnel task create gcp-example.json
<TASK ID>

4. Query Task

funnel task get <TASK ID>

{
  "executors": [
    {
      "command": [
        "sha256sum",
        "/mnt/disks/tes-batch-integration/README.md | tee /mnt/disks/tes-batch-integration/README.md.sha256"
      ],
      "image": "alpine"
    }
  ],
  "id": "d6f0tgpurbu7o23pgj20",
  "inputs": [
    {
      "path": "/mnt/disks/tes-batch-integration/README.md",
      "url": "gs://tes-batch-integration/README.md"
    }
  ],
  "name": "GCP Batch Task Example",
  "outputs": [
    {
      "path": "/mnt/disks/tes-batch-integration/README.md.sha256",
      "url": "gs://tes-batch-integration/README.md.sha256"
    }
  ],
  "state": "COMPLETE"
}

5. Verify Outputs

gsutil cat gs://tes-batch-integration/README.md.sha256
9b9916cea5348edd6ad78893231edb81fc96772d1dd99fae9c2a64f84646cb1c  /mnt/disks/tes-batch-integration/README.md

Additional Resources