Skip to main content
Version: 26.2

General

When working with Seqera Platform, you might encounter the following issues.

Common errors​

timeout is not an integer or out of range​

This error occurs when your Redis instance is older than the version Seqera Platform requires. Seqera Platform 24.2 to 25.3 requires Redis 6.2 or later. Seqera Platform 26.1 and later does not support Redis 6.x. To resolve, upgrade your Redis instance to a supported version according to your cloud provider's instructions. From 26.1, use Redis 7.2 or 7.4, or migrate to Valkey 7.x. See Cache layer changes for the full support matrix.

Unknown pipeline repository or missing credentials from public GitHub repositories​

GitHub imposes rate limits on repository pulls, including public repositories. GitHub caps unauthenticated requests at 60 per hour and authenticated requests at 5000 per hour. The 60-per-hour cap usually causes this error. For the same error on a private organization-owned repository, see the following entry.

To resolve:

  1. Ensure there's at least one GitHub credential in your workspace's Credentials tab.

  2. Ensure the Access token field of every GitHub credential is populated with a personal access token and not a user password. GitHub personal access tokens (PATs) are typically longer than passwords and include a ghp_ prefix. For example: ghp_IqIMNOZH6zOwIEB4T9A2g4EHMy8Ji42q4HA

  3. Confirm that your PAT provides the elevated threshold and that transactions are charged against it:

    curl -H "Authorization: token ghp_LONG_ALPHANUMERIC_PAT" -H "Accept: application/vnd.github.v3+json" https://api.github.com/rate_limit

Unknown pipeline repository or expired Git credentials from private organization-owned GitHub repositories​

GitHub organizations can restrict token access independently of your own repository access. GitHub returns 404 for private repositories that a token can't access. Seqera reports 401, 403, and 404 responses as this same error, even when the workspace credential works for user-owned repositories.

Check the following:

  1. The repository URL is exactly https://github.com/<org>/<repo>, with no .git suffix or /tree/<branch> segment.
  2. The token can access the organization's repositories:
    • Fine-grained tokens must be created with the organization as the resource owner. The organization must allow fine-grained tokens and may need to approve yours.
    • Classic tokens require the repo scope and, if the organization enforces SAML single sign-on (SSO), SSO authorization for that organization.
    • GitHub Apps must be installed on the organization with access to the repository.

To test the token stored in the workspace credential:

curl -sS -D - -o /dev/null \
  -H "Authorization: Bearer <your_access_token>" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/repos/<org>/<repo>

A 404 response means the token can't see the repository. To resolve, re-authorize or re-scope the token in GitHub. A 403 response with an X-GitHub-SSO header means the token needs SSO authorization. A 200 response means the token is valid. Check which credential the workspace selects in Multiple credential filtering.

For the same error on a public repository, see the previous entry.

Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect)​

This error occurs when incorrect configuration values are assigned to the backend and cron containers' MICRONAUT_ENVIRONMENTS environment variable. You might see other unexpected behavior, such as two exact copies of the same Nextflow job submitted to the executor for scheduling.

Verify the following:

  1. The MICRONAUT_ENVIRONMENTS environment variable associated with the backend container:
    • Contains prod,redis,ha
    • Does not contain cron
  2. The MICRONAUT_ENVIRONMENTS environment variable associated with the cron container:
    • Contains prod,redis,cron
    • Does not contain ha
  3. You don't have another copy of the MICRONAUT_ENVIRONMENTS environment variable defined elsewhere in your application (such as a tower.env file or Kubernetes ConfigMap).
  4. If you're using a separate container/pod to execute migrate-db.sh, ensure there's no MICRONAUT_ENVIRONMENTS environment variable assigned to it.

No such variable​

This error occurs when you execute a DSL1-based Nextflow workflow with Nextflow 22.03.0-edge or later.

"<parameter>" must be string when launching a pipeline​

This error occurs when a parameter nested inside an object-typed schema group has a null value. Seqera Platform tolerates null and blank values for top-level parameters only. The error message concatenates the group and parameter names. Omit optional nested parameters instead of setting them to null:

# Fails validation
alignment:
  aligner: bwa
  reference: null

# Passes validation
alignment:
  aligner: bwa

Nextflow resolves a missing key to null at runtime. Your pipeline logic behaves the same. For more information, see Pipeline schema.

Sleep commands in Nextflow workflows​

The behavior of sleep commands in your Nextflow workflows depends on where they are used:

  • In an errorStrategy block, Nextflow uses the Groovy sleep function, which takes its value in milliseconds.
  • In a process script block, that language's sleep binary or method is used. For example, this bash script uses the bash sleep binary, which takes its value in seconds.

Limit of N running workflows reached​

This error occurs at launch when your organization reaches its quota of concurrent active pipeline runs. All runs in SUBMITTED or RUNNING status across the organization's workspaces count toward the quota. Your Seqera license sets the limit. The related error Organizational quota limit fully restricts usage of running workflows means the quota blocks all launches in the organization.

To resolve:

  1. Cancel any runs stuck in SUBMITTED status. Active runs occupy quota until they complete or you cancel them.
  2. Contact Seqera support to raise the quota.

Seqera Platform retrieves license changes when it next polls the license server, every 24 hours by default.

Containers​

Use rootless containers in Nextflow pipelines​

Most containers use the root user by default. Some users prefer a non-root user in the container to minimize the risk of privilege escalation. Because Nextflow and its tasks use a shared work directory to manage input and output data, rootless containers can cause file permission errors in some environments:

touch: cannot touch '/fsx/work/ab/27d78d2b9b17ee895b88fcee794226/.command.begin': Permission denied

This should not occur with AWS Batch from Seqera version 22.1.0. In other cases, force all task containers to run as root. Add one of the following to your Nextflow configuration:

// cloud executors
process.containerOptions = "--user 0:0"

// Kubernetes
k8s.securityContext = [
  "runAsUser": 0,
  "runAsGroup": 0
]

Databases​

Database connection failure in Seqera Enterprise 22.2.0​

Seqera Enterprise 22.2.0 introduced a breaking change: TOWER_DB_DRIVER must now be org.mariadb.jdbc.Driver.

If you use Amazon Aurora as your database, you might encounter a java.sql.SQLNonTransientConnectionException: ... could not load system variables error, likely because of a known error tracked in the MariaDB project.

To resolve, modify the Seqera Enterprise configuration:

  1. Ensure your TOWER_DB_DRIVER uses the specified MariaDB URI.
  2. Modify your TOWER_DB_URL to: TOWER_DB_URL=jdbc:mysql://<domain>:<port>/<database-name>?usePipelineAuth=false&useBatchMultiSend=false

java.sql.SQLException time zone errors on login​

After login authentication, Seqera presents an Unexpected error while processing error, with java.sql.SQLException errors related to the server time zone in the backend log:

Error log
io.micronaut.transaction.exceptions.CannotCreateTransactionException: Could not open Hibernate Session for transaction
…
Caused by: org.hibernate.exception.GenericJDBCException: Unable to acquire JDBC Connection
…
java.sql.SQLException: The server time zone value 'CEST' is unrecognized or represents more than one time zone. You must configure either the server or JDBC driver (via the 'serverTimezone' configuration property) to use a more specific time zone value if you want to utilize time zone support.
…

Seqera can't connect to the database because the JDBC client doesn't specify a time zone. Set it with the serverTimezone property.

To resolve, append serverTimezone to TOWER_DB_URL. For the Europe/Amsterdam time zone:

export TOWER_DB_URL="jdbc:mysql://<database-ip>:3306/tower?permitMysqlScheme=true&serverTimezone=Europe/Amsterdam"

java.io.IOException: Unsupported protocol version 252​

When a service is restarted or otherwise interrupted, it can create invalid entries that corrupt your installation's Redis cache. Completed or terminated runs then display as in progress. To resolve, delete the key with the invalid entry (replace <container-name> with your container name):

## Check if the key exists
docker exec -ti <container-name> redis-cli keys \* | grep workflow

## Show the hash contents of the key
docker exec -ti <container-name> redis-cli hgetall "workflow/modified"

## Delete the key
docker exec -ti <container-name> redis-cli del "workflow/modified"

Email and TLS​

TLS errors​

Nextflow and Seqera Platform can both interact with email providers on your behalf. These providers often require TLS connections, many now requiring at least TLSv1.2.

TLS connection errors can occur because of variability in the default TLS version specified by your JDK distribution. If you encounter any of the following errors, there is likely a mismatch between your default TLS version and what the email provider supports:

  • Unexpected error sending mail ... TLS 1.0 and 1.1 are not supported. Please upgrade/update your client to support TLS 1.2
  • ERROR nextflow.script.WorkflowMetadata - Failed to invoke 'workflow.onComplete' event handler ... javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)

To resolve:

  1. Set a JDK environment variable to force Nextflow and Seqera containers to use TLSv1.2 by default:

    export JAVA_OPTS="-Dmail.smtp.ssl.protocols=TLSv1.2"

  2. Add this parameter to your nextflow.config file:

    mail {
        smtp.ssl.protocols = 'TLSv1.2'
    }

  3. Ensure these values are also set for Nextflow and Seqera:

    • mail.smtp.starttls.enable=true
    • mail.smtp.starttls.required=true

Git integration​

Get branches operation not supported by BitbucketServerRepositoryProvider provider​

If you supplied the correct Bitbucket credentials and URL details in your tower.yml and still see this error, upgrade to at least v22.3.0. This version addresses SCM provider authentication issues and likely resolves the retrieval failure.

Cannot invoke "String.toCharArray()" because "password" is null​

This error occurs when a GitLab credential has an Access token but no Password. When cloning the repository, Nextflow releases before 26.04.0 authenticate with the username and password only and ignore the access token.

To resolve, enter your token value in both the Password and Access token fields of your GitLab credential. Nextflow accepts the access token without a Password from 26.04.0 onward.

Healthcheck​

Seqera Platform API healthcheck endpoint​

To implement automated healthcheck functionality, use Seqera's service-info endpoint. For example:

curl -o /dev/null -s -w "%{http_code}\n" --connect-timeout 2  "https://api.cloud.seqera.io/service-info"  -H "Accept: application/json"
200

Login​

Login fails: screen frozen at /auth?success=true​

From version 22.1, Seqera Enterprise implements stricter cookie security by default and only sends an auth cookie if the client is connected over HTTPS. Login attempts over HTTP fail by default.

To resolve, set the environment variable TOWER_ENABLE_UNSAFE_MODE=true to allow HTTP connectivity to Seqera (not recommended for production environments).

Restrict Seqera access to a set of email addresses​

Removing the email section from the login page is not currently supported. You can, however, restrict which email identities can log in to your Seqera Enterprise instance with the trustedEmails configuration parameter in your tower.yml file:

# tower.yml
tower:
  trustedEmails:
    # Any email address pattern which matches will have automatic access.
    - '*@seqera.io'
    - 'named_user@example.com'

    # Alternatively, specify a single entry to deny access to all other emails.
    - 'fake_email_address_which_cannot_be_accessed@your_domain.org'

Users with email addresses outside the trustedEmails list undergo an approval process on the Profile > Admin > Users page. This is an effective backup method when SSO becomes unavailable.

note
  1. You must rebuild your containers (docker compose down) to force Seqera to implement this change. Ensure your database is persistent before you issue the teardown command. See Docker Compose for more information.
  2. All login attempts are visible to the root user at Profile > Admin panel > Users.
  3. Any user logged in before the restriction is not subject to the new restriction. An organization admin should remove users that previously logged in with an untrusted email from the Admin panel users list. This restarts the approval process before they can log in by email.

Login fails: admin approval required with Entra ID OIDC​

The Entra ID app integrated with Seqera must have user consent settings configured to "Allow user consent for apps" so that admin approval is not required for each application login. See User consent settings.

Username and Password not accepted with Google SMTP​

Seqera Enterprise email integration with Google SMTP can fail as of May 30, 2022, because of a security posture change by Google.

To re-establish email connectivity, follow these instructions to provision an app password. Update your TOWER_SMTP_PASSWORD environment variable with the app password, then restart the application.

Logging​

Broken Nextflow log file in v22.3.1​

A Seqera Launcher issue affects the Nextflow log file download in version 22.3.1. Version 22.3.2 fixes it. Update to version 22.3.2 or later.

Miscellaneous​

Maximum parallel Seqera browser tabs​

Because of a limitation in server-side event technology in HTTP/1.1, up to five tabs can be open simultaneously per browser product. Additional tabs remain stuck in a loading state.

Monitoring​

Integrate third-party Java Application Performance Monitoring (APM) solutions​

Mount the APM solution's JAR file in Seqera's backend container and set the agent JVM option through the JAVA_OPTS environment variable.

Retrieve the trace logs for a workflow run​

You can't download the trace logs directly through Seqera, but you can configure your workflow to export the file to persistent storage:

  1. Set this block in your nextflow.config:

    trace {
        enabled = true
    }

  2. Add a copy command to your pipeline's Advanced options > Post-run script field:

    aws s3 cp ./trace.txt s3://<bucket>/trace/trace.txt

Seqera Platform intermittently reports Live events sync offline​

Seqera Platform uses server-sent events to push real-time updates to your browser. The client must connect to the server's /api/live endpoint to start the data stream, and this connection can occasionally fail because of factors like network latency.

To resolve, reload the Platform browser tab to re-establish the client's connection to the server. If reloading fails, contact Seqera support for help adjusting webserver timeout settings.

Networking​

503 errors during pipeline execution​

A 503 error indicates that one or more services that Seqera Enterprise contacts during workflow execution are unavailable. Database connectivity is a common cause.

To resolve, ensure all required services are running and available.

SocketTimeoutException: connect timed out with self-hosted Git servers​

You might see connection timeout errors when launching workflows from a self-hosted Git server, such as Bitbucket or GitLab. If you configured the correct Git credentials in Seqera Enterprise, this error means the backend/cron container can't connect to the Git remote host, often because of a missing or incorrect proxy configuration.

Error log

ERROR i.s.t.c.GlobalErrorController - Unexpected error while processing - Error ID: 6h3HBUkaPe03vgzoDPc5HO
java.net.SocketTimeoutException: connect timed out
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.base/java.net.Socket.connect(Socket.java:609)
        at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:289)
        at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
        at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
        at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
        at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:203)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:189)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
        at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:334)
        at nextflow.scm.RepositoryProvider.checkResponse(RepositoryProvider.groovy:167)
        at nextflow.scm.RepositoryProvider.invoke(RepositoryProvider.groovy:136)
        at nextflow.scm.RepositoryProvider.memoizedMethodPriv$invokeAndParseResponseString(RepositoryProvider.groovy:218)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1259)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
        at nextflow.scm.RepositoryProvider$_closure2.doCall(RepositoryProvider.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at groovy.lang.Closure.call(Closure.java:412)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at groovy.lang.Closure.call(Closure.java:428)
        at nextflow.scm.RepositoryProvider.invokeAndParseResponse(RepositoryProvider.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:193)
        at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:61)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:185)
        at nextflow.scm.BitbucketRepositoryProvider.getCloneUrl(BitbucketRepositoryProvider.groovy:114)
        at nextflow.scm.AssetManager.memoizedMethodPriv$getGitRepositoryUrl(AssetManager.groovy:394)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1259)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1029)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1012)
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:101)
        at nextflow.scm.AssetManager$_closure1.doCall(AssetManager.groovy)
        at nextflow.scm.AssetManager$_closure1.doCall(AssetManager.groovy)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1026)
        at groovy.lang.Closure.call(Closure.java:412)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
        at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
        at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
        at groovy.lang.Closure.call(Closure.java:406)
        at nextflow.scm.AssetManager.getGitRepositoryUrl(AssetManager.groovy)

To resolve, update the HTTP proxy configuration in the backend and cron environment with your proxy details:

export http_proxy="http://<proxy-server>:<port>"
export https_proxy="https://<proxy-server>:<port>"

Optimization​

OutOfMemoryError: Container killed due to memory usage​

Nextflow can underestimate the memory allocation for containerized tasks. As a workaround, add a retry error strategy to the failing process that increases the allocated memory on each retry:

process {
    errorStrategy = 'retry'
    maxRetries    = 3
    memory = { 1.GB * task.attempt }
}

Plugins​

Use the Nextflow SQL DB plugin to query AWS Athena​

From Nextflow 22.05.0-edge, your Nextflow pipelines can query data from AWS Athena. Add these items to your nextflow.config. Secrets are optional:

plugins {
  id 'nf-sqldb@0.4.0'
}

sql {
    db {
        'athena' {
              url = 'jdbc:awsathena://AwsRegion=<region>;S3OutputLocation=s3://<s3-bucket>'
              user = secrets.ATHENA_USER
              password = secrets.ATHENA_PASSWORD
            }
    }
}

Then call the functionality in your workflow:

channel.sql.fromQuery("select * from test", db: "athena", emitColumns:true).view()
note

This example uses the legacy nf-sqldb@0.4.0 syntax. Newer plugin versions use an explicit include { fromQuery } from 'plugin/nf-sqldb' statement instead. See the nf-sqldb documentation.

See the nf-sqldb discussion for more information.

Repositories​

Private Docker registry integration​

Seqera-invoked jobs can pull container images from private Docker registries, such as JFrog Artifactory. The method depends on your computing platform.

For AWS Batch, modify your EC2 launch template using these AWS instructions.

note

This solution requires Docker Engine 17.07 or later to use --password-stdin.

You might need to add commands to your launch template, depending on your security posture:

cp /root/.docker/config.json /home/ec2-user/.docker/config.json && chmod 777 /home/ec2-user/.docker/config.json

For Azure Batch, create a Container registry-type credential in your Seqera workspace and associate it with the Azure Batch compute environment in the same workspace.

For Kubernetes, use an imagePullSecret, per #2827.

Remote resource not found​

This error occurs when the Nextflow head job fails to retrieve the repository credentials from Seqera. If your Nextflow log contains an entry like DEBUG nextflow.scm.RepositoryProvider - Request [credentials -:-], check the protocol of your instance's TOWER_SERVER_URL value. It must be set to https rather than http, unless you use TOWER_ENABLE_UNSAFE_MODE to allow HTTP connections to Seqera in a test environment.

Secrets​

Missing AWS execution role arn during launch​

The ECS agent must have access to retrieve secrets from AWS Secrets Manager. Secrets-using pipelines launched in an AWS Batch compute environment encounter this error when an IAM execution role is not provided. See Secrets.

AWS Batch task failures with secrets​

You might encounter errors when executing pipelines that use secrets on AWS Batch:

  • If you use nf-sqldb version 0.4.1 or earlier and have secrets in your nextflow.config, you might see nextflow.secret.MissingSecretException: Unknown config secret errors in your Nextflow log.

To resolve, explicitly define the xpack-amzn plugin in your configuration:

plugins {
  id 'xpack-amzn'
  id 'nf-sqldb'
}

  • If you have two or more processes that use the same container image but only some of them use secrets, your secret-using processes might fail during the initial run and then succeed when resumed. This is caused by a bug in how Nextflow (22.07.1-edge and earlier) registers jobs with AWS Batch.

    To resolve, upgrade Nextflow to version 22.08.0-edge or later. If you can't upgrade, use one of these workarounds:

    • Use a different container image for each process.
    • Define the same set of secrets in each process that uses the same container image.

Tower Agent​

Unexpected Exception in WebSocket … Operation timed out​

Tower Agent reconnection logic was improved in version 0.5.0. Update your Tower Agent before relaunching your pipeline.

Reattach to a running agent​

When you SSH back to the login node, attach to the agent session at any time:

tmux attach -t tower-agent

You can see the current log output. Detach again with Ctrl-b, then d, to leave the agent running.

Agent process stopped​

If tmux ls shows no sessions, or attaching reveals the agent has exited, restart it as in Tower Agent setup. Common causes are a login node reboot, a process killed for exceeding login-node resource limits, or a revoked access token.

Agent shows as disconnected in Seqera Platform​

If Seqera Platform shows the agent as disconnected while it's running on the cluster, verify that the Agent Connection ID in your workspace credential exactly matches the argument you passed to tw-agent.

Timeout waiting for command response when adding a pipeline​

Before version 26.2, adding a pipeline to an agent-backed compute environment could fail intermittently with this error, even when the agent was running and connected:

ERROR: Timeout waiting for command response - Check Tower Agent is running and connected

The failure was more likely for file:// pipeline repositories, pipelines with several includeConfig statements, and concurrent adds. The agent answered every command, but Seqera Platform dropped some responses before they reached the request. You don't need to investigate the agent. Raising tower.agent.execution-timeout doesn't help.

Version 26.2 fixes the issue. On earlier versions, retry the failed add. Because the failure isn't deterministic, a retry usually succeeds.

Authentication errors on agent startup​

Personal access tokens can be revoked or expire. If the agent logs authentication errors, generate a new token in Seqera Platform and restart the agent with the updated TOWER_ACCESS_TOKEN value.

Permission denied on the work directory​

The agent needs read and write access to the work directory. If launches fail with permission errors, confirm that the directory exists and is owned by the user running the agent:

mkdir -p ~/work

Enable trace logging​

To diagnose connection or execution issues in detail, enable trace-level logging:

export TOWER_ACCESS_TOKEN=<YOUR TOKEN>
export LOGGER_LEVELS_IO_SEQERA_TOWER_AGENT=TRACE
./tw-agent <YOUR CONNECTION ID>

Trace logging shows WebSocket connection details, message exchanges, reconnection attempts, command execution details and exit codes, and full stack traces for errors.

Google​

Spot VM preemption causes task interruptions​

Spot VMs reduce cost but increase the likelihood that a task is interrupted before completion. When Google Cloud reclaims a Spot VM, Google Cloud Batch terminates the task with exit code 50001. Add a retry strategy to your Nextflow configuration so that Nextflow automatically re-executes interrupted tasks. See Spot Instances in the Nextflow documentation. For example:

process {
  errorStrategy = { task.exitStatus == 50001 ? 'retry' : 'finish' }
  maxRetries    = 5
}

Seqera service account permissions for Google Cloud Batch​

Grant the following roles to the custom service account that submits Batch jobs:

  • Batch Agent Reporter (roles/batch.agentReporter)
  • Batch Job Editor (roles/batch.jobsEditor)
  • Logs Writer (roles/logging.logWriter)
  • Logs Viewer (roles/logging.logViewer)
  • Service Account User (roles/iam.serviceAccountUser)
  • Storage Admin (roles/storage.admin), or bucket-level Storage access

For detailed setup instructions, see Service account permissions.

Kubernetes​

Invalid value: "xxx": must be less or equal to memory limit​

This error can occur when you specify a value in the Head Job memory field while creating a Kubernetes-type compute environment.

If you receive an error that includes field: spec.containers[x].resources.requests and message: Invalid value: "xxx": must be less than or equal to memory limit, your Kubernetes cluster might be configured with system resource limits that deny the Nextflow head job's resource request. To isolate the component causing the problem, launch a pod directly on your cluster through your Kubernetes administration solution. For example:

---
apiVersion: v1
kind: Pod
metadata:
  name: debug
  labels:
    app: debug
spec:
  containers:
    - name: debug
      image: busybox
      command: ["sh", "-c", "sleep 10"]
      resources:
        requests:
          memory: "xxxMi" # or "xxxGi"
  restartPolicy: Never

On-premises HPC​

java: command not found​

When submitting jobs to your on-premises HPC (using either SSH or Tower Agent authentication), the following error might appear in your Nextflow logs, even with Java on your PATH environment variable:

java: command not found
Nextflow is trying to use the Java VM defined for the following environment variables:
  JAVA_CMD: java
  NXF_OPTS:

Possible causes:

  1. The queue where the Nextflow head job runs is in a different environment or node than your login node userspace.
  2. If your HPC cluster uses modules, the Java module might not be loaded by default.

To troubleshoot:

  1. Open an interactive session with the head job queue.
  2. Launch the Nextflow job from the interactive session.
  3. If your cluster uses modules, add module load <java-module> in the Advanced options > Pre-run script field when creating your HPC compute environment in Seqera.
  4. If your cluster doesn't use modules, source an environment with Java and Nextflow in the Advanced options > Pre-run script field when creating your HPC compute environment in Seqera.

Pipeline submissions to HPC clusters fail for some users​

Nextflow launcher scripts fail if processed by a non-Bash shell, such as zsh or tcsh. You can identify this problem from these error entries:

  1. Your .nextflow.log contains an error like Invalid workflow status - expected: SUBMITTED; current: FAILED.
  2. Your Seqera Error report tab contains an error like:
Slurm job submission failed
- command: mkdir -p /home//\<username\>//scratch; cd /home//\<username\>//scratch; echo <long-base64-string> | base64 -d > nf-<run-id>.launcher.sh; sbatch ./nf-<run-id>.launcher.sh
- exit   : 1
- message: Submitted batch job <#>

Connect to the head node over SSH and run ps -p $$ to verify your default shell. If you see an entry other than Bash, fix it as follows:

  1. Check which shells are available: cat /etc/shells
  2. Change your shell: chsh -s /usr/bin/bash (the path to the binary might differ, depending on your HPC configuration).
  3. If submissions continue to fail after the shell change, ask your Seqera Platform admin to restart the backend and cron containers, then submit again.

Execution logs don't update in real time for HPC compute environments​

While a task runs on an HPC compute environment (such as Slurm, Grid Engine, LSF, or PBS Pro), the Execution log tab on the run details page does not refresh automatically.

This is expected behavior. Real-time log streaming is supported only for compute environments that stream logs from a cloud logging service: AWS Batch, Azure Batch, Google Cloud Batch, Kubernetes, and the AWS Cloud and Azure Cloud environments. For HPC compute environments, Seqera Platform retrieves the task log from the task work directory (the task's .command.log file) instead of streaming it.

To load the latest log content, change tabs or refresh the page.

Other run details, such as run status, task counters, and metrics, update in real time regardless of the compute environment type.

On this Page