[turborepo] `turbo build` hangs on linux #5908

alvesvaren · 2023-09-08T13:37:31Z

What version of Turborepo are you using?

1.10.13

What package manager are you using / does the bug impact?

pnpm

What operating system are you using?

Linux

Describe the Bug

Turbo hangs on all tasks for me, (at least for ~5 minutes, haven't waited longer than that).

I tried v1.9.5 and with that one it runs basically instantly.

Expected Behavior

Turbo tasks run as usual

To Reproduce

Create a new from the basic template
Try to run any task (turbo build)
It gets stuck

Reproduction Repo

No response

_TURBO-1398

The text was updated successfully, but these errors were encountered:

gsoltis · 2023-09-08T20:27:40Z

I was not able to reproduce this, using Ubuntu 20.04, pnpm, and [email protected]. Can you provide more details? What do the logs look like, can you run with -vvv?

All works as expected:

npx -y [email protected] my-turborepo pnpm
cd my-turborepo
pnpm build

Do you have different steps to reproduce the issue?

nathanhammond · 2023-09-08T23:59:01Z

Can you try pinning to 1.10.3?

alvesvaren · 2023-09-09T09:37:19Z

Installed 1.10.3 and tried again with -vvv:

2023-09-09T11:32:02.166+0200 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.3
2023-09-09T11:32:02.167+0200 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /home/alve/dev/project-name/node_modules/turbo-linux-64/bin/turbo
2023-09-09T11:32:02.167+0200 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /home/alve/dev/project-name/node_modules/turbo/node_modules/turbo-linux-64/bin/turbo
2023-09-09T11:32:02.167+0200 [DEBUG] turborepo_lib::shim: Repository Root: /home/alve/dev/project-name
╭────────────────────────────────────────────────────────────────────────╮
│                                                                        │
│                  Update available v1.10.3 ≫ v1.10.13                   │
│    Changelog: https://github.com/vercel/turbo/releases/tag/v1.10.13    │
│               Run "npx @turbo/codemod update" to update                │
│                                                                        │
│      Follow @turborepo for updates: https://twitter.com/turborepo      │
╰────────────────────────────────────────────────────────────────────────╯
2023-09-09T11:32:02.167+0200 [DEBUG] turborepo_lib::shim: Running command as global turbo
2023-09-09T11:32:02.167+0200 [TRACE] log: registering event source with poller: token=Token(2147483649), interests=READABLE
2023-09-09T11:32:02.168+0200 [DEBUG] turborepo_lib::cli: pkg_inference_root set to ""
2023-09-09T11:32:02.169+0200 [TRACE] turborepo_lib::execution_state: Found pnpm as package manager
2023-09-09T11:32:02.169+0200 [DEBUG] turbo: Found go binary at "/home/alve/.local/share/pnpm/global/5/.pnpm/[email protected]/node_modules/turbo/bin/go-turbo"
2023-09-09T11:32:02.169+0200 [TRACE] turbo: Invoking go binary with {"api_client_config":{"token":null,"team_id":null,"team_slug":null,"api_url":"https://vercel.com/api","use_preflight":false,"timeout":20},"package_manager":"pnpm","cli_args":{"api":null,"color":false,"cpu_profile":null,"cwd":"/home/alve/dev/project-name","heap":null,"login":null,"no_color":false,"preflight":false,"remote_cache_timeout":null,"team":null,"token":null,"trace":null,"verbosity":3,"test_run":false,"command":{"Run":{"cache_dir":null,"cache_workers":10,"concurrency":null,"continue_execution":false,"dry_run":null,"single_package":false,"filter":[],"force":null,"framework_inference":true,"global_deps":[],"graph":null,"env_mode":"Infer","ignore":[],"include_dependencies":false,"no_cache":false,"no_daemon":false,"no_deps":false,"output_logs":null,"only":false,"parallel":false,"pkg_inference_root":"","profile":null,"remote_only":false,"scope":[],"since":null,"summarize":null,"log_prefix":null,"tasks":["build"],"pass_through_args":[],"experimental_space_id":null}}}}
2023-09-09T11:32:02.173+0200 [DEBUG] turbo: build tag: rust
2023-09-09T11:32:02.196+0200 [DEBUG] turbo: running in daemon mode
2023-09-09T11:32:02.196+0200 [DEBUG] turbo: global hash env vars: vars=[]
2023-09-09T11:32:02.424+0200 [DEBUG] turbo: global hash: value=ec8aba6e703c91fa
2023-09-09T11:32:02.424+0200 [DEBUG] turbo: local cache folder: path=""
** gets stuck here for at least 5 minutes**

I can't reproduce it when creating a new repo, so it's probably something with how I've set it up (nothing weird, just a normal monorepo with a few apps/packages)

alvesvaren · 2023-09-09T09:40:02Z

turbo.json from my repository that doesn't work:

{
  "$schema": "https://turbo.build/schema.json",
  "globalDependencies": ["**/.env.*local"],
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": [".next/**", "!.next/cache/**"]
    },
    "lint": {},
    "dev": {
      "cache": false,
      "persistent": true
    },
    "db:generate": { "cache": false },
    "db:push": { "cache": false },
    "db:deploy": { "cache": false },
    "db:migrate": { "cache": false }
  }
}

I am on arch with node v18.17.1 and used pnpm i -g [email protected] to install turbo
which turbo => /home/alve/.local/share/pnpm/turbo

nathanhammond · 2023-09-11T01:49:30Z

Well, since we're here, in 1.10 we shipped better handling for .env files.

Your turbo.json should likely look like this:

{
  "$schema": "https://turbo.build/schema.json",
  "globalDotEnv": [".env"],
  "pipeline": {
    "build": {
      "dotEnv": [".env.production.local", ".env.local", ".env.production", ".env"],
      "dependsOn": ["^build"],
      "outputs": [".next/**", "!.next/cache/**"]
    },
    "lint": {},
    "dev": {
      "cache": false,
      "persistent": true
    },
    "db:generate": { "cache": false },
    "db:push": { "cache": false },
    "db:deploy": { "cache": false },
    "db:migrate": { "cache": false }
  }
}

That (shouldn't be) related to this change, but with you patiently getting us more information I'm pleased that we can at least help with one thing while looking for the other issue. 😅

nathanhammond · 2023-09-11T02:59:16Z

Actually, looking at this again, "globalDependencies": ["**/.env.*local"], is triggering a full walk of the directory. Dropping that plus moving to 1.10.3 may be a temporary workaround while we address the root cause.

alvesvaren · 2023-09-11T07:33:00Z

Tried with your json file, but it didn't seem to help, still stuck running turbo db:generate:

2023-09-11T09:28:41.257+0200 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.3
2023-09-11T09:28:41.258+0200 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /home/alve/dev/projectname/node_modules/turbo-linux-64/bin/turbo
2023-09-11T09:28:41.258+0200 [DEBUG] turborepo_lib::shim: No local turbo binary found at: /home/alve/dev/projectname/node_modules/turbo/node_modules/turbo-linux-64/bin/turbo
2023-09-11T09:28:41.258+0200 [DEBUG] turborepo_lib::shim: Repository Root: /home/alve/dev/projectname
╭────────────────────────────────────────────────────────────────────────╮
│                                                                        │
│                  Update available v1.10.3 ≫ v1.10.13                   │
│    Changelog: https://github.com/vercel/turbo/releases/tag/v1.10.13    │
│               Run "npx @turbo/codemod update" to update                │
│                                                                        │
│      Follow @turborepo for updates: https://twitter.com/turborepo      │
╰────────────────────────────────────────────────────────────────────────╯
2023-09-11T09:28:41.258+0200 [DEBUG] turborepo_lib::shim: Running command as global turbo
2023-09-11T09:28:41.258+0200 [TRACE] log: registering event source with poller: token=Token(2147483649), interests=READABLE
2023-09-11T09:28:41.259+0200 [DEBUG] turborepo_lib::cli: pkg_inference_root set to "apps/composer"
2023-09-11T09:28:41.260+0200 [TRACE] turborepo_lib::execution_state: Found pnpm as package manager
2023-09-11T09:28:41.260+0200 [DEBUG] turbo: Found go binary at "/home/alve/.local/share/pnpm/global/5/.pnpm/[email protected]/node_modules/turbo/bin/go-turbo"
2023-09-11T09:28:41.260+0200 [TRACE] turbo: Invoking go binary with {"api_client_config":{"token":null,"team_id":null,"team_slug":null,"api_url":"https://vercel.com/api","use_preflight":false,"timeout":20},"package_manager":"pnpm","cli_args":{"api":null,"color":false,"cpu_profile":null,"cwd":"/home/alve/dev/projectname","heap":null,"login":null,"no_color":false,"preflight":false,"remote_cache_timeout":null,"team":null,"token":null,"trace":null,"verbosity":3,"test_run":false,"command":{"Run":{"cache_dir":null,"cache_workers":10,"concurrency":null,"continue_execution":false,"dry_run":null,"single_package":false,"filter":[],"force":null,"framework_inference":true,"global_deps":[],"graph":null,"env_mode":"Infer","ignore":[],"include_dependencies":false,"no_cache":false,"no_daemon":true,"no_deps":false,"output_logs":null,"only":false,"parallel":false,"pkg_inference_root":"apps/web","profile":null,"remote_only":false,"scope":[],"since":null,"summarize":null,"log_prefix":null,"tasks":["db:generate"],"pass_through_args":[],"experimental_space_id":null}}}}
2023-09-11T09:28:41.263+0200 [DEBUG] turbo: build tag: rust
2023-09-11T09:28:41.285+0200 [DEBUG] turbo: Using apps/web as a basis for selecting packages
2023-09-11T09:28:41.285+0200 [DEBUG] turbo: global hash env vars: vars=[]
2023-09-11T09:28:41.285+0200 [DEBUG] turbo: global hash: value=beb986baeb38a1b8
2023-09-11T09:28:41.285+0200 [DEBUG] turbo: local cache folder: path=""

This was with 1.10.3. Also tried removing "globalDotEnv" but no difference.

alvesvaren · 2023-09-11T07:37:50Z

I also got this a few times when running the commands:

2023-09-11T09:35:30.975+0200 [DEBUG] turbo.TurbodClient: starting turbod binary /home/alve/.local/share/pnpm/global/5/.pnpm/[email protected]/node_modules/turbo/bin/turbo
2023-09-11T09:35:31.976+0200 [DEBUG] turbo.TurbodClient: starting turbod binary /home/alve/.local/share/pnpm/global/5/.pnpm/[email protected]/node_modules/turbo/bin/turbo
2023-09-11T09:35:32.976+0200 [DEBUG] turbo.TurbodClient: starting turbod binary /home/alve/.local/share/pnpm/global/5/.pnpm/[email protected]/node_modules/turbo/bin/turbo
2023-09-11T09:35:33.976+0200 [WARN]  turbo: :
  warning=
  | failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed.
  |     To quickly resolve the issue, try running:
  |     - $ turbo daemon clean
  | 
  |     To debug further - please ensure the following:
  |     - the process identified by the pid in the file at /tmp/turbod/5d0b08731af479bc/turbod.pid is not running, and remove /tmp/turbod/5d0b08731af479bc/turbod.pid
  |     - check the logs at /home/alve/.local/share/turborepo/logs/5d0b08731af479bc-semanttic-composer-mono.log
  |     - the unix domain socket at /tmp/turbod/5d0b08731af479bc/turbod.sock has been removed
  | 
  |     You can also run without the daemon process by passing --no-daemon
  
 WARNING  failed to contact turbod. Continuing in standalone mode: connection to turbo daemon process failed.
        To quickly resolve the issue, try running:
        - $ turbo daemon clean

        To debug further - please ensure the following:
        - the process identified by the pid in the file at /tmp/turbod/5d0b08731af479bc/turbod.pid is not running, and remove /tmp/turbod/5d0b08731af479bc/turbod.pid
        - check the logs at /home/alve/.local/share/turborepo/logs/5d0b08731af479bc-semanttic-composer-mono.log
        - the unix domain socket at /tmp/turbod/5d0b08731af479bc/turbod.sock has been removed

        You can also run without the daemon process by passing --no-daemon

None of the specified .pid-files exist and running turbo daemon clean doesn't fix it, however running with --no-daemon works as it did before (gets stuck)

consoleaf · 2023-09-29T08:51:25Z

Hi! I have a similar issue (Turbo just hangs), and after a manual "bisect" over recent versions, for me the issue is introduced in v1.10.2. Please let me know if I can help investigate this further if needed.

--no-daemon and --no-cache don't affect it in my case, daemon doesn't fail, it just doesn't do anything. CPU and RAM consumption is next to zero

mpereira · 2023-10-03T01:57:49Z

I'm also seeing this, on macOS.

$ ./node_modules/turbo-darwin-arm64/bin/turbo -vvv run build --filter=<app>
2023-10-03T03:53:34.790+0200 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.14
2023-10-03T03:53:34.798+0200 [DEBUG] turborepo_lib::shim: Local turbo path: <REDACTED>/node_modules/turbo-darwin-arm64/bin/turbo
2023-10-03T03:53:34.798+0200 [DEBUG] turborepo_lib::shim: Local turbo version: 1.10.14
2023-10-03T03:53:34.798+0200 [DEBUG] turborepo_lib::shim: Repository Root: <REDACTED>
2023-10-03T03:53:34.798+0200 [DEBUG] turborepo_lib::shim: Running local turbo binary in <REDACTED>/node_modules/turbo-darwin-arm64/bin/turbo

2023-10-03T03:53:34.798+0200 [DEBUG] turborepo_lib::shim: supports_skip_infer_and_single_package true
2023-10-03T03:53:34.801+0200 [DEBUG] turborepo_lib::shim: Global turbo version: 1.10.14
2023-10-03T03:53:34.804+0200 [DEBUG] turborepo_lib::cli: pkg_inference_root set to ""
2023-10-03T03:53:34.806+0200 [TRACE] turborepo_lib::execution_state: Found npm as package manager
2023-10-03T03:53:34.807+0200 [DEBUG] turbo: Found go binary at "<REDACTED>/node_modules/turbo-darwin-arm64/bin/go-turbo"
2023-10-03T03:53:34.824+0200 [TRACE] turbo: Invoking go binary with {"api_client_config":{"token":null,"team_id":null,"team_slug":null,"api_url":"https://vercel.com/api","use_preflight":false,"timeout":20},"spaces_api_client_config":{"token":null,"team_id":null,"team_slug":null,"api_url":"https://vercel.com/api","use_preflight":false,"timeout":20},"package_manager":"npm","cli_args":{"api":null,"color":false,"cpu_profile":null,"cwd":"<REDACTED>","heap":null,"login":null,"no_color":false,"preflight":false,"remote_cache_timeout":null,"team":null,"token":null,"trace":null,"verbosity":3,"test_run":false,"command":{"Run":{"cache_dir":null,"cache_workers":10,"concurrency":null,"continue_execution":false,"dry_run":null,"single_package":false,"filter":["<REDACTED>"],"force":null,"framework_inference":true,"global_deps":[],"graph":null,"env_mode":"Infer","ignore":[],"include_dependencies":false,"no_cache":false,"no_daemon":false,"no_deps":false,"output_logs":null,"log_order":"auto","only":false,"parallel":false,"pkg_inference_root":"","profile":null,"remote_only":false,"scope":[],"since":null,"summarize":null,"log_prefix":"auto","tasks":["build"],"pass_through_args":[],"experimental_space_id":null,"experimental_rust_codepath":false}}}}
2023-10-03T03:53:34.889+0200 [DEBUG] turbo: build tag: rust
2023-10-03T03:53:34.914+0200 [DEBUG] turbo: running in daemon mode
2023-10-03T03:53:34.917+0200 [DEBUG] turbo: global hash env vars: vars=[]
2023-10-03T03:53:34.926+0200 [DEBUG] turbo: global hash: value=374b21a612331788
2023-10-03T03:53:34.926+0200 [DEBUG] turbo: local cache folder: path=""
<...hangs forever...>

Update: I found the problem. There's a directory under my app directory with many PNG files. If that directory is not in .gitignore, the command hangs. Adding it to .gitignore fixes it.

gsoltis · 2023-10-04T21:06:36Z

@alvesvaren Do you have multiple workspaces with the db:generate script defined? Is there a particular workspace that exhibits the behavior when you run turbo db:generate? Or does it happen when you try it with any of them?

alvesvaren · 2023-10-10T08:47:41Z

It's just one package that has that script defined. It hangs when I run any pipeline, not just db:generate, for example when running turbo build.

We have refactored a lot and created a new turborepo, which fixed it in our project, but the old turborepo is still broken

gsoltis · 2023-10-10T20:14:40Z

Ok, I'm going to close this as it seems like you have a working setup. Happy to reopen if you've got something from the old setup that we can repro though. It sounds like there's either some large directory tree that's being traversed, or some filesystem oddity.

alvesvaren added kind: bug needs: triage owned-by: turborepo labels Sep 8, 2023

gsoltis added needs: reproduction and removed needs: triage labels Sep 8, 2023

nathanhammond mentioned this issue Sep 9, 2023

[turborepo] file hashing taking 2 minutes before failing #5773

Open

nathanhammond mentioned this issue Sep 25, 2023

[turborepo] broken input globs cause turbo to hang #6016

Closed

nathanhammond added the linear: turbo label Oct 3, 2023

gsoltis self-assigned this Oct 4, 2023

gsoltis closed this as completed Oct 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[turborepo] `turbo build` hangs on linux #5908

[turborepo] `turbo build` hangs on linux #5908

alvesvaren commented Sep 8, 2023 •

edited by padmaia

Loading

gsoltis commented Sep 8, 2023

nathanhammond commented Sep 8, 2023

alvesvaren commented Sep 9, 2023

alvesvaren commented Sep 9, 2023 •

edited

Loading

nathanhammond commented Sep 11, 2023 •

edited

Loading

nathanhammond commented Sep 11, 2023

alvesvaren commented Sep 11, 2023

alvesvaren commented Sep 11, 2023

consoleaf commented Sep 29, 2023 •

edited

Loading

mpereira commented Oct 3, 2023 •

edited

Loading

gsoltis commented Oct 4, 2023

alvesvaren commented Oct 10, 2023

gsoltis commented Oct 10, 2023

[turborepo] turbo build hangs on linux #5908

[turborepo] turbo build hangs on linux #5908

Comments

alvesvaren commented Sep 8, 2023 • edited by padmaia Loading

What version of Turborepo are you using?

What package manager are you using / does the bug impact?

What operating system are you using?

Describe the Bug

Expected Behavior

To Reproduce

Reproduction Repo

gsoltis commented Sep 8, 2023

nathanhammond commented Sep 8, 2023

alvesvaren commented Sep 9, 2023

alvesvaren commented Sep 9, 2023 • edited Loading

nathanhammond commented Sep 11, 2023 • edited Loading

nathanhammond commented Sep 11, 2023

alvesvaren commented Sep 11, 2023

alvesvaren commented Sep 11, 2023

consoleaf commented Sep 29, 2023 • edited Loading

mpereira commented Oct 3, 2023 • edited Loading

gsoltis commented Oct 4, 2023

alvesvaren commented Oct 10, 2023

gsoltis commented Oct 10, 2023

[turborepo] `turbo build` hangs on linux #5908

[turborepo] `turbo build` hangs on linux #5908

alvesvaren commented Sep 8, 2023 •

edited by padmaia

Loading

alvesvaren commented Sep 9, 2023 •

edited

Loading

nathanhammond commented Sep 11, 2023 •

edited

Loading

consoleaf commented Sep 29, 2023 •

edited

Loading

mpereira commented Oct 3, 2023 •

edited

Loading