Frequent build errors on Netlify #204

marcoSven · 2020-10-30T18:45:21Z

I experience intermittently build errors. They do not appear to be related to CMS content changes or code changes, it just randomly, unfortunately frequently, fails.

The following changes did not improve or worsen the fail rate - I think
I removed netlify-plugin-gatsby-cache - #44
I removed gatsby-rehype-inline-images - #202

Clearing the cache before a deploy is not helping in my opinion.

The error is always something like this
...
11:17:15 AM: success run static queries - 0.204s - 16/16 78.48/s
11:17:18 AM: error [object Object] ENOENT: no such file or directory, unlink '/opt/build/repo/public/static/c79ed69a45dfa3eb3fa94442cc61454a/photo-1499479387933-4567e710809c.jpg'
11:17:18 AM:
11:17:18 AM:
11:17:18 AM: Error: ENOENT: no such file or directory, unlink '/opt/build/repo/public/stati c/c79ed69a45dfa3eb3fa94442cc61454a/photo-1499479387933-4567e710809c.jpg'
11:17:18 AM:
11:17:18 AM: not finished Generating image thumbnails - 13.367s
11:17:18 AM: not finished run page queries - 2.574s
...
11:17:18 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2
11:17:18 AM: Finished saving boot dependencies
11:17:18 AM: Started saving go dependencies
11:17:18 AM: Finished saving go dependencies
11:17:18 AM: Build failed due to a user error: Build script returned non-zero exit code: 2
11:17:18 AM: Failing build: Failed to build site
11:17:19 AM: Finished processing build request in 47.137682029s

marcoSven · 2020-10-30T18:50:12Z

here is another one

11:47:09 AM: success run static queries - 0.106s - 16/16 150.38/s
11:47:13 AM: error [object Object] ENOENT: no such file or directory, unlink '/opt/build/repo/public/static/be02825633540f618f4010844e830d2f/photo-1541337326331-c1bd6156f6f6.jpg'
11:47:13 AM:
11:47:13 AM:
11:47:13 AM: Error: ENOENT: no such file or directory, unlink '/opt/build/repo/public/stati c/be02825633540f618f4010844e830d2f/photo-1541337326331-c1bd6156f6f6.jpg'
11:47:13 AM:
11:47:13 AM: not finished Generating image thumbnails - 14.152s
11:47:13 AM: not finished run page queries - 3.623s
11:47:13 AM: (sharp:1364): GLib-CRITICAL **: 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:47:13 AM: (sharp:1364): GLib-CRITICAL **: 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:47:13 AM: (sharp:1364): GLib-CRITICAL **: 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:47:13 AM: (sharp:1364): GLib-CRITICAL **: 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:47:13 AM: (sharp:1364): GLib-CRITICAL **: 18:47:13.145: g_hash_table_insert_internal: assertion 'hash_table != NULL' failed
11:47:13 AM: (sharp:1364): GLib-CRITICAL **: 18:47:13.145: g_hash_table_lookup: assertion 'hash_table != NULL' failed
11:47:16 AM: bash: line 1: 1364 Segmentation fault (core dumped) gatsby build --log-pages --verbose
11:47:16 AM:
11:47:16 AM: ┌─────────────────────────────┐
11:47:16 AM: │ "build.command" failed │
11:47:16 AM: └─────────────────────────────┘
...
11:47:16 AM: Finished saving go dependencies
11:47:16 AM: Build failed due to a user error: Build script returned non-zero exit code: 2
11:47:16 AM: Failing build: Failed to build site
11:47:16 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2
11:47:16 AM: Finished processing build request in 48.689597638s

styxlab · 2020-10-30T21:32:52Z

The sharp errors look scary but can be ignored (there is an async issue in the gatsby-plugin-sharp plugin, but the images are still generated). The other build errors, however, should not happen and I am surprised to see those given that I also deploy on Netlify without problems. Could you please first test, if you also see those errors when building locally? Maybe it is an issue with the volume, how many images are you processing?

marcoSven · 2020-10-30T23:25:49Z

Funny I just had the first time the same error on my local.

yarn clean
yarn build

I looked for the file in demo/public/static and it did exist. I ran yarn build and everything was fine 🙃. Any advice on what to look for? I am just starting out with Gatsby,

wondering if

this is related to the async issue in the gatsby-plugin-sharp, though I do not see the error every time the build fails.
memory issue

how many images are you processing?

3.406837832 Number of node types: 16. Nodes per type: SitePage: 41, SitePlugin: 64, Site: 1, SiteBuildMetadata: 1, Directory: 3, File: 32, ImageSharp: 28, GhostTag: 14, GhostSettings: 1, GhostAuthor: 1, GhostPage: 4, HtmlRehype: 15, GhostPost: 11, ghostConfig: 1, ContactPage: 1, TopicsPage: 1

marcoSven · 2020-10-31T04:42:59Z

So I signed up for Gatsby, I made 3 deploys so far, and they always succeded in Gatsby but failed on Netlify.

I experienced the issues once more on my local, so I am not 100% confident that building on Gatsby solves my issue.

styxlab · 2020-10-31T09:33:04Z

I also build on Gatsby Cloud and not directly on Netlify, but I am also not seeing the issue locally. Could you please provide your repo with (temporary) API keys, so I can test your site locally on my machine? This is most likely either a problem with the used Gatsby version or related to your (local/Netlify) build system.

Apart from clearing the cache with yarn clean, did you also

remove your public folder before building?
remove node_modules + yarn.lock, then issue yarn to recreate?

The error messages points to dangling cache files.

marcoSven · 2020-10-31T13:26:42Z

I also build on Gatsby Cloud and not directly on Netlify

Not sure if I like that dependency 😄 but it works flawlessly so far. I am currently building simultaneously on both and Netlify always fails. I have removed the lock file btw and the build installs the packages with npm. I use yarn locally.

remove your public folder before building?
remove node_modules + yarn.lock, then issue yarn to recreate?

I tried a lot of things replicated the same steps after a fail, but since the build time takes a while this is a painful time suck.
So far I have not been able to consistently reproduce the issue on my local. The public folder is removed when you run clean, no?

This is most likely either a problem with the used Gatsby version or related to your (local/Netlify) build system.

Local - Gatsby CLI version: 2.12.107 node v12.19.0
Netlify - [email protected] node v12.18.0

Gatsby - Gatsby CLI version: 2.12.113 Node v12.19.0

I still think it's related to image processing and running out of memory, and considering the lighthouse results mentioned in #203 images not correctly sized etc maybe improvements will help with this issue.

Error: ENOENT: no such file or directory, chmod '/opt/build/repo/public/static

Thoughts on how I can trace the error? All I was able to confirm is that the file does exist at least after the failed build, not sure where the error is thrown and how to debug.

styxlab · 2020-11-01T17:10:16Z

I'd be surprised if the error is related to image processing and running out of memory. You can turn off Gatsby image processing entirely by:

removing gatsby-rehype-inline-images (for inline images)
setting the flag gatsbyImages to false in mediaConfig.js (for feature images).

This could be a first step in trying to localize the issue.

marcoSven · 2020-11-01T21:44:51Z

I already removed gatsby-rehype-inline-images and I will give removing the feature images a try

marcoSven · 2020-11-02T00:09:04Z

While I looked at #202 I cloned this repository
changed content API settings in /dev/gatsby-theme-try-ghost/packages/gatsby-theme-try-ghost/src/utils/.ghost.json

installed packages with yarn and ran yarn build

and got this error Error: ENOENT: no such file or directory, chmod '/dev/gatsby-theme-try-ghost/demo/public/static/d5aa518ea784e1b659 b6bf9a69bd6f76/site-meta.png'

yarn build right after no error

styxlab · 2020-11-02T07:46:19Z

and got this error Error: ENOENT: no such file or directory, chmod '/dev/gatsby-theme-try-ghost/demo/public/static/d5aa518ea784e1b659 b6bf9a69bd6f76/site-meta.png'

Sounds like a duplicate of #188 if you are building on macOS. I think this error is different from the ones showing the unlink error message, although they could have the same root cause. My hunch is that Gatsby has a problem with with cache invalidation. If we can fully replicate the issues we can report them upstream. Before, I will update the project to the latest gatsby version, so we don't report intermittent problems.

marcoSven · 2020-11-14T08:06:19Z

If we can fully replicate the issues we can report them upstream

Try
https://gitpod.io/#snapshot/142e3a5b-3c8c-46c5-ad99-6fbaa85836e7

run yarn develop

styxlab · 2020-11-14T20:32:37Z

After clearing the cache, I can see the issue now. Actually, I think it is the same as gatsby/issues/27984.

styxlab · 2020-11-23T09:23:51Z

Fun fact: this error tells you something about the speed of your file-system. The faster your file writes the less likely you run into this error. Final fix is on its way...

styxlab · 2020-11-23T19:17:30Z

@marcoSven: Could you please verify that the reported build errors are fixed?

marcoSven · 2020-11-24T05:04:50Z

I had a hunch something hardware is related to this intermittent issue. Tells you something about Netlify vs Gatsby Cloud - kind of annoyed this was frequently happening on my Mac ;)

Looks like it is fixed!!! 🎉

styxlab · 2020-11-24T08:34:33Z

Perfect... and thanks for bringing this up. Both this project and official Gatsby benefited from it.

marcoSven mentioned this issue Nov 3, 2020

gatsby-rehype-inline-images missing images #202

Closed

styxlab mentioned this issue Nov 5, 2020

fix some wrong/missing asyncs and bump gatsby tp latest version #223

Merged

styxlab mentioned this issue Nov 17, 2020

[gatsby-plugin-ghost-images]: Uncritical build errors on macOS #188

Closed

styxlab mentioned this issue Nov 23, 2020

Fix 204 #236

Merged

styxlab closed this as completed Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Frequent build errors on Netlify #204

Frequent build errors on Netlify #204

marcoSven commented Oct 30, 2020

marcoSven commented Oct 30, 2020 •

edited

Loading

styxlab commented Oct 30, 2020 •

edited

Loading

marcoSven commented Oct 30, 2020 •

edited

Loading

marcoSven commented Oct 31, 2020

styxlab commented Oct 31, 2020 •

edited

Loading

marcoSven commented Oct 31, 2020

styxlab commented Nov 1, 2020

marcoSven commented Nov 1, 2020

marcoSven commented Nov 2, 2020

styxlab commented Nov 2, 2020

marcoSven commented Nov 14, 2020

styxlab commented Nov 14, 2020 •

edited

Loading

styxlab commented Nov 23, 2020 •

edited

Loading

styxlab commented Nov 23, 2020

marcoSven commented Nov 24, 2020

styxlab commented Nov 24, 2020

Frequent build errors on Netlify #204

Frequent build errors on Netlify #204

Comments

marcoSven commented Oct 30, 2020

marcoSven commented Oct 30, 2020 • edited Loading

styxlab commented Oct 30, 2020 • edited Loading

marcoSven commented Oct 30, 2020 • edited Loading

marcoSven commented Oct 31, 2020

styxlab commented Oct 31, 2020 • edited Loading

marcoSven commented Oct 31, 2020

styxlab commented Nov 1, 2020

marcoSven commented Nov 1, 2020

marcoSven commented Nov 2, 2020

styxlab commented Nov 2, 2020

marcoSven commented Nov 14, 2020

styxlab commented Nov 14, 2020 • edited Loading

styxlab commented Nov 23, 2020 • edited Loading

styxlab commented Nov 23, 2020

marcoSven commented Nov 24, 2020

styxlab commented Nov 24, 2020

marcoSven commented Oct 30, 2020 •

edited

Loading

styxlab commented Oct 30, 2020 •

edited

Loading

marcoSven commented Oct 30, 2020 •

edited

Loading

styxlab commented Oct 31, 2020 •

edited

Loading

styxlab commented Nov 14, 2020 •

edited

Loading

styxlab commented Nov 23, 2020 •

edited

Loading