cannot allocate memory after opening truncated diff (possible memory leak)

[mrexodia]

• Gitea version (or commit ref): 1.1.0+635-gefa482d (built version of Rewrite migrations to not depend on future code changes #2604)
• Git version: 2.1.4
• Operating system: Raspian (Raspberry Pi Zero W)
• Database
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

• I have a low amount of ram (512mb)
• Memory obtained stays very high and causes nothing to work
• Current memory usage stays very high (even though GC already ran)
• You need to view this commit (it's 36mb diff, so it's not shown): https://try.gitea.io/mrexodia/DarkSouls3.TextViewer/commit/629cf9b3d6b295bbcddf76d1f6167259b764d9dc
• Line where it happens:

  gitea/modules/context/repo.go

  Line 318 in 92123fe

  ctx.Handle(500, "GetTags", err)

Log

2017/10/06 21:18:00 [I] Log Mode: File(Info)
2017/10/06 21:18:01 [I] XORM Log Mode: File(Info)
2017/10/06 21:18:01 [I] Cache Service Enabled
2017/10/06 21:18:01 [I] Session Service Enabled
2017/10/06 21:18:01 [I] Mail Service Enabled
2017/10/06 21:18:01 [I] Register Mail Service Enabled
2017/10/06 21:18:01 [I] Notify Mail Service Enabled
2017/10/06 21:18:01 [I] Git Version: 2.1.4
2017/10/06 21:18:01 [I] SQLite3 Supported
2017/10/06 21:18:01 [I] Run Mode: Production
2017/10/06 21:18:04 [I] Listen: http://0.0.0.0:3000
2017/10/06 21:35:22 [...ules/context/repo.go:318 func1()] [E] GetTags: fork/exec /usr/bin/git: cannot allocate memory
2017/10/06 21:35:26 [...ules/context/repo.go:318 func1()] [E] GetTags: fork/exec /usr/bin/git: cannot allocate memory
2017/10/06 21:36:37 [...ules/context/repo.go:318 func1()] [E] GetTags: fork/exec /usr/bin/git: cannot allocate memory
2017/10/06 21:36:38 [...ules/context/repo.go:318 func1()] [E] GetTags: fork/exec /usr/bin/git: cannot allocate memory
2017/10/06 21:36:39 [...ules/context/repo.go:318 func1()] [E] GetTags: fork/exec /usr/bin/git: cannot allocate memory

Screenshots

Here is a video showing exactly how I reproduced the problem on a just-started gitea instance. I tried this about 4 times and it happens every time. It might be possible to reproduce on a system with more memory by cloning this repo multiple times and opening the offending diff a bunch of times as well, but I didn't test this yet.

https://youtu.be/RynWvzilbaQ

[mrexodia]

The main cause of the issue I described here is fixed #2990, however I found that Blob.Data is another cause of excessive memory usage. This is a list of things that use this function

• IsImageFile
• GetEditorconfig (should this respect Settings.UI.MaxDisplayFileSize?)
• ServeBlob
• editFile (should this respect Settings.UI.MaxDisplayFileSize?)
• getFileContentFromDefaultBranch (should this respect Settings.UI.MaxDisplayFileSize?)
• renderDirectory (readme, @lafriks said this should respect Settings.UI.MaxDisplayFileSize)
• renderFile (this reads all of the blob before checking Settings.UI.MaxDisplayFileSize so it might as well not check)
• wiki.go (should this respect Settings.UI.MaxDisplayFileSize?) see wiki.go should respect Settings.UI.MaxDisplayFileSize #5289

Currently I cannot really continue before go-gitea/git#94 is merged, but I will prepare a branch on my repo with all the things fixed so people can test/review my changes. You can find my current fixes here.

[lafriks]

go-gitea/git#94 is merged

[lunny]

@mrexodia should this be closed and make a new issue for the only one wiki.go issue?

[mrexodia]

No problem for me, whatever you think is better for the project :)