LFS gives 404 error, after migrating gitea service from server to another server

[skoenen]

Description

Hi there,

recently we had to move our gitea service from one machine to a newer machine.
Process:

1. Setup VM
2. Setup Gitea in docker rootless variant
3. Declare volumes (on nfs shares) for:

• gitea_conf => /etc/gitea
• gitea_data => /var/lib/gitea
• gitea_ssh => /var/lib/gitea/home/.ssh
• repositories => /var/lib/gitea/repositories
• lfs => /var/lib/gitea/lfs

4. Populate the gitea database (postgres) from postgres dump file
5. check that the mounts are readable and writable by container git user:

• docker-compose exec -u git gitea touch .../test.txt

Normal repositories working fine so far. But lfs files pose a problem:

CMD: git lfs fetch --all
Error: [...] Not Found: [404] Not Found
gitea log: No errors mentioned, authentication works router: completed POST

After that i tried the "Associate accessible 1 OIDs" in the repo settings > lfs.

That resulted in 500 Internal Server Error
gitea log:

2023/08/21 11:43:08 ...eb/routing/logger.go:102:func1() [I] router: completed GET /user/events for 192.168.176.5:45578, 200 OK in 1081356.3ms @ events/events.go:18(events.Events)
2023/08/21 11:43:08 ...dules/git/command.go:281:Run() [D] git.Command.RunDir(/var/lib/gitea/repositories/stefan.koenen/lfs-test.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= rev-parse
2023/08/21 11:43:08 ...dules/git/command.go:281:Run() [D] git.Command.RunDir(/var/lib/gitea/repositories/stefan.koenen/lfs-test.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch
2023/08/21 11:43:08 ...dules/git/command.go:281:Run() [D] git.Command.RunDir(/var/lib/gitea/repositories/stefan.koenen/lfs-test.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= cat-file --batch-check
2023/08/21 11:43:08 ...dules/git/command.go:281:Run() [D] git.Command.RunDir(/var/lib/gitea/repositories/stefan.koenen/lfs-test.git): /usr/bin/git -c protocol.version=2 -c credential.helper= -c filter.lfs.required= -c filter.lfs.smudge= -c filter.lfs.clean= for-each-ref "--format=%(objectname) %(refname)" refs/heads/ --sort=-committerdate
2023/08/21 11:43:08 ...ters/web/repo/lfs.go:551:LFSAutoAssociate() [E] LFSAutoAssociate: unable collect all LFS objects from database, expected 1, actually 0
2023/08/21 11:43:08 .../context_response.go:74:HTML() [D] Template: status/500
2023/08/21 11:43:08 ...eb/routing/logger.go:102:func1() [I] router: completed POST /stefan.koenen/lfs-test/settings/lfs/pointers/associate for 192.168.176.5:35444, 500 Internal Server Error in 74.5ms @ repo/lfs.go:527(repo.LFSAutoAssociate)

Seems to be related to #17207

I check manually for the files and they are accessible by git user and also the folder is writable. So it seems there is something in the database missing.

gitea data:

Name	Value
Site Title	GIT of AIS-Lab
Gitea Version	1.20.0 built with GNU Make 4.4.1, go1.20.6 : bindata, timetzdata, sqlite, sqlite_unlock_notify
Configuration File Path	/etc/gitea/app.ini
Gitea Base URL	https://git.ais-lab.eu/
Server Domain	git.ais-lab.eu
Local Mode	Disable Router Log
Run As Username	git
Run Mode	Prod
Git Version	2.40.1, Wire Protocol Version 2 Enabled
Repository Root Path	/var/lib/gitea/repositories
Static File Root Path	/var/lib/gitea
Custom File Root Path	/var/lib/gitea/custom
Log Path	/var/lib/gitea/log
Script Type	bash
Reverse Authentication User	X-WEBAUTH-USER
SSH Configuration
Enabled	true
Use Built-In Server	true
SSH Server Domain	git.ais-lab.eu
Port	22
Listen Port	22
LFS Configuration
Enabled	true
LFS Content Path	{"Type":"local","Path":"/var/lib/gitea/lfs","TemporaryPath":"/var/lib/gitea/lfs/tmp","MinioConfig":{"UseSSL":false,"InsecureSkipVerify":false,"ServeDirect":false}}
LFS HTTP Auth Expiry	24h0m0s

Gitea Version

1.20.0

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

2.40.1, Wire Protocol Version 2 Enabled

Operating System

Fedora Server 37

How are you running Gitea?

Docker Image rootless version
With several NFS-Shares for large data

Database

PostgreSQL

[lunny]

What's your Gitea version?

[skoenen]

Gitea Version: 1.20.0

[lunny]

Can you upgrade to 1.20.3 and try again?

[zjjhot]

hello，please try to transfer these settings from your old machine's app.ini to new machine.

[skoenen]

I will check these values again, but the app.ini is the same as it was on the old server. The app.ini is on its own nfs share and is just mounted to the new server.

[Woodpecker106]

I'm interested to know if you got to the bottom of this as I have the same issue. I just tried copying the LFS secret from my old app.ini to the new one but the issue still persists.

[skoenen]

I did not manage to solve that automatically. I did manual change the corresponding files/database entries.

[Endll]

Same issue, copying LFS secrets doesn't work either . So is there a solution? @Woodpecker106 @zjjhot I dumped Gitea server from one Windows server to another . Then I got 404 error when I checkout a new branch .

[lunny]

Maybe you can try to use an admin user to do that? Looks like there is a bug here.

[Endll]

Maybe you can try to use an admin user to do that? Looks like there is a bug here.

Thanks for the reply, but I didn't understand what to do with the admin account. I registered Gitea as a win service. I modified the LFS secret in app.ini and then restarted the service and it didn't work.

[Endll]

Maybe you can try to use an admin user to do that? Looks like there is a bug here.

In addition, 404 is a git lfs error instead of a gitea error. When pulling a remote branch, if the remote branch didn't have a git lfs commit, everything is normal. However, if the remote branch has a lfs commit, then an error will occur:
$ git-lfs.exe filter-process
Error downloading object:......
[404] Not Found
[3e8666794abcf279e893f5cec0c54f52e00837bf226c96f5e2aa9c054a0ce538] Not Found
github.com/git-lfs/git-lfs/errors.newWrappedError