Windows Desktop App often uses upload/delete instead of move #7838
Comments
|
@joholm2 thank you very much for your feedback. Please provide a few lines of the client debug log from the time of the incident. (see https://doc.owncloud.org/desktop/2.6/troubleshooting.html#log-files) |
|
Hi, I enabled logging as you said and carried out a short test...
The activity log shows both "Moves" and "Upload/Deletes" - where just Moves should have been used.
I don't yet understand the logging process but appear to have acquired multiple log files; attached here... Edit... removed numerous log files and will post more specific case below.. |
Desktop app checks for new changes after 30 second wait period. From every sync run, the log is written to a new file. |
|
I don't see any 30 delay in syncing. When I create a "New folder" and rename it as fast as I can, the App always jumps in and uploads "New folder" and then uploads the renamed one and deletes "New folder". Better if it did wait a few moments. But how ar the log files? Do they show why it does so many unnecessary upload & deletes? |
Sorry, I didn't explain precise. Local changes get synced immediately. Desktop app always checks for new server-side changes after 30 second wait period.
Didn't check the log files yet. |
|
One new log file attached. Two step test... (what I did)
Result (what Desktop client did) The log covers the period of Step 2. I notice that a rename is attempted but fails... Any help, advice or suggestions would be much appreciated. |
|
I collected logs from both client and server during a simple test similar to the one above; in this case just renaming a file (with result upload/delete). I see the same "Can't rename because the etag has....." in the client log; and on the server there is an equivalent error when trying to access the file to be renamed.... ...."message":"Exception: HTTP/1.1 404 File with name {filename} could not be located: One thing I don't understand in the server log is large numbers of pairs of lines... updateToken","method":"DELETE","url":"{url-to-file}","message":"updating token ddd, last check is now dddddddddd"} The method (DELETE in example above) varies between DELETE, PROPFIND and PUT; but the pattern is the same. Can anyone say whether these getToken responses are normal or an error? |
|
Hi, There doesn't seem to be much in the way or responses or help here. Is this all I can expect from OwnCloud? For us It's a very big issue - if there is no help or support what else can we do? |
|
Team is busy with other tasks like 2.6.3 and 2.7.0 release and customer requests, and this problem is nothing that can be solved in the short term. For professional support with guaranteed response times, you'll find options on the bottom of this page: |
|
Thanks, but i don't need it to be solved in the short term. I just need some help on the way. Anyone who understands these logs could give me some hints as to what they mean in a couple of minutes. |
|
Just to add that we have set up an owncloud docker image server and repeated the tests. The results are the same, the issue exists even there; so presumably that rules out any server misconfiguration as the cause of the issue. |
|
Another test - I installed the latest Windows client on another PC, connected to the same docker image mentioned in my last post and sync'd with the test folder. a) Same results when moving or renaming folders and files. Most often an upload & delete is used instead of the expected move. b) When one sync'd PC does the unwanted upload & delete the second PC does a corresponding download & delete. This is not good - to say the least! |
|
I have to ask again... It really is an intolerable fault - I just needed to rename a directory containing 4.37 GB of pictures. I've learnt from previous experience that it's best to let it complete it's wasteful upload - in the past, interrupting it has caused even more trouble. |
Sorry, as mentioned before, nothing planned before 2.6.3 and 2.7.0 release. ownCloud desktop sync client relies heavily on filesystem events to detect local changes. There are edge cases where the client does not get reliable information. Quite some effort has already been done to detect changes (e.g. checksum comparison), but the remaining cases are really hard to solve. Ensuring the integrity of the data is the higher goal than saving some GB. I performed a quick test and it detected simple Start DockerStart sync clientDebug log outputYou can filter for the folderwatcher Simple log in sync dirThis log isn't really useful for serious debugging Docker logInteresting in your log
|
|
Hi Michael, Thanks, we really appreciate your reply. It's a shame you tested on macOS - we are using Windows so I don't know how closely they match. Interesting you mention SMB storage. Are you referring to the PC files or the sever file system? On the PCs we have mainly files on the single hard disk, but one is on a Micro-SD card in a "built in" slot. On the sever we have a separate Raid storage unit mounted via SMB. In our setup using Docker we have sync'd only local HD storage on the PC and presumably Docker uses purely local storage on the sever. Since the Docker setup shows the same problems it's hard to draw any conclusions. |
Strange, in my Docker tests, I wasn't able to reproduce. I'll need to have another look when time allows… |
|
Thanks, as I mentioned you used a new client on macOS and we have 2.6.1 client on Windows10 - so not directly comparable. |
|
It's now 3 weeks since I renamed a directory containing about 5 GB of files on my windows PC, and OwnCloud client decided to re-upload the whole lot rather than just rename/move the directory on the server. I use a program called FreeFileSync to keep a local backup up to date and run that sporadically - often every few weeks. So almost 3 weeks after the "rename" I ran FreeFileSync and watched to see what it did with the directory. With no problem at all it saw it as a "move" and simply moved it on the backup drive. So if FreeFileSync can figure it out correctly after 3 weeks it can't be that hard for OwnCloud client after just a few seconds. Maybe some developer here can take a look at the FreeFileSync code. |
|
Hi, We've been waiting now over 7 months and so far no useful help or solution. We really need to be able to sort out our photo collection without Owncloud continuously up & down-loading gigabytes of files every time we move or rename a folder. If Owncloud can't function properly we'll need to move on to something which can. |
|
Today on Windows 10 the ownCloud App updated to 2.1.1 I made a quick test of the synchronisation to my test area in a completely standard Docker/ownCloud. Moving a sub-folder containing files from one folder to another (in the same sync folder) resulted in a move on the sever. Good - although it beats my why the log shows both folder move and then individual file moves - surely carrying out a proper move of a folder means all the files are included anyway. But next - move the sub-folder onward to another folder (still within same sync area). This time all the files are upload as if new to the server and the originals are deleted. Bad! Third try. move te sub-folder back to te starting folder - another upload/delete session - completely unnecessary. I made several more moves in Windows but no sign of any move operations in the sync log - always uploads and deletes. So there is no improvement this new version of the Windows app. This is a critical issue - when will it be fixed? |
|
I have just completed a switch from Owncloud to Nextcloud. Same server Tested renaming and moving of folders and files and Nextcloud gets it right every time. Goodbye Owncloud !!! |
|
Hi all, This behavior is still an issue and I think it is an important one that affects a core functionality of OwnCloud. The unreliable detection of move/rename operations is a significant efficiency reduction. While I understand the raised point that data consistency is more important than sparing bandwidth, the issue also affects consistency.
Depending on the sync status when shutting down client 1, the delete operation of the old folder will be processed by the server or not. This can result in the moved files being reuploaded by client 2 to the old location and hence duplicate data. On another note, the reupload process also destroys the moved files history (because they get deleted and recreated). IMO, these are two major problems (inconsistent data, corrupted file history, along with unnecessary syncing times) for OwnClouds core functionality that should be addressed, even if they are complicated to resolve. @joholm2 Can you confirm that this still works reliably with NextCloud? There seem to be similar open issues in the tracker (e.g, nextcloud/desktop#2132). |
|
Hi,
Yes, we can confirm that move and rename work reliably (for us) with
Nextcloud.
Since we switched from Owncloud to Nextcloud we have had no such
problems at all.
We can rename folders containing gigabytes of data and/or large numbers
of files without fear that they are re-uploaded (and re-downloaded to
other clients).
One noticeable thing is that upon renaming (or moving) a folder, the
NextCloud Windows client dialog shows that is runs through every
underlying folder & file. The counter shows progress and it doesn't
normally take long (eg 2 or 3 seconds for a hundred files). We have
also double checked that it is not "secretly" uploading he files during
that slight delay.
So the Windows "dynamic log" which pops up when you left-click the
NextCloud icon shows a rename operation for each and every file within
the renamed/moved folder - which seems a bit unnecessary. But the web
interface "Activity" log show simple "You renamed <folder-X> to
<folder-Y>.
I checked the case #2132 you mentioned and confirm we don't see that
problem at all. I see that issue was reported on server 19.0.0 and we
actually started and 20.0.02 (now on 20.0.7).
Good luck - it would be interesting to hear how things work out...
Regards
David Bloomfield
+46705315381
Johannisholm Adventure AB
+4625060000
http://joholm.se <http://joholm.se/>
[email protected]
…------ Original Message ------
From: "Alex Schmitz" <[email protected]>
To: "owncloud/client" <[email protected]>
Cc: "joholm2" <[email protected]>; "Mention" <[email protected]>
Sent: 2021-02-18 08:58:35
Subject: Re: [owncloud/client] Windows Desktop App often uses
upload/delete instead of move (#7838)
Hi all,
as there has not been any action for some time, I decided to add my two
cents here.
This behavior is still an issue and I think it is an important one that
affects a core functionality of OwnCloud.
The unreliable detection of move/rename operations is a significant
efficiency reduction. While I understand the raised point that data
consistency is more important than sparing bandwidth, the issue also
affects consistency.
Take the following example case:
Move a folder with some file on client A to a different folder (such
that reuploading gets triggered).
Abort the reupload process by shutting down (or whatever other way).
Open OwnCloud on a second synced client.
Depending on the sync status when shutting down client 1, the delete
operation of the old folder will be processed by the server or not. This
can result in the moved files being reuploaded by client 2 to the old
location and hence duplicate data.
On another note, the reupload process also destroys the moved files
history (because they get deleted and recreated).
IMO, these are two major problems (inconsistent data, corrupted file
history, along with unnecessary syncing times) for OwnClouds core
functionality that should be addressed, even if they are complicated to
resolve.
However, moving files around is working perfectly nice with other
clients (e.g. Dropbox, Seafile) so detection based on OS behavior should
be possible in principle.
@joholm2 <https://github.com/joholm2> Can you confirm that this still
works reliably with NextCloud? There seem to be similar open issues in
the tracker (e.g, nextcloud/desktop#2132
<nextcloud/desktop#2132>).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#7838 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOT5L3JXA6XC22BBS7MXXFLS7TCCXANCNFSM4L6SI5VQ>.
|
|
@joholm2 Could you please confirm that for the last tests the ownCloud client was updated to 2.7.1.. Which ownCloud server version was installed? |
Yes. I see there is a typo in my comment Nov 2020 where it says 2.1.1. In earlier posts i had already mentioned using 2.6.1, so the November version was undoubtedly 2.7.1. Owncloud server was always kept up to date, so was whatever was current in November 2020. |
|
Thank you for the quick response! |
|
Just to add to this: I also tried playing around with the various ways of moving files in Windows Explorer, but couldn't find any way to work around the issue. Namely, neither using the context menu |
|
Don't forget the simple renaming of a folder containing sub-folders and files. There is less scope for variation in how that is done.
Med vänlig hälsning
David Bloomfield
Johannisholm Adventure AB
0705315381
On 22 February 2021 14:30:16 CET, Alex Schmitz <[email protected]> wrote:
Just to add to this:
I last experienced the issue yesterday, running client 2.7.6 and server
10.6.0.5.
I also tried playing around with the various ways of moving files in
Windows Explorer, but couldn't find any way to work around the issue.
Namely, neither using the context menu `Cut -> Paste` nor the shortcut
`Strg+X` + `Strg+V` nor simple `drag and drop` is working consistently.
It also does not seem to matter whether the move operation takes place
inside the same Explorer window or across two windows.
…--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#7838 (comment)
|
|
@sfadschm Thank you for the info. |
I tend to test my code, I definitely fixed a bug. If the issue would continue we would have more than one bug involved. @jnweiger your discovery is unrelated to this issue
We couldn't find any information about the file we detected as move origin in the db.
This one indicates that by the time we want to populate the move the move was already reverted by your script
If you find out that move detection on linux is unreliable please open a new issue. |
I might be blind, but the latest testpilot and daily builds I can find are from 21 February for 2.7.7 and 08 February for 2.7.6. |
|
Mac & Win built this morning. Looks like an issue with the Linux builds: |
|
Okay, will try to test them.
(Sorted by modification date or name ascending.) |
|
Issue persists with |
|
It seems the daily build isn't working. There are no new builds with the bug-fix available. Confirmed the bug-fix with a local build from github on Linux: |
|
No, my test is valid for this issue. The problem with upload/delete using a folder sync connection is reproducible on Linux. I got similar error messages '404 Not Found' and 'Can't rename because the etag has changed or the directory is gone...' in the logfile. 20210301_0637_client_2-7-6.log.0.gz Testing the client compiled from source code containing the bug-fix on Linux shows the desired behavior, only moves resp. renames are done (-> Rename detected (up)...). |
|
The up-to-date daily win-build will be available soon. |
|
Can confirm the issue to be resolved on Win10 using |
|
Hi there, A few days ago I recognized that my selfhosted OwnCloud on a Raspi 4 within my home network got very slow download. I have both, the Raspi and my Desktop on a 1Gbit-Switch, but the download drops (partly) under 1 MB/s. I saw that there is a new OwnCloud Version, so I started the update to latest stable (10.8.0.4). After that it started to happen, that instead of moving files from folder to folder, it starts to delete and reupload them. Renaming files still works as "move". I'm using the Client Version 2.8.2 (build 4246) btw. If I move the files on the webDAV, the Client works fine and moves my files on my Desktop as well. I never had this problem before in the 3-4 years I use Owncloud. Is there any solution to solve this? |
|
@Darkexy the problem described in this issue was fixed in the 2.8.2 release. Please update. If you still see problems, please open a new issues, to help the team to better understand your problem. |
@michaelstingl Hi, like I wrote, I'm using Client version 2.8.2 (build 4246). I even reinstalled it yesterday since I thought this propably can fix it, but unfortuantly it didn't. |
|
I could move a folder on iPhone files app. There is a curl command here if needed: curl -u user:pass -X MOVE --header 'Destination: https://example.com/owncloud/remote.php/dav/files/USERNAME/target.jpg' https://example.com/owncloud/remote.php/dav/files/USERNAME/source.jpg |
Expected behaviour
In Windows explorer make any change to a file or folder, within the same sync area, which can be carried out by a move/rename.
eg...
All of these can be accomplished by a simple move operation.
Actual behaviour
Sometimes a “move” is used
Sometimes files are deleted on the server and re-uploaded.
I see no obvious pattern in how it chooses what method to use.
I thought maybe it depended on file size, but even moving a large (500 MB file) into a sub-folder and later back again resulted in a “move” first (Ok) and later a totally unnecessary upload/delete on returning to the original folder.
This is very inefficient and there is no obvious reason for it.
It can also lead to a very bad situation when renaming a folder containing a large quantity of data.
In cases where Owncloud chooses the upload/delete route the resulting upload can take hours - all totally unnecessary. Aborting such an upload leads to a further catastrophe with potential loss of data.
Steps to reproduce
As described above in Expected behaviour.
Check OwnCloud "Activity" in desktop app.
Example here is for creation of a new (empty) folder and repeated renaming (with idle time of a few seconds between each step). Activity log ordering - bottom to top.
In this case adding the "_test" suffix results in a Move and removing the "_test" suffix results in an Upload followed by a Delete.
Server configuration
Operating system:Ubuntu 18.04
Web server: Apache/2.4.29 (Ubuntu)
Database: mySQL Ver 15.1 Distrib 10.1.43-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
PHP version: 7.2.24 and now 7.3.16-1
ownCloud version: 10.3.2.2 and now 10.4.0.4
Storage backend (external storage): SMB attached drive
Client configuration
Client version: 2.6.0 and now 2.6.1
Operating system: Windows 10
The text was updated successfully, but these errors were encountered: