[unison-users] Unison: File has been modified during synchronization. Transfer aborted.

[plataleas plataleas [e-mail] [unison-users]]

Hi Unison Users

We like the unison project and using the tool for many applications. If big
folders are synchronized we encounter the following errors frequently:


*Failed: The source file /data/filexyhas been modified during
synchronization. Transfer aborted.*

* Failed: The source file /data/filexyzhas been modified during
synchronization. Transfer aborted. *
*Synchronization incomplete at 10:05:07 (124 items transferred, 1 skipped,
77 failed)*

Is it possible to skip files that have changed during synchronization? If
we sync big shares, it is probable that files may change during changes.

We run the script with the following options:

*unison 'ssh://host1//data' '/data' -ui text -batch -owner -group
-numericids -times -prefer newer -logfile /var/log/unison.log *

Thanks
martin

['Benjamin C. Pierce' [e-mail] [unison-users]]

Do you mean “Skip the files that get changed during synchronization but process all the rest”? That’s what Unison is doing.

- B

Post by plataleas plataleas ***@gmail.com [unison-users]
Hi Unison Users
Failed: The source file /data/filexy
has been modified during synchronization. Transfer aborted.
Failed: The source file /data/filexyz
has been modified during synchronization. Transfer aborted.
Synchronization incomplete at 10:05:07 (124 items transferred, 1 skipped, 77 failed)
Is it possible to skip files that have changed during synchronization? If we sync big shares, it is probable that files may change during changes.
unison 'ssh://host1//data' '/data <ssh://host1//data'%20'/data>' -ui text -batch -owner -group -numericids -times -prefer newer -logfile /var/log/unison.log
Thanks
martin

[plataleas plataleas [e-mail] [unison-users]]

Hi Benjamin

Yes exactly! We would expect that files that get changed during
synchronization are skipped.

We have the following entries in unison.log:

1) Failed

*Failed: The source file /data/filexyhas been modified during
synchronization. Transfer aborted.*

2) Skipped
skipped: *data/file123* (Failed to fingerprint file " *data/file123* ":
the file keeps on changing)

What is the difference between 1) and 2) ?

Why unison is failing on file "1)" and skipping on file "2)" ?

#unison -version
#unison version 2.48.3

regards
Martin

Post by 'Benjamin C. Pierce' ***@cis.upenn.edu [unison-users]
Do you mean “Skip the files that get changed during synchronization but
process all the rest”? That’s what Unison is doing.
- B
Hi Unison Users
We like the unison project and using the tool for many applications. If
*Failed: The source file /data/filexyhas been modified during
synchronization. Transfer aborted.*
*Failed: The source file /data/filexyzhas been modified during
synchronization. Transfer aborted.*
*Synchronization incomplete at 10:05:07 (124 items transferred, 1
skipped, 77 failed)*
Is it possible to skip files that have changed during synchronization? If
we sync big shares, it is probable that files may change during changes.
*unison 'ssh://host1//data' '/data' -ui text -batch -owner -group
-numericids -times -prefer newer -logfile /var/log/unison.log *
Thanks
martin

['Benjamin C. Pierce' [e-mail] [unison-users]]

Ah: (1) and (2) both indicate that a single file has been skipped. In (1), the file is skipped because it is being modified so often that Unison can’t even fingerprint it. In (2), Unison succeeds in fingerprinting and later starts to transfer it, but notices that it has changed during the transfer.

- B

Post by plataleas plataleas ***@gmail.com [unison-users]
Hi Benjamin
Yes exactly! We would expect that files that get changed during synchronization are skipped.
1) Failed
Failed: The source file /data/filexy
has been modified during synchronization. Transfer aborted.
2) Skipped
skipped: data/file123 (Failed to fingerprint file " data/file123 ": the file keeps on changing)
What is the difference between 1) and 2) ?
Why unison is failing on file "1)" and skipping on file "2)" ?
#unison -version
#unison version 2.48.3
regards
Martin
Do you mean “Skip the files that get changed during synchronization but process all the rest”? That’s what Unison is doing.
- B

Post by plataleas plataleas ***@gmail.com [unison-users]
Hi Unison Users
Failed: The source file /data/filexy
has been modified during synchronization. Transfer aborted.
Failed: The source file /data/filexyz
has been modified during synchronization. Transfer aborted.
Synchronization incomplete at 10:05:07 (124 items transferred, 1 skipped, 77 failed)
Is it possible to skip files that have changed during synchronization? If we sync big shares, it is probable that files may change during changes.
unison 'ssh://host1//data' '/data <>' -ui text -batch -owner -group -numericids -times -prefer newer -logfile /var/log/unison.log
Thanks
martin

[plataleas plataleas [e-mail] [unison-users]]

Thanks for clarification, that helps a lot.

Does the unison log makes sense?
- If skipped due to fingerprint not possible > Failed
- If skipped due to "file changed during transfer" > Skipped

In both cases the file is skipped!

The status report is misleading (7 files are skipped, right?):
Synchronization complete at 13:30:52 (218 items transferred, 5 skipped,
2 failed)

Thanks
martin

Post by 'Benjamin C. Pierce' ***@cis.upenn.edu [unison-users]
Ah: (1) and (2) both indicate that a single file has been skipped. In
(1), the file is skipped because it is being modified so often that Unison
can’t even fingerprint it. In (2), Unison succeeds in fingerprinting and
later starts to transfer it, but notices that it has changed during the
transfer.
- B
Hi Benjamin
Yes exactly! We would expect that files that get changed during
synchronization are skipped.
1) Failed
*Failed: The source file /data/filexyhas been modified during
synchronization. Transfer aborted.*
2) Skipped
the file keeps on changing)
What is the difference between 1) and 2) ?
Why unison is failing on file "1)" and skipping on file "2)" ?
#unison -version
#unison version 2.48.3
regards
Martin

Post by 'Benjamin C. Pierce' ***@cis.upenn.edu [unison-users]
Do you mean “Skip the files that get changed during synchronization but
process all the rest”? That’s what Unison is doing.
- B
Hi Unison Users
We like the unison project and using the tool for many applications. If
*Failed: The source file /data/filexyhas been modified during
synchronization. Transfer aborted.*
*Failed: The source file /data/filexyzhas been modified during
synchronization. Transfer aborted.*
*Synchronization incomplete at 10:05:07 (124 items transferred, 1
skipped, 77 failed)*
Is it possible to skip files that have changed during synchronization? If
we sync big shares, it is probable that files may change during changes.
*unison 'ssh://host1//data' '/data' -ui text -batch -owner -group
-numericids -times -prefer newer -logfile /var/log/unison.log *
Thanks
martin

[Alan Schmitt [e-mail] [unison-users]]

Post by plataleas plataleas ***@gmail.com [unison-users]
Does the unison log makes sense?
- If skipped due to fingerprint not possible > Failed
- If skipped due to "file changed during transfer" > Skipped

It's the other way around (skipped means it was not attempted to
transfer because fingerprinting failed, and failed means that when
checking after the transfer, a different fingerprint was found).

Post by plataleas plataleas ***@gmail.com [unison-users]
In both cases the file is skipped!

There are additional cases when a file is skipped, for instance when the
user says so.

Post by plataleas plataleas ***@gmail.com [unison-users]
Synchronization complete at 13:30:52 (218 items transferred, 5 skipped,
2 failed)

Not really. 7 files were not transferred: 5 were not tried, and 2 were
attempted but failed.

Best,

Alan

[Martin Vahi [e-mail] [unison-users]]

Post by plataleas plataleas ***@gmail.com [unison-users]
Hi Benjamin
Yes exactly! We would expect that files that get changed during
synchronization are skipped.
...

I haven't tested the idea, but MAY BE a workaround
to try MIGHT be that You use a copy-on-write file system,
in which case it is extremely cheap to do unmodified
recursive copies and then use the Unison on a copy.
You might write a script that does the copy-on-write clone
and then runs Unison.

As of the time of the writing of the current letter
I haven't tested the DragonFlyBSD based idea, but
the DragonFlyBSD with its Hummer file system
MIGHT be used ONLY in the role of a file server that
gets mounted by other computers with sshfs. That
way the lack of DragonFlyBSD packages/ports MIGHT
not be a problem. I do not know, if Unison can be
compiled on the DragonFlyBSD, but if the Unison runs
some Linux on the same LAN than the DragonFlyBSD machine
runs, then MAYBE the solution isn't as sluggish as
one might suspect. I haven't tested it. It's just
a wild idea for a workaround.

Thank You for reading my letter :-)