"cannot receive: invalid record type" when receiving raw incremental stream with encrypted filesystem and redundant snapshots

[justinlovinger]

System information

Type	Version/Name
Distribution Name	NixOS
Distribution Version	20.09
Linux Kernel	5.4.69
Architecture	x86-64
ZFS Version	0.8.4-1
SPL Version	0.8.4-1

Describe the problem you're observing

ZFS fails to receive incremental stream with error "cannot receive: invalid record type", if filesystem contains an encrypted child and some snapshots in incremental stream are already on receiving filesystem.

Describe how to reproduce the problem

# dd if=/dev/zero of=/tmp/poolfile1 bs=1M count=100
# zpool create test1 /tmp/poolfile1
# zfs create -o encryption=on -o keyformat=passphrase test1/fs1
# touch /test1/fs1/a
# zfs snapshot -r test1@a
# touch /test1/fs1/b
# zfs snapshot -r test1@b
# touch /test1/fs1/c
# zfs snapshot -r test1@c
# dd if=/dev/zero of=/tmp/poolfile2 bs=1M count=100
# zpool create test2 /tmp/poolfile2
# zfs send -R --raw test1@b | sudo zfs receive -v -F test2
# zfs send -R --raw -I test1@a test1@c | sudo zfs receive -v -F test2

Include any warning/errors/backtraces from the system logs

The full log of the reproduction case is

# dd if=/dev/zero of=/tmp/poolfile1 bs=1M count=100
# zpool create test1 /tmp/poolfile1
# zfs create -o encryption=on -o keyformat=passphrase test1/fs1
Enter passphrase:
Re-enter passphrase:
# touch /test1/fs1/a
# zfs snapshot -r test1@a
# touch /test1/fs1/b
# zfs snapshot -r test1@b
# touch /test1/fs1/c
# zfs snapshot -r test1@c
# dd if=/dev/zero of=/tmp/poolfile2 bs=1M count=100
# zpool create test2 /tmp/poolfile2
# zfs send -R --raw test1@b | sudo zfs receive -v -F test2
receiving full stream of test1@a into test2@a
received 44.6K stream in 1 seconds (44.6K/sec)
receiving incremental stream of test1@b into test2@b
received 312B stream in 1 seconds (312B/sec)
receiving full stream of test1/fs1@a into test2/fs1@a
received 18.8K stream in 1 seconds (18.8K/sec)
receiving incremental stream of test1/fs1@b into test2/fs1@b
received 10.1K stream in 1 seconds (10.1K/sec)
# zfs send -R --raw -I test1@a test1@c | sudo zfs receive -v -F test2
receiving incremental stream of test1@b into test2@b
snap test2@b already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of test1@c into test2@c
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of test1/fs1@b into test2/fs1@b
snap test2/fs1@b already exists; ignoring
cannot receive: invalid record type
cannot restore to test2/fs1@b: destination already exists

Additional notes

If you rollback the receiving filesystem, like zfs rollback -r test2@a and zfs rollback -r test2/fs1@a, the incremental receive will succeed.

With -F, the receiving filesystem will delete snapshots missing from the stream, even though it fails to receive.

[spmfox]

Just hit this today, took me a while to figure out it was a issue and not something I was doing wrong. I've been looking through the man pages to see if ignoring existing snapshots is expected behavior or not. I guess I took this for granted because I based my replication system around the assumption that ZFS ignored existing snapshots on the receiving side. With this, it will break an entire recursive replication for one snapshot already existing. Particularly this impacts resuming a mirror that had some of its snapshots transferred but not all of them.

Its outside my skill level to figure out, but can someone confirm is this a "bug" or is it a unavoidable side effect of sending with --raw? Can it be fixed somehow or is this just a limitation like the copies=3 is a limitation of encryption?

(thanks for the hard work, I cannot live with ZFS)

[shodanshok]

I was just hit by this issue without using raw send/recv (it was a plain incremental send/recv). I am using ZFS 2.0.5 and this is the first time I encountered the issue; before I was running ZFS 0.8.6 on both source and destination without problem.

Even after removing the snapshot causing cannot receive: invalid record type, the transfer could not be restored unless I rolled back the destination to the last available snapshot (it aborted with the common destination has been modified warning).

I don't know if this is a coincidence of if the upgrade to ZFS 2.0.5 is related. I'll keep the machine under monitor and report back any finding.

[shodanshok]

I was just hit by this issue without using raw send/recv (it was a plain incremental send/recv). I am using ZFS 2.0.5 and this is the first time I encountered the issue; before I was running ZFS 0.8.6 on both source and destination without problem.

Even after removing the snapshot causing cannot receive: invalid record type, the transfer could not be restored unless I rolled back the destination to the last available snapshot (it aborted with the common destination has been modified warning).

I don't know if this is a coincidence of if the upgrade to ZFS 2.0.5 is related. I'll keep the machine under monitor and report back any finding.

I was again hit by this issue. To solve it I had, again, to rollback to the latest snapshot (this time I tried with "zfs recv -F" and it worked). Anyone having some idea on what is going on?

EDIT: Checking the code, the only relevant reference I can find is in libzfs_sendrecv.c, function recv_skip at lines 3827-3831, where it is the default case of a switch statement - ie: a unknow record.

But I don't know what was causing the unknown/unexpected record in the first place...

[scineram]

There is similar error in the function zfs_receive_one except that also prints out the incorrect drr_type. Could you patch your version to do the same in recv_skip? That should be the case anyway. Maybe that could help debug this.

[shodanshok]

@scineram thanks for your suggestion. I opened a dedicated issue here

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[montanaviking]

Hi all,
I still see the above issue popping up randomly when I try to send snapshots to my ZFS backup drive. I'm using
version: 2.1.6-0york1~18.04
srcversion: EBD93AD6D6AD43867A67BFA
vermagic: 5.4.0-144-generic SMP mod_unload modversions
So far, this would definitely complicate my efforts to automate my ZFS backups. I find that, in general, I need to delete the snapshots up the snapshot that reported the error: example:

snap zbackupc2/carbonics@zfs-auto-snap_weekly-2023-03-13-1546 already exists; ignoring
cannot receive: invalid recorsnap zbackupc2/carbonics@zfs-auto-snap_weekly-2023-03-13-1546 already exists; ignoring
cannot receive: invalid recorsnap zbackupc2/carbonics@zfs-auto-snap_weekly-2023-03-13-1546 already exists; ignoring
cannot receive: invalid record type
cannot restore to zbackupc2/carbonics@zfs-auto-snap_weekly-2023-03-13-1546: destination already exists

Thanks again for all your hard work on ZFS!
Phil

[montanaviking]

I also notice that if, during the send, I get the message that
received xB stream in ....
and x>0 then the send will work without incident.
Thanks,
Phil