Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

autoreplace not working correctly with ZFS 0.8.x #10577

Closed
shodanshok opened this issue Jul 15, 2020 · 1 comment
Closed

autoreplace not working correctly with ZFS 0.8.x #10577

shodanshok opened this issue Jul 15, 2020 · 1 comment

Comments

@shodanshok
Copy link
Contributor

System information

Type Version/Name
Distribution Name CentOS
Distribution Version 7.8
Linux Kernel 3.10.0-1127.10.1.el7.x86_64
Architecture x86_64
ZFS Version 0.8.4-1
SPL Version 0.8.4-1

Describe the problem you're observing

autoreplacing is not working correctly on ZFS 0.8.x: when inserting a new blank disk, zpool status show the device as UNAVAIL. zed is running and autoreplace=on. To start the resilver, one as to export/import the pool or manually run a zpool replace. Using WWN/id/sd* names make no difference.

Debugging the issue, I found the following zfs events:

TIME                           CLASS
Jul 13 2020 18:16:18.069000000 sysevent.fs.zfs.history_event
Jul 13 2020 18:16:18.127000000 sysevent.fs.zfs.config_sync
Jul 13 2020 18:16:18.127000000 sysevent.fs.zfs.pool_import
Jul 13 2020 18:16:18.131000000 sysevent.fs.zfs.history_event
Jul 13 2020 18:16:18.213000000 sysevent.fs.zfs.config_sync
Jul 13 2020 18:16:40.553502714 resource.fs.zfs.statechange <--- remove
Jul 13 2020 18:16:40.622501866 sysevent.fs.zfs.config_sync
Jul 13 2020 18:17:15.479073873 resource.fs.zfs.statechange <--- attach
Jul 13 2020 18:17:15.496073664 sysevent.fs.zfs.vdev_online
Jul 13 2020 18:17:15.558072903 sysevent.fs.zfs.config_sync

ZED debug:

zed_disk_event:
        class: EC_dev_add
        subclass: disk
        dev_name: /dev/sdb
        path: /devices/pci0000:00/0000:00:02.6/0000:07:00.0/virtio5/host0/target0:0:0/0:0:0:1/block/sdb
        devid: scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1
        phys_path: pci-0000:07:00.0-scsi-0:0:0:1
        dev_size: 2147483648
zfs_slm_event: EC_dev_add.disk
zfs_deliver_add: adding scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-1 (pci-0000:07:00.0-scsi-0:0:0:1) (is_slice 0)
zfs_iter_pool: evaluating vdevs on tank (by devid)
zfs_iter_pool: evaluating vdevs on tank (by phys_path)
  zfs_iter_vdev: matched phys_path on pci-0000:07:00.0-scsi-0:0:0:1
zfs_process_add: pool 'tank' vdev '/dev/disk/by-path/pci-0000:07:00.0-scsi-0:0:0:1-part1', phys 'pci-0000:07:00.0-scsi-0:0:0:1' wholedisk 1, dm 0 (7228918098681742888)

It seems that the problem lies in the missing partition (part1) on the newly inserted disk. In other words, ZED tries to bring the vdev online before creating the required *-part1 block device. In turn ZFS detect the missing partition/block device and report it as UNAVAIL.

By comparison, these are the zpool events logged on an otherwise identical VM with CentOS 7.7 and ZFS 0.7.13:

TIME                           CLASS
Jul 13 2020 18:01:04.958000000 sysevent.fs.zfs.history_event
Jul 13 2020 18:01:05.054000000 sysevent.fs.zfs.config_sync
Jul 13 2020 18:01:05.054000000 sysevent.fs.zfs.pool_import
Jul 13 2020 18:01:05.062000000 sysevent.fs.zfs.history_event
Jul 13 2020 18:01:05.175000000 sysevent.fs.zfs.config_sync
Jul 13 2020 18:05:55.190363080 ereport.fs.zfs.io           <--- remove
Jul 13 2020 18:05:55.190363080 ereport.fs.zfs.io
Jul 13 2020 18:05:55.190363080 ereport.fs.zfs.io
Jul 13 2020 18:05:55.190363080 ereport.fs.zfs.probe_failure
Jul 13 2020 18:05:55.257362073 ereport.fs.zfs.vdev.too_small
Jul 13 2020 18:05:55.257362073 resource.fs.zfs.statechange
Jul 13 2020 18:07:29.741645829 sysevent.fs.zfs.config_sync <--- attach
Jul 13 2020 18:07:29.875643033 resource.fs.zfs.statechange
Jul 13 2020 18:07:29.930641886 resource.fs.zfs.statechange
Jul 13 2020 18:07:29.993640572 sysevent.fs.zfs.config_sync
Jul 13 2020 18:07:30.024639925 sysevent.fs.zfs.vdev_attach
Jul 13 2020 18:07:30.116638006 sysevent.fs.zfs.resilver_start
Jul 13 2020 18:07:30.116638006 sysevent.fs.zfs.history_event
Jul 13 2020 18:07:30.788623987 sysevent.fs.zfs.history_event
Jul 13 2020 18:07:30.788623987 sysevent.fs.zfs.resilver_finish
Jul 13 2020 18:07:30.792623903 sysevent.fs.zfs.config_sync
Jul 13 2020 18:07:30.907621504 sysevent.fs.zfs.vdev_remove
Jul 13 2020 18:07:30.909621463 sysevent.fs.zfs.history_event
Jul 13 2020 18:07:30.986619856 sysevent.fs.zfs.config_sync
Jul 13 2020 18:07:35.907517196 sysevent.fs.zfs.history_event

As you can see, there are many more events (and vdev_attach sounds as a obvious thing missing from the reported ZFS 0.8.4 events).

Any suggestion?
Thanks.

Describe how to reproduce the problem

  1. create a simple mirrored pool with a single top-level vdev
  2. detach a disk from the system
  3. zpool status show the disk as OFFLINE (it was zed which offlined the removed disk)
  4. insert a new blank disk (or the same disk after being wiped on another system)
  5. zpool status now show the disk as UNAVAIL
  6. to start resilver, one has to manually export/import the pool or issue zpool replace
shodanshok added a commit to shodanshok/zfs that referenced this issue Jul 20, 2020
@shodanshok
zed: do not offline a missing device if no spare is available
232529c 
Due to commit bd813523 a removed device is now explicitly offlined
by zed if no spare is available, rather than the letting ZFS detecting
it as UNAVAIL. This however broke autoreplacing of whole-disk devices,
as decribed here: openzfs#10577
In short: when a new device is reinserted in the same slot, zed will
try to ONLINE it without letting ZFS recreate the necessary partition
table.

The following change simply avoids to OFFLINE the removed device
if no spare is available (or if `spare_on_remove` is false).

Signed-off-by: Gionatan Danti <[email protected]>
@shodanshok shodanshok mentioned this issue Jul 20, 2020
Closed
12 tasks
@shodanshok
Copy link
Contributor Author

@loli10K @behlendorf Upon futher debug, it turned out that the main issue was not related to device attach, rather to device removal. In short: when a new device is reinserted in the same slot, zed will try to ONLINE it without letting ZFS recreate the necessary partition table.

For further info, please give a look here: #10599

behlendorf added a commit to behlendorf/zfs that referenced this issue Aug 18, 2020
@behlendorf
ZED: Do not offline a missing device if no spare is available
d0d32ca 
Due to commit d48091d a removed device is now explicitly offlined by
the ZED if no spare is available, rather than the letting ZFS detect
it as UNAVAIL. This broke auto-replacing of whole-disk devices, as
decribed in issue openzfs#10577.  In short, when a new device is reinserted
in the same slot, the ZED will try to ONLINE it without letting ZFS
recreate the necessary partition table.

This change simply avoids setting the device OFFLINE when removed if
no spare is available (or if spare_on_remove is false).  This change
has been left minimal to allow it to be backported to 0.8.x release.
The auto_offline_001_pos ZTS test has been updated accordingly.

Some follow up work is planned to update the ZED so it transitions
the vdev to a REMOVED state.  This is a state which has always
existed but there is no current interface the ZED can use to
accomplish this.  Therefore it's being left to a follow up PR.

Co-authored-by: Gionatan Danti <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Issue openzfs#10577
@behlendorf behlendorf mentioned this issue Aug 18, 2020
Merged
12 tasks
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Sep 16, 2020
@behlendorf @tonyhutter
ZED: Do not offline a missing device if no spare is available
a72ae9b 
Due to commit d48091d a removed device is now explicitly offlined by
the ZED if no spare is available, rather than the letting ZFS detect
it as UNAVAIL. This broke auto-replacing of whole-disk devices, as
described in issue openzfs#10577.  In short, when a new device is reinserted
in the same slot, the ZED will try to ONLINE it without letting ZFS
recreate the necessary partition table.

This change simply avoids setting the device OFFLINE when removed if
no spare is available (or if spare_on_remove is false).  This change
has been left minimal to allow it to be backported to 0.8.x release.
The auto_offline_001_pos ZTS test has been updated accordingly.

Some follow up work is planned to update the ZED so it transitions
the vdev to a REMOVED state.  This is a state which has always
existed but there is no current interface the ZED can use to
accomplish this.  Therefore it's being left to a follow up PR.

Reviewed-by: Gionatan Danti <[email protected]>
Co-authored-by: Gionatan Danti <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes openzfs#10577
Closes openzfs#10730
jsai20 pushed a commit to jsai20/zfs that referenced this issue Mar 30, 2021
@behlendorf @jsai20
ZED: Do not offline a missing device if no spare is available
00d2ce8 
Due to commit d48091d a removed device is now explicitly offlined by
the ZED if no spare is available, rather than the letting ZFS detect
it as UNAVAIL. This broke auto-replacing of whole-disk devices, as
described in issue openzfs#10577.  In short, when a new device is reinserted
in the same slot, the ZED will try to ONLINE it without letting ZFS
recreate the necessary partition table.

This change simply avoids setting the device OFFLINE when removed if
no spare is available (or if spare_on_remove is false).  This change
has been left minimal to allow it to be backported to 0.8.x release.
The auto_offline_001_pos ZTS test has been updated accordingly.

Some follow up work is planned to update the ZED so it transitions
the vdev to a REMOVED state.  This is a state which has always
existed but there is no current interface the ZED can use to
accomplish this.  Therefore it's being left to a follow up PR.

Reviewed-by: Gionatan Danti <[email protected]>
Co-authored-by: Gionatan Danti <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes openzfs#10577
Closes openzfs#10730
sempervictus pushed a commit to sempervictus/zfs that referenced this issue May 31, 2021
@behlendorf
ZED: Do not offline a missing device if no spare is available
e259638 
Due to commit d48091d a removed device is now explicitly offlined by
the ZED if no spare is available, rather than the letting ZFS detect
it as UNAVAIL. This broke auto-replacing of whole-disk devices, as
described in issue openzfs#10577.  In short, when a new device is reinserted
in the same slot, the ZED will try to ONLINE it without letting ZFS
recreate the necessary partition table.

This change simply avoids setting the device OFFLINE when removed if
no spare is available (or if spare_on_remove is false).  This change
has been left minimal to allow it to be backported to 0.8.x release.
The auto_offline_001_pos ZTS test has been updated accordingly.

Some follow up work is planned to update the ZED so it transitions
the vdev to a REMOVED state.  This is a state which has always
existed but there is no current interface the ZED can use to
accomplish this.  Therefore it's being left to a follow up PR.

Reviewed-by: Gionatan Danti <[email protected]>
Co-authored-by: Gionatan Danti <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes openzfs#10577
Closes openzfs#10730
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

zed: do not offline a missing device if no spare is available shodanshok/zfs
1 participant
@shodanshok