[bug] Failed to limit disk quota when using driver grpquota

[kuei-chi]

Ⅰ. Issue Description

I tried to limit my container disk quota with driver grpquota. But it was failed and raised error in log. My kernel version and quota-tools version are larger than requirements.

Ⅱ. Describe what happened

I launched pouchd with comand pouchd --quota-driver grpquota and used the example command in [document] (https://github.com/alibaba/pouch/blob/6913f762e0a0acb0aca0a20677068858f734f8a4/docs/features/pouch_with_diskquota.md#container-rootfs-diskquota) to test it.

But the rootfs was not limited in container.

root@ubuntu1804:~# pouch run -ti --disk-quota /=10g registry.hub.docker.com/library/busybox:latest df -h
registry.hub.docker.com/library/busybox:latest:                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:061ca9704a714ee3e8b80523ec720c64f6209ad3f97c0ff7cb9ec7d19f15149f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:4415a904b1aca178c2450fd54928ab362825e863c0ad5452fd020e92f7a6a47e: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:697743189b6d255069caf6c455be10c7f8cae8076c6f94d224ae15cd41420e87:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:d8233ab899d419c58cf3634c0df54ff5d8acc28f8173f09c21df4a07229e1205:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 14.7s                                                                    total:  4.6 Ki (321.0 B/s)
Filesystem                Size      Used Available Use% Mounted on
overlay                  29.0G      2.6G     24.9G   9% /
tmpfs                   492.6M         0    492.6M   0% /dev
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/sda3                29.0G      2.6G     24.9G   9% /etc/hostname
/dev/sda3                29.0G      2.6G     24.9G   9% /etc/hosts
/dev/sda3                29.0G      2.6G     24.9G   9% /etc/resolv.conf
tmpfs                   492.6M         0    492.6M   0% /sys/fs/cgroup
tmpfs                   492.6M         0    492.6M   0% /proc/kcore
tmpfs                   492.6M         0    492.6M   0% /proc/timer_list
tmpfs                   492.6M         0    492.6M   0% /proc/sched_debug

And pouchd log shows error:

ERRO[2019-04-02T02:27:33.489656898-07:00] failed to getfattr, dir: (/var/lib/pouch/containerd/root/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs), stdout: (), stderr: (/var/lib/pouch/containerd/root/io.containerd.snapshotter.v1.overlayfs/snapshots/3/fs: system.subtree: Operation not supported
), exit: (1), err: (exit status 1)

Ⅲ. Describe what you expected to happen

I excepted disk quota was limited like document shows.

$ pouch run -ti --disk-quota /=10g registry.hub.docker.com/library/busybox:latest df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  10.0G     24.0K     10.0G   0% /
tmpfs                    64.0M         0     64.0M   0% /dev
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                    64.0M         0     64.0M   0% /run
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     1.9G         0      1.9G   0% /sys/firmware
tmpfs                     1.9G         0      1.9G   0% /proc/scsi

Ⅳ. How to reproduce it (as minimally and precisely as possible)

1. Run pouchd with command: pouchd --quota-driver grpquota
2. Follow document command to limit disk size: pouch run -ti --disk-quota /=10g registry.hub.docker.com/library/busybox:latest df -h

Ⅴ. Anything else we need to know?

I'm using vagrant VM as test environment. My Vagrantfile is

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure("2") do |config|
  config.vm.box = "generic/ubuntu1804"
  config.vm.network "private_network", ip: "192.168.55.66"
  config.vm.provider "virtualbox" do |vb|
    vb.gui = true
    vb.memory = "1024"
    disk1 = 'disk1.vdi'
    unless File.exist?(disk1)
      # Create a 50GB additional disk for test.
      vb.customize ['createhd', '--filename', disk1, '--size', 50 * 1024]
    end
    vb.customize ['storageattach', :id, '--storagectl', 'IDE Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', disk1]

  end
end

Ⅵ. Environment:

• pouch version (use pouch version):

Arch:            amd64
BuildTime:       2019-02-14T02:21:34+00:00
GitCommit:       1.2.0
GoVersion:       go1.10.4
KernelVersion:   4.15.0-46-generic
Os:              linux
Version:         1.2.0
APIVersion:      1.24

• OS (e.g. from /etc/os-release):

Distributor ID: Ubuntu
Description:    Ubuntu 18.04.2 LTS
Release:        18.04
Codename:       bionic

• Kernel (e.g. uname -a):

Linux ubuntu1804.localdomain 4.15.0-46-generic #49-Ubuntu SMP Wed Feb 6 09:33:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

root@ubuntu1804:~# dpkg -l | grep quota
ii  quota                                 4.04-2ubuntu0.1                   amd64     
ii  quotatool                             1:1.4.12-2build1                  amd64       

• Others:
  My filesystem is ext4.

/dev/sda3 / ext4 rw,relatime,grpquota,errors=remount-ro,data=ordered 0 0
/dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0

Besides, I tried to add a new disk and mount it to /var/lib/pouch and prevented to use rootfs. But it still didn't work.

/dev/sda3 / ext4 rw,relatime,grpquota,errors=remount-ro,data=ordered 0 0
/dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0
/dev/sdb /var/lib/pouch ext4 rw,relatime,grpquota,data=ordered 0 0

Thanks for your help.

[kuei-chi]

Hi, do you have any idea about this problem? Thanks!!!

[rudyfly]

delele this option --quota-driver grpquota when run pouchd

[kuei-chi]

Hi master @rudyfly,

If I remove --quota-driver grpquota when run pouchd, it would use prjquota by default. Setting quota would failed because I didn't format my disk with project,quota option.

root@ubuntu1804:~# pouch run -ti --disk-quota /=10g registry.hub.docker.com/library/busybox:latest df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  29.0G      3.1G     24.5G  11% /
tmpfs                   492.6M         0    492.6M   0% /dev
shm                      64.0M         0     64.0M   0% /dev/shm
/dev/sda3                29.0G      3.1G     24.5G  11% /etc/hostname
/dev/sda3                29.0G      3.1G     24.5G  11% /etc/hosts
/dev/sda3                29.0G      3.1G     24.5G  11% /etc/resolv.conf
tmpfs                   492.6M         0    492.6M   0% /sys/fs/cgroup
tmpfs                   492.6M         0    492.6M   0% /proc/kcore
tmpfs                   492.6M         0    492.6M   0% /proc/timer_list
tmpfs                   492.6M         0    492.6M   0% /proc/sched_debug

pouchd error:

failed to remount prjquota, mountpoint: (/), stdout: (), stderr: (mount: /: mount point not mounted or bad option.

Because our services are running online now, it's difficult to format every servers so we want to use grpquota to limit the disk size of containers.

May I know if Alibaba uses grpquota on its online services?

Thank you very much.

[rudyfly]

So, you should reformat your disk with project,quota, this doc maybe helpful for you, https://blog.csdn.net/luckyapple1028/article/details/75754591, also you can see kernel docs.