Make taskqueues mimic loopback device's -20 nice value

[ryao]

A comparison of swap on zvols with swap on the loopback device revealed
that the loopback device sets the nice value to -20. An audit of the
code does not reveal any downside to doing that, so we can do it too.

[ryao]

My earlier tests with dd if=/dev/zero of=/test bs=4096 count=1048576 showed 220MB/sec sequential write performance on a 6-disk raidz2 vdev. Now I am seeing 286MB/sec. I believe that this patch caused it, although I have not had time to formally test my server with and without this patch to be certain.

[behlendorf]

Interesting, I'm surprised there's such a large effect. If true this is certainly some low hanging fruit worth investigating. However, rather than applying this policy to all taskq threads it would would be better to use the existing provided Solaris interface. The taskq_create() function takes a priority which is roughly translated in to a niceness value. I would try and target only a subset of the threads by passing a priority of MAX_RT_PRIO.

[behlendorf]

I did some quick testing of this change on a larger 3 groups of 8+2raidz configuration and didn't see any performance improvement. While this was just a quick test we're going to need to run some more careful targeted tests before merging this to determine if this improves things or not.