Linux-VServer - User contributions [en]

util-vserver:Cgroups

2010-02-18T13:51:04Z

Bobnormal: /* Prerequisites */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you need to have <tt>/dev/cgroup</tt> mounted.

Recent versions of <tt>util-vserver</tt> sort this out for you by including the appropriate mount command in the <tt>util-vserver</tt> <tt>init</tt> (ie: runlevel) script included in the <tt>util-vserver</tt> distribution, however this apparently only works for the <tt>sysv</tt> <tt>init</tt> script, and not the Debian or Gentoo ones.

If you were to mount the <tt>cgroup</tt> Control Groups filesystem manually, you would use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o ''<subsystems>'' /dev/cgroup</tt>

Where <tt>''<subsystems>''</tt> is something like <tt>cpuset,memory</tt>.

To avoid the need for manual configuration after reboot, on Gentoo you may wish to add the cgroup mount to <tt>/etc/fstab</tt>. For Debian see the live examples section at the bottom of this page.
<pre>
none /dev/cgroup cgroup cpu,cpuset,memory 0 2
</pre>

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T13:50:42Z

Bobnormal: /* Prerequisites */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you need to have <tt>/dev/cgroup</tt> mounted.

Recent versions of <tt>util-vserver</tt> sort this out for you by including the appropriate mount command in the <tt>util-vserver</tt> <tt>init</tt> (ie: runlevel) script included in the <tt>util-vserver</tt> distribution, however this apparently only works for the <tt>sysv</tt> <tt>init</tt> script, and not the Debian or Gentoo ones.

If you were to mount the <tt>cgroup</tt> Control Groups filesystem manually, you would use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o ''<subsystems>'' /dev/cgroup</tt>

Where <tt>''<subsystems>''</tt> is something like <tt>cpuset,memory</tt>.

To avoid the need for manual configuration after reboot, on Gentoo you may wish to add the cgroup mount to <tt>/etc/fstab</tt>. For Debian see the live examples section at the bottom of this page.
: <tt>none /dev/cgroup cgroup cpu,cpuset,memory 0 2</tt>

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T13:50:26Z

Bobnormal: /* Prerequisites */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you need to have <tt>/dev/cgroup</tt> mounted.

Recent versions of <tt>util-vserver</tt> sort this out for you by including the appropriate mount command in the <tt>util-vserver</tt> <tt>init</tt> (ie: runlevel) script included in the <tt>util-vserver</tt> distribution, however this apparently only works for the <tt>sysv</tt> <tt>init</tt> script, and not the Debian or Gentoo ones.

If you were to mount the <tt>cgroup</tt> Control Groups filesystem manually, you would use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o ''<subsystems>'' /dev/cgroup</tt>

Where <tt>''<subsystems>''</tt> is something like <tt>cpuset,memory</tt>.

To avoid the need for manual configuration after reboot, on Gentoo you may wish to add the cgroup mount to <tt>/etc/fstab</tt>. For Debian see the live examples section at the bottom of this page.
: <tt>none /dev/cgroup cgroup cpu,cpuset,memory</tt>

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:51:46Z

Bobnormal: /* Prerequisites */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you need to have <tt>/dev/cgroup</tt> mounted.

Recent versions of <tt>util-vserver</tt> sort this out for you by including the appropriate mount command in the <tt>util-vserver</tt> <tt>init</tt> (ie: runlevel) script included in the <tt>util-vserver</tt> distribution.

If you were to mount the <tt>cgroup</tt> Control Groups filesystem manually, you would use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o ''<subsystems>'' /dev/cgroup</tt>

Where <tt>''<subsystems>''</tt> is something like <tt>cpuset,memory</tt>.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:37:41Z

Bobnormal: /* Prerequisites */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you will need to have /dev/cgroup mounted.

Recent versions of <tt>util-vserver</tt> sort this out for you by including the appropriate mount command in the <tt>init</tt> (runlevel) script included in the <tt>util-vserver</tt> distribution.

To mount manually, you could use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o cpuset,memory /dev/cgroup</tt>

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:37:14Z

Bobnormal: /* Kernel configuration */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Prerequisites ==

To use <tt>util-vserver</tt>'s Control Groups (<tt>cgroups</tt>) support, you will need to have /dev/cgroup mounted. Recent versions of util-vserver sort this out for you by including the appropriate mount command in the <tt>init</tt> (runlevel) script. To mount manually, you could use something like:
: <tt># mkdir /dev/cgroup
: # mount -t cgroup -o cpuset,memory /dev/cgroup</tt>

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:20:45Z

Bobnormal: /* Real world Examples of Scheduling */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place.

Please add the following information for each example you put here (use <tt>vserver-info</tt>).
* Base kernel version
* vServer version
* Other kernel patches in use (<tt>grsec</tt>, etc.)
* <tt>util-vserver</tt> release

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:19:45Z

Bobnormal: /* real world exemples of scheduling */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= Real world Examples of Scheduling =

This section is for working and tested examples you have put in place. Please add the vServer patch, base kernel version, util-vserver release for each example you put here (use <tt>vserver-info</tt>).

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:17:43Z

Bobnormal: /* Kernel configuration */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('''CGroup Namespaces''') is unset for the time being.

'''CGroup Namespaces''' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= real world exemples of scheduling =

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:17:20Z

Bobnormal: /* Kernel configuration */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> (''CGroup Namespaces'') is unset for the time being.

''CGroup Namespaces'' are a different approach to namespaces than that used by Linux vServer, and are not currently supported.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= real world exemples of scheduling =

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:16:49Z

Bobnormal: /* Kernel configuration */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> (''CGroup Namespaces'') is unset for the time being (''CGroup Namespaces'' are a different approach to namespaces than that used by Linux vServer, and are not currently supported)

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= real world exemples of scheduling =

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-18T10:16:33Z

Bobnormal: /* Kernel configuration */

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure <tt>CONFIG_CGROUP_NS</tt> ('CGroup Namespaces') is unset for the time being ('CGroup Namespaces' are a different approach to namespaces than that used by Linux vServer, and are not currently supported)

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= real world exemples of scheduling =

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

Resource Limits

2010-02-18T10:12:52Z

Bobnormal: add leading reference to cgroups

'''NOTE: This page describes Linux <tt>rlimit</tt> support (an SVr4/BSD4.3/POSIX.1-2001 standard). vServers can also be managed using the Linux Control Group (<tt>cgroup</tt>) system, for information on Control Group based resource management, see [[util-vserver:Cgroups|util-vserver - Cgroups]].'''

Most properties related to system resources, might it be the memory consumption, the number of processes or file-handles, qualify for imposing limits on them.

The Linux kernel provides the getrlimit and setrlimit system calls to get and set resource limits ''per process''. Each resource has an associated soft and hard limit. The soft limit is the value that the kernel enforces for the corresponding resource. The hard limit acts as a ceiling for the soft limit: an unprivileged process may only set its soft limit to a value in the range from 0 up to the hard limit, and (irreversibly) lower its hard limit. A privileged process (one with the CAP_SYS_RESOURCE capability) may make arbitrary changes to either limit value.

The Linux-VServer kernel extends this system to provide resource limits for ''whole contexts'', not just single processes. Additionally a few new limit types missing in the vanilla kernel were introduced.

Additionally the context limit system keeps track of observed maxima and resource limit hits, to provide some feedback for the administrator. See [[Context Accounting]] for details.

__FORCETOC__

== List of Resource Limits ==

Below is a list of resource limits used for contexts and processes within. The tables contain the following information:

; ID : Resource limit ID
; Name : Human readable identifier used in userspace utilities
; procfs : Name used in /proc/virtual/*/limit
; ulimit : command line switch for the ulimit utility
; Unit : Aprropriate unit for the limit
; Tag : Special resource limit code to denote if resources are accounted, enforced (see below)
; Description : Description of capability/flag effects

=== Special Resource Limit Codes ===

The tag column may contain one or more of the following tags:

{| class="wikitable_inline"
! Tag
! Description
|-
| A
| Limit will be accounted
|-
| E
| Limit will be enforced
|}

=== Linux Resource Limits ===

Below is a list of resource limits available in vanilla Linux (>=2.6.18).

{| class="wikitable_list"
! ID
! Name
! procfs
! ulimit
! Tag
! Unit
! Description
|-
| 0
| CPU
| --
| -t
|
| ms
| CPU time in ms
|-
| 1
| FSIZE
| --
| -f
|
| KiB
| Maximum file size
|-
| 2
| DATA
| --
| -d
|
| KiB
| Maximum size of the data segment
|-
| 3
| STACK
| --
| -s
|
| KiB
| Maximum stack size
|-
| 4
| CORE
| --
| -c
|
| KiB
| Maximum core file size
|-
| 5
| RSS
| RSS
| -m
| AE
| page
| Maximum resident set size
|-
| 6
| NPROC
| PROC
| -u
| AE
| 1
| Maximum number of processes
|-
| 7
| NOFILE
| FILES
| -n
| AE
| 1
| Maximum number of open files
|-
| 8
| MEMLOCK
| VML
| -l
| AE
| page
| Maximum locked-in-memory address space
|-
| 9
| AS
| VM
| -v
| AE
| page
| Maximum address space size
|-
| 10
| LOCKS
| LOCKS
| -x
| AE
| 1
| Maximum file locks held
|-
| 11
| SIGPENDING
| --
| -i
|
| 1
| Maximum number of pending signals
|-
| 12
| MSGQUEUE
| MSGQ
| -q
| AE
| Byte
| Maximum bytes in POSIX mqueues
|-
| 13
| NICE
| --
|
|
| 1
| Maximum nice prio allowed to raise to
|-
| 14
| RTPRIO
| --
| -r
|
| 1
| Maximum realtime priority
|}

=== Linux-VServer Resource Limits ===

Below is a list of additional resource limits available in the Linux-VServer kernel.

{| class="wikitable_list"
! ID
! Name
! procfs
! Tag
! Unit
! Description
|-
| 16
| NSOCK
| SOCK
| AE
| 1
| Maximum number of open sockets
|-
| 17
| OPENFD
| OFD
| AE
| 1
| Maximum number of open file descriptors
|-
| 18
| ANON
| ANON
| A
| page
| Maximum anonymous memory size
|-
| 19
| SHMEM
| SHM
| AE
| page
| Maximum shared memory size
|-
| 20
| SEMARY
| SEMA
| A
| 1
| Maximum number of semaphore arrays
|-
| 21
| NSEMS
| SEMS
| A
| 1
| Maximum number of semaphores
|-
| 22
| DENTRY
| DENT
| AE
| 1
| Maximum number of dentry structs
|}

== Determining the Page Size ==

You can use the following program to determine the page size for your architecture (if it supports the getpagesize() function)

<pre>
#include <stdlib.h>
#include <stdint.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
int page_size = getpagesize();
printf("The page size is %d\n", page_size);
exit(0);
}
</pre>

Here's how to compile and run it (assuming you save it as pagesize.c):

<pre>
# gcc pagesize.c -o pagesize
# ./pagesize
The page size is 4096
</pre>

'''Note''': Whether getpagesize() is present as a Linux system call depends on the architecture. If it is, it returns the kernel symbol PAGE_SIZE, which is architecture and machine model dependent. Generally, one uses binaries that are architecture but not machine model dependent, in order to have a single binary distribution per architecture. This means that a user program should not find PAGE_SIZE at compile time from a header file, but use an actual system call, at least for those architectures (like sun4) where this dependency exists. Here libc4, libc5, glibc 2.0 fail because their getpagesize() returns a statically derived value, and does not use a system call. Things are OK in glibc 2.1.

If you prefer, you can get the pagesize using python, just start a python console and write:

<pre>
>>> import resource
>>> resource.getpagesize()
4096
</pre>

Here are some typical page sizes ...

{| class="wikitable_list"
! arch
! pagesize
|-
| arm, extensa, s390, sh64, x86, x86_64, v850
| 4k
|-
| alpha, cris
| 8k
|-
| m68k, sparc
| 4k, 8k
|-
| powerpc
| 4k, 64k
|-
| parisc
| 4k, 16k, 64k
|-
| ia64, mips
| 4k, 8k, 16k, 64k
|}

== Configure Resource Limits in util-vserver ==

For example, if the adress space size (AS) and the resident set size (RSS) should be limited, the appropriate config files should be the following. You might have to create parent directories first.

<pre>
# ls -al /etc/vservers/myguest/rlimits
total 28
drwxr-xr-x 2 root root 4096 2005-08-24 12:37 .
drwxr-xr-x 5 root root 4096 2005-08-24 00:22 ..
-rw-r--r-- 1 root root 6 2005-08-24 12:43 as
-rw-r--r-- 1 root root 6 2005-08-24 12:37 rss

# cat /etc/vservers/myguest/rlimits/as
90000

# cat /etc/vservers/myguest/rlimits/rss
10000
</pre>

util-vserver:Cheatsheet

2010-02-16T16:56:22Z

Bobnormal: /* What are the services that need to be running in a basic setup ? */

This is a cheatsheet for util-vserver. Some of these recipes have been arrived at by users, using trial and error, so they may not always be the "official" solutions.

== Where to find the latest util-vserver ? ==

:Look here:

:<pre>http://people.linux-vserver.org/~dhozac/t/uv-testing/</pre>

:or checkout subversion repository:

:<pre>svn co http://svn.linux-vserver.org/svn/util-vserver/trunk util-vserver</pre>

:And for the lastest kernel patch:

:<pre>http://vserver.13thfloor.at/Experimental/</pre>

== How to add an IP address to a live guest ?==

# create the address on the host: <pre>ifconfig eth0:10 172.16.0.145/12</pre> or <pre>ip addr add dev eth0 172.16.0.145/12</pre> or even <pre>ip addr add dev dummy0 172.16.0.145/12</pre>
# add the adress:<pre>naddress --nid guestname --add --ip 172.16.0.145 --bcast 172.31.255.255</pre>
#* Depending on what you use the vserver for, you may not want the broadcast thing at all; you can also use a /32 mask for the IP in this case.
# add the ip to the config directory to make it stick if you restart <br/>
# enter the guest and verify services that need to listen on the ip, restart if necessary (for me it was not).

== How to change the machine type live ?==

: For this you can use the '''vuname''' command,
:: <pre>sudo vuname --xid 40133 -s -t 'machine=i686'</pre>

: Possible values for TAG are:
:: <pre>context, sysname, nodename, release, version, machine, domainname</pre>

:To make those change permanent you have to modify the files in your /etc/vservers/<vserver-name>/uts/machine file see the great flower page for details.

== What are the main vutils commands ?==

: '''Main ones are''' :

:# vserver : to create enter , start, stop, destroy etc the vserver guests.
:# vuname : to change live guest : context, sysname, nodename, release, version, machine or domainname.
:# vattribute: to change capabilities of a live guest (bcap, ccap and flags).
:# nattribute: to change network capabilities and flags of a live guest (ncaps and flags).
:# vlimit : to change the limits of a live guest (cpu, fsize, data, stack, core, rss, nproc,nofile, memlock, as, locks, msgqueue, nsock, openfd, anon, shmem, semary,nsems, and dentry).
:# vsched : to change schheduling parameters of live guest.
:# vdevmap : to remap device or give permission to a guest to create/read/open specific devices.
:# vdlimit : to change the directory limits of a live guest.
:: those parameters can be '''setup permanently''' in the configuration files, see the vutil flower page for details ([http://www.nongnu.org/util-vserver/doc/conf/configuration.html The Great Flower Page]).

:'''Other usefull vutils''' :

:# vapt-get : to run apt-get from the host in one or more guest.
:# vemerge : to run emerge from the host in one or more guest.
:# vmount : to mount file systems inside guest from the host.
:# vsomething : to run one command to all vserver in one go.

: '''daily usage one''':

:# vps : ps in a guest aware way (see all the process of host+guest).
:# vtop : top in a guest aware way (see all the process of host+guest).
:# vserver-info : gives version and other usefull information about the utils installed.
:# vserver-stat : gives informations about running guests.

== What are the services that need to be running in a basic setup ?==

: '''Services to allow on boot''' :

:# vprocunhide : this one is mandatory to allow access to required parts of /proc on the guest
:# vservers-default: this one will start the vservers on reboot, most users need this one
:# util-vserver : ? do not know what it does.

: this article in incomplete, please do not hesistate to add your comments !

util-vserver:Cgroups

2010-02-16T10:51:22Z

Bobnormal: /* cgroup and CFS based CPU hard limiting that replaces sched_hard */ formatting, link to prerelease util-vserver, grammar, etc.

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure CONFIG_CGROUP_NS is unset so guests start properly for the time being.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

===References===
You can find documentation about the CFS hard limiting in <tt>Documentation/scheduler/sched-cfs-hard-limits.txt</tt> inside your kernel source dir.

===Requirements===
This feature is currently available in <tt>patch-2.6.31.2-vs2.3.0.36.15.diff</tt> and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vServer start you need a recent utils package. It worked for me with: <tt>0.30.216-pre2864</tt>. (Download from [http://people.linux-vserver.org/~dhozac/t/uv-testing/ util-vserver prereleases])

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

===Configuration===
Example for an upper bound of 2/5th (or 40%) of the all CPU power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the resources used by the cgroup.

If you set both CPU share AND hard limit the system will do fine but hard limits takes priority over CPU share scheduling, so CPU share will do the job but each cgroup will have an upper bound that it cannot cross even if the CPU share you gave it is higher.

The hard limit feature adds 3 cgroup files for the CFS group scheduler:
* <tt>cfs_runtime_us</tt>: Hard limit for the group in microseconds.
* <tt>cfs_period_us</tt>: Time period in microseconds within which hard limits is enforced.
* <tt>cfs_hard_limit</tt>: The control file to enable or disable hard limiting for the group.

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= real world exemples of scheduling =

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

util-vserver:Cgroups

2010-02-16T09:44:16Z

Bobnormal: joke

Bears run away when you yell at them, even <tt>lynx</tt>es.

== Kernel configuration ==

When configuring your kernel for cgroups with util-vserver you must make sure CONFIG_CGROUP_NS is unset so guests start properly for the time being.

== Draft - Distributing cpu shares with cgroups ==

From what i gathered in sched-design-CFS.txt [http://people.redhat.com/mingo/cfs-scheduler/sched-design-CFS.txt]

This is simply done by adjusting the cpu.shares. Just do:

echo '512' > /dev/cgroup/<guest name>/cpu.shares

The share you get is equal to the guest's share divided by the sum of the cpu shares of all the guest. So for exemple :

<pre>
vserver guest 1 => 512
vserver guest 2 => 512
vserver guest 3 => 2048
vserver guest 4 => 512
</pre>

so you have a total of 3584 cpu shares (2048+512+512+512) , then you get :

<pre>
vserver guest 1 => 512 / 3584 = 14% cpu
vserver guest 2 => 512 / 3584 = 14% cpu
vserver guest 3 => 2048 / 3584 = 57% cpu
vserver guest 4 => 512 / 3584 = 14% cpu
</pre>
<br/>

Note that this is fair scheduling and this will not enfore HARD limit (as far as i know).

== Making share permanent with util vserver ==

You must use the "cgroup" directory. You can apply defaults to all vservers or choose different settings for each guest:

* /etc/vservers/.default/cgroup , this directory contains settings applying to all guest when they start
* /etc/vservers/<guestname>/cgroup , this directory contains settings for the guest when it starts.

Example :

<pre>
mkdir /etc/vservers/.defaults/cgroup
mkdir /etc/vservers/<guestname>/cgroup
echo '2048' > /etc/vservers/<guestname>/cgroup/cpu.shares
# List of CPUs
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.cpus
# NUMA nodes
echo 1 > /etc/vservers/<guestname>/cgroup/cpuset.mems
</pre>

Note that /etc/vservers is an example, in my Aqueos install i use /usr/local/etc/vservers but /etc/vservers seems to be the defaults for the classic installs.

Regards,
Ghislain.

== cgroup and CFS based CPU hard limiting that replaces sched_hard ==

You can find documentation about the cfs hard limiting in Documentation/scheduler/sched-cfs-hard-limits.txt inside your kernel source dir.

This feature is currently available in patch-2.6.31.2-vs2.3.0.36.15.diff and is in testing phase as of this patch set so report any bugs to the mailing list.

To get the hard limit setup on every vserver start you need a recent utils package. It worked for me with: 0.30.216-pre2864.

Before trying to setup limits for one guest you should mount the cgroup filesystem:

[ -d /dev/cgroup ] || mkdir /dev/cgroup
mount -t cgroup -ocpu none /dev/cgroup

Example for an upper bound of 2/5th (or 40%) of the all cpu power that a guest/cgroup can use :

<pre>
# force CFS hard limit (only needed for older kernel versions)
# echo 1 > /etc/vservers/<guestname>/cgroup/cpu.cfs_hard_limit
# time assigned to guest (in microseconds) 200000 = 0,2 sec
echo 200000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_runtime_us
# in each specified period (in microseconds) 500000 = 0,5 sec
echo 500000 > /etc/vservers/<guestname>/cgroup/cpu.cfs_period_us
</pre>

This limit is an hard limit, see it like an upper wall for the ressources used by the cgroup.
If you set both cpu share AND hard limit the system will do fine but hard limits takes priority over cpu share scheduling, so cpu share will do the job but each cgroup will have an upper bound that it cannot cross even if the cpu share you gived it is higher.

<pre>
Hard limit feature adds 3 cgroup files for CFS group scheduler:
cfs_runtime_us: Hard limit for the group in microseconds.
cfs_period_us: Time period in microseconds within which hard limits is enforced.
cfs_hard_limit: The control file to enable or disable hard limiting for the group.
</pre>

<br/>

== using cgroup to enforce memory limits ==

in linux-vserver patch version vs2.3.0.36.29 memory limiting by cgroup is introduced. to use it you need to have the following config lines in your kernel build (aditionally to the others mentioned for cgroup cpu limits):

* CONFIG_RESOURCE_COUNTERS=y
* CONFIG_CGROUP_MEM_RES_CTLR=y
* CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

make sure /dev/cgroup is mounted with -o...,memory to be able to use this feature. The following files let you adjust memory limits of a running vserver (create them in /etc/vservers/-vserver-name-
/cgroup/ to make them permanent):

* memory.memsw.limit_in_bytes the total memory limit (memory+swap) of your cgroup context
* memory.limit_in_bytes the total memory limit

values are stored in bytes. When writing to those files you can use suffixes: K,M,G.

Note: cgroup memory limits are to replace rss.soft and rss.hard some time in the future.
As for vs2.3.0.36.29 the tools top and free do not show the limited memory amount asigned to the guest.

For a deeper understanding check out Documentation/cgroups/memory.txt of your kernel source tree.

= real world exemples of scheduling =

this part is to be filled with exemple you have put in place and are working and have been tested, please add the patch and kernel version for each exemple you put here.

== Ben's install on Debian Lenny ==

I used the kernels from [http://repo.psand.net], described at [http://kernels.bristolwireless.net/]. I've done this on a few versions, works for 2.6.31.7 with patch vs2.3.0.36.27 on amd64, also 2.6.31.11 with patch vs2.3.0.36.28. I used the stock Lenny util-vserver, patched as described below. The kernel config is critically important, with specific cgroup options necessary in order to get cgroups working in this way. Check the configs for the [http://repo.psand.net] kernels to see which ones I used.

==== Getting Lenny Ready ====

There's a very old version of util-vserver on Lenny, it needs this patch applying before it will set the cgroups properly, it basically only adds one line:

--- /usr/lib/util-vserver/vserver.suexec.orig 2008-12-12 22:56:25.000000000 -0600
+++ /usr/lib/util-vserver/vserver.suexec 2009-08-20 02:11:42.000000000 -0500
@@ -22,7 +22,8 @@ test -z "$is_stopped" -o "$OPTION_INSECU
exit 1
}
generateOptions "$VSERVER_DIR"
-addtoCPUSET "$VSERVER_DIR"
+addtoCPUSET "$VSERVER_DIR"
+attachToCgroup "$VSERVER_DIR"

user=$1
shift

Next I added a correctly mounted cgroup file system on /dev/cgroup/.

$ mkdir /dev/cgroup
$ mount -t cgroup vserver /dev/cgroup

For the util-vserver to do the right thing, this directory needed adding too:

$ mkdir /etc/vservers/.defaults/cgroup

==== Sharing out the CPU between guest servers ====

I have a few test guests hanging around that I play with, call onetime, twotime, threetime, fourtime and fivetime. I order to set the shares for each guest I did this:

mkdir /etc/vservers/fivetime/cgroup/ /etc/vservers/fourtime/cgroup/ /etc/vservers/threetime/cgroup/ /etc/vservers/twotime/cgroup/ /etc/vservers/twotime/cgroup/
echo "512" > /etc/vservers/fivetime/cgroup/cpu.shares
echo "1024" > /etc/vservers/fourtime/cgroup/cpu.shares
echo "1024" > /etc/vservers/threetime/cgroup/cpu.shares
echo "1536" > /etc/vservers/twotime/cgroup/cpu.shares
echo "1024" > /etc/vservers/onetime/cgroup/cpu.shares

Then started the guests. When the system was loaded (I used one instance of cpuburn on each server - not advised but a useful test) they each should have got the following percentage of CPU.

{| class="wikitable"
! Guest Name !! cpu.share given !! percentage of cpu
|-
| fivetime || 512 || 10%
|-
| fourtime || 1024 || 20%
|-
| threetime || 1024 || 20%
|-
| twotime || 1536 || 30%
|-
| onetime || 1024 || 20%
|}

This didn't quite happen, as each process could migrate to other CPUs. When I fixed every guest to use only one of the available CPUs (see below how I did this) the percentage of processing time alloted to each guest were then pretty much exact! Each process was given exactly it's designated percentage of time according to vtop.

==== Dishing out different processors sets to different guest servers ====

The "cpuset" for each guest is the subset of CPUs which it is permitted to use. I found out the number of CPUs available on my system by doing this:

$ cat /dev/cgroup/cpuset.cpus

This gave me the result 0-1, meaning that the overall set for my cgroups consists of CPUs 0 and 1 (for a quad core system one would expect the result 0-3, or for quad core with HT, 0-7). I stopped my guests, then for each guest specified a cpuset containing only CPU 0 for each of them:

$ echo "0" > /etc/vservers/onetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/twotime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/threetime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fourtime/cgroup/cpuset.cpus
$ echo "0" > /etc/vservers/fivetime/cgroup/cpuset.cpus

On restarting the guest, I could see (using vtop) that these guest were only using the CPU 0 (the column "Last used cpu (SMP)" needs to be on in vtop in order to see this). This set up isn't particularly useful, but did allow me to check that the cpu.shares I specified for my guest were working as expected.

==== Doing this to servers live ====

The parameters in the last two sections can be set when the servers are running. For example to move the guest "threetime" so that it could use both CPUs I did this:

$ echo "0-1" > /dev/cgroup/threetime/cpuset.cpus

The processes running on threetime instantly were allocated cycle on both CPUs. Then:

$ echo "1" > /dev/cgroup/threetime/cpuset.cpus

Shifts them all to CPU 1. One can change where cycles are allocated with impunity. The same with CPU shares:

$ echo "4096" > /dev/cgroup/threetime/cpu.shares

Gave threetime a much bigger slice of the processors when it was under load.

'''NOTE''': The range "0-1" is not the only way of specifying a set of CPUs, I could have used "0,1". On bigger systems, with say 8 CPUs one could use "0-2,4,5", which would be the same as "0,1,2,4,5" or "0-2,4-5".

==== Making sure all of this gets set up after a reboot ====

This process will make sure /dev/cgroup is present at boot and correctly mounted:

* patch util-vserver (see above)
* mkdir /etc/vservers/.defaults/cgroup
* mkdir /lib/udev/devices/cgroup (this will mean that the /dev/cgroup is created early in the boot process)
* add the following line to /etc/fstab
vserver /dev/cgroup cgroup defaults 0 0

Applying CPU Limits

2010-02-16T09:40:03Z

Bobnormal: /* Kernel config */ zcat > gzip -d

= Applying CPU limits =

This is a short summary on how to apply limits. This doesn't cover how to choose the values but just on how to apply the numbers in a running vserver without saving the configuration.
note: you can find soft/hard cpu limit with cgroups for vserver2.3 on [[util-vserver:Cgroups]]

== Kernel config ==

First check whether the kernel is with '''CONFIG_VSERVER_HARDCPU''' enabled:

# cat /boot/config-blah | grep CONFIG_VSERVER_HARDCPU
CONFIG_VSERVER_HARDCPU=y

Note: With newer kernels, the config of the currently running kernel can be found with `zcat /proc/config.gz | grep VSERVER`.

== Context flag ==

Then check if the vserver you want to apply limits has the [http://linux-vserver.org/Capabilities_and_Flags#Context_flags_.28cflags.29 sched_hard] context flag enabled by checking the contents of ''/proc/virtual/<xid>/status''. The sched_hard flag is ''0x00000100'', so you have to find this value summed up in the vserver flag (see [http://www.mail-archive.com/vserver@list.linux-vserver.org/msg10448.html this message] for details). If its not enabled, use the following command (more info [http://www.mail-archive.com/vserver@list.linux-vserver.org/msg08385.html here]):

vattribute --set --xid <xid> --flag sched_hard

== Setting the limits ==

Then you should be able to set the limits with a command like this one (more info [http://oldwiki.linux-vserver.org/vsched+explained here]):

vsched --xid <xid> --fill-rate <fill-rate> --interval <interval> \
--fill-rate2 <fill-rate2> --interval2 <interval2> \
--tokens-max <tokens-max> --tokens-min <tokens-min> --idle-time --force

it's important to use the option ''--force'' so the limit is applied at the same time in the cpus. If you want to enable fair share (i.e, idle-time) for the vserver, make sure to use the ''--idle-time'' option at the previous command.

Then you should check at ''/proc/virtual/<xid>/sched'' (more info [http://linux-vserver.org/ProcFS#sched here]). Try a command like

watch -n1 cat /proc/virtual/<xid>/sched

If the collum "Ticks spent on hold" is greater than zero and the flag "On hold (H) or running (R)" assumes the hold value, then its working :)

If you want to remove the scheduling, use the command

vattribute --set --xid <xid> --flag ~sched_hard

But remember that these instructions just cover setting limits in a running vserver. To save the values you choose please refer to the [http://www.nongnu.org/util-vserver/doc/conf/configuration.html great flower page].

Capabilities and Flags

2010-02-15T16:01:19Z

Bobnormal: /* Network context flags (nflags) */

In computer science, a capability is a token used by a process to prove that it is allowed to perform an operation on an object. The Linux Capability System is based on "POSIX Capabilities", a somewhat different concept, designed to split up the all powerful root privilege into a set of distinct privileges.

== The Capability/Flag System ==

=== POSIX Capabilities ===

A process has three sets of bitmaps called the inheritable(I), permitted(P), and effective(E) capabilities. Each capability is implemented as a bit in each of these bitmaps that is either set or unset.

When a process tries to do a privileged operation, the operating system will check the appropriate bit in the effective set of the process (instead of checking whether the effective uid of the process is 0 as is normally done).

For example, when a process tries to set the clock, the Linux kernel will check that the process has the CAP_SYS_TIME bit (which is currently bit 25) set in its effective set.

The permitted set of the process indicates the capabilities the process can use. The process can have capabilities set in the permitted set that are not in the effective set.

This indicates that the process has temporarily disabled this capability. A process is allowed to set a bit in its effective set only if it is available in the permitted set. The distinction between effective and permitted exists so that processes can "bracket" operations that need privilege.

The inheritable capabilities are the capabilities of the current process that should be inherited by a program executed by the current process. The permitted set of a process is masked against the inheritable set during exec(). Nothing special happens during fork() or clone(). Child processes and threads are given an exact copy of the capabilities of the parent process.

The implementation in Linux stopped at this point, whereas POSIX Capabilities require the addition of capability sets to files too, to replace the SUID flag (at least for executables)

=== Upper Bound for Capabilities ===

Because the current Linux Capability system does not implement the filesystem related portions of POSIX Capabilities which would make setuid and setgid executables secure, and because it is much safer to have a secure upper bound for all processes within a context, an additional per-context capability mask has been added to limit all processes belonging to that context to this mask.
The meaning of the individual caps (bits) of the capability bound mask is exactly the same as with the permitted capability set.

=== Context Capabilities ===

As the Linux capabilities have almost reached the maximum number that is possible without heavy modifications to the kernel, it was a natural step to add a context-specific capability system.

The Linux-VServer context capability set acts as a mechanism to fine tune existing Linux capabilities. It is not visible to the processes within a context, as they would not know how to modify or verify it.

In general there are two ways to use those capabilities:
* Require one or a number of context capabilities to be set in addition to a given Linux capability, each one controlling a distinct part of the functionality. For example the CAP_NET_ADMIN could be split into RAW and PACKET sockets, so you could take away each of them separately by not providing the required context capability.
* Consider the context capability sufficient for a specified functionality, even if the Linux Capability says something different. For example mount() requires CAP_SYS_ADMIN which adds a dozen other things we do not want, so we define VXC_SECURE_MOUNT to allow mounts for certain contexts.

The difference between the context flags and the context capabilities is more an abstract logical separation than a functional one, because they are handled in a very similar way.

== List of capabilities/flags ==

Below is a list of capabilities and flags used for contexts and processes within. The tables contain the following information:

; Bit : The bit number to enable the capability/flag
; Mask : The bit number in hexadecimal notation
; Name : Human readable identifier used in userspace utilities
; Tag : Special capability/flag code to denote special behaviour, legacy usage and others (see below)
; Description : Description of capability/flag effects

=== Special capability/flags codes ===

The tag column may contain one or more of the following tags:

{| class="wikitable_inline"
! Tag
! Description
|-
| I
| Internal use only
|-
| L
| Only supported with legacy enabled
|-
| O
| One time capability/flag (once it's cleared, it can't be re-enabled again)
|-
| U
| Unsupported
|-
| X
| Slightly different meaning in legacy
|}

=== Context capabilities (ccaps) ===

The set of available context capabilities is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of capabilities currently available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| SET_UTSNAME
|
| Allow setdomainname(2) and sethostname(2)
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| SET_RLIMIT
|
| Allow setrlimit(2)
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| FS_SECURITY
|
| Allow setxattr for security attributes
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| TIOCSTI
|
| Allow the tiocsti ioctl (fake input character)
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
| L
| Allow usage of raw ICMP sockets
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| SYSLOG
|
| Allow syslog(2)
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| OOM_ADJUST
|
| Allow 'safe' oom adjustments
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| AUDIT_CONTROL
|
| Allow loginuid write (for auditing)
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SECURE_MOUNT
|
| Allow secure mount(2)
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SECURE_REMOUNT
|
| Allow secure remount
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| BINARY_MOUNT
|
| Allow binary/network mounts
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| QUOTA_CTL
|
| Allow quota ioctls
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| ADMIN_MAPPER
|
| Allow access to device mapper
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| ADMIN_CLOOP
|
| Allow access to loop devices
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| KTHREAD
|
| Allow creating kernel threads
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| NAMESPACE
|
| Allow namespace related operations
|}

=== Context flags (cflags) ===

The set of available context flags is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| INFO_SCHED
| L
| Account all processes as one
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| INFO_NPROC
| L
| Apply process limits to context
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
| L
| Context cannot be entered
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| INFO_INIT
| X
| Show a fake init process
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| INFO_HIDE
| X
| Hide context information in task status
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| INFO_ULIMIT
| L
| Apply ulimits to context
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| INFO_NSPACE
| L
| Use private namespace
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SCHED_HARD
|
| Enable hard scheduler
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| SCHED_PRIO
|
| Enable priority scheduler
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| SCHED_PAUSE
|
| Pause context (unschedule)
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00010000
| VIRT_MEM
|
| Virtualize memory information
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00020000
| VIRT_UPTIME
|
| Virtualize uptime information
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00040000
| VIRT_CPU
|
| Virtualize cpu usage information
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00080000
| VIRT_LOAD
|
| Virtualize load average information
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x00100000
| VIRT_TIME
|
| Allow per guest time offsets
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x01000000
| HIDE_MOUNT
|
| Hide entries in /proc/$pid/mounts
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
| L
| Hide foreign network interfaces
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_VINFO
|
| Hide context information in task status
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 33
| style="font-family: monospace" | 0x0002<<32
| STATE_INIT
| IO
| Enable init state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 37
| style="font-family: monospace" | 0x0020<<32
| REBOOT_KILL
|
| Kill all processes on reboot(2)
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make context persistent
|-
| style="text-align: right" | 48
| style="font-family: monospace" | 0x0001<<48
| FORK_RSS
|
| Block fork when RSS limit is exceeded
|-
| style="text-align: right" | 49
| style="font-family: monospace" | 0x0002<<48
| PROLIFIC
|
| Allow context to create new contexts
|-
| style="text-align: right" | 52
| style="font-family: monospace" | 0x0010<<48
| IGNEG_NICE
|
| Ignore priority raise
|}

=== Network capabilities (ncaps) ===
The set of available network capabilities is specific to Linux-VServer and applied to all processes contained within a network context. Below is a list of flags available since ''at least'' 2.3.0.34

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| TUN_CREATE
|
| Allows the guest to open TUN devices (from the kernel's TUN/TAP-interface / network devices which link to userspace). ie: <code>tun_set_iff()</code>
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
|
| Allow usage of raw ICMP sockets. (Makes <code>ping</code> work)
|}

=== Network context flags (nflags) ===

The set of available network context flags is specific to Linux-VServer and applied to all processes contained within a network context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
|
| Context cannot be entered
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SINGLE_IP
|
| Enable special handling of network contexts with a single IP only
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LBACK_REMAP
|
| use loopback-virtualisation (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| LBACK_ALLOW
|
| if set, allows guests without LBACK_REMAP to connect to 127.0.0.0/8
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
|
| Hide foreign network interfaces (ie: network interfaces not carrying and IP belonging to the guest)
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_LBACK
|
| hides the real loopback-address from the guest (rewrites to 127.0.0.1 when queried) (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make network context persistent. Allows you to configure a network context to use a certain IP with specific flags, and quickly run processes at later points with long pauses between them, rather than re-creating it every time a process needs to be executed.
|}

=== System capabilities (bcaps) ===

The set of available system capabilities is inherited from the Linux kernel and applied to all processes contained within a context. Below is a list of capabilities currently available in the vanilla kernel.

<div style="color: red; font-weight: bold;">
BIG FAT WARNING: Adding any system capability to your virtual server WILL reduce security. Do not change the default values unless you absolutely know what you are doing!
</div>

{| class="wikitable_list"
! Bit
! Mask
! Name
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| CHOWN
| In a system with the [_POSIX_CHOWN_RESTRICTED] option defined, this overrides the restriction of changing file ownership and group ownership.
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| DAC_OVERRIDE
| Override all DAC access, including ACL execute access if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| DAC_READ_SEARCH
| Overrides all DAC restrictions regarding read and search on files and directories, including ACL restrictions if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| FOWNER
| Overrides all restrictions about allowed operations on files, where file owner ID must be equal to the user ID, except where CAP_FSETID is applicable. It doesn't override MAC and DAC restrictions.
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| FSETID
| Overrides the following restrictions that the effective user ID shall match the file owner ID when setting the S_ISUID and S_ISGID bits on that file; that the effective group ID (or one of the supplementary group IDs) shall match the file owner ID when setting the S_ISGID bit on that file; that the S_ISUID and S_ISGID bits are cleared on successful return from chown(2) (not implemented).
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| KILL
| Overrides the restriction that the real or effective user ID of a process sending a signal must match the real or effective user ID of the process receiving the signal.
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| SETGID
|
* Allows setgid(2) manipulation
* Allows setgroups(2)
* Allows forged gids on socket credentials passing.
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| SETUID
|
* Allows set*uid(2) manipulation (including fsuid).
* Allows forged pids on socket credentials passing.
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SETPCAP
| Transfer any capability in your permitted set to any pid, remove any capability in your permitted set from any pid
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LINUX_IMMUTABLE
| Allow modification of S_IMMUTABLE and S_APPEND file attributes
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| NET_BIND_SERVICE
|
* Allows binding to TCP/UDP sockets below 1024
* Allows binding to ATM VCIs below 32
|-
| style="text-align: right" | 11
| style="font-family: monospace" | 0x00000800
| NET_BROADCAST
| Allow broadcasting, listen to multicast
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| NET_ADMIN
|
* Allow interface configuration
* Allow administration of IP firewall, masquerading and accounting
* Allow setting debug option on sockets
* Allow modification of routing tables
* Allow setting arbitrary process / process group ownership on sockets
* Allow binding to any address for transparent proxying
* Allow setting TOS (type of service)
* Allow setting promiscuous mode
* Allow clearing driver statistics
* Allow multicasting
* Allow read/write of device-specific registers
* Allow activation of ATM control sockets
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| NET_RAW
|
* Allow use of RAW sockets
* Allow use of PACKET sockets
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| IPC_LOCK
|
* Allow locking of shared memory segments
* Allow mlock and mlockall (which doesn't really have anything to do with IPC)
|-
| style="text-align: right" | 15
| style="font-family: monospace" | 0x00008000
| IPC_OWNER
| Override IPC ownership checks
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SYS_MODULE
|
* Insert and remove kernel modules - modify kernel without limit
* Modify cap_bset
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SYS_RAWIO
|
* Allow ioperm/iopl access
* Allow sending USB messages to any device via /proc/bus/usb
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| SYS_CHROOT
| Allow use of chroot()
|-
| style="text-align: right" | 19
| style="font-family: monospace" | 0x00080000
| SYS_PTRACE
| Allow ptrace() of any process
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| SYS_PACCT
| Allow configuration of process accounting
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| SYS_ADMIN
|
* Allow configuration of the secure attention key
* Allow administration of the random device
* Allow examination and configuration of disk quotas
* Allow configuring the kernel's syslog (printk behaviour)
* Allow setting the domainname
* Allow setting the hostname
* Allow calling bdflush()
* Allow mount() and umount(), setting up new smb connection
* Allow some autofs root ioctls
* Allow nfsservctl
* Allow VM86_REQUEST_IRQ
* Allow to read/write pci config on alpha
* Allow irix_prctl on mips (setstacksize)
* Allow flushing all cache on m68k (sys_cacheflush)
* Allow removing semaphores (Used instead of CAP_CHOWN to "chown" IPC message queues, semaphores and shared memory)
* Allow locking/unlocking of shared memory segment
* Allow turning swap on/off
* Allow forged pids on socket credentials passing
* Allow setting readahead and flushing buffers on block devices
* Allow setting geometry in floppy driver
* Allow turning DMA on/off in xd driver
* Allow administration of md devices (mostly the above, but some extra ioctls)
* Allow tuning the ide driver
* Allow access to the nvram device
* Allow administration of apm_bios, serial and bttv (TV) device
* Allow manufacturer commands in isdn CAPI support driver
* Allow reading non-standardized portions of pci configuration space
* Allow DDI debug ioctl on sbpcd driver
* Allow setting up serial ports
* Allow sending raw qic-117 commands
* Allow enabling/disabling tagged queuing on SCSI controllers and sending arbitrary SCSI commands
* Allow setting encryption key on loopback filesystem
* Allow setting zone reclaim policy
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| SYS_BOOT
| Allow use of reboot()
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00800000
| SYS_NICE
|
* Allow raising priority and setting priority on other (different UID) processes
* Allow use of FIFO and round-robin (realtime) scheduling on own processes and setting the scheduling algorithm used by another process.
* Allow setting cpu affinity on other processes
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| SYS_RESOURCE
|
* Override resource limits. Set resource limits.
* Override quota limits.
* Override reserved space on ext2 filesystem
* Modify data journaling mode on ext3 filesystem (uses journaling resources)
* ''NOTE:'' ext2 honors fsuid when checking for resource overrides, so you can override using fsuid too
* Override size restrictions on IPC message queues
* Allow more than 64hz interrupts from the real-time clock
* Override max number of consoles on console allocation
* Override max number of keymaps
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| SYS_TIME
|
* Allow manipulation of system clock
* Allow irix_stime on mips
* Allow setting the real-time clock
|-
| style="text-align: right" | 26
| style="font-family: monospace" | 0x04000000
| SYS_TTY_CONFIG
|
* Allow configuration of tty devices
* Allow vhangup() of tty
|-
| style="text-align: right" | 27
| style="font-family: monospace" | 0x08000000
| MKNOD
| Allow the privileged aspects of mknod()
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x10000000
| LEASE
| Allow taking of leases on files
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x20000000
| AUDIT_WRITE
| ??
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x40000000
| AUDIT_CONTROL
| ??
|}

== Setting flags and capabilities ==
To see how to set the flags and capabilities, see [[util-vserver:Capabilities and Flags]] if you're using util-vserver.

If you would like to edit those flags without restarting the vservers, you can use vattribute and nattribute. See [[util-vserver:Cheatsheet]]

Capabilities and Flags

2010-02-15T15:18:27Z

Bobnormal: /* Network capabilities (ncaps) */

In computer science, a capability is a token used by a process to prove that it is allowed to perform an operation on an object. The Linux Capability System is based on "POSIX Capabilities", a somewhat different concept, designed to split up the all powerful root privilege into a set of distinct privileges.

== The Capability/Flag System ==

=== POSIX Capabilities ===

A process has three sets of bitmaps called the inheritable(I), permitted(P), and effective(E) capabilities. Each capability is implemented as a bit in each of these bitmaps that is either set or unset.

When a process tries to do a privileged operation, the operating system will check the appropriate bit in the effective set of the process (instead of checking whether the effective uid of the process is 0 as is normally done).

For example, when a process tries to set the clock, the Linux kernel will check that the process has the CAP_SYS_TIME bit (which is currently bit 25) set in its effective set.

The permitted set of the process indicates the capabilities the process can use. The process can have capabilities set in the permitted set that are not in the effective set.

This indicates that the process has temporarily disabled this capability. A process is allowed to set a bit in its effective set only if it is available in the permitted set. The distinction between effective and permitted exists so that processes can "bracket" operations that need privilege.

The inheritable capabilities are the capabilities of the current process that should be inherited by a program executed by the current process. The permitted set of a process is masked against the inheritable set during exec(). Nothing special happens during fork() or clone(). Child processes and threads are given an exact copy of the capabilities of the parent process.

The implementation in Linux stopped at this point, whereas POSIX Capabilities require the addition of capability sets to files too, to replace the SUID flag (at least for executables)

=== Upper Bound for Capabilities ===

Because the current Linux Capability system does not implement the filesystem related portions of POSIX Capabilities which would make setuid and setgid executables secure, and because it is much safer to have a secure upper bound for all processes within a context, an additional per-context capability mask has been added to limit all processes belonging to that context to this mask.
The meaning of the individual caps (bits) of the capability bound mask is exactly the same as with the permitted capability set.

=== Context Capabilities ===

As the Linux capabilities have almost reached the maximum number that is possible without heavy modifications to the kernel, it was a natural step to add a context-specific capability system.

The Linux-VServer context capability set acts as a mechanism to fine tune existing Linux capabilities. It is not visible to the processes within a context, as they would not know how to modify or verify it.

In general there are two ways to use those capabilities:
* Require one or a number of context capabilities to be set in addition to a given Linux capability, each one controlling a distinct part of the functionality. For example the CAP_NET_ADMIN could be split into RAW and PACKET sockets, so you could take away each of them separately by not providing the required context capability.
* Consider the context capability sufficient for a specified functionality, even if the Linux Capability says something different. For example mount() requires CAP_SYS_ADMIN which adds a dozen other things we do not want, so we define VXC_SECURE_MOUNT to allow mounts for certain contexts.

The difference between the context flags and the context capabilities is more an abstract logical separation than a functional one, because they are handled in a very similar way.

== List of capabilities/flags ==

Below is a list of capabilities and flags used for contexts and processes within. The tables contain the following information:

; Bit : The bit number to enable the capability/flag
; Mask : The bit number in hexadecimal notation
; Name : Human readable identifier used in userspace utilities
; Tag : Special capability/flag code to denote special behaviour, legacy usage and others (see below)
; Description : Description of capability/flag effects

=== Special capability/flags codes ===

The tag column may contain one or more of the following tags:

{| class="wikitable_inline"
! Tag
! Description
|-
| I
| Internal use only
|-
| L
| Only supported with legacy enabled
|-
| O
| One time capability/flag (once it's cleared, it can't be re-enabled again)
|-
| U
| Unsupported
|-
| X
| Slightly different meaning in legacy
|}

=== Context capabilities (ccaps) ===

The set of available context capabilities is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of capabilities currently available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| SET_UTSNAME
|
| Allow setdomainname(2) and sethostname(2)
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| SET_RLIMIT
|
| Allow setrlimit(2)
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| FS_SECURITY
|
| Allow setxattr for security attributes
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| TIOCSTI
|
| Allow the tiocsti ioctl (fake input character)
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
| L
| Allow usage of raw ICMP sockets
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| SYSLOG
|
| Allow syslog(2)
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| OOM_ADJUST
|
| Allow 'safe' oom adjustments
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| AUDIT_CONTROL
|
| Allow loginuid write (for auditing)
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SECURE_MOUNT
|
| Allow secure mount(2)
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SECURE_REMOUNT
|
| Allow secure remount
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| BINARY_MOUNT
|
| Allow binary/network mounts
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| QUOTA_CTL
|
| Allow quota ioctls
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| ADMIN_MAPPER
|
| Allow access to device mapper
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| ADMIN_CLOOP
|
| Allow access to loop devices
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| KTHREAD
|
| Allow creating kernel threads
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| NAMESPACE
|
| Allow namespace related operations
|}

=== Context flags (cflags) ===

The set of available context flags is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| INFO_SCHED
| L
| Account all processes as one
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| INFO_NPROC
| L
| Apply process limits to context
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
| L
| Context cannot be entered
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| INFO_INIT
| X
| Show a fake init process
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| INFO_HIDE
| X
| Hide context information in task status
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| INFO_ULIMIT
| L
| Apply ulimits to context
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| INFO_NSPACE
| L
| Use private namespace
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SCHED_HARD
|
| Enable hard scheduler
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| SCHED_PRIO
|
| Enable priority scheduler
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| SCHED_PAUSE
|
| Pause context (unschedule)
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00010000
| VIRT_MEM
|
| Virtualize memory information
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00020000
| VIRT_UPTIME
|
| Virtualize uptime information
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00040000
| VIRT_CPU
|
| Virtualize cpu usage information
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00080000
| VIRT_LOAD
|
| Virtualize load average information
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x00100000
| VIRT_TIME
|
| Allow per guest time offsets
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x01000000
| HIDE_MOUNT
|
| Hide entries in /proc/$pid/mounts
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
| L
| Hide foreign network interfaces
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_VINFO
|
| Hide context information in task status
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 33
| style="font-family: monospace" | 0x0002<<32
| STATE_INIT
| IO
| Enable init state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 37
| style="font-family: monospace" | 0x0020<<32
| REBOOT_KILL
|
| Kill all processes on reboot(2)
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make context persistent
|-
| style="text-align: right" | 48
| style="font-family: monospace" | 0x0001<<48
| FORK_RSS
|
| Block fork when RSS limit is exceeded
|-
| style="text-align: right" | 49
| style="font-family: monospace" | 0x0002<<48
| PROLIFIC
|
| Allow context to create new contexts
|-
| style="text-align: right" | 52
| style="font-family: monospace" | 0x0010<<48
| IGNEG_NICE
|
| Ignore priority raise
|}

=== Network capabilities (ncaps) ===
The set of available network capabilities is specific to Linux-VServer and applied to all processes contained within a network context. Below is a list of flags available since ''at least'' 2.3.0.34

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| TUN_CREATE
|
| Allows the guest to open TUN devices (from the kernel's TUN/TAP-interface / network devices which link to userspace). ie: <code>tun_set_iff()</code>
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
|
| Allow usage of raw ICMP sockets. (Makes <code>ping</code> work)
|}

=== Network context flags (nflags) ===

The set of available network context flags is specific to Linux-VServer and applied to all processes contained within a network context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
|
| Context cannot be entered
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SINGLE_IP
|
| Enable special handling of network contexts with a single IP only
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LBACK_REMAP
|
| use loopback-virtualisation (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| LBACK_ALLOW
|
| if set, allows guests without LBACK_REMAP to connect to 127.0.0.0/8
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
|
| Hide foreign network interfaces (ie: network interfaces not carrying and IP belonging to the guest)
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_LBACK
|
| hides the real loopback-address from the guest (rewrites to 127.0.0.1 when queried) (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make network context persistent
|}

=== System capabilities (bcaps) ===

The set of available system capabilities is inherited from the Linux kernel and applied to all processes contained within a context. Below is a list of capabilities currently available in the vanilla kernel.

<div style="color: red; font-weight: bold;">
BIG FAT WARNING: Adding any system capability to your virtual server WILL reduce security. Do not change the default values unless you absolutely know what you are doing!
</div>

{| class="wikitable_list"
! Bit
! Mask
! Name
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| CHOWN
| In a system with the [_POSIX_CHOWN_RESTRICTED] option defined, this overrides the restriction of changing file ownership and group ownership.
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| DAC_OVERRIDE
| Override all DAC access, including ACL execute access if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| DAC_READ_SEARCH
| Overrides all DAC restrictions regarding read and search on files and directories, including ACL restrictions if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| FOWNER
| Overrides all restrictions about allowed operations on files, where file owner ID must be equal to the user ID, except where CAP_FSETID is applicable. It doesn't override MAC and DAC restrictions.
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| FSETID
| Overrides the following restrictions that the effective user ID shall match the file owner ID when setting the S_ISUID and S_ISGID bits on that file; that the effective group ID (or one of the supplementary group IDs) shall match the file owner ID when setting the S_ISGID bit on that file; that the S_ISUID and S_ISGID bits are cleared on successful return from chown(2) (not implemented).
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| KILL
| Overrides the restriction that the real or effective user ID of a process sending a signal must match the real or effective user ID of the process receiving the signal.
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| SETGID
|
* Allows setgid(2) manipulation
* Allows setgroups(2)
* Allows forged gids on socket credentials passing.
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| SETUID
|
* Allows set*uid(2) manipulation (including fsuid).
* Allows forged pids on socket credentials passing.
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SETPCAP
| Transfer any capability in your permitted set to any pid, remove any capability in your permitted set from any pid
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LINUX_IMMUTABLE
| Allow modification of S_IMMUTABLE and S_APPEND file attributes
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| NET_BIND_SERVICE
|
* Allows binding to TCP/UDP sockets below 1024
* Allows binding to ATM VCIs below 32
|-
| style="text-align: right" | 11
| style="font-family: monospace" | 0x00000800
| NET_BROADCAST
| Allow broadcasting, listen to multicast
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| NET_ADMIN
|
* Allow interface configuration
* Allow administration of IP firewall, masquerading and accounting
* Allow setting debug option on sockets
* Allow modification of routing tables
* Allow setting arbitrary process / process group ownership on sockets
* Allow binding to any address for transparent proxying
* Allow setting TOS (type of service)
* Allow setting promiscuous mode
* Allow clearing driver statistics
* Allow multicasting
* Allow read/write of device-specific registers
* Allow activation of ATM control sockets
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| NET_RAW
|
* Allow use of RAW sockets
* Allow use of PACKET sockets
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| IPC_LOCK
|
* Allow locking of shared memory segments
* Allow mlock and mlockall (which doesn't really have anything to do with IPC)
|-
| style="text-align: right" | 15
| style="font-family: monospace" | 0x00008000
| IPC_OWNER
| Override IPC ownership checks
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SYS_MODULE
|
* Insert and remove kernel modules - modify kernel without limit
* Modify cap_bset
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SYS_RAWIO
|
* Allow ioperm/iopl access
* Allow sending USB messages to any device via /proc/bus/usb
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| SYS_CHROOT
| Allow use of chroot()
|-
| style="text-align: right" | 19
| style="font-family: monospace" | 0x00080000
| SYS_PTRACE
| Allow ptrace() of any process
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| SYS_PACCT
| Allow configuration of process accounting
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| SYS_ADMIN
|
* Allow configuration of the secure attention key
* Allow administration of the random device
* Allow examination and configuration of disk quotas
* Allow configuring the kernel's syslog (printk behaviour)
* Allow setting the domainname
* Allow setting the hostname
* Allow calling bdflush()
* Allow mount() and umount(), setting up new smb connection
* Allow some autofs root ioctls
* Allow nfsservctl
* Allow VM86_REQUEST_IRQ
* Allow to read/write pci config on alpha
* Allow irix_prctl on mips (setstacksize)
* Allow flushing all cache on m68k (sys_cacheflush)
* Allow removing semaphores (Used instead of CAP_CHOWN to "chown" IPC message queues, semaphores and shared memory)
* Allow locking/unlocking of shared memory segment
* Allow turning swap on/off
* Allow forged pids on socket credentials passing
* Allow setting readahead and flushing buffers on block devices
* Allow setting geometry in floppy driver
* Allow turning DMA on/off in xd driver
* Allow administration of md devices (mostly the above, but some extra ioctls)
* Allow tuning the ide driver
* Allow access to the nvram device
* Allow administration of apm_bios, serial and bttv (TV) device
* Allow manufacturer commands in isdn CAPI support driver
* Allow reading non-standardized portions of pci configuration space
* Allow DDI debug ioctl on sbpcd driver
* Allow setting up serial ports
* Allow sending raw qic-117 commands
* Allow enabling/disabling tagged queuing on SCSI controllers and sending arbitrary SCSI commands
* Allow setting encryption key on loopback filesystem
* Allow setting zone reclaim policy
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| SYS_BOOT
| Allow use of reboot()
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00800000
| SYS_NICE
|
* Allow raising priority and setting priority on other (different UID) processes
* Allow use of FIFO and round-robin (realtime) scheduling on own processes and setting the scheduling algorithm used by another process.
* Allow setting cpu affinity on other processes
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| SYS_RESOURCE
|
* Override resource limits. Set resource limits.
* Override quota limits.
* Override reserved space on ext2 filesystem
* Modify data journaling mode on ext3 filesystem (uses journaling resources)
* ''NOTE:'' ext2 honors fsuid when checking for resource overrides, so you can override using fsuid too
* Override size restrictions on IPC message queues
* Allow more than 64hz interrupts from the real-time clock
* Override max number of consoles on console allocation
* Override max number of keymaps
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| SYS_TIME
|
* Allow manipulation of system clock
* Allow irix_stime on mips
* Allow setting the real-time clock
|-
| style="text-align: right" | 26
| style="font-family: monospace" | 0x04000000
| SYS_TTY_CONFIG
|
* Allow configuration of tty devices
* Allow vhangup() of tty
|-
| style="text-align: right" | 27
| style="font-family: monospace" | 0x08000000
| MKNOD
| Allow the privileged aspects of mknod()
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x10000000
| LEASE
| Allow taking of leases on files
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x20000000
| AUDIT_WRITE
| ??
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x40000000
| AUDIT_CONTROL
| ??
|}

== Setting flags and capabilities ==
To see how to set the flags and capabilities, see [[util-vserver:Capabilities and Flags]] if you're using util-vserver.

If you would like to edit those flags without restarting the vservers, you can use vattribute and nattribute. See [[util-vserver:Cheatsheet]]

Capabilities and Flags

2010-02-15T15:17:17Z

Bobnormal: /* Network context flags (nflags) */

In computer science, a capability is a token used by a process to prove that it is allowed to perform an operation on an object. The Linux Capability System is based on "POSIX Capabilities", a somewhat different concept, designed to split up the all powerful root privilege into a set of distinct privileges.

== The Capability/Flag System ==

=== POSIX Capabilities ===

A process has three sets of bitmaps called the inheritable(I), permitted(P), and effective(E) capabilities. Each capability is implemented as a bit in each of these bitmaps that is either set or unset.

When a process tries to do a privileged operation, the operating system will check the appropriate bit in the effective set of the process (instead of checking whether the effective uid of the process is 0 as is normally done).

For example, when a process tries to set the clock, the Linux kernel will check that the process has the CAP_SYS_TIME bit (which is currently bit 25) set in its effective set.

The permitted set of the process indicates the capabilities the process can use. The process can have capabilities set in the permitted set that are not in the effective set.

This indicates that the process has temporarily disabled this capability. A process is allowed to set a bit in its effective set only if it is available in the permitted set. The distinction between effective and permitted exists so that processes can "bracket" operations that need privilege.

The inheritable capabilities are the capabilities of the current process that should be inherited by a program executed by the current process. The permitted set of a process is masked against the inheritable set during exec(). Nothing special happens during fork() or clone(). Child processes and threads are given an exact copy of the capabilities of the parent process.

The implementation in Linux stopped at this point, whereas POSIX Capabilities require the addition of capability sets to files too, to replace the SUID flag (at least for executables)

=== Upper Bound for Capabilities ===

Because the current Linux Capability system does not implement the filesystem related portions of POSIX Capabilities which would make setuid and setgid executables secure, and because it is much safer to have a secure upper bound for all processes within a context, an additional per-context capability mask has been added to limit all processes belonging to that context to this mask.
The meaning of the individual caps (bits) of the capability bound mask is exactly the same as with the permitted capability set.

=== Context Capabilities ===

As the Linux capabilities have almost reached the maximum number that is possible without heavy modifications to the kernel, it was a natural step to add a context-specific capability system.

The Linux-VServer context capability set acts as a mechanism to fine tune existing Linux capabilities. It is not visible to the processes within a context, as they would not know how to modify or verify it.

In general there are two ways to use those capabilities:
* Require one or a number of context capabilities to be set in addition to a given Linux capability, each one controlling a distinct part of the functionality. For example the CAP_NET_ADMIN could be split into RAW and PACKET sockets, so you could take away each of them separately by not providing the required context capability.
* Consider the context capability sufficient for a specified functionality, even if the Linux Capability says something different. For example mount() requires CAP_SYS_ADMIN which adds a dozen other things we do not want, so we define VXC_SECURE_MOUNT to allow mounts for certain contexts.

The difference between the context flags and the context capabilities is more an abstract logical separation than a functional one, because they are handled in a very similar way.

== List of capabilities/flags ==

Below is a list of capabilities and flags used for contexts and processes within. The tables contain the following information:

; Bit : The bit number to enable the capability/flag
; Mask : The bit number in hexadecimal notation
; Name : Human readable identifier used in userspace utilities
; Tag : Special capability/flag code to denote special behaviour, legacy usage and others (see below)
; Description : Description of capability/flag effects

=== Special capability/flags codes ===

The tag column may contain one or more of the following tags:

{| class="wikitable_inline"
! Tag
! Description
|-
| I
| Internal use only
|-
| L
| Only supported with legacy enabled
|-
| O
| One time capability/flag (once it's cleared, it can't be re-enabled again)
|-
| U
| Unsupported
|-
| X
| Slightly different meaning in legacy
|}

=== Context capabilities (ccaps) ===

The set of available context capabilities is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of capabilities currently available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| SET_UTSNAME
|
| Allow setdomainname(2) and sethostname(2)
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| SET_RLIMIT
|
| Allow setrlimit(2)
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| FS_SECURITY
|
| Allow setxattr for security attributes
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| TIOCSTI
|
| Allow the tiocsti ioctl (fake input character)
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
| L
| Allow usage of raw ICMP sockets
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| SYSLOG
|
| Allow syslog(2)
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| OOM_ADJUST
|
| Allow 'safe' oom adjustments
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| AUDIT_CONTROL
|
| Allow loginuid write (for auditing)
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SECURE_MOUNT
|
| Allow secure mount(2)
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SECURE_REMOUNT
|
| Allow secure remount
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| BINARY_MOUNT
|
| Allow binary/network mounts
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| QUOTA_CTL
|
| Allow quota ioctls
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| ADMIN_MAPPER
|
| Allow access to device mapper
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| ADMIN_CLOOP
|
| Allow access to loop devices
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| KTHREAD
|
| Allow creating kernel threads
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| NAMESPACE
|
| Allow namespace related operations
|}

=== Context flags (cflags) ===

The set of available context flags is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| INFO_SCHED
| L
| Account all processes as one
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| INFO_NPROC
| L
| Apply process limits to context
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
| L
| Context cannot be entered
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| INFO_INIT
| X
| Show a fake init process
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| INFO_HIDE
| X
| Hide context information in task status
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| INFO_ULIMIT
| L
| Apply ulimits to context
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| INFO_NSPACE
| L
| Use private namespace
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SCHED_HARD
|
| Enable hard scheduler
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| SCHED_PRIO
|
| Enable priority scheduler
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| SCHED_PAUSE
|
| Pause context (unschedule)
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00010000
| VIRT_MEM
|
| Virtualize memory information
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00020000
| VIRT_UPTIME
|
| Virtualize uptime information
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00040000
| VIRT_CPU
|
| Virtualize cpu usage information
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00080000
| VIRT_LOAD
|
| Virtualize load average information
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x00100000
| VIRT_TIME
|
| Allow per guest time offsets
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x01000000
| HIDE_MOUNT
|
| Hide entries in /proc/$pid/mounts
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
| L
| Hide foreign network interfaces
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_VINFO
|
| Hide context information in task status
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 33
| style="font-family: monospace" | 0x0002<<32
| STATE_INIT
| IO
| Enable init state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 37
| style="font-family: monospace" | 0x0020<<32
| REBOOT_KILL
|
| Kill all processes on reboot(2)
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make context persistent
|-
| style="text-align: right" | 48
| style="font-family: monospace" | 0x0001<<48
| FORK_RSS
|
| Block fork when RSS limit is exceeded
|-
| style="text-align: right" | 49
| style="font-family: monospace" | 0x0002<<48
| PROLIFIC
|
| Allow context to create new contexts
|-
| style="text-align: right" | 52
| style="font-family: monospace" | 0x0010<<48
| IGNEG_NICE
|
| Ignore priority raise
|}

=== Network capabilities (ncaps) ===
The set of available network capabilities is specific to Linux-VServer and applied to all processes contained with a network context. Below is a list of flags available since ''at least'' 2.3.0.34

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| TUN_CREATE
|
| Allows the guest to open TUN devices (from the kernel's TUN/TAP-interface / network devices which link to userspace). ie: <code>tun_set_iff()</code>
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
|
| Allow usage of raw ICMP sockets. (Makes <code>ping</code> work)
|}

=== Network context flags (nflags) ===

The set of available network context flags is specific to Linux-VServer and applied to all processes contained within a network context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
|
| Context cannot be entered
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SINGLE_IP
|
| Enable special handling of network contexts with a single IP only
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LBACK_REMAP
|
| use loopback-virtualisation (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| LBACK_ALLOW
|
| if set, allows guests without LBACK_REMAP to connect to 127.0.0.0/8
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
|
| Hide foreign network interfaces (ie: network interfaces not carrying and IP belonging to the guest)
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_LBACK
|
| hides the real loopback-address from the guest (rewrites to 127.0.0.1 when queried) (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make network context persistent
|}

=== System capabilities (bcaps) ===

The set of available system capabilities is inherited from the Linux kernel and applied to all processes contained within a context. Below is a list of capabilities currently available in the vanilla kernel.

<div style="color: red; font-weight: bold;">
BIG FAT WARNING: Adding any system capability to your virtual server WILL reduce security. Do not change the default values unless you absolutely know what you are doing!
</div>

{| class="wikitable_list"
! Bit
! Mask
! Name
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| CHOWN
| In a system with the [_POSIX_CHOWN_RESTRICTED] option defined, this overrides the restriction of changing file ownership and group ownership.
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| DAC_OVERRIDE
| Override all DAC access, including ACL execute access if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| DAC_READ_SEARCH
| Overrides all DAC restrictions regarding read and search on files and directories, including ACL restrictions if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| FOWNER
| Overrides all restrictions about allowed operations on files, where file owner ID must be equal to the user ID, except where CAP_FSETID is applicable. It doesn't override MAC and DAC restrictions.
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| FSETID
| Overrides the following restrictions that the effective user ID shall match the file owner ID when setting the S_ISUID and S_ISGID bits on that file; that the effective group ID (or one of the supplementary group IDs) shall match the file owner ID when setting the S_ISGID bit on that file; that the S_ISUID and S_ISGID bits are cleared on successful return from chown(2) (not implemented).
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| KILL
| Overrides the restriction that the real or effective user ID of a process sending a signal must match the real or effective user ID of the process receiving the signal.
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| SETGID
|
* Allows setgid(2) manipulation
* Allows setgroups(2)
* Allows forged gids on socket credentials passing.
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| SETUID
|
* Allows set*uid(2) manipulation (including fsuid).
* Allows forged pids on socket credentials passing.
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SETPCAP
| Transfer any capability in your permitted set to any pid, remove any capability in your permitted set from any pid
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LINUX_IMMUTABLE
| Allow modification of S_IMMUTABLE and S_APPEND file attributes
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| NET_BIND_SERVICE
|
* Allows binding to TCP/UDP sockets below 1024
* Allows binding to ATM VCIs below 32
|-
| style="text-align: right" | 11
| style="font-family: monospace" | 0x00000800
| NET_BROADCAST
| Allow broadcasting, listen to multicast
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| NET_ADMIN
|
* Allow interface configuration
* Allow administration of IP firewall, masquerading and accounting
* Allow setting debug option on sockets
* Allow modification of routing tables
* Allow setting arbitrary process / process group ownership on sockets
* Allow binding to any address for transparent proxying
* Allow setting TOS (type of service)
* Allow setting promiscuous mode
* Allow clearing driver statistics
* Allow multicasting
* Allow read/write of device-specific registers
* Allow activation of ATM control sockets
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| NET_RAW
|
* Allow use of RAW sockets
* Allow use of PACKET sockets
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| IPC_LOCK
|
* Allow locking of shared memory segments
* Allow mlock and mlockall (which doesn't really have anything to do with IPC)
|-
| style="text-align: right" | 15
| style="font-family: monospace" | 0x00008000
| IPC_OWNER
| Override IPC ownership checks
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SYS_MODULE
|
* Insert and remove kernel modules - modify kernel without limit
* Modify cap_bset
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SYS_RAWIO
|
* Allow ioperm/iopl access
* Allow sending USB messages to any device via /proc/bus/usb
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| SYS_CHROOT
| Allow use of chroot()
|-
| style="text-align: right" | 19
| style="font-family: monospace" | 0x00080000
| SYS_PTRACE
| Allow ptrace() of any process
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| SYS_PACCT
| Allow configuration of process accounting
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| SYS_ADMIN
|
* Allow configuration of the secure attention key
* Allow administration of the random device
* Allow examination and configuration of disk quotas
* Allow configuring the kernel's syslog (printk behaviour)
* Allow setting the domainname
* Allow setting the hostname
* Allow calling bdflush()
* Allow mount() and umount(), setting up new smb connection
* Allow some autofs root ioctls
* Allow nfsservctl
* Allow VM86_REQUEST_IRQ
* Allow to read/write pci config on alpha
* Allow irix_prctl on mips (setstacksize)
* Allow flushing all cache on m68k (sys_cacheflush)
* Allow removing semaphores (Used instead of CAP_CHOWN to "chown" IPC message queues, semaphores and shared memory)
* Allow locking/unlocking of shared memory segment
* Allow turning swap on/off
* Allow forged pids on socket credentials passing
* Allow setting readahead and flushing buffers on block devices
* Allow setting geometry in floppy driver
* Allow turning DMA on/off in xd driver
* Allow administration of md devices (mostly the above, but some extra ioctls)
* Allow tuning the ide driver
* Allow access to the nvram device
* Allow administration of apm_bios, serial and bttv (TV) device
* Allow manufacturer commands in isdn CAPI support driver
* Allow reading non-standardized portions of pci configuration space
* Allow DDI debug ioctl on sbpcd driver
* Allow setting up serial ports
* Allow sending raw qic-117 commands
* Allow enabling/disabling tagged queuing on SCSI controllers and sending arbitrary SCSI commands
* Allow setting encryption key on loopback filesystem
* Allow setting zone reclaim policy
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| SYS_BOOT
| Allow use of reboot()
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00800000
| SYS_NICE
|
* Allow raising priority and setting priority on other (different UID) processes
* Allow use of FIFO and round-robin (realtime) scheduling on own processes and setting the scheduling algorithm used by another process.
* Allow setting cpu affinity on other processes
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| SYS_RESOURCE
|
* Override resource limits. Set resource limits.
* Override quota limits.
* Override reserved space on ext2 filesystem
* Modify data journaling mode on ext3 filesystem (uses journaling resources)
* ''NOTE:'' ext2 honors fsuid when checking for resource overrides, so you can override using fsuid too
* Override size restrictions on IPC message queues
* Allow more than 64hz interrupts from the real-time clock
* Override max number of consoles on console allocation
* Override max number of keymaps
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| SYS_TIME
|
* Allow manipulation of system clock
* Allow irix_stime on mips
* Allow setting the real-time clock
|-
| style="text-align: right" | 26
| style="font-family: monospace" | 0x04000000
| SYS_TTY_CONFIG
|
* Allow configuration of tty devices
* Allow vhangup() of tty
|-
| style="text-align: right" | 27
| style="font-family: monospace" | 0x08000000
| MKNOD
| Allow the privileged aspects of mknod()
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x10000000
| LEASE
| Allow taking of leases on files
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x20000000
| AUDIT_WRITE
| ??
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x40000000
| AUDIT_CONTROL
| ??
|}

== Setting flags and capabilities ==
To see how to set the flags and capabilities, see [[util-vserver:Capabilities and Flags]] if you're using util-vserver.

If you would like to edit those flags without restarting the vservers, you can use vattribute and nattribute. See [[util-vserver:Cheatsheet]]

Networking vserver guests

2010-02-15T14:22:44Z

Bobnormal: /* Guests */

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. '''There is no other use for this step''', so feel free to skip it if you are not worried about sharing packet counters. (Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker and this will actually matter are slim to none and you should feel free to skip this section.)

====Process====
First you need to load the dummy interface driver (requires CONFIG_DUMMY=m in your kernel configuration)
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

====Optional Extra Step====
It is possible to facilitate further segregation (such that even dummy packet counters are never shared between guests). To do so, use:
# modprobe dummy numdummies=<number-of-devices-required>
... and associate each vServer with a unique dummy device.

===Guests===
Set up each guest vserver. If you skipped the host configuration above, you will need to replace dummy0 with the name of your host's primary network interface, for example eth0.

cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix

Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Networking vserver guests

2010-02-15T14:22:06Z

Bobnormal: /* Process */

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. '''There is no other use for this step''', so feel free to skip it if you are not worried about sharing packet counters. (Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker and this will actually matter are slim to none and you should feel free to skip this section.)

====Process====
First you need to load the dummy interface driver (requires CONFIG_DUMMY=m in your kernel configuration)
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

====Optional Extra Step====
It is possible to facilitate further segregation (such that even dummy packet counters are never shared between guests). To do so, use:
# modprobe dummy numdummies=<number-of-devices-required>
... and associate each vServer with a unique dummy device.

===Guests===
Set up each guest vserver. If you skipped the above step, replace dummy0 with the name of your host's primary network interface, for example eth0.

cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix

Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Networking vserver guests

2010-02-15T14:16:38Z

Bobnormal: /* Configuration */

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. '''There is no other use for this step''', so feel free to skip it if you are not worried about sharing packet counters. (Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker and this will actually matter are slim to none and you should feel free to skip this section.)

====Process====
First you need to load the dummy interface driver (requires CONFIG_DUMMY=m in your kernel configuration)
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

===Guests===
Set up each guest vserver. If you skipped the above step, replace dummy0 with the name of your host's primary network interface, for example eth0.

cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix

Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Networking vserver guests

2010-02-15T13:55:52Z

Bobnormal: /* Guests */

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. There is no other use for this step, so feel free to skip it if you are not worried about sharing packet counters. Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker are slim to none and you should feel free to skip this section.

====Process====
First you need to load the dummy interface driver
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

===Guests===
Set up each guest vserver. If you skipped the above step, replace dummy0 with the name of your host's primary network interface, for example eth0.

cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix

Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Networking vserver guests

2010-02-15T13:55:45Z

Bobnormal: /* Guests */

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. There is no other use for this step, so feel free to skip it if you are not worried about sharing packet counters. Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker are slim to none and you should feel free to skip this section.

====Process====
First you need to load the dummy interface driver
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

===Guests===
Set up each guest vserver. If you skipped the above step, replace dummy0 with the name of your host's primary network interface.

cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix

Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Networking vserver guests

2010-02-15T13:55:35Z

Bobnormal: /* Guests */

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. There is no other use for this step, so feel free to skip it if you are not worried about sharing packet counters. Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker are slim to none and you should feel free to skip this section.

====Process====
First you need to load the dummy interface driver
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

===Guests===
Set up each guest vserver. If you skipped the above step, replace dumm0 with the name of your host's primary network interface.

cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix

Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Frequently Asked Questions

2010-02-15T13:54:10Z

Bobnormal: add pointer to http://linux-vserver.org/Networking_vserver_guests for packet counter hide

<div style="margin: 2em auto 2em auto; padding: 10px; background-color: #F9ECCD; border: 1px solid #004433; text-align: center;">
[[Image:Icon-Caution.png|left]]
We currently migrate to MediaWiki from our old installation, but not all content has been migrated yet. Take a look at the [[Wiki Team]] page for instructions how to help or look at the [http://oldwiki.linux-vserver.org old wiki] to find the information not migrated yet.

'''To ease migration we created a [[List of old Documentation pages]].'''
</div>

CURRENTLY THE CONTENT OF THE OLD WIKI FAQ (AND MORE) IS BEING MIGRATED TO THIS PAGE (TASK: DERJOHN)

__TOC__

== General ==

{{Question
|Question=What is a 'Guest'?
||Details=To talk about stuff, we need some naming. The physical machine is called 'Host' and the 'main' context running the Host Distro is called 'Host Context'. The virtual machine/distro is called 'Guest' and basically is a Distribution (Userspace) running inside a 'Guest Context'.
|Signature=derjohn}}

{{Question
|Question=What kind of Operating System (OS) can I run as guest?
||Details= With VServer you can only run Linux guests. The trick is that a guest does not run a kernel on its own (as XEN and UML do), it merely uses a virtualized host kernel-interface. VServer offers so called security contexts which make it possible to separate one guest from each other, i.e. they cannot get data from each other. Imagine it as a chroot environment with much more security and features.
|Signature=derjohn}}

{{Question
|Question=Is this a new project? When was it started?
||Details=The first public occurrence of Linux-VServer was Oct 2001. The initial mail can be found here: http://www.cs.helsinki.fi/linux/linux-kernel/2001-40/1065.html
So you can expect a mature software product which does its magic quite well (And hey, we have a version > 2.0!)
|Signature=derjohn}}

{{Question
|Question=Which distributions did you test?
||Details=Some. Check out the wiki for ready-made guest images. But you can easily build own guest images, e.g. with Debian's debootstrap. Checkout [[Building Guest Systems]] how to do that.
|Signature=derjohn}}

{{Question
|Question=Is VServer comparable to XEN/UML/QEMU?
||Details=Nope. XEN/UML/QEMU and VServer are just good friends. Because you ask, you probably know what XEN/UML/QEMU are. VServer in contrary to XEN/UML/QEMU does not "emulate" any hardware you run a kernel on. The purpose of Linux VServer is to isolate (groups of) applications. The isolation is done by the kernel (see [[Overview]] for a more detailed comparison). You can run a VServer kernel in a XEN/UML/QEMU guest. This is confirmed to work at least with Linux 2.6/vs2.0.
|Signature=derjohn}}

{{Question
|Question=With which version should I begin?
||Details=If you are new to VServer I recommend to try the latest stable kernel patch, and the latest util-vserver "alpha" release.
|Signature=derjohn}}

{{Question
|Question=Is VServer secure?
||Details=We hope so. It should be as least as secure as Linux is. We consider it much much more secure though.
|Signature=derjohn}}

{{Question
|Question=Performance?
||Details=For a single guest, we basically have native performance. Some tests showed insignificant overhead (about 1-2%) others ran faster than on an unpatched kernel. This is IMVHO significantly less than other solutions waste, especially if you have more than a single guest (because of the resource sharing).
|Signature=derjohn}}

{{Question
|Question=What is the "great flower page"?
||Details=Well, [http://www.nongnu.org/util-vserver/doc/conf/configuration.html this page] contains all configuration options for util-vserver. The name of the page is derived from the stylesheet(s) it contains.
|Signature=derjohn}}

== Resources usage ==

{{Question
|Question=Resource sharing?
||Details=Yes ....
* memory: Dynamically.
* CPU usage: Dynamically (token bucket)
|Signature=derjohn}}

{{Question
|Question=Resource limiting?
||Details=You can put limits per guest on different subsystems.
* using ulimits and rlimits (rlimit is a new feature of kernel 2.6/vs2.0.) per guest, to limit the memory consumption, the number of processes or file-handles, ... : see [[Resource Limits]]
* CPU usage : see [[CPU Scheduler]]
* disk space usage : see [[Disk Limits and Quota]]
Note that you can only offer guaranteed resource availability with some ticks at the time.
|Signature=derjohn&xm}}

{{Question
|Question=How do I limit a guests RAM? I want to prevent OOM situations on the host!
||Details=First you can read [http://linux-vserver.org/Memory+Allocation] and [[Memory Limits]].
If you want a recipe, do this:
# Check the size of memory pages. On x86 and x86_64 is usually 4 KB per page.
# Create /etc/vserver/<guest>/rlimits/
# Check your physical memory size on the host, e.g. with "free -m". maxram = kilobytes/pagesize.
# Limit the guests physical RAM to value smaller then maxram:<pre>echo %%insertYourPagesHereSmallerThanMaxram%% > /etc/vserver/<guest>/rlimits/rss </pre>
# Check your swapspace, e.g. with 'swapon -s'. maxswap = swapkilobytes/pagesize.
# Limit the guest's maximum number of as pages to a value smaller than (maxram+maxswap): <pre> echo %%desiredvalue%% > /etc/vserver/<guest>/rlimits/as </pre>
# Correctly display the memory information inside the guest:<pre>echo "VIRT_MEM" >> /etc/vservers/<guest>/flags</pre>
It should be clear this can still lead to OOM situations. Example: You have two guests and your as limit per guest is greater than 50% of (maxram+maxswap). If both guests request their maximum at the same point in time, there will be not enough mem .....
|Signature=derjohn}}

{{Question
|Question=Disk I/O limiting? Is that possible?
||Details=Well, since vs2.1.1 Linux-VServer supports a mechanism called 'I/O scheduling', which appeared in the 2.6 mainline some time ago. The mainline kernel offers several I/O schedulers:
<pre>
# cat /sys/block/hdc/queue/scheduler
noop [anticipatory] deadline cfq
</pre>

The default is anticipatory a.k.a. "AS". When running several guests on a host you probably want the I/O performance shared in a fair way among the different guests. The kernel comes with a "completely fair queueing" scheduler, CFQ, which can do that. (More on schedulers can be found at http://lwn.net/Articles/114770/)
This is how to set the scheduler to "cfq" manually:
<pre>
root# echo "cfq" > /sys/block/hdc/queue/scheduler
root# cat /sys/block/hdc/queue/scheduler
noop anticipatory deadline [cfq]
</pre>
Keep in mind that you have to do it on all physical discs. So if you run an md-softraid, do it to all physical /dev/hdXYZ discs!
If you run Debian there is a predefined way to set the /sys values at boot-time:
<pre>
# apt-get install sysfsutils
[...]

# grep cfq /etc/sysfs.conf
block/sda/queue/scheduler = cfq
block/sdc/queue/scheduler = cfq

# /etc/init.d/sysfsutils restart
</pre>

For non-vserver processes and CFQ you can set by which key the kernel decides about the fairness:
<pre>
cat /sys/block/hdc/queue/iosched/key_type
pgid [tgid] uid gid
</pre>
Hint: The 'key_type'-feature has been removed in the mainline kernel recently. Don't look for it any longer :(

The default is tgid, which means to share fairly among process groups. Think every guest is treated like a own process group. It's not possible to set a scheduler strategy within a guest. All processes belonging to the same guest are treated like "noop" within the guest. So: If you run apache and some ftp-server within the _same_ guest, there is no fair scheduling between them, but there is fair scheduling between the whole guest and all other guests.

And: It's possible to tune the scheduler parameters in several ways. Have a look at /sys/block/hdc/queue/....
|Signature=derjohn}}

{{Question
|Question=Nice disk I/O scheduling, is that possible?
||Details=Well, since linux 2.6.13 processess have another priority next to the cpu nice scheduling hint, it's called io nice.
It's split into three groups, called real-time, best effort and idle. The default is best-effort, but within best-effort, you can have a niceness from 0 to and including 7.
You can set this niceness by the tool ionice, which for debian is either in the package util-linux or schedutils.
To change the io-niceness you need the <tt>CAP_SYS_NICE</tt>, '''and''' need to have the same uid as the processe you want to ionice.

:'''Note:''' If you want to use any schedulung other than best-effort you will also need the <tt>CAP_SYS_ADMIN</tt>-flag. Be warned that this gives quite some capabilities to the vserver, not just for I/O scheduling!

If you want to increase the niceness of an I/O hogging process within a vserver you need to do:
<PRE>
chcontext --xid sponlp1 sudo -u '#2089' ionice -c2 -n5 -p24409
</PRE>
with sudo and ionice installed on the root server to increase the *nice*ness of pid 24409, with uid 2089
|Signature=Groteblup}}

== Unification ==

{{Question
|Question=What is unification (vunify)?
||Details=Unification is Hard Links on Steroids. Guests can 'share' common files (usually binaries and libraries) in a secure way, by creating hard links with special properties (immutable but unlinkable (removable)). The tool to identify common files and to unify them is called vunify.
|Signature=derjohn}}

{{Question
|Question=What is vhashify?
||Details=The successor of vunify, a tool which does unification based on hash values (which allows to find common files in arbitrary paths.)
It creates hardlinks to files named after a hash of the content of the file. If you have a recent version of the vserver patch (2.2+), with CONFIG_VSERVER_COWBL enabled, you can even modify the hardlinked files inside the vservers and the links will be broken automatically.
There seems to be a catch when a hashified file has multiple hardlinks inside a guest, or when another internal hardlink is added after hashification. Link breaking will remove all the internal hardlinks too, so the guest will end up with different copies of the original file. The correct solution would be to not hashify files that have multiple links prior to hashification, and to break the link to the hashified version when a new internal hardlink is created. Apparently, this is not implemented yet (?).
|Signature=Guy-}}

{{Question
|Question=How do I manage a multi-guest setup with vhashify?
||Details=For 'vhashify', just do these once:
<pre>
mkdir /etc/vservers/.defaults/apps/vunify/hash /vservers/.hash
ln -s /vservers/.hash /etc/vservers/.defaults/apps/vunify/hash/root
</pre>
Then, do this one line per vserver:
<pre>
mkdir /etc/vservers/<vservername>/apps/vunify # vhashify reuses vunify configuration
</pre>
To hashify a running vserver, do (possibly from a cronjob):
<pre>
vserver name-of-guest hashify
</pre>

The guest needs to be running because vhashify tries to figure out what files not to hashify by calling the package manager of the guest via <tt>vserver enter</tt>.

In order for the OS cache to benefit from the hardlinking, you'll have to restart the vservers.

To clean up hashified files that are no longer referenced by any vserver, do (possibly from a cronjob):
<pre>
find /vservers/.hash -type f -links 1 -print0 | xargs -0 rm
</pre>

Until you do this, the files still take up place even though no vservers need them.
|Signature=Guy-}}

== Filesystem usage ==

{{Question
|Question=Is there a way to implement "user/group quota" per VServer?
||Details=Yes, but not on a shared partition for now. You need to put the guest on a separate partition, setup a vroot device (to make the quota access secure), copy that into the guest, and adjust the mtab line inside the guest. If 8 vroot device is not enough for you, you can add more with the kernel parameter max_vroot (exemple for built in kernel vroot /vmlinuz-2.6.31.6-vs2.3.0.36.26aq root=/dev/md1 ro max_vroot=20 ). If vroot is a module you'd actually want to put for exemple "options vroot max_vroot=20" in /etc/modprobe.conf and then just do modprobe vroot
|Signature=derjohn,gadnet}}

{{Question
|Question=What about "Quota" for a context? Howto limit disk usage?
||Details=Context quotas are now called Disk Limits (so that we can tell them apart from the user/group quotas :). They are supported out of the box (with vs2.0+) for all major filesystems (ext2/3, ReiserFS, JFS). You need to tag the FS with XID (see below). Please read [[Disk Limits and Quota]] for more information.
|Signature=derjohn}}

{{Question
|Question=How do I tag a guest's directory with xid?
||Details=Tagging the guest's files gives you several advantages, e.g. the accounting will work properly.
Filesystem XID tagging only works on supported filesystem. Those are currently: ext2/3, reiserfs/reiser3, xfs and jfs.
To activate the XID tagging you have to mount the filesystem with "-o tag" (former tagxid is outdated since VS2.2). Attention: It's _not_ possible to "-o remount,tag", you have to mount it freshly. The guests will tag their files automatiaclly. If you copy files in from the host, you have to tag them manually like this:
<pre>
chxid -c xid -R /var/lib/vservers/<guest>
</pre>
Note: Context 0 and 1 will see all files, guests will only be able to access untagged files and their own XID. They can see other XID files but no information about the file, e.g. no owner, no group, no permissions.
Note: It is not advised to tag the root filesystem, as [http://www.paul.sladen.org/vserver/archives/200602/0020.html explained by Herbert] : trying to do so will expose you to some troubles !
|Signature=derjohn_and_gonzo_and_are}}

{{Question
|Question=How can I copy anything from host to guest partition, normally unvisible on host?
||Details=You should just change namespace, e.g.:
<pre>
vnamespace --enter <xid> -- /bin/bash
</pre>
and then use standard cp or rsync programs.
|Signature=SergiuszPawlowicz}}

== Network ==

{{Question
|Question=Does it support IPv6?
||Details=Currently it requires an additional patch, but the functionality should be available in 2.3+ soon. [[IPv6]] has more information.
|Signature=derjohn}}

{{Question
|Question=I can't do all I want with the network interfaces inside the guest?
||Details=For now the networking is 'Host Business' -- the host is a router, and each guest is a server. You can set the capability ICMP_RAW in the context of the guest, or even the capability CAP_NET_RAW (which would even allow to sniff interfaces of other guests!). Likely to change with ngnet.
|Signature=derjohn}}

{{Question
|Question=How do I add several IPs to a vserver?
||Details=First of all a single guest vserver only supports up to 16 IPs (There is a 64-IP patch available, which is in "derjohn's kernel").
Here is a little helper-script that adds a list of IPs defined in a text file, one per line.
<pre>
#!/bin/bash
j=1
for i in `cat myiplist`; do
j=$(($j+1))
mkdir $j
echo $i > $j/ip
echo "24" > $j/prefix
done
</pre>
|Signature=derjohn}}

{{Question
|Question=How do I assign a new IP address to a running guest?
||Details=This is done from the host server:
* add the ip on the host, for example
<pre>
ip addr add 194.169.123.23/24 dev eth0
</pre>
* add the ip to the guest's network context (a guests NID is the same as the XID {context ID})
<pre>
naddress --add --nid <nid> --ip 194.169.123.23/24
</pre>
* enter the guest (best via ssh)
* restart the services that need to make use of the new address if required
* update the config in ''/etc/vserver/<servername>/interfaces'' to reflect the changes for the next guest restart (if desired)
|Signature=BenjaminGreen}}

{{Question
|Question=If my host has only one a single public IP, can I use RFC1918 IP (e.g. 192.168.foo.bar) for the guest vservers?
||Details=Yes, use iptables with SNAT to masquerade it.
<pre>
iptables -t nat -I POSTROUTING -s $VSERVER_NETZ ! -d $VSERVER_NETZ -j SNAT --to $EXT_IP
</pre>
See: [[HowtoPrivateNetworking]] and
http://www.tgunkel.de/it/software/doc/linux_server.en#h3-VServer_Masquerading_SNAT (THX, [MUPPETS]Gonzo)
|Signature=derjohn}}

{{Question
|Question=If I shut down my vserver guest, the whole Internet interface ethX on the host is shut down. What happened?
||Details=When you shut down a guest (''i.e. vserver foo stop''), the IP is brought down on the host also. If this IP happens to be the primary IP of the host, the kernel will not only bring down the primary IP, but also all secondary IP addresses. But in very recent kernels, there is an option ''settable'' which prevents that nasty feature. It's called "alias promotion". You may set it via sysctl by adding ''net.ipv4.conf.all.promote_secondaries=1'' in /etc/sysctl.conf or via sysctl command line.
|Signature=derjohn}}

{{Question
|Question=Can I run an OpenVPN Server in a guest?
||Details=
Yes. To get a OpenVPN Server running in a guest, all networking setup has to be done on the host. This answer describes the common case and shows some pitfalls, for detailled information about OpenVPN, please consult the appropriate documentation on the OpenVPN homepage.
This is the minimal OpenVPN configuration for the Server which will be used to demonstrate how to get it running in a client:
<pre>
# Networking setup
server 192.168.16.0 255.255.255.0
dev tun16
ifconfig-noexec
comp-lzo
# Certificates
dh ...
ca ...
cert ...
key ...
# Management
persist-key
keepalive 10 60
verb 4
</pre>
First of all you have to prepare the host with a persistent interface in the right mode and with the right settings. This is easily done by using openvpn and the ip and route tools.
<pre>
# openvpn --mktun --dev tun16
# ip link set dev tun16 txqueuelen 100
# ifconfig tun16 192.168.16.1 pointopoint 192.168.16.2 mtu 1500
# route add -net 192.168.16.0 netmask 255.255.255.0 gw 192.168.16.2
</pre>
If you need different settings, openvpn will tell you the ifconfig and route commands it uses to configure the interface when being started on the host with the original config file, but without ifconfig-noexec.
Additionally, the guest needs /dev/net/tun to make OpenVPN happy. This can be created with MAKEDEV:
<pre>
# cd /var/lib/vserver/<myopenvpnserver>/dev/
# ./MAKEDEV tun
(creates the dev/net/tun device accessible by the guest - even a tap interface needs /dev/net/tun !)
</pre>
Finally, the guest needs to have the tun device assigned:
<pre>
# head /etc/vservers/<myopenvpnserver>/interfaces/1/*
==> /etc/vservers/<myopenvpnserver>/interfaces/1/ip <==
192.168.16.1

==> /etc/vservers/<myopenvpnserver>/interfaces/1/nodev <==
tun16

==> /etc/vservers/<myopenvpnserver>/interfaces/1/prefix <==
24
#
</pre>
The client's conf may look like that:
<pre>
# Basic setup
client
proto tcp-client
dev tun
remote <ipaddress>
comp-lzo
verb 4

# Certificate
ca ...
</pre>

[ Based on derJohn's original answer, all errors mine ]
|Signature=DavidS}}

{{Question
|Question=Trying to connect to a vserver from the host or another vserver on the same host fails
||Details=strace shows
<pre>
sin_addr=inet_addr("xx.xx.xx.xx")}, yy) = -1 EINVAL (Invalid argument)
</pre>
A: The host/guest cannot communicate with another guest on same host.
* check all netmasks on all interfaces (do they overlap) ?
* check policy routing (disable it temporary) ?
* check that lo is up (Networking within a host/guest always uses lo interface)
|Signature=CommonProblems}}

{{Question
|Question=Can I use iptables ?
||Details=Yes but right now only on the host (rootserver). Please realize that all traffic is local and will not touch the forward chain.
|Signature=BeginnerFAQ}}

{{Question
|Question=Is it possible to prevent guest from bringing down primary ip?
||Details=Yes. Remove /etc/vservers/<guest>/interfaces/X/dev, and touch /etc/vservers/<guest>/interfaces/X/nodev
|Signature=Daniel&Serge}}

{{Question
|Question=Is it possible to provide a different MAC address per vServer?
||Details=Short answer - yes but it's a hassle.

Real answer from '''_are_''':
<pre>
When I once needed 'real' seperate MAC-addresses I used TAP-devices and VDE2 ([http://vde.sourceforge.net/ Virtual Distributed Ethernet]).
Basically vServer is an isolation of existing resources, not a virtualization of 'new' devices.
Without extra fuss you can't add a 'new' network interface to a vServer, no matter if it is eth* or tap*, you always add it to the host and give the vServer access to it.
I got the TAP+VDE2 up and running, but I think it is too much trouble for basically the simple adding of IPs to a vServer unless you really need the MAC address separate.
</pre>
|Signature=bobnormal}}

{{Question
|Question=Is it possible to hide packet counters on the host network interface from vServer guests?
||Details=Yes, see [[Networking_vserver_guests|Networking vServer Guests]]
|Signature=bobnormal}}

== Administration tools ==

{{Question
|Question=Which guest vservers are running?
||Details=Use vserver-stat to find out. Example output:
<pre>
CTX PROC VSZ RSS userTIME sysTIME UPTIME NAME
0 77 965.1M 334.6M 14m14s18 2m28s69 1h33m46 root server
49152 7 14M 5.2M 0m00s40 0m00s30 1h30m15 chiffon
</pre>
|Signature=derjohn}}

{{Question
|Question=Is there a web-based interface for vserver that will allow creation/deletion/configuration etc. of vserver guests?
||Details=
* http://OpenVPS.org which is a set of scripts with a web-interface for webhosters/ISPs
* http://Openvcp.org which is a distributed system (agent!) with a web-interface, with which you can build/remove guests
* http://vsmon.revolutionlinux.com/ is a distributed monitoring-only solution that allows you to search for a particular vserver in your park.
|Signature=derjohn}}

== Hosting foreign distributions ==

{{Question
|Question=I run a Debian host and want to build an Ubuntu guest. Howto?
||Details=Simple ;) Assume you want to build a breezy guest on a sid host with IP 192.168.0.2 and hostname vubuntu, then do:
<pre>
vserver vubuntu build --force -m debootstrap --hostname vubuntu.myvservers.net --netdev eth0 --interface 192.168.0.2/24 \
--context 42 -- -d breezy -m http://de.archive.ubuntu.com/ubuntu
</pre>

[UPDATE] Currently there are problems in building breezy under unclear circumstances, which seems to have to do with udev. If the above didnt work, try:
<pre>
vserver vubuntu build --force -m debootstrap --hostname vubuntu.myvservers.net --netdev eth0 --interface 192.168.0.2/24 \
--context 42 -- -d breezy -m http://de.archive.ubuntu.com/ubuntu -- --exclude=udev
</pre>
In very recent versions of the utils, the problem should not occur anymore (it has to do with the 'secure-mount' if you look in the MLs)

Well, sid's debootstrap knows how to bootstrap Ubuntu linux. Make sure to have a current debootstrap package:
<pre>
apt-get update
apt-get install debootstrap
</pre>
The knowledge how to build ubuntu 'breezy badger' (which you probably want to be your guest at the time of writing) has been added recently.
|Signature=derjohn}}

{{Question
|Question=I want to build a Gentoo guest. Howto?
||Details=Even simpler ;) See http://www.gentoo.org/proj/en/vps/vserver-howto.xml#doc_chap3 .
|Signature=gcc}}

== Application level problems ==

{{Question
|Question=I did everything right, but the application foo does not start. What's up there?
||Details=Before asking on the IRC channel, please check out the 'problematic programs' page:
[[Problematic Programs]]
|Signature=derjohn}}

{{Question
|Question=When I try to ssh to the guest, I log into the host, even if I installed sshd on the guest. What's wrong here?
||Details=Look at /etc/ssh/sshd_config of the host:
<pre>
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
</pre>
And now change the setting to
<pre>
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
ListenAddress your.hosts.ip.here # not the guests IP!
</pre>
Then '/etc/init.d/ssh restart' on the host, after that on the guest (if you did apt-get install ssh on the guest already.)
Do I have to explain more? If the hosts sshd binds all available IP addresses on port 22 (The hosts 'sees' even all addresses of the guests!). So if the guest starts its sshd, it can't bind to port 22 any more. You need to change that setting only on the host.
(BTW: A similar approach has to be done for a lot of daemons, e.g. Apache. If the daemon does not support an explicit bind, you may use the chbind command to 'hide' IP addresses from the daemon before starting.)|Signature=derjohn}}

{{Question
|Question=Bind9 does not like to start in my guest.
||Details=Check out the [[Problematic Programs]] page and/or get my [http://linux-vserver.derjohn.de/bind9-packages/bind9-capacheck_9.3.2-2_i386.deb vserver-guest-ready Debian package] for Debian Sid guests and check out the [http://linux-vserver.derjohn.de/bind9-packages/README.txt readme]. (Hint: This is fresh stuff. Please give me feedback)

[UPDATE] Since VServer Devel 2.1.1-rc18 you do not need to patch the userland tools anymore. The capabilities are masked.
|Signature=derjohn}}

{{Question
|Question=My mysqld running in a guest behaves strangely and is awfully slow/locks up
||Details=This can be related to /tmp being too small. mysqld stores temporary tables in /tmp and as such, if a lot of queries happen and /tmp runs full this can cause one query to lock up whilst creating the tmp table and all other queries waiting to acquire the lock. There are two possible solutions to that problem: a.) Modify /etc/vservers/vserver-name/fstab and assign more memory to the tmpfs of /tmp and b.) remove the /tmp entry from /etc/vservers/vserver-name/fstab completly. Especially on database servers with a rather high load the second one might be the preferred method.|Signature=sp}}

{{Question
|Question=Pure-FTP does not run inside a VServer?
||Details=That's because it has capabilities enabled, make sure you rebuild your distro's package passing also the `--without-capabilities` flag to configure.
|Signature=Pedro Algarvio, aka, s0undt3ch}}

{{Question
|Question=Why do neither sshd nor crond (vixie-cron) work correctly in my CentOS / Fedora guest? I get 'pam_loginuid(crond:session): set_loginuid failed opening loginuid' and similar lines in my logs.
||Details=Took me a while to figure this out, and it turned out to be mentioned in the old wiki. Here is the solution on how to solve a common problem with sshd / crond, somehow related to selinux and auditing:

pam authentication (also used with openssh) enables "pam_loginuid.so" in the /etc/pam.d/* files. Comment those out as they are not necessary and will not load within a guest anyway. This probably is also necessary on updates later on, if the configs get changed. You therefore may add the following command line to a cronjob file or your software update script:
<pre>
/bin/sed --in-place -e "s/^session.*required.*pam_loginuid.so/# session\trequired\tpam_loginuid.so/g" /etc/pam.d/*
</pre>
|Signature=patrick}}

{{Question
|Question=How do i install nagios-plugins on a Gentoo guest?
||Details=Unfortunately, the nagios-plugins ./configure scripts wants to ping 127.0.0.1 which is not available inside a guest. Therefore you have to build nagios-plugins outside the guest.
The easiest way to do this from the host (assuming the guest is running) is:
<pre>
vnamespace -e <xid> -- chroot /vservers/<name> emerge nagios-plugins -va
</pre>
|Signature=Hollow}}

{{Question
|Question=Somebody runs ntpd in guest and you can't use ntpdate in host?
||Details=Try to run ntpdate with options -u :
ntpdate -u ntp.domain.xy
or you can use command:
chbind --nid 42 --ip 1.2.3.4 -- ntpdate ntp.domain.xy
where IP will be the IP of host.
|Signature=Punkie/Bertl}}

== Start / Stop a VServer ==

{{Question
|Question=How do I make a vserver guest start by default?
||Details=At least on Debian, I can tell you how to do it with the new-style config. If your guest is called "derjohn" and you want it to be started somewhere at the of your bootstrap process, then do:
<pre>
echo "default" > /etc/vservers/derjohn/apps/init/mark
</pre>
If you want to start it earlier, please read the init script "/etc/init.d/util-vserver" to find out how to do it. In most cases you don't need to change this. On Debian the vservers are started at "20", so after most other stuff is up (networking etc.).

Besides that I created a small helper script for managing the autostart foo: ((vserver-autostart))|Signature=derjohn}}

{{Question
|Question=My host works, but when I start a guest it says that it has a problem with chbind.
||Details=You are probably using util-vserver <= 0.30.209, which does use dynamic network contexts internally (With 0.30.210 this fact changed). So if you compiled your kernel without dynamic contexts, you may start guests, but you can't use the network context.The solution is either to switch to .210 util (or Hollow's toolset) or compile the kernel with dynamic network contexts.
SE Keyword: invalid option `nid' testme.sh
|Signature=derjohn}}

{{Question
|Question=What is old-style and new-style config?
||Details=Old-style config refers to a single text-file that contains all the configuration settings. With new-style config the configuration is split into several directories and files. You should probably go for new-style config if you are asking.
|Signature=derjohn}}

{{Question
|Question=How can I reboot/halt guests?
||Details=It depends.
For legacy Linux-VServer (i.e. 1.2.x), you have to replace /sbin/halt in the guests with vreboot and start rebootmgr in the host. You also need to have a <guest>.conf file in /etc/vservers for each guest. Please have a look at /etc/init.d/rebootmgr.
For Linux-VServer 2.0+, sys_reboot has been virtualized to do the right thing. No changes are needed in guests. Please note that some things depend on the init style used by the guest : read [[util-vserver:InitStyles]]
|Signature=derjohn}}

{{Question
|Question=What is the initial PATH?
||Details=By default, vserver uses the 'sysv' startup style, which mimics the init process by running the 3rd runlevel through '/etc/init.d/rc 3' (or '/etc/rc.d/rc 3'). Usually this 'rc' script uses a hard-coded PATH. In the case it doesn't, util-vserver also mimics init's default PATH through /etc/vservers/.defaults/apps/init/environment, or if not present /usr/local/lib/util-vserver/defaults/environment. Beware that all those default PATH usually do not include /usr/local.
|Signature=daniel_hozac&Beuc}}

{{Question
|Question=When I try to start a guest i get this message "/proc/uptime can not be accessed. Usually, this is caused by procfs-security. Please read the FAQ for more details"?
||Details=After a reboot you need to run the vprocunhide script. If running this script causes many errors to print on the screen, try checking the kernel you have booted with (perhaps it does not have the linux-vserver extensions enabled).
|Signature=mattzerah}}

== Kernel ==

{{Question
|Question=Is SMP Supported?
||Details=Yes, on all SMP capable kernel architectures.
|Signature=derjohn}}

{{Question
|Question=Do I really need the legacy-interfaces? What are these legacy-interfaces?
||Details=Since Linux-VServer is an ongoing project, new features might replace old ones, some might require a development version. Legacy-interfaces are available for backward compability (which might be removed someday) with Linux-VServer 1.2.x.
|Signature=derjohn}}

{{Question
|Question=I have a vserver running on a Linux kernel with preemption. Is VServer "preempt" safe?
||Details=There are no known issues about running vserver on a preemption enabled kernel. I would like to add, that the vserver kernelhackers would probably exclude that option in 'make menuconfig' if there would be an incompatibility. Just my $.02 :)
|Signature=derjohn}}

{{Question
|Question=32 vs 64 Bit? What should I take?
||Details=If you have the choice make the host a 64 bit one. You can run a guest as 32 bit or as 64 bit on a 64 bit host. To run it as 32 bit, you need to compile the x86_64 (a.k.a. AMD64) with the following options:
<pre>
[*] Kernel support for ELF binaries
<M> Kernel support for MISC binaries
[*] IA32 Emulation <---- without that, the entire 32bit API is not present
<M> IA32 a.out support
</pre>
You can force the guest to behave like a 32 environment like this:
<pre>
echo linux_32bit > /etc/vservers/$NAME/personality
echo i686 > /etc/vservers/$NAME/uts/machine
</pre>
(thanks cehteh for the hint!)

But you can force debootstrap to put 32 bit binaries into the guest by 'export ARCH=i386';
<pre>
export ARCH=i386 ; vserver build ....
</pre>

On debian when using the newvserver script "export ARCH=i386" has no effect, just use:
<pre>
newvserver --arch i386 ...
</pre>
|Signature=derjohn}}

== Distribution specific questions ==

{{Question
|Question=VServer is included in the stable Debian GNU/Linux for years now. What VS version did they include?
||Details=At the time of writing, Debian Lenny is the stable release of Debian and includes a 2.6.26 based kernel-package called 2.6.26-2-vserver-ARCH. This currently contains VServer 2.3.0.35 (according to changelog.Debian.gz in the Debian package).
|Signature=scientes}}

{{Question
|Question=Were can I get newer versions of VServer as ready made packages for Debian?
||Details=Here you go: http://linux-vserver.derjohn.de/ . There is also some stuff on backports.org, but my kernels are always 'devel' branch.
|Signature=derjohn}}

== Misc ==

{{Question
|Question=Why isn't there a device /dev/xyz within a guest?
||Details=Device nodes allow userspace to access hardware (or virtual resources). Creating a device node inside the guest's namespace will give access to that device, so for security reasons, the number of 'given' devices is small.
|Signature=derjohn}}

{{Question
|Question=I want to (re)mount a partition in a running guest ... but the guest has no rights (capability) to (re)mount?
||Details=I'll explain. I take as example your /tmp partition within the guest is too small, what will be likely the case if you stay with the 16MB default (vserver build mounts /tmp as 16 MB tmpfs!).
<pre>
# vnamespace -e XID mount -t tmpfs -o remount,size=256m,mode=1777 none /var/lib/vservers/<guest>/tmp/
</pre>
(if there's a problem, try expanding the symlinks in the mount path)
Be warned that the guest will not recognize the change, as the /etc/mtab file is not updated when you mount like this. To permanently change the mount, edit /etc/vserver/<guest>/fstab on the host.

If you get:
<pre>
mount: can't find /var/lib/vservers/<guest>/tmp in /etc/fstab or /etc/mtab
</pre>
then try instead:
<pre>
vnamespace -e builder chroot /var/lib/vservers/<guest>/ mount -o remount,size=64m,mode=1777 /tmp
</pre>

Note that this not work for adding a bindmount (<tt>-o bind</tt>) of a directory outside of a vserver into the vserver. For this, there is no alternative but restarting the vserver.
|Signature=derjohn}}

{{Question
|Question=Does anyone know how to increase the size of /tmp within a vserver w/o restarting?
||Details=Use the remount option for mount.
# vnamespace -e XID mount -n -t tmpfs -o remount,size=32m tmpfs /<vdir>/<guest>/tmp
or something like that. The arguments are needed since mount is not going to be using /etc/fstab for the information and the version of /proc/mounts is best understood by
# vnamespace -e XID cat /proc/mounts.
See [[Frequently_Asked_Questions#I want to (re)mount a partition in a running guest ... but the guest has no rights (capability) to (re)mount?]]
|Signature=derjohn/dhozac}}

{{Question
|Question=#1 ERROR: capset(): Operation not permitted
||Details=capabilities are not enabled in kernel-setup
please check that CONFIG_SECURITY_CAPABILITIES is loaded or included in the kernel. ( check with "cat /path_to_kernel/.config | grep -i cap ")
(2.6.11.5-vs-1.9.5 + 0.30-205)
|Signature=IrcQuestions}}

{{Question
|Question=How can I make 'vserver start' mount the root filesystem?
||Details=Mount it via /etc/vservers/vserver-name/fstab, make sure to set the option 'dev' e.g.:
<pre>/dev/drbd0 / xfs rw,dev 0 0</pre>
|Signature=AdrianReyer}}

{{Question
|Question=I deleted a guest's directory without shutting it down. Now I have a "ghost" running. Is there any possibility to get it out of proc without rebooting?
||Details=
vkill --xid <xid> -s 15; sleep 2; vkill --xid <xid> -s 9

You will also need to remove guest's ip, for example with:
ip addr del <ip> dev eth0
|Signature=daniel_hozac & gebura }}

{{Question
|Question=When using nice and su (for example, in the updatedb cron job), I get: su: Permission denied. What does it mean?
||Details=A guest cannot lower its nice value - and that's what 'su' does through pam_limits which sets a nice value of 0. You can see it through strace:
$ strace nice su nobody
[...]
setpriority(PRIO_PROCESS, 0, 0) = -1 EACCES (Permission denied)
You can use 'su nobody -c nice some_cmd' instead.
(Now there's the question of why a guest process cannot lower its nice value.)
|Signature=daniel_hozac&Beuc}}

{{Question
|Question=How do I handle NFS mounts within in a guest?
||Details=There are three ways.

'''1)''' Mount the NFS share from the host OS and let vserver guest access it as part of it's file system.

''mount --bind'' may also be beneficial in this scenario.

'''2)''' Use util-vserver and create a ''fstab.remote'' file in the /etc/vserver/<vserver_name> directory. Populate this with the NFS shares and they will be mounted in the context of the vserver guest.

See http://www.nongnu.org/util-vserver/doc/conf/configuration.html

'''3)''' Add capabilities to the vserver guest instance to grant sufficient rights to allow NFS mounts.

Add the following to /etc/vserver/<vserver_name>/bcapabilities
SYS_ADMIN
Add the following to /etc/vserver/<vserver_name>/ccapabilities
SECURE_MOUNT
BINARY_MOUNT

See [[Capabilities_and_Flags]] for more information about vserver capabilities.

If you want the NFS shares to be mounted when the guest starts, add them to /etc/vserver/<vserver_name>/fstab

||Signature=martindk}}

{{Question
|Question=vserver start/stop/enter fails with something like "vnamespace: execvp("/usr/sbin/vserver"): No such file or directory" ?
||Details=Check whether ''/usr'' is mounted in the namespace you are working with.
<pre>vnamespace -e <guest> cat /proc/mounts</pre>
If there is no ''/usr'', you can fix your problem with simply mounting it using the following command:
<pre>vnamespace -e <guest> mount /dev/<device> /usr</pre>

||Signature=sim0n}}

{{Question
|Question=The command vserver <$server> start gives '/etc/init.d/rc: line 74: /etc/default/rcS: No such file or directory', what do I do?
||Details=The vserver has not been correctly installed, this has several reasons
check your install log and it should tell you something about that your server didn't get installed properly
* use stable distribution of debian as server (debootstrap may be different over the versions)
* deny_mount, deny_caps and deny_pivot should be off if your running grsec.

||Signature=Dude}}

{{Question
|Question=How could I rename a vserver directory?
||Details=Please note : this procedure renames the '''directory''', not the '''hostname''' !
#Stop the vserver in question
#rename the <tt>/vservers/<server name></tt> directory
#rename the <tt>/etc/vservers/<server name></tt> directory
#update link: <tt>/etc/vservers/<server name>/run</tt> → <tt>/var/run/vservers/<server name></tt>
#update link: <tt>/etc/vservers/<server name>/vdir</tt> → <tt>/etc/vservers/.defaults/vdirbase/<server name></tt>
#update link: <tt>/etc/vservers/<server name>/cache</tt> → <tt>/etc/vservers/.defaults/cachebase/<server name></tt>
#update link: <tt>/var/run/vservers.rev/<server XID></tt> → <tt>/etc/vservers/<server name></tt>
#Start the vserver in question. It should start properly.

|Signature=FlorianD (from ''hillct'' page in old wiki)}}

== Upgrade from 2.0 to 2.2 ==

{{Question
|Question=I now get errors like "ncontext: vc_net_create(): Invalid argument; dynamic contexts disabled." on startup. Vservers are not started
||Details=Dynamic context are disabled by default and are deprecated. For example, tagxid and network checks won't be useable with dynamic ids. Now you should manually assign a explicit context to your vservers, like
echo 101 > /etc/vservers/myvserv/context
ADDENDUM: please consider that valid static contexts are between 2 and 49151 ( daniel_hozac on IRC ) otherwise you will end up with unexplainable error "ncontext: vc_net_migrate(): No such process" when trying to start the vserver.

|Signature=daniel_hozac&Beuc}}

{{Question
|Question=How do I assign a static context to an existing vserver?
||Details=Simple ;) See the answer above.
|Signature=gcc}}

{{Question
|Question=Since upgrading to a newer VS version my guest complains about "vsched: non-numeric value specified for '--priority_bias" at start time. What's wrong?
||Details=The scheduler paramters changed.You can use this (ugly) script to convert them or do it by hand:
<pre>
# cat /usr/local/sbin/vserver-convert-schedule-to-scheddir
#/bin/sh
mkdir /etc/vservers/$1/sched
sed -e 1p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/fill-rate
sed -e 2p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/interval
sed -e 3p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens
sed -e 4p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens-min
sed -e 5p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens-max

mv /etc/vservers/$1/schedule /etc/vservers/$1/schedule.converted.see.scheddir

# see: http://oldwiki.linux-vserver.org/Scheduler+Parameters
# see: http://www.nongnu.org/util-vserver/doc/conf/configuration.html#sched
</pre>
||Signature=derjohn}}

{{Question
|Question=Since upgrading to a newer VS version my guest doesn't have the amount of shared memory (SHM / SHMMAX / SHMALL ) as it had in the former version. What changed?
||Details=Every VS version that runs on a kernel >= 2.6.19 offers sysctl values per guest. This has to do with the 'ipc namespace' feature that was added to the mainline kernel in version 2.6.19. Linux-VServer uses that feature to give each guest a separate 'ipc namespace' and thus 'own' sysctl values per guest. Because shmmax is such a sysctl value, you have to set it per guest.
Here is an example how to do so:

<pre>
# mkdir /etc/vservers/<vserver>/sysctl/0 -p
# echo kernel.shmall > /etc/vservers/<vserver>/sysctl/0/setting
# echo 134217728 > /etc/vservers/<vserver>/sysctl/0/value
# mkdir /etc/vservers/<vserver>/sysctl/1 -p
# echo kernel.shmmax > /etc/vservers/<vserver>/sysctl/1/setting
# echo 134217728 > /etc/vservers/<vserver>/sysctl/1/value
</pre>
It's also explained on the geat flower page:
# see: http://www.nongnu.org/util-vserver/doc/conf/configuration.html -> Look for "sysctl".

After changing those values, restart your guest, enter it and check if the values are set:
<pre>
# sysctl -a | grep shm
...
kernel.shmall = 134217728
kernel.shmmax = 134217728
</pre>

To change a value for a running guest, on the host use:
<pre>
vspace -e CONTEXTID --ipc sysctl -w kernel.shmall=134217728
vspace -e CONTEXTID --ipc sysctl -w kernel.shmmax=134217728
</pre>

||Signature=derjohn
}}

Networking vserver guests

2010-02-15T13:52:21Z

Bobnormal: /* Host */ add explanation of optional dummy device procedure

Setting up network access to and from your vserver guests.
__TOC__
==Introduction==

Lets imagine, you have only one external IP -- <code>$EXTIP</code>.

You want to have several vservers running without worrying about port overlapping.

Example:

Two vservers run a default webserver, running on port 80. If each "guest" vserver shares an IP with the host, then the two webservers will conflict.

One solution is:

* All vservers are contained in a "virtual lan", say 192.168.1.x
* Each vserver has its own IP
* Control port forwarding on "parent" host. That is, run a router.

==Configuration==
===Host===

===Optional: Load 'dummy' device(s)===

====Why?====
This driver allows you to associate vServer guests with a ''virtual'' network device rather than the host's real network interface, which functions to hide packet counters from vServer guests. There is no other use for this step, so feel free to skip it if you are not worried about sharing packet counters. Packet counts ''may'' be useful to an attacker with control of a vServer guest who wishes to perform side-channel attacks during cryptanalysis, or network traffic analysis against your host or other vServer guests. If you did not understand the last sentence and your installation is not particularly security sensitive, then the chances that you will encounter a skilled attacker are slim to none and you should feel free to skip this section.

====Process====
First you need to load the dummy interface driver
# modprobe dummy

To have it automatically loaded on startup, add
"loopback" to /etc/modules

Set up <code>dummy0</code> interface on the parent host
# /etc/network/interfaces on a Debian box,
# configure on other distros with your preferred way
auto dummy0
iface dummy0 inet static
address 192.168.1.250
netmask 255.255.255.0

===Guests===
Set up each guest vserver:
cd /etc/vservers/$VSERVER/interfaces/0
echo dummy0 > dev
echo 192.168.1.1 > ip
echo 1 > name
echo 24 > prefix
Consider using a value of <code>name</code> equal to the last digit of the IP for easy separation.

===Host as router===
Configure the host to act as a router.

For internal packets going outside, pretend each packet came from our external IP (put it in one line without backslash):
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 \
-d ! 192.168.1.0/24 -j SNAT --to-source $EXTIP
For each service that runs on a vserver, map it to an external port. Vserver local address <code>$VHOST</code> and port <code>$INTPORT</code> you select one external port <code>$EXTPORT</code> and run the following (put it in one line without backslash):
# iptables -t nat -A PREROUTING -s ! 192.168.1.0/24 \
-m tcp -p tcp --dport $EXTPORT
-j DNAT --to-destination $VHOST:$INTPORT
That's all!

==Verifying==
Try <code>ping pool.ntp.org</code> from your vserver -- it should ping fine.

Try to connect to your <code>$EXTIP:$EXTPORT</code> (from another external host) -- you will successfully connect to service running on a guest vserver.
==See also==
* [[Frequently_Asked_Questions#If_my_host_has_only_one_a_single_public_IP.2C_can_I_use_RFC1918_IP_.28e.g._192.168.foo.bar.29_for_the_guest_vservers.3F |FAQ on private networking]]
* [[Frequently_Asked_Questions#When_I_try_to_ssh_to_the_guest.2C_I_log_into_the_host.2C_even_if_I_installed_sshd_on_the_guest._What.27s_wrong_here.3F |Permit guest sshd to bind to its IP address's port 22]]
* [[Networking_vserver_guests_RHEL|Networking_vserver_guests_RHEL]]

[[Category:HowTo]]
[[Category:Network]]

Capabilities and Flags

2010-02-15T13:39:35Z

Bobnormal: /* Network context flags (nflags) */ slightly clearer explanation of hide_netif and hide_lback

In computer science, a capability is a token used by a process to prove that it is allowed to perform an operation on an object. The Linux Capability System is based on "POSIX Capabilities", a somewhat different concept, designed to split up the all powerful root privilege into a set of distinct privileges.

== The Capability/Flag System ==

=== POSIX Capabilities ===

A process has three sets of bitmaps called the inheritable(I), permitted(P), and effective(E) capabilities. Each capability is implemented as a bit in each of these bitmaps that is either set or unset.

When a process tries to do a privileged operation, the operating system will check the appropriate bit in the effective set of the process (instead of checking whether the effective uid of the process is 0 as is normally done).

For example, when a process tries to set the clock, the Linux kernel will check that the process has the CAP_SYS_TIME bit (which is currently bit 25) set in its effective set.

The permitted set of the process indicates the capabilities the process can use. The process can have capabilities set in the permitted set that are not in the effective set.

This indicates that the process has temporarily disabled this capability. A process is allowed to set a bit in its effective set only if it is available in the permitted set. The distinction between effective and permitted exists so that processes can "bracket" operations that need privilege.

The inheritable capabilities are the capabilities of the current process that should be inherited by a program executed by the current process. The permitted set of a process is masked against the inheritable set during exec(). Nothing special happens during fork() or clone(). Child processes and threads are given an exact copy of the capabilities of the parent process.

The implementation in Linux stopped at this point, whereas POSIX Capabilities require the addition of capability sets to files too, to replace the SUID flag (at least for executables)

=== Upper Bound for Capabilities ===

Because the current Linux Capability system does not implement the filesystem related portions of POSIX Capabilities which would make setuid and setgid executables secure, and because it is much safer to have a secure upper bound for all processes within a context, an additional per-context capability mask has been added to limit all processes belonging to that context to this mask.
The meaning of the individual caps (bits) of the capability bound mask is exactly the same as with the permitted capability set.

=== Context Capabilities ===

As the Linux capabilities have almost reached the maximum number that is possible without heavy modifications to the kernel, it was a natural step to add a context-specific capability system.

The Linux-VServer context capability set acts as a mechanism to fine tune existing Linux capabilities. It is not visible to the processes within a context, as they would not know how to modify or verify it.

In general there are two ways to use those capabilities:
* Require one or a number of context capabilities to be set in addition to a given Linux capability, each one controlling a distinct part of the functionality. For example the CAP_NET_ADMIN could be split into RAW and PACKET sockets, so you could take away each of them separately by not providing the required context capability.
* Consider the context capability sufficient for a specified functionality, even if the Linux Capability says something different. For example mount() requires CAP_SYS_ADMIN which adds a dozen other things we do not want, so we define VXC_SECURE_MOUNT to allow mounts for certain contexts.

The difference between the context flags and the context capabilities is more an abstract logical separation than a functional one, because they are handled in a very similar way.

== List of capabilities/flags ==

Below is a list of capabilities and flags used for contexts and processes within. The tables contain the following information:

; Bit : The bit number to enable the capability/flag
; Mask : The bit number in hexadecimal notation
; Name : Human readable identifier used in userspace utilities
; Tag : Special capability/flag code to denote special behaviour, legacy usage and others (see below)
; Description : Description of capability/flag effects

=== Special capability/flags codes ===

The tag column may contain one or more of the following tags:

{| class="wikitable_inline"
! Tag
! Description
|-
| I
| Internal use only
|-
| L
| Only supported with legacy enabled
|-
| O
| One time capability/flag (once it's cleared, it can't be re-enabled again)
|-
| U
| Unsupported
|-
| X
| Slightly different meaning in legacy
|}

=== Context capabilities (ccaps) ===

The set of available context capabilities is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of capabilities currently available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| SET_UTSNAME
|
| Allow setdomainname(2) and sethostname(2)
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| SET_RLIMIT
|
| Allow setrlimit(2)
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| FS_SECURITY
|
| Allow setxattr for security attributes
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| TIOCSTI
|
| Allow the tiocsti ioctl (fake input character)
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| RAW_ICMP
| L
| Allow usage of raw ICMP sockets
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| SYSLOG
|
| Allow syslog(2)
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| OOM_ADJUST
|
| Allow 'safe' oom adjustments
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| AUDIT_CONTROL
|
| Allow loginuid write (for auditing)
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SECURE_MOUNT
|
| Allow secure mount(2)
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SECURE_REMOUNT
|
| Allow secure remount
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| BINARY_MOUNT
|
| Allow binary/network mounts
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| QUOTA_CTL
|
| Allow quota ioctls
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| ADMIN_MAPPER
|
| Allow access to device mapper
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| ADMIN_CLOOP
|
| Allow access to loop devices
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| KTHREAD
|
| Allow creating kernel threads
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| NAMESPACE
|
| Allow namespace related operations
|}

=== Context flags (cflags) ===

The set of available context flags is specific to Linux-VServer and applied to all processes contained within a context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| INFO_SCHED
| L
| Account all processes as one
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| INFO_NPROC
| L
| Apply process limits to context
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
| L
| Context cannot be entered
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| INFO_INIT
| X
| Show a fake init process
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| INFO_HIDE
| X
| Hide context information in task status
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| INFO_ULIMIT
| L
| Apply ulimits to context
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| INFO_NSPACE
| L
| Use private namespace
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SCHED_HARD
|
| Enable hard scheduler
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| SCHED_PRIO
|
| Enable priority scheduler
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| SCHED_PAUSE
|
| Pause context (unschedule)
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00010000
| VIRT_MEM
|
| Virtualize memory information
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00020000
| VIRT_UPTIME
|
| Virtualize uptime information
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00040000
| VIRT_CPU
|
| Virtualize cpu usage information
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00080000
| VIRT_LOAD
|
| Virtualize load average information
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x00100000
| VIRT_TIME
|
| Allow per guest time offsets
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x01000000
| HIDE_MOUNT
|
| Hide entries in /proc/$pid/mounts
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
| L
| Hide foreign network interfaces
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_VINFO
|
| Hide context information in task status
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 33
| style="font-family: monospace" | 0x0002<<32
| STATE_INIT
| IO
| Enable init state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 37
| style="font-family: monospace" | 0x0020<<32
| REBOOT_KILL
|
| Kill all processes on reboot(2)
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make context persistent
|-
| style="text-align: right" | 48
| style="font-family: monospace" | 0x0001<<48
| FORK_RSS
|
| Block fork when RSS limit is exceeded
|-
| style="text-align: right" | 49
| style="font-family: monospace" | 0x0002<<48
| PROLIFIC
|
| Allow context to create new contexts
|-
| style="text-align: right" | 52
| style="font-family: monospace" | 0x0010<<48
| IGNEG_NICE
|
| Ignore priority raise
|}

=== Network context flags (nflags) ===

The set of available network context flags is specific to Linux-VServer and applied to all processes contained within a network context. Below is a list of flags available in 2.1.1 and above.

{| class="wikitable_list"
! Bit
! Mask
! Name
! Tag
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| INFO_LOCK
| L
| Prohibit further context migration
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| INFO_PRIVATE
|
| Context cannot be entered
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SINGLE_IP
|
| Enable special handling of network contexts with a single IP only
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LBACK_REMAP
|
| use loopback-virtualisation (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| LBACK_ALLOW
|
| if set, allows guests without LBACK_REMAP to connect to 127.0.0.0/8
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x02000000
| HIDE_NETIF
|
| Hide foreign network interfaces (ie: network interfaces not carrying and IP belonging to the guest)
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x04000000
| HIDE_LBACK
|
| hides the real loopback-address from the guest (rewrites to 127.0.0.1 when queried) (will only work in 2.3.0.xx or greater)
|-
| style="text-align: right" | 32
| style="font-family: monospace" | 0x0001<<32
| STATE_SETUP
| IO
| Enable setup state
|-
| style="text-align: right" | 34
| style="font-family: monospace" | 0x0004<<32
| STATE_ADMIN
| O
| Enable admin state
|-
| style="text-align: right" | 36
| style="font-family: monospace" | 0x0010<<32
| SC_HELPER
| I
| Enable state change helper
|-
| style="text-align: right" | 38
| style="font-family: monospace" | 0x0040<<32
| PERSISTENT
|
| Make network context persistent
|}

=== System capabilities (bcaps) ===

The set of available system capabilities is inherited from the Linux kernel and applied to all processes contained within a context. Below is a list of capabilities currently available in the vanilla kernel.

<div style="color: red; font-weight: bold;">
BIG FAT WARNING: Adding any system capability to your virtual server WILL reduce security. Do not change the default values unless you absolutely know what you are doing!
</div>

{| class="wikitable_list"
! Bit
! Mask
! Name
! Description
|-
| style="text-align: right" | 0
| style="font-family: monospace" | 0x00000001
| CHOWN
| In a system with the [_POSIX_CHOWN_RESTRICTED] option defined, this overrides the restriction of changing file ownership and group ownership.
|-
| style="text-align: right" | 1
| style="font-family: monospace" | 0x00000002
| DAC_OVERRIDE
| Override all DAC access, including ACL execute access if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 2
| style="font-family: monospace" | 0x00000004
| DAC_READ_SEARCH
| Overrides all DAC restrictions regarding read and search on files and directories, including ACL restrictions if [_POSIX_ACL] is defined. Excluding DAC access covered by CAP_LINUX_IMMUTABLE.
|-
| style="text-align: right" | 3
| style="font-family: monospace" | 0x00000008
| FOWNER
| Overrides all restrictions about allowed operations on files, where file owner ID must be equal to the user ID, except where CAP_FSETID is applicable. It doesn't override MAC and DAC restrictions.
|-
| style="text-align: right" | 4
| style="font-family: monospace" | 0x00000010
| FSETID
| Overrides the following restrictions that the effective user ID shall match the file owner ID when setting the S_ISUID and S_ISGID bits on that file; that the effective group ID (or one of the supplementary group IDs) shall match the file owner ID when setting the S_ISGID bit on that file; that the S_ISUID and S_ISGID bits are cleared on successful return from chown(2) (not implemented).
|-
| style="text-align: right" | 5
| style="font-family: monospace" | 0x00000020
| KILL
| Overrides the restriction that the real or effective user ID of a process sending a signal must match the real or effective user ID of the process receiving the signal.
|-
| style="text-align: right" | 6
| style="font-family: monospace" | 0x00000040
| SETGID
|
* Allows setgid(2) manipulation
* Allows setgroups(2)
* Allows forged gids on socket credentials passing.
|-
| style="text-align: right" | 7
| style="font-family: monospace" | 0x00000080
| SETUID
|
* Allows set*uid(2) manipulation (including fsuid).
* Allows forged pids on socket credentials passing.
|-
| style="text-align: right" | 8
| style="font-family: monospace" | 0x00000100
| SETPCAP
| Transfer any capability in your permitted set to any pid, remove any capability in your permitted set from any pid
|-
| style="text-align: right" | 9
| style="font-family: monospace" | 0x00000200
| LINUX_IMMUTABLE
| Allow modification of S_IMMUTABLE and S_APPEND file attributes
|-
| style="text-align: right" | 10
| style="font-family: monospace" | 0x00000400
| NET_BIND_SERVICE
|
* Allows binding to TCP/UDP sockets below 1024
* Allows binding to ATM VCIs below 32
|-
| style="text-align: right" | 11
| style="font-family: monospace" | 0x00000800
| NET_BROADCAST
| Allow broadcasting, listen to multicast
|-
| style="text-align: right" | 12
| style="font-family: monospace" | 0x00001000
| NET_ADMIN
|
* Allow interface configuration
* Allow administration of IP firewall, masquerading and accounting
* Allow setting debug option on sockets
* Allow modification of routing tables
* Allow setting arbitrary process / process group ownership on sockets
* Allow binding to any address for transparent proxying
* Allow setting TOS (type of service)
* Allow setting promiscuous mode
* Allow clearing driver statistics
* Allow multicasting
* Allow read/write of device-specific registers
* Allow activation of ATM control sockets
|-
| style="text-align: right" | 13
| style="font-family: monospace" | 0x00002000
| NET_RAW
|
* Allow use of RAW sockets
* Allow use of PACKET sockets
|-
| style="text-align: right" | 14
| style="font-family: monospace" | 0x00004000
| IPC_LOCK
|
* Allow locking of shared memory segments
* Allow mlock and mlockall (which doesn't really have anything to do with IPC)
|-
| style="text-align: right" | 15
| style="font-family: monospace" | 0x00008000
| IPC_OWNER
| Override IPC ownership checks
|-
| style="text-align: right" | 16
| style="font-family: monospace" | 0x00010000
| SYS_MODULE
|
* Insert and remove kernel modules - modify kernel without limit
* Modify cap_bset
|-
| style="text-align: right" | 17
| style="font-family: monospace" | 0x00020000
| SYS_RAWIO
|
* Allow ioperm/iopl access
* Allow sending USB messages to any device via /proc/bus/usb
|-
| style="text-align: right" | 18
| style="font-family: monospace" | 0x00040000
| SYS_CHROOT
| Allow use of chroot()
|-
| style="text-align: right" | 19
| style="font-family: monospace" | 0x00080000
| SYS_PTRACE
| Allow ptrace() of any process
|-
| style="text-align: right" | 20
| style="font-family: monospace" | 0x00100000
| SYS_PACCT
| Allow configuration of process accounting
|-
| style="text-align: right" | 21
| style="font-family: monospace" | 0x00200000
| SYS_ADMIN
|
* Allow configuration of the secure attention key
* Allow administration of the random device
* Allow examination and configuration of disk quotas
* Allow configuring the kernel's syslog (printk behaviour)
* Allow setting the domainname
* Allow setting the hostname
* Allow calling bdflush()
* Allow mount() and umount(), setting up new smb connection
* Allow some autofs root ioctls
* Allow nfsservctl
* Allow VM86_REQUEST_IRQ
* Allow to read/write pci config on alpha
* Allow irix_prctl on mips (setstacksize)
* Allow flushing all cache on m68k (sys_cacheflush)
* Allow removing semaphores (Used instead of CAP_CHOWN to "chown" IPC message queues, semaphores and shared memory)
* Allow locking/unlocking of shared memory segment
* Allow turning swap on/off
* Allow forged pids on socket credentials passing
* Allow setting readahead and flushing buffers on block devices
* Allow setting geometry in floppy driver
* Allow turning DMA on/off in xd driver
* Allow administration of md devices (mostly the above, but some extra ioctls)
* Allow tuning the ide driver
* Allow access to the nvram device
* Allow administration of apm_bios, serial and bttv (TV) device
* Allow manufacturer commands in isdn CAPI support driver
* Allow reading non-standardized portions of pci configuration space
* Allow DDI debug ioctl on sbpcd driver
* Allow setting up serial ports
* Allow sending raw qic-117 commands
* Allow enabling/disabling tagged queuing on SCSI controllers and sending arbitrary SCSI commands
* Allow setting encryption key on loopback filesystem
* Allow setting zone reclaim policy
|-
| style="text-align: right" | 22
| style="font-family: monospace" | 0x00400000
| SYS_BOOT
| Allow use of reboot()
|-
| style="text-align: right" | 23
| style="font-family: monospace" | 0x00800000
| SYS_NICE
|
* Allow raising priority and setting priority on other (different UID) processes
* Allow use of FIFO and round-robin (realtime) scheduling on own processes and setting the scheduling algorithm used by another process.
* Allow setting cpu affinity on other processes
|-
| style="text-align: right" | 24
| style="font-family: monospace" | 0x01000000
| SYS_RESOURCE
|
* Override resource limits. Set resource limits.
* Override quota limits.
* Override reserved space on ext2 filesystem
* Modify data journaling mode on ext3 filesystem (uses journaling resources)
* ''NOTE:'' ext2 honors fsuid when checking for resource overrides, so you can override using fsuid too
* Override size restrictions on IPC message queues
* Allow more than 64hz interrupts from the real-time clock
* Override max number of consoles on console allocation
* Override max number of keymaps
|-
| style="text-align: right" | 25
| style="font-family: monospace" | 0x02000000
| SYS_TIME
|
* Allow manipulation of system clock
* Allow irix_stime on mips
* Allow setting the real-time clock
|-
| style="text-align: right" | 26
| style="font-family: monospace" | 0x04000000
| SYS_TTY_CONFIG
|
* Allow configuration of tty devices
* Allow vhangup() of tty
|-
| style="text-align: right" | 27
| style="font-family: monospace" | 0x08000000
| MKNOD
| Allow the privileged aspects of mknod()
|-
| style="text-align: right" | 28
| style="font-family: monospace" | 0x10000000
| LEASE
| Allow taking of leases on files
|-
| style="text-align: right" | 29
| style="font-family: monospace" | 0x20000000
| AUDIT_WRITE
| ??
|-
| style="text-align: right" | 30
| style="font-family: monospace" | 0x40000000
| AUDIT_CONTROL
| ??
|}

== Setting flags and capabilities ==
To see how to set the flags and capabilities, see [[util-vserver:Capabilities and Flags]] if you're using util-vserver.

If you would like to edit those flags without restarting the vservers, you can use vattribute and nattribute. See [[util-vserver:Cheatsheet]]

Frequently Asked Questions

2010-02-15T10:55:54Z

Bobnormal: add new MAC per vServer info from _are_ on IRC

<div style="margin: 2em auto 2em auto; padding: 10px; background-color: #F9ECCD; border: 1px solid #004433; text-align: center;">
[[Image:Icon-Caution.png|left]]
We currently migrate to MediaWiki from our old installation, but not all content has been migrated yet. Take a look at the [[Wiki Team]] page for instructions how to help or look at the [http://oldwiki.linux-vserver.org old wiki] to find the information not migrated yet.

'''To ease migration we created a [[List of old Documentation pages]].'''
</div>

CURRENTLY THE CONTENT OF THE OLD WIKI FAQ (AND MORE) IS BEING MIGRATED TO THIS PAGE (TASK: DERJOHN)

__TOC__

== General ==

{{Question
|Question=What is a 'Guest'?
||Details=To talk about stuff, we need some naming. The physical machine is called 'Host' and the 'main' context running the Host Distro is called 'Host Context'. The virtual machine/distro is called 'Guest' and basically is a Distribution (Userspace) running inside a 'Guest Context'.
|Signature=derjohn}}

{{Question
|Question=What kind of Operating System (OS) can I run as guest?
||Details= With VServer you can only run Linux guests. The trick is that a guest does not run a kernel on its own (as XEN and UML do), it merely uses a virtualized host kernel-interface. VServer offers so called security contexts which make it possible to separate one guest from each other, i.e. they cannot get data from each other. Imagine it as a chroot environment with much more security and features.
|Signature=derjohn}}

{{Question
|Question=Is this a new project? When was it started?
||Details=The first public occurrence of Linux-VServer was Oct 2001. The initial mail can be found here: http://www.cs.helsinki.fi/linux/linux-kernel/2001-40/1065.html
So you can expect a mature software product which does its magic quite well (And hey, we have a version > 2.0!)
|Signature=derjohn}}

{{Question
|Question=Which distributions did you test?
||Details=Some. Check out the wiki for ready-made guest images. But you can easily build own guest images, e.g. with Debian's debootstrap. Checkout [[Building Guest Systems]] how to do that.
|Signature=derjohn}}

{{Question
|Question=Is VServer comparable to XEN/UML/QEMU?
||Details=Nope. XEN/UML/QEMU and VServer are just good friends. Because you ask, you probably know what XEN/UML/QEMU are. VServer in contrary to XEN/UML/QEMU does not "emulate" any hardware you run a kernel on. The purpose of Linux VServer is to isolate (groups of) applications. The isolation is done by the kernel (see [[Overview]] for a more detailed comparison). You can run a VServer kernel in a XEN/UML/QEMU guest. This is confirmed to work at least with Linux 2.6/vs2.0.
|Signature=derjohn}}

{{Question
|Question=With which version should I begin?
||Details=If you are new to VServer I recommend to try the latest stable kernel patch, and the latest util-vserver "alpha" release.
|Signature=derjohn}}

{{Question
|Question=Is VServer secure?
||Details=We hope so. It should be as least as secure as Linux is. We consider it much much more secure though.
|Signature=derjohn}}

{{Question
|Question=Performance?
||Details=For a single guest, we basically have native performance. Some tests showed insignificant overhead (about 1-2%) others ran faster than on an unpatched kernel. This is IMVHO significantly less than other solutions waste, especially if you have more than a single guest (because of the resource sharing).
|Signature=derjohn}}

{{Question
|Question=What is the "great flower page"?
||Details=Well, [http://www.nongnu.org/util-vserver/doc/conf/configuration.html this page] contains all configuration options for util-vserver. The name of the page is derived from the stylesheet(s) it contains.
|Signature=derjohn}}

== Resources usage ==

{{Question
|Question=Resource sharing?
||Details=Yes ....
* memory: Dynamically.
* CPU usage: Dynamically (token bucket)
|Signature=derjohn}}

{{Question
|Question=Resource limiting?
||Details=You can put limits per guest on different subsystems.
* using ulimits and rlimits (rlimit is a new feature of kernel 2.6/vs2.0.) per guest, to limit the memory consumption, the number of processes or file-handles, ... : see [[Resource Limits]]
* CPU usage : see [[CPU Scheduler]]
* disk space usage : see [[Disk Limits and Quota]]
Note that you can only offer guaranteed resource availability with some ticks at the time.
|Signature=derjohn&xm}}

{{Question
|Question=How do I limit a guests RAM? I want to prevent OOM situations on the host!
||Details=First you can read [http://linux-vserver.org/Memory+Allocation] and [[Memory Limits]].
If you want a recipe, do this:
# Check the size of memory pages. On x86 and x86_64 is usually 4 KB per page.
# Create /etc/vserver/<guest>/rlimits/
# Check your physical memory size on the host, e.g. with "free -m". maxram = kilobytes/pagesize.
# Limit the guests physical RAM to value smaller then maxram:<pre>echo %%insertYourPagesHereSmallerThanMaxram%% > /etc/vserver/<guest>/rlimits/rss </pre>
# Check your swapspace, e.g. with 'swapon -s'. maxswap = swapkilobytes/pagesize.
# Limit the guest's maximum number of as pages to a value smaller than (maxram+maxswap): <pre> echo %%desiredvalue%% > /etc/vserver/<guest>/rlimits/as </pre>
# Correctly display the memory information inside the guest:<pre>echo "VIRT_MEM" >> /etc/vservers/<guest>/flags</pre>
It should be clear this can still lead to OOM situations. Example: You have two guests and your as limit per guest is greater than 50% of (maxram+maxswap). If both guests request their maximum at the same point in time, there will be not enough mem .....
|Signature=derjohn}}

{{Question
|Question=Disk I/O limiting? Is that possible?
||Details=Well, since vs2.1.1 Linux-VServer supports a mechanism called 'I/O scheduling', which appeared in the 2.6 mainline some time ago. The mainline kernel offers several I/O schedulers:
<pre>
# cat /sys/block/hdc/queue/scheduler
noop [anticipatory] deadline cfq
</pre>

The default is anticipatory a.k.a. "AS". When running several guests on a host you probably want the I/O performance shared in a fair way among the different guests. The kernel comes with a "completely fair queueing" scheduler, CFQ, which can do that. (More on schedulers can be found at http://lwn.net/Articles/114770/)
This is how to set the scheduler to "cfq" manually:
<pre>
root# echo "cfq" > /sys/block/hdc/queue/scheduler
root# cat /sys/block/hdc/queue/scheduler
noop anticipatory deadline [cfq]
</pre>
Keep in mind that you have to do it on all physical discs. So if you run an md-softraid, do it to all physical /dev/hdXYZ discs!
If you run Debian there is a predefined way to set the /sys values at boot-time:
<pre>
# apt-get install sysfsutils
[...]

# grep cfq /etc/sysfs.conf
block/sda/queue/scheduler = cfq
block/sdc/queue/scheduler = cfq

# /etc/init.d/sysfsutils restart
</pre>

For non-vserver processes and CFQ you can set by which key the kernel decides about the fairness:
<pre>
cat /sys/block/hdc/queue/iosched/key_type
pgid [tgid] uid gid
</pre>
Hint: The 'key_type'-feature has been removed in the mainline kernel recently. Don't look for it any longer :(

The default is tgid, which means to share fairly among process groups. Think every guest is treated like a own process group. It's not possible to set a scheduler strategy within a guest. All processes belonging to the same guest are treated like "noop" within the guest. So: If you run apache and some ftp-server within the _same_ guest, there is no fair scheduling between them, but there is fair scheduling between the whole guest and all other guests.

And: It's possible to tune the scheduler parameters in several ways. Have a look at /sys/block/hdc/queue/....
|Signature=derjohn}}

{{Question
|Question=Nice disk I/O scheduling, is that possible?
||Details=Well, since linux 2.6.13 processess have another priority next to the cpu nice scheduling hint, it's called io nice.
It's split into three groups, called real-time, best effort and idle. The default is best-effort, but within best-effort, you can have a niceness from 0 to and including 7.
You can set this niceness by the tool ionice, which for debian is either in the package util-linux or schedutils.
To change the io-niceness you need the <tt>CAP_SYS_NICE</tt>, '''and''' need to have the same uid as the processe you want to ionice.

:'''Note:''' If you want to use any schedulung other than best-effort you will also need the <tt>CAP_SYS_ADMIN</tt>-flag. Be warned that this gives quite some capabilities to the vserver, not just for I/O scheduling!

If you want to increase the niceness of an I/O hogging process within a vserver you need to do:
<PRE>
chcontext --xid sponlp1 sudo -u '#2089' ionice -c2 -n5 -p24409
</PRE>
with sudo and ionice installed on the root server to increase the *nice*ness of pid 24409, with uid 2089
|Signature=Groteblup}}

== Unification ==

{{Question
|Question=What is unification (vunify)?
||Details=Unification is Hard Links on Steroids. Guests can 'share' common files (usually binaries and libraries) in a secure way, by creating hard links with special properties (immutable but unlinkable (removable)). The tool to identify common files and to unify them is called vunify.
|Signature=derjohn}}

{{Question
|Question=What is vhashify?
||Details=The successor of vunify, a tool which does unification based on hash values (which allows to find common files in arbitrary paths.)
It creates hardlinks to files named after a hash of the content of the file. If you have a recent version of the vserver patch (2.2+), with CONFIG_VSERVER_COWBL enabled, you can even modify the hardlinked files inside the vservers and the links will be broken automatically.
There seems to be a catch when a hashified file has multiple hardlinks inside a guest, or when another internal hardlink is added after hashification. Link breaking will remove all the internal hardlinks too, so the guest will end up with different copies of the original file. The correct solution would be to not hashify files that have multiple links prior to hashification, and to break the link to the hashified version when a new internal hardlink is created. Apparently, this is not implemented yet (?).
|Signature=Guy-}}

{{Question
|Question=How do I manage a multi-guest setup with vhashify?
||Details=For 'vhashify', just do these once:
<pre>
mkdir /etc/vservers/.defaults/apps/vunify/hash /vservers/.hash
ln -s /vservers/.hash /etc/vservers/.defaults/apps/vunify/hash/root
</pre>
Then, do this one line per vserver:
<pre>
mkdir /etc/vservers/<vservername>/apps/vunify # vhashify reuses vunify configuration
</pre>
To hashify a running vserver, do (possibly from a cronjob):
<pre>
vserver name-of-guest hashify
</pre>

The guest needs to be running because vhashify tries to figure out what files not to hashify by calling the package manager of the guest via <tt>vserver enter</tt>.

In order for the OS cache to benefit from the hardlinking, you'll have to restart the vservers.

To clean up hashified files that are no longer referenced by any vserver, do (possibly from a cronjob):
<pre>
find /vservers/.hash -type f -links 1 -print0 | xargs -0 rm
</pre>

Until you do this, the files still take up place even though no vservers need them.
|Signature=Guy-}}

== Filesystem usage ==

{{Question
|Question=Is there a way to implement "user/group quota" per VServer?
||Details=Yes, but not on a shared partition for now. You need to put the guest on a separate partition, setup a vroot device (to make the quota access secure), copy that into the guest, and adjust the mtab line inside the guest. If 8 vroot device is not enough for you, you can add more with the kernel parameter max_vroot (exemple for built in kernel vroot /vmlinuz-2.6.31.6-vs2.3.0.36.26aq root=/dev/md1 ro max_vroot=20 ). If vroot is a module you'd actually want to put for exemple "options vroot max_vroot=20" in /etc/modprobe.conf and then just do modprobe vroot
|Signature=derjohn,gadnet}}

{{Question
|Question=What about "Quota" for a context? Howto limit disk usage?
||Details=Context quotas are now called Disk Limits (so that we can tell them apart from the user/group quotas :). They are supported out of the box (with vs2.0+) for all major filesystems (ext2/3, ReiserFS, JFS). You need to tag the FS with XID (see below). Please read [[Disk Limits and Quota]] for more information.
|Signature=derjohn}}

{{Question
|Question=How do I tag a guest's directory with xid?
||Details=Tagging the guest's files gives you several advantages, e.g. the accounting will work properly.
Filesystem XID tagging only works on supported filesystem. Those are currently: ext2/3, reiserfs/reiser3, xfs and jfs.
To activate the XID tagging you have to mount the filesystem with "-o tag" (former tagxid is outdated since VS2.2). Attention: It's _not_ possible to "-o remount,tag", you have to mount it freshly. The guests will tag their files automatiaclly. If you copy files in from the host, you have to tag them manually like this:
<pre>
chxid -c xid -R /var/lib/vservers/<guest>
</pre>
Note: Context 0 and 1 will see all files, guests will only be able to access untagged files and their own XID. They can see other XID files but no information about the file, e.g. no owner, no group, no permissions.
Note: It is not advised to tag the root filesystem, as [http://www.paul.sladen.org/vserver/archives/200602/0020.html explained by Herbert] : trying to do so will expose you to some troubles !
|Signature=derjohn_and_gonzo_and_are}}

{{Question
|Question=How can I copy anything from host to guest partition, normally unvisible on host?
||Details=You should just change namespace, e.g.:
<pre>
vnamespace --enter <xid> -- /bin/bash
</pre>
and then use standard cp or rsync programs.
|Signature=SergiuszPawlowicz}}

== Network ==

{{Question
|Question=Does it support IPv6?
||Details=Currently it requires an additional patch, but the functionality should be available in 2.3+ soon. [[IPv6]] has more information.
|Signature=derjohn}}

{{Question
|Question=I can't do all I want with the network interfaces inside the guest?
||Details=For now the networking is 'Host Business' -- the host is a router, and each guest is a server. You can set the capability ICMP_RAW in the context of the guest, or even the capability CAP_NET_RAW (which would even allow to sniff interfaces of other guests!). Likely to change with ngnet.
|Signature=derjohn}}

{{Question
|Question=How do I add several IPs to a vserver?
||Details=First of all a single guest vserver only supports up to 16 IPs (There is a 64-IP patch available, which is in "derjohn's kernel").
Here is a little helper-script that adds a list of IPs defined in a text file, one per line.
<pre>
#!/bin/bash
j=1
for i in `cat myiplist`; do
j=$(($j+1))
mkdir $j
echo $i > $j/ip
echo "24" > $j/prefix
done
</pre>
|Signature=derjohn}}

{{Question
|Question=How do I assign a new IP address to a running guest?
||Details=This is done from the host server:
* add the ip on the host, for example
<pre>
ip addr add 194.169.123.23/24 dev eth0
</pre>
* add the ip to the guest's network context (a guests NID is the same as the XID {context ID})
<pre>
naddress --add --nid <nid> --ip 194.169.123.23/24
</pre>
* enter the guest (best via ssh)
* restart the services that need to make use of the new address if required
* update the config in ''/etc/vserver/<servername>/interfaces'' to reflect the changes for the next guest restart (if desired)
|Signature=BenjaminGreen}}

{{Question
|Question=If my host has only one a single public IP, can I use RFC1918 IP (e.g. 192.168.foo.bar) for the guest vservers?
||Details=Yes, use iptables with SNAT to masquerade it.
<pre>
iptables -t nat -I POSTROUTING -s $VSERVER_NETZ ! -d $VSERVER_NETZ -j SNAT --to $EXT_IP
</pre>
See: [[HowtoPrivateNetworking]] and
http://www.tgunkel.de/it/software/doc/linux_server.en#h3-VServer_Masquerading_SNAT (THX, [MUPPETS]Gonzo)
|Signature=derjohn}}

{{Question
|Question=If I shut down my vserver guest, the whole Internet interface ethX on the host is shut down. What happened?
||Details=When you shut down a guest (''i.e. vserver foo stop''), the IP is brought down on the host also. If this IP happens to be the primary IP of the host, the kernel will not only bring down the primary IP, but also all secondary IP addresses. But in very recent kernels, there is an option ''settable'' which prevents that nasty feature. It's called "alias promotion". You may set it via sysctl by adding ''net.ipv4.conf.all.promote_secondaries=1'' in /etc/sysctl.conf or via sysctl command line.
|Signature=derjohn}}

{{Question
|Question=Can I run an OpenVPN Server in a guest?
||Details=
Yes. To get a OpenVPN Server running in a guest, all networking setup has to be done on the host. This answer describes the common case and shows some pitfalls, for detailled information about OpenVPN, please consult the appropriate documentation on the OpenVPN homepage.
This is the minimal OpenVPN configuration for the Server which will be used to demonstrate how to get it running in a client:
<pre>
# Networking setup
server 192.168.16.0 255.255.255.0
dev tun16
ifconfig-noexec
comp-lzo
# Certificates
dh ...
ca ...
cert ...
key ...
# Management
persist-key
keepalive 10 60
verb 4
</pre>
First of all you have to prepare the host with a persistent interface in the right mode and with the right settings. This is easily done by using openvpn and the ip and route tools.
<pre>
# openvpn --mktun --dev tun16
# ip link set dev tun16 txqueuelen 100
# ifconfig tun16 192.168.16.1 pointopoint 192.168.16.2 mtu 1500
# route add -net 192.168.16.0 netmask 255.255.255.0 gw 192.168.16.2
</pre>
If you need different settings, openvpn will tell you the ifconfig and route commands it uses to configure the interface when being started on the host with the original config file, but without ifconfig-noexec.
Additionally, the guest needs /dev/net/tun to make OpenVPN happy. This can be created with MAKEDEV:
<pre>
# cd /var/lib/vserver/<myopenvpnserver>/dev/
# ./MAKEDEV tun
(creates the dev/net/tun device accessible by the guest - even a tap interface needs /dev/net/tun !)
</pre>
Finally, the guest needs to have the tun device assigned:
<pre>
# head /etc/vservers/<myopenvpnserver>/interfaces/1/*
==> /etc/vservers/<myopenvpnserver>/interfaces/1/ip <==
192.168.16.1

==> /etc/vservers/<myopenvpnserver>/interfaces/1/nodev <==
tun16

==> /etc/vservers/<myopenvpnserver>/interfaces/1/prefix <==
24
#
</pre>
The client's conf may look like that:
<pre>
# Basic setup
client
proto tcp-client
dev tun
remote <ipaddress>
comp-lzo
verb 4

# Certificate
ca ...
</pre>

[ Based on derJohn's original answer, all errors mine ]
|Signature=DavidS}}

{{Question
|Question=Trying to connect to a vserver from the host or another vserver on the same host fails
||Details=strace shows
<pre>
sin_addr=inet_addr("xx.xx.xx.xx")}, yy) = -1 EINVAL (Invalid argument)
</pre>
A: The host/guest cannot communicate with another guest on same host.
* check all netmasks on all interfaces (do they overlap) ?
* check policy routing (disable it temporary) ?
* check that lo is up (Networking within a host/guest always uses lo interface)
|Signature=CommonProblems}}

{{Question
|Question=Can I use iptables ?
||Details=Yes but right now only on the host (rootserver). Please realize that all traffic is local and will not touch the forward chain.
|Signature=BeginnerFAQ}}

{{Question
|Question=Is it possible to prevent guest from bringing down primary ip?
||Details=Yes. Remove /etc/vservers/<guest>/interfaces/X/dev, and touch /etc/vservers/<guest>/interfaces/X/nodev
|Signature=Daniel&Serge}}

{{Question
|Question=Is it possible to provide a different MAC address per vServer?
||Details=Short answer - yes but it's a hassle.

Real answer from '''_are_''':
<pre>
When I once needed 'real' seperate MAC-addresses I used TAP-devices and VDE2 ([http://vde.sourceforge.net/ Virtual Distributed Ethernet]).
Basically vServer is an isolation of existing resources, not a virtualization of 'new' devices.
Without extra fuss you can't add a 'new' network interface to a vServer, no matter if it is eth* or tap*, you always add it to the host and give the vServer access to it.
I got the TAP+VDE2 up and running, but I think it is too much trouble for basically the simple adding of IPs to a vServer unless you really need the MAC address separate.
</pre>
|Signature=bobnormal}}

== Administration tools ==

{{Question
|Question=Which guest vservers are running?
||Details=Use vserver-stat to find out. Example output:
<pre>
CTX PROC VSZ RSS userTIME sysTIME UPTIME NAME
0 77 965.1M 334.6M 14m14s18 2m28s69 1h33m46 root server
49152 7 14M 5.2M 0m00s40 0m00s30 1h30m15 chiffon
</pre>
|Signature=derjohn}}

{{Question
|Question=Is there a web-based interface for vserver that will allow creation/deletion/configuration etc. of vserver guests?
||Details=
* http://OpenVPS.org which is a set of scripts with a web-interface for webhosters/ISPs
* http://Openvcp.org which is a distributed system (agent!) with a web-interface, with which you can build/remove guests
* http://vsmon.revolutionlinux.com/ is a distributed monitoring-only solution that allows you to search for a particular vserver in your park.
|Signature=derjohn}}

== Hosting foreign distributions ==

{{Question
|Question=I run a Debian host and want to build an Ubuntu guest. Howto?
||Details=Simple ;) Assume you want to build a breezy guest on a sid host with IP 192.168.0.2 and hostname vubuntu, then do:
<pre>
vserver vubuntu build --force -m debootstrap --hostname vubuntu.myvservers.net --netdev eth0 --interface 192.168.0.2/24 \
--context 42 -- -d breezy -m http://de.archive.ubuntu.com/ubuntu
</pre>

[UPDATE] Currently there are problems in building breezy under unclear circumstances, which seems to have to do with udev. If the above didnt work, try:
<pre>
vserver vubuntu build --force -m debootstrap --hostname vubuntu.myvservers.net --netdev eth0 --interface 192.168.0.2/24 \
--context 42 -- -d breezy -m http://de.archive.ubuntu.com/ubuntu -- --exclude=udev
</pre>
In very recent versions of the utils, the problem should not occur anymore (it has to do with the 'secure-mount' if you look in the MLs)

Well, sid's debootstrap knows how to bootstrap Ubuntu linux. Make sure to have a current debootstrap package:
<pre>
apt-get update
apt-get install debootstrap
</pre>
The knowledge how to build ubuntu 'breezy badger' (which you probably want to be your guest at the time of writing) has been added recently.
|Signature=derjohn}}

{{Question
|Question=I want to build a Gentoo guest. Howto?
||Details=Even simpler ;) See http://www.gentoo.org/proj/en/vps/vserver-howto.xml#doc_chap3 .
|Signature=gcc}}

== Application level problems ==

{{Question
|Question=I did everything right, but the application foo does not start. What's up there?
||Details=Before asking on the IRC channel, please check out the 'problematic programs' page:
[[Problematic Programs]]
|Signature=derjohn}}

{{Question
|Question=When I try to ssh to the guest, I log into the host, even if I installed sshd on the guest. What's wrong here?
||Details=Look at /etc/ssh/sshd_config of the host:
<pre>
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
</pre>
And now change the setting to
<pre>
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
ListenAddress your.hosts.ip.here # not the guests IP!
</pre>
Then '/etc/init.d/ssh restart' on the host, after that on the guest (if you did apt-get install ssh on the guest already.)
Do I have to explain more? If the hosts sshd binds all available IP addresses on port 22 (The hosts 'sees' even all addresses of the guests!). So if the guest starts its sshd, it can't bind to port 22 any more. You need to change that setting only on the host.
(BTW: A similar approach has to be done for a lot of daemons, e.g. Apache. If the daemon does not support an explicit bind, you may use the chbind command to 'hide' IP addresses from the daemon before starting.)|Signature=derjohn}}

{{Question
|Question=Bind9 does not like to start in my guest.
||Details=Check out the [[Problematic Programs]] page and/or get my [http://linux-vserver.derjohn.de/bind9-packages/bind9-capacheck_9.3.2-2_i386.deb vserver-guest-ready Debian package] for Debian Sid guests and check out the [http://linux-vserver.derjohn.de/bind9-packages/README.txt readme]. (Hint: This is fresh stuff. Please give me feedback)

[UPDATE] Since VServer Devel 2.1.1-rc18 you do not need to patch the userland tools anymore. The capabilities are masked.
|Signature=derjohn}}

{{Question
|Question=My mysqld running in a guest behaves strangely and is awfully slow/locks up
||Details=This can be related to /tmp being too small. mysqld stores temporary tables in /tmp and as such, if a lot of queries happen and /tmp runs full this can cause one query to lock up whilst creating the tmp table and all other queries waiting to acquire the lock. There are two possible solutions to that problem: a.) Modify /etc/vservers/vserver-name/fstab and assign more memory to the tmpfs of /tmp and b.) remove the /tmp entry from /etc/vservers/vserver-name/fstab completly. Especially on database servers with a rather high load the second one might be the preferred method.|Signature=sp}}

{{Question
|Question=Pure-FTP does not run inside a VServer?
||Details=That's because it has capabilities enabled, make sure you rebuild your distro's package passing also the `--without-capabilities` flag to configure.
|Signature=Pedro Algarvio, aka, s0undt3ch}}

{{Question
|Question=Why do neither sshd nor crond (vixie-cron) work correctly in my CentOS / Fedora guest? I get 'pam_loginuid(crond:session): set_loginuid failed opening loginuid' and similar lines in my logs.
||Details=Took me a while to figure this out, and it turned out to be mentioned in the old wiki. Here is the solution on how to solve a common problem with sshd / crond, somehow related to selinux and auditing:

pam authentication (also used with openssh) enables "pam_loginuid.so" in the /etc/pam.d/* files. Comment those out as they are not necessary and will not load within a guest anyway. This probably is also necessary on updates later on, if the configs get changed. You therefore may add the following command line to a cronjob file or your software update script:
<pre>
/bin/sed --in-place -e "s/^session.*required.*pam_loginuid.so/# session\trequired\tpam_loginuid.so/g" /etc/pam.d/*
</pre>
|Signature=patrick}}

{{Question
|Question=How do i install nagios-plugins on a Gentoo guest?
||Details=Unfortunately, the nagios-plugins ./configure scripts wants to ping 127.0.0.1 which is not available inside a guest. Therefore you have to build nagios-plugins outside the guest.
The easiest way to do this from the host (assuming the guest is running) is:
<pre>
vnamespace -e <xid> -- chroot /vservers/<name> emerge nagios-plugins -va
</pre>
|Signature=Hollow}}

{{Question
|Question=Somebody runs ntpd in guest and you can't use ntpdate in host?
||Details=Try to run ntpdate with options -u :
ntpdate -u ntp.domain.xy
or you can use command:
chbind --nid 42 --ip 1.2.3.4 -- ntpdate ntp.domain.xy
where IP will be the IP of host.
|Signature=Punkie/Bertl}}

== Start / Stop a VServer ==

{{Question
|Question=How do I make a vserver guest start by default?
||Details=At least on Debian, I can tell you how to do it with the new-style config. If your guest is called "derjohn" and you want it to be started somewhere at the of your bootstrap process, then do:
<pre>
echo "default" > /etc/vservers/derjohn/apps/init/mark
</pre>
If you want to start it earlier, please read the init script "/etc/init.d/util-vserver" to find out how to do it. In most cases you don't need to change this. On Debian the vservers are started at "20", so after most other stuff is up (networking etc.).

Besides that I created a small helper script for managing the autostart foo: ((vserver-autostart))|Signature=derjohn}}

{{Question
|Question=My host works, but when I start a guest it says that it has a problem with chbind.
||Details=You are probably using util-vserver <= 0.30.209, which does use dynamic network contexts internally (With 0.30.210 this fact changed). So if you compiled your kernel without dynamic contexts, you may start guests, but you can't use the network context.The solution is either to switch to .210 util (or Hollow's toolset) or compile the kernel with dynamic network contexts.
SE Keyword: invalid option `nid' testme.sh
|Signature=derjohn}}

{{Question
|Question=What is old-style and new-style config?
||Details=Old-style config refers to a single text-file that contains all the configuration settings. With new-style config the configuration is split into several directories and files. You should probably go for new-style config if you are asking.
|Signature=derjohn}}

{{Question
|Question=How can I reboot/halt guests?
||Details=It depends.
For legacy Linux-VServer (i.e. 1.2.x), you have to replace /sbin/halt in the guests with vreboot and start rebootmgr in the host. You also need to have a <guest>.conf file in /etc/vservers for each guest. Please have a look at /etc/init.d/rebootmgr.
For Linux-VServer 2.0+, sys_reboot has been virtualized to do the right thing. No changes are needed in guests. Please note that some things depend on the init style used by the guest : read [[util-vserver:InitStyles]]
|Signature=derjohn}}

{{Question
|Question=What is the initial PATH?
||Details=By default, vserver uses the 'sysv' startup style, which mimics the init process by running the 3rd runlevel through '/etc/init.d/rc 3' (or '/etc/rc.d/rc 3'). Usually this 'rc' script uses a hard-coded PATH. In the case it doesn't, util-vserver also mimics init's default PATH through /etc/vservers/.defaults/apps/init/environment, or if not present /usr/local/lib/util-vserver/defaults/environment. Beware that all those default PATH usually do not include /usr/local.
|Signature=daniel_hozac&Beuc}}

{{Question
|Question=When I try to start a guest i get this message "/proc/uptime can not be accessed. Usually, this is caused by procfs-security. Please read the FAQ for more details"?
||Details=After a reboot you need to run the vprocunhide script. If running this script causes many errors to print on the screen, try checking the kernel you have booted with (perhaps it does not have the linux-vserver extensions enabled).
|Signature=mattzerah}}

== Kernel ==

{{Question
|Question=Is SMP Supported?
||Details=Yes, on all SMP capable kernel architectures.
|Signature=derjohn}}

{{Question
|Question=Do I really need the legacy-interfaces? What are these legacy-interfaces?
||Details=Since Linux-VServer is an ongoing project, new features might replace old ones, some might require a development version. Legacy-interfaces are available for backward compability (which might be removed someday) with Linux-VServer 1.2.x.
|Signature=derjohn}}

{{Question
|Question=I have a vserver running on a Linux kernel with preemption. Is VServer "preempt" safe?
||Details=There are no known issues about running vserver on a preemption enabled kernel. I would like to add, that the vserver kernelhackers would probably exclude that option in 'make menuconfig' if there would be an incompatibility. Just my $.02 :)
|Signature=derjohn}}

{{Question
|Question=32 vs 64 Bit? What should I take?
||Details=If you have the choice make the host a 64 bit one. You can run a guest as 32 bit or as 64 bit on a 64 bit host. To run it as 32 bit, you need to compile the x86_64 (a.k.a. AMD64) with the following options:
<pre>
[*] Kernel support for ELF binaries
<M> Kernel support for MISC binaries
[*] IA32 Emulation <---- without that, the entire 32bit API is not present
<M> IA32 a.out support
</pre>
You can force the guest to behave like a 32 environment like this:
<pre>
echo linux_32bit > /etc/vservers/$NAME/personality
echo i686 > /etc/vservers/$NAME/uts/machine
</pre>
(thanks cehteh for the hint!)

But you can force debootstrap to put 32 bit binaries into the guest by 'export ARCH=i386';
<pre>
export ARCH=i386 ; vserver build ....
</pre>

On debian when using the newvserver script "export ARCH=i386" has no effect, just use:
<pre>
newvserver --arch i386 ...
</pre>
|Signature=derjohn}}

== Distribution specific questions ==

{{Question
|Question=VServer is included in the stable Debian GNU/Linux for years now. What VS version did they include?
||Details=At the time of writing, Debian Lenny is the stable release of Debian and includes a 2.6.26 based kernel-package called 2.6.26-2-vserver-ARCH. This currently contains VServer 2.3.0.35 (according to changelog.Debian.gz in the Debian package).
|Signature=scientes}}

{{Question
|Question=Were can I get newer versions of VServer as ready made packages for Debian?
||Details=Here you go: http://linux-vserver.derjohn.de/ . There is also some stuff on backports.org, but my kernels are always 'devel' branch.
|Signature=derjohn}}

== Misc ==

{{Question
|Question=Why isn't there a device /dev/xyz within a guest?
||Details=Device nodes allow userspace to access hardware (or virtual resources). Creating a device node inside the guest's namespace will give access to that device, so for security reasons, the number of 'given' devices is small.
|Signature=derjohn}}

{{Question
|Question=I want to (re)mount a partition in a running guest ... but the guest has no rights (capability) to (re)mount?
||Details=I'll explain. I take as example your /tmp partition within the guest is too small, what will be likely the case if you stay with the 16MB default (vserver build mounts /tmp as 16 MB tmpfs!).
<pre>
# vnamespace -e XID mount -t tmpfs -o remount,size=256m,mode=1777 none /var/lib/vservers/<guest>/tmp/
</pre>
(if there's a problem, try expanding the symlinks in the mount path)
Be warned that the guest will not recognize the change, as the /etc/mtab file is not updated when you mount like this. To permanently change the mount, edit /etc/vserver/<guest>/fstab on the host.

If you get:
<pre>
mount: can't find /var/lib/vservers/<guest>/tmp in /etc/fstab or /etc/mtab
</pre>
then try instead:
<pre>
vnamespace -e builder chroot /var/lib/vservers/<guest>/ mount -o remount,size=64m,mode=1777 /tmp
</pre>

Note that this not work for adding a bindmount (<tt>-o bind</tt>) of a directory outside of a vserver into the vserver. For this, there is no alternative but restarting the vserver.
|Signature=derjohn}}

{{Question
|Question=Does anyone know how to increase the size of /tmp within a vserver w/o restarting?
||Details=Use the remount option for mount.
# vnamespace -e XID mount -n -t tmpfs -o remount,size=32m tmpfs /<vdir>/<guest>/tmp
or something like that. The arguments are needed since mount is not going to be using /etc/fstab for the information and the version of /proc/mounts is best understood by
# vnamespace -e XID cat /proc/mounts.
See [[Frequently_Asked_Questions#I want to (re)mount a partition in a running guest ... but the guest has no rights (capability) to (re)mount?]]
|Signature=derjohn/dhozac}}

{{Question
|Question=#1 ERROR: capset(): Operation not permitted
||Details=capabilities are not enabled in kernel-setup
please check that CONFIG_SECURITY_CAPABILITIES is loaded or included in the kernel. ( check with "cat /path_to_kernel/.config | grep -i cap ")
(2.6.11.5-vs-1.9.5 + 0.30-205)
|Signature=IrcQuestions}}

{{Question
|Question=How can I make 'vserver start' mount the root filesystem?
||Details=Mount it via /etc/vservers/vserver-name/fstab, make sure to set the option 'dev' e.g.:
<pre>/dev/drbd0 / xfs rw,dev 0 0</pre>
|Signature=AdrianReyer}}

{{Question
|Question=I deleted a guest's directory without shutting it down. Now I have a "ghost" running. Is there any possibility to get it out of proc without rebooting?
||Details=
vkill --xid <xid> -s 15; sleep 2; vkill --xid <xid> -s 9

You will also need to remove guest's ip, for example with:
ip addr del <ip> dev eth0
|Signature=daniel_hozac & gebura }}

{{Question
|Question=When using nice and su (for example, in the updatedb cron job), I get: su: Permission denied. What does it mean?
||Details=A guest cannot lower its nice value - and that's what 'su' does through pam_limits which sets a nice value of 0. You can see it through strace:
$ strace nice su nobody
[...]
setpriority(PRIO_PROCESS, 0, 0) = -1 EACCES (Permission denied)
You can use 'su nobody -c nice some_cmd' instead.
(Now there's the question of why a guest process cannot lower its nice value.)
|Signature=daniel_hozac&Beuc}}

{{Question
|Question=How do I handle NFS mounts within in a guest?
||Details=There are three ways.

'''1)''' Mount the NFS share from the host OS and let vserver guest access it as part of it's file system.

''mount --bind'' may also be beneficial in this scenario.

'''2)''' Use util-vserver and create a ''fstab.remote'' file in the /etc/vserver/<vserver_name> directory. Populate this with the NFS shares and they will be mounted in the context of the vserver guest.

See http://www.nongnu.org/util-vserver/doc/conf/configuration.html

'''3)''' Add capabilities to the vserver guest instance to grant sufficient rights to allow NFS mounts.

Add the following to /etc/vserver/<vserver_name>/bcapabilities
SYS_ADMIN
Add the following to /etc/vserver/<vserver_name>/ccapabilities
SECURE_MOUNT
BINARY_MOUNT

See [[Capabilities_and_Flags]] for more information about vserver capabilities.

If you want the NFS shares to be mounted when the guest starts, add them to /etc/vserver/<vserver_name>/fstab

||Signature=martindk}}

{{Question
|Question=vserver start/stop/enter fails with something like "vnamespace: execvp("/usr/sbin/vserver"): No such file or directory" ?
||Details=Check whether ''/usr'' is mounted in the namespace you are working with.
<pre>vnamespace -e <guest> cat /proc/mounts</pre>
If there is no ''/usr'', you can fix your problem with simply mounting it using the following command:
<pre>vnamespace -e <guest> mount /dev/<device> /usr</pre>

||Signature=sim0n}}

{{Question
|Question=The command vserver <$server> start gives '/etc/init.d/rc: line 74: /etc/default/rcS: No such file or directory', what do I do?
||Details=The vserver has not been correctly installed, this has several reasons
check your install log and it should tell you something about that your server didn't get installed properly
* use stable distribution of debian as server (debootstrap may be different over the versions)
* deny_mount, deny_caps and deny_pivot should be off if your running grsec.

||Signature=Dude}}

{{Question
|Question=How could I rename a vserver directory?
||Details=Please note : this procedure renames the '''directory''', not the '''hostname''' !
#Stop the vserver in question
#rename the <tt>/vservers/<server name></tt> directory
#rename the <tt>/etc/vservers/<server name></tt> directory
#update link: <tt>/etc/vservers/<server name>/run</tt> → <tt>/var/run/vservers/<server name></tt>
#update link: <tt>/etc/vservers/<server name>/vdir</tt> → <tt>/etc/vservers/.defaults/vdirbase/<server name></tt>
#update link: <tt>/etc/vservers/<server name>/cache</tt> → <tt>/etc/vservers/.defaults/cachebase/<server name></tt>
#update link: <tt>/var/run/vservers.rev/<server XID></tt> → <tt>/etc/vservers/<server name></tt>
#Start the vserver in question. It should start properly.

|Signature=FlorianD (from ''hillct'' page in old wiki)}}

== Upgrade from 2.0 to 2.2 ==

{{Question
|Question=I now get errors like "ncontext: vc_net_create(): Invalid argument; dynamic contexts disabled." on startup. Vservers are not started
||Details=Dynamic context are disabled by default and are deprecated. For example, tagxid and network checks won't be useable with dynamic ids. Now you should manually assign a explicit context to your vservers, like
echo 101 > /etc/vservers/myvserv/context
ADDENDUM: please consider that valid static contexts are between 2 and 49151 ( daniel_hozac on IRC ) otherwise you will end up with unexplainable error "ncontext: vc_net_migrate(): No such process" when trying to start the vserver.

|Signature=daniel_hozac&Beuc}}

{{Question
|Question=How do I assign a static context to an existing vserver?
||Details=Simple ;) See the answer above.
|Signature=gcc}}

{{Question
|Question=Since upgrading to a newer VS version my guest complains about "vsched: non-numeric value specified for '--priority_bias" at start time. What's wrong?
||Details=The scheduler paramters changed.You can use this (ugly) script to convert them or do it by hand:
<pre>
# cat /usr/local/sbin/vserver-convert-schedule-to-scheddir
#/bin/sh
mkdir /etc/vservers/$1/sched
sed -e 1p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/fill-rate
sed -e 2p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/interval
sed -e 3p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens
sed -e 4p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens-min
sed -e 5p -n /etc/vservers/$1/schedule > /etc/vservers/$1/sched/tokens-max

mv /etc/vservers/$1/schedule /etc/vservers/$1/schedule.converted.see.scheddir

# see: http://oldwiki.linux-vserver.org/Scheduler+Parameters
# see: http://www.nongnu.org/util-vserver/doc/conf/configuration.html#sched
</pre>
||Signature=derjohn}}

{{Question
|Question=Since upgrading to a newer VS version my guest doesn't have the amount of shared memory (SHM / SHMMAX / SHMALL ) as it had in the former version. What changed?
||Details=Every VS version that runs on a kernel >= 2.6.19 offers sysctl values per guest. This has to do with the 'ipc namespace' feature that was added to the mainline kernel in version 2.6.19. Linux-VServer uses that feature to give each guest a separate 'ipc namespace' and thus 'own' sysctl values per guest. Because shmmax is such a sysctl value, you have to set it per guest.
Here is an example how to do so:

<pre>
# mkdir /etc/vservers/<vserver>/sysctl/0 -p
# echo kernel.shmall > /etc/vservers/<vserver>/sysctl/0/setting
# echo 134217728 > /etc/vservers/<vserver>/sysctl/0/value
# mkdir /etc/vservers/<vserver>/sysctl/1 -p
# echo kernel.shmmax > /etc/vservers/<vserver>/sysctl/1/setting
# echo 134217728 > /etc/vservers/<vserver>/sysctl/1/value
</pre>
It's also explained on the geat flower page:
# see: http://www.nongnu.org/util-vserver/doc/conf/configuration.html -> Look for "sysctl".

After changing those values, restart your guest, enter it and check if the values are set:
<pre>
# sysctl -a | grep shm
...
kernel.shmall = 134217728
kernel.shmmax = 134217728
</pre>

To change a value for a running guest, on the host use:
<pre>
vspace -e CONTEXTID --ipc sysctl -w kernel.shmall=134217728
vspace -e CONTEXTID --ipc sysctl -w kernel.shmmax=134217728
</pre>

||Signature=derjohn
}}