Running virtual machines or Docker containers on Nodegrid using the main OS disk can be problematic if the virtual machines or containers will be making many writes to the disk. A busy VM or container can significantly degrade the lifespan of the main OS disk. Therefore, I recommend to use a second disk when using VMs or Docker containers. If you have Nodegrid device with internal second disk or if you are using an external USB disk, these can be utilized as storage locations for VMs and Docker Containers.
The leading practices for VM and Docker Storage Location that I recommend are:
- Storage Disk Layout
- Single partition utilizing maximum disk space
- Mount partition in unchanging location: /var/local/mnt
- Filesystem Layout
- Create directories in the new partition root:
- For virtual machines: virtual-machines
- For docker containers: docker-root
- If using for additional purposes, use other directory names
- Make KVM virtual machines use 'virtual-machines' directory
- Make Docker use 'docker-root' directory
Note: This article assumes Linux knowledge on how to manage disk partitions with fdisk and mount them in fstab.
[Coming Soon, a walk-through of this article showing step-by-step beginning to end]
Storage Disk Layout
In this practical, I'm going to recommend a disk layout with a single partition utilizing the entire disk, and formatted as Linux ext4 filesystem type. As root, use the builtin command fdisk to wipe any existing partitions and create a new Linux partition.
Mount Point
Nodegrid has an auto-mount feature such that any recognized formatted partition of a disk or inserted USB disk will be automatically mounted dynamically under /run/media/sd##, where the hashes are replaced with the device identifier of the disk. With VMs or Docker containers, you will want this mount point to always be the same even if it is removed and later plugged in. Instead I recommend using a consistent mount point for your disk. This can be achieved by editing /etc/fstab to mount your specific disk to the location you desire. A good location is /var/local/mnt. Create the directory /var/local/mnt and then you can use this as your mount point root.
Filesystem Layout
Once you have created, formatted, and mounted the partition, change to the /var/local/mnt directory. If it is mounted, you will see 'lost+found' as one of directory entries when you type the 'ls' command. From here create a directory to hold virtual machine images: virtual-machines, and then create a directory for your docker containers and images: docker-root.
KVM Virtual Machine Image Location
KVM uses /var/lib/libvirt/images as the location to store VMs by default. The easiest way to make KVM use virtual-machines directory we created above is to use mount bind feature to literally bind /var/lib/libvirt/images to /var/local/mnt/virtual-machines. This means when you create a file or VM under /var/lib/libvirt/images, the physical location it is stored will be in /var/local/mnt/virtual-machines. You put the mount bind option into /etc/fstab.
Docker Data Root Location
With docker, it is much easier to redirect docker to a new location. Edit the file /etc/docker/daemon.json and add a 'data-root' key to the JSON dictionary. For our case, it would look like this:
- {
"ip-forward": false,
"data-root": "/var/local/mnt/docker-root"
}
Before you edit the file, first disable docker service in Nodegrid UI, this is done under "Security-->Services". Then make the change and restart the service. If there are no errors, you will see the Docker app in Nodegrid UI and if you look at the contents of /var/local/mnt/docker-root, you will see new directories and files created by docker. If there is an error, you can check the log /var/log/dockerd for what is wrong, most likely a typo.
Done
Why do I recommend using a single partition?
Because you can get the maximum disk space usage to avoid making the partition too small and later you have to grow it.
Why do I recommend mounting in /var/local/mnt?
The default mount /run/media is dynamically created. If you use dynamic mounted disk, you don't get to choose the mount location. It can change if you remove the disk, insert a different disk and then insert your VM/Docker storage disk. Creating a directory in /var/local for your mounted disk and modifying /etc/fstab to always mount the disk here prevents this problem.
With docker, I can already use mount bind points for my container data, why do I need to move the data-root?Docker images you download such as Alpine Linux, Nginx, Apache, etc., these are all stored on the main OS disk by default. If download new images, erase them, make custom changes using docker-compose, these activities all contribute to the disk degradation. The best option is to move the data-root entirely.