I’m working on a new approach in my distributed systems class. In the past student groups all did the same project using a simulated network library. This year I got funding to try something different. I want to have them work on a larger project where each group takes responsibility for a different component of the project. The project will use UDP as well as some DDS (cycloneDDS, over multicast UDP).
The host for the class is an Ubuntu 20.04.06 system with several Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (total 63 cores or 128 threads), 256G Ram, 4T disk. While this is a relatively powerful server, I’m expecting over 80 students, and each student group (2 or 3) will need a sandbox to test their component as well as the integration sandbox. Also for DFAI reasons, we have to have some control over visibility between groups.
One consideration is docker, but the main concern is that the students have not had any prior formal exposure to containers, and rootless docker isn’t exactly strait forward especially with isolated virtual networks. Some students may have investigated a container system on their own, I want to minimize the time on the lab environment and max the time on the distributed system part of the course.
I was looking into LXC, and have read the articles on the website, and have done some basic testing. I would like some advice if anyone is willing to give it.
The general setup I was thinking of:
One virtual network (libvirt based) for each group as a separate sandbox. My initial tests of sample code seems to work well. Currently I’ve set the libvirt networks to assign ip addresses based on mac addresses of the containers. I considered multiple lxc Bridges, but I’m not sure how to set up the dhcp flags to set different addresses per bridge, or how to set up the dnsmasq for the same names of containers on different bridges. I was thinking of having a single dns server running on all virtual networks and giving each network a separate domain name, but I haven’t tested that.
The general idea is that the students develop and compile on the host linux box, and copy the executables into the containers to run. I plan to develop some shell scripts to bring their copy of the entire project up on their virtual nework during testing.
Questions I have:
- Ram usage - Compared to docker, LXC containers seem to be more complete, and are running multiple services. While I’m assuming that the underlying host VM will share as much as it can (e.g. text pages), what is the best way to pare down a distribution to the minimum needed to run some simple UDP code. If the full project has 10 components, and I have 20 groups (two instances of the project - two integration sandboxes), then I have 202 containers running. Do I use distribution builder to build a custom local? Use an existing image and remove unneeded systems?
- Sharing containers. Ideally, I’d like the students in a given group to work with the same containers on the same virtual network. My idea was to put them in a common group and share the container folder (instead of ~/.local/share/lxc) using the -P flag. For this to work though the group of the container would have to be the group shared group. When I changed the group id of the container, the other use was able to use lxc-ls to see it, but lxc-start by either user failed with quota exceeded for the network. Our authentication is Kerberos based, and setting up group accounts will be a pain.
Thanks for any feedback you can provide.