Summary
Mycelium is a peer to peer distributed database. Full code can be found at this repository We are going to be using Docker-Compose as a tool to develop this databases network behavior. Docker-Compose is a tool that deploys docker containers. You use it for deploying many containers at a time. It is very useful if you have auth services, back-end API, databases, and more you want run together. In this case, we are using Docker-Compose to deploy Mycelium nodes in a simulated LAN environment.
Goal
Mycelium uses UDP multicast to find and configure instances of itself over the LAN. This configuration is also be used for distribution of user actions over TCP as some user actions will result in further multicast messages and additional operations.
Here we will be looking at using UDP multicast to send a message whenever an instance of Mycelium comes online.
In this message we will be sharing information with online nodes about each other. Eventually we will be using UDP to setup an event message channel that will be used to keep nodes in sync with system events. System events in Mycelium are things like the creation of new data containers and will result in indexes getting updated over all the nodes. Our goal is to use these indexes to handle searching and replication of data.
Creating an Image
We will be using Docker, Docker-Compose and Rust. Follow the guide here for getting started with Docker. Follow the guide here for Docker-Compose. Follow the guide here for getting started with Rust. Additionally since we are using Rust we can build for targets other than theplatform we are developing on. In this case we will also be getting the build chain for x86_64 MUSL with the following:
$ rustup target add x86_64-unknown-linux-musl
In the Mycelium project code we have the following Dockerfile:
FROM alpine:latest
COPY target/x86_64-unknown-linux-musl/release/mycelium /usr/local/bin/mycelium
EXPOSE 34120
EXPOSE 34121
CMD ["mycelium"]
Here we are telling docker to get the latest Alpine Linux image, copy our MUSL executable onto it, open a couple of ports, and finally start the executable we copied. You could choose to instead copy the source for Mycelium into a container, build, and finally copy the executable from that build into a final container. Generally, to keep containers as small as possible you want to use an intermediate build image. In this case we will skip all of that and just target MUSL from our host system and copy the executable.
Once we have our image we can setup a docker-compose.yml file:
version: "3.7"
services:
node0:
image: mycelium:0.1.0
ports:
- "34120:34120"
node1:
image: mycelium:0.1.0
node2:
image: mycelium:0.1.0
Using compose we are setting up 3 nodes which will all be using the image we had just created above. On one of those images we expose the TCP port 34120 to our host system.
To make building/deploying easier:
#!/bin/bash cargo build --bin mycelium --release --target x86_64-unknown-linux-musl docker-compose stop -t 2 docker-compose rm -f docker image build -t mycelium:0.1.0 ./ docker-compose up
Now when this runs you should see something like:
node2_1 | Message from: 172.18.0.2, tags: []
node0_1 | Message from: 172.18.0.3, tags: []
node1_1 | Message from: 172.18.0.4, tags: []
Using a TCP client we can then run a Mycelium SQL statement (there is a WIP Mycelium client in the source directory that can be used or a command line utility):
insert [0] into dog
This creates a new container with a node that contains this byte array. This will result in an updated message from the 3 running Mycelium nodes:
node2_1 | Message from: 172.18.0.2, tags: []
node0_1 | Message from: 172.18.0.3, tags: ["dog"]
node1_1 | Message from: 172.18.0.4, tags: []
Now that we have a environment setup where we can run nodes networked together we can move on to create the behavior and that is what we will do in the next post.