Howdy, and Welcome!
I am Matt and I am new to blogging! In fact, this is my second ever post. I made my first a few weeks ago about networking in Mycelium. After that article I realized I have not provided a good background to the project and its goals. So here we go! The goal of the Mycelium project is to develop a decentralized distributed p2p database. More technical descriptions can be found in the repository or the wiki. This will be a less technical introduction.
What is Mycelium?
In nature, mycelium describes a collection of different organisms that live underground. Everyone is familiar with the fruits of some of those organisms as mushrooms. As well as fruiting mycelium transfers information between plants it connects. It is a distributed information network! Mycelium takes it's name from this natural example of networking.
Mycelium the software project.
Mycelium will run on everything from a drone to a server. Currently it runs on x86 and Raspberry Pi. It's designed use is in a disrupted environment using Bluetooth and WiFi for networking. The idea is that data will be distributed over a mesh of loosely connected devices. Networking happens when they come within BlueTooth proximity or connect to a LAN. Some devices like phones, tablets, or drones will create peer to peer networks as they come into proximity with other devices. Another scenario is small team or business. There is a team size where you need a database, but do not have the resources for a centralized solution. Mycelium will behave like a centralized database that naturally distributes data over several devices. This distribution should create a natural amount of data replication for redundancy and allow concurrent use of data without corrupting or locking network files.
So what are you using?
Mycelium uses nothing outside the ecosystem of the Rust programming language. Everything will be built using the tools and packages available within that system. Rust was picked for a couple of reasons:
First, the ecosystem of Rust is young. A young ecosystem means less baggage and fewer abandoned paths. This makes the whole ecosystem easier to comprehend. However, we are still waiting for a few features in the language like a good async/await story.
Second, we have a parallel/concurrent first attitude for this system. Rust has built-in support for preventing certain classifications of concurrency bugs.
Finally, documentation is the primary reason for picking Rust. Having a book, an API reference, and an advanced book all built into the compiler is an ergonomic wonderland for development. While exploring new ways of programming you can waste a lot of time looking through reference material. Building documentation on the spot for the exact libraries and platform is amazing.
So how now brown cow?
One step at a time... So something like developing a database will be a little less than easy...
Mycelium has started with a very naive implementation of something a hair more complex than a key/value store. We will build on top of the filesystem and use standard libraries or common crates to create our MVP. Once we have the MVP we will be able to use benchmarking and our unit tests to refactor.
Data
Let's start with a little about the structure of Mycelium data:
Mycelium core is the library about data and that data is organized in buckets called tags. Indexes are created over those tags back to the location in the bucket. Tag structure is a very shallow tree. Mycelium uses Crossbeam and it's libraries for SkipList, Map, and Sets. These libraries are not yet a part of the crossbeam crate. Until they are the source files for these data structures will be a part of Myceliums source. Once these are available in the crossbeam crates they will be referenced there instead.
On top of Mycelium Core we have Mycelium Index. Mycelium Index will be used to create relationships like foreign keys, or parent/child. Using those relationships we can then perform operations using a SQL like language. Indexes relate items within a Tag or between Tags.
Networking
The key feature of this database is that it will distribute marked information over a mesh of networked nodes without direction. To start off with here Mycelium uses Tokio for TCP and UDP currently and will move to rust-libp2p soon. Tokio will remain as an executor.
Interpreter
Mycelium currently uses a pseudo interpreter and will continue to do that for the time being. The SQL like project will start after the MVP versions of core, index, and networking are done.