M-o-a-T

The MoaT

«MoaT» is the «Manager of all Things».

A moat is also the bog around the castle where those who want to tackle its walls get mired in, if not drown. The conclusions to be drawn from this are hardly surprising, thus left to the esteemed reader.

Quick heads-up: if you want to know what's new, there's something like a mostly-infrequently-updated blog here.

Project Goals

The MoaT's design is guided by a few principles which, in combination, mean that there's no existing project which matches its (eventual …) features.

  • 100% open source.

  • Ability to control a variety of devices which are not networked.

  • On-line configuration changes: no need to restart services.

  • No requirement to use a cloud service.

  • 100% redundancy: MoaT shall be a system without a single point of failure, as far as possible.

  • The protocols must be programming language agnostic.

  • Easy to script / program

    • no requirement to restart the system when you update a script

  • Reasonably modern code base

    • thus, not written in ancient Perl (cf. FHEM)

  • Reasonably low resource usage

    • thus, not written in Java (cf. openHAB)

    • usable on a Raspberry Pi 2

  • API before command line before user interface

  • MoaT does not poll devices: all changes need to be pushed.

  • MoaT supports "simple" wired systems.

  • There is a single device status database which records the states of, well, everything.

  • Installation of MoaT-compatible hardware components does not depend on Cat5 wiring or WiFi.

Design Overview

At the lowest level, there are external sensors.

Sensors are connected to some low-power controller. MoaT supports a few different ones, and its code is modular enough to be easily portable. On the hardware side there's a common plug+socket for warm-swappability.

These controllers need to be configured and updated, then send and receive messages. They can do this via a low-speed bus with two to four data wires.

Next, a computer acting as bus master connects the bus to the local IP network. For redundany there may be up to three masters. The master is responsible for bus address allocation, device configuration, forwarding relevant messages, etc..

As messages describe state changes, the actual device states need to be stored somewhere. The MoaT uses DistKV, a zero-master hierarchical key-value storage system. DistKV uses MQTT or Serf to exchange messages between its nodes.

DistKV also holds whole the installation's configuration, as well as its automation scripts.

There is an MQTT adaption layer for DistKV so that IoT devices (ESPHome) or visualizations (Home Assistant) can use it transparently.

Automation

DistKV supports remote code execution (async-enabled Python 3.7+). Its built-in helpers allow you to quickly deploy and change small (or not-so-small) automation scripts, or integrations between DistKV and other protocols.

Scripts can be parameterized, allowing a fair amount of code re-use.

All values in DistKV are typed; there's a translation layer to MQTT. Your scripts are going to deal with True, False, Numbers and dates; "ON" vs. "on" vs. "1" is handled in one place only: the MQTT adapter's configuration.

DistKV

The DistKV system is the central building block of this system. It's basically a simple key-value system, but with a few unique twists:

  • There's no master.

  • There's no storage.

  • There's no single point of failure.

  • There's no consensus protocol.

  • There's no requirement to encode everything.

OK, the latter is a lie: it's a network protocol, so it needs to encode your data to something; however, DistKV does that for you transparently. Behind the scenes it uses MsgPack.

On the other hand, there are some built-in smarts:

  • Of course there's authentication and stuff.

  • You can register data translators. Home Assistant wants its configuration data as JSON, MQTT typically uses "ON" and "OFF" instead of True and False? Not a problem, native DistKV code will deal with real data structures, numbers, and booleans.

  • Of course there's storage, you need to do something about power failures. DistKV records the current state plus all changes to disk. Bonus feature: if you just deleted your entire configuration by mistake, restoring it from the log is trivial. Well, almost, but you get the point.

  • Rebooting, on the other hand, is handled by simply contacting one of the not-rebooted nodes and fetching the current state.

  • Network partitions (switch failure, crashed MQTT server) are not a problem either. DistKV transparently recovers and ensures that the system returns to a consistent state.

  • The client protocol is documented and reasonably easy to implement, so using DistKV from something-that's-not-Python3 is possible. Contributions welcome.

Future goals

World domination.

Seriously, though, MoaT does not try to replace your home or office control system. It does however offer a unique, usable substrate environment on which to build those parts, esp. those which should be redundant.

Notes

Base technology

  • Python 3.7, Trio

  • State retention: distkv

Trio is a Python framework for reasonable async programming. No, asyncio isn't reasonable.

User interface

The MoaT is not in the business of reinventing Home Assistant, FHEM, OpenHAB, or any other frontend. It is a manager, not an overlord. ;-)

The obvious choice of user interface is likely to be Home Assistant, as it has reasonable support for MQTT and supports autoconfiguration.

Hacking

Collaboration is always appreciated.

History

The MoaT began its life as «HomEvenT», an obvious contraction of "home event", if funkily capitalized. The first version used a home-grown python-interpreted async scripting language. Version 2, unfinished, tried to replace that with native Python scripts and fine-grained dynamic state storage based on etcd, which was an interesting idea but ultimately much too complex.

The current version distills appropriate lessons of simplicity from this experience. The author's home runs on six Raspberry Pi machines plus KNX plus 1wire plus a heap of ESP8266 switches.

As the MoaT is not intended just for managing events in a home, it was time for a name change.