HouseMon – about the ‘default’ decoder circuit

In this post, I’ll cover the anatomy of the ‘default’ decoder circuit provided with the current build of HouseMon. After we are done, you should have a reasonable understanding of its current capabilities and function. I’ll also allude to some potential improvements that can be made by using some revised/additional gadgets.

The basic decoder circuit is quite generic and can process data from various input sources, although because of the inclusive nodeMap gadget and other specifics it is a little biased to RFM12 packet decoding as provided by the RFM12Demo sketch.

Here is the circuit definition:

# pipeline used for decoding RF12demo data and storing it in the database
circuits.rf12toDatabase =
  gadgets: [
    { name: "st", type: "SketchType" }
    { name: "d1", type: "Dispatcher" }
    { name: "nm", type: "nodesJeeLabs" }
    { name: "d2", type: "Dispatcher" }
    { name: "rd", type: "Readings" }
    { name: "ss", type: "PutReadings" }
    { name: "f2", type: "FanOut" }
    { name: "sr", type: "SplitReadings" }
    { name: "db", type: "LevelDB" }
  wires: [
    { from: "st.Out", to: "d1.In" }
    { from: "d1.Out", to: "nm.In" }
    { from: "nm.Out", to: "d2.In" }
    { from: "d2.Out", to: "rd.In" }
    { from: "rd.Out", to: "ss.In" }
    { from: "ss.Out", to: "f2.In" }
    { from: "f2.Out:sr", to: "sr.In" }
    { from: "f2.Out:db", to: "db.In" }
    { from: "sr.Out", to: "db.In" }
  feeds: [
    { data: "Sketch-", to: "d1.Prefix" }
    { data: "Node-", to: "d2.Prefix" }
  labels: [
    { external: "In", internal: "st.In" }

At the ‘start’ of this circuit is the ‘SketchType’ gadget. It expects a stream of data to enter its .In pin, and this is generally provided by the ‘Serial’ gadget or the ‘replay’ circuit (which emits data that; to all intents and purposes looks like it came from a serial device).

‘SketchType’ looks at each message it receives for any text surrounded by square braces ‘[]’ and extracts that text. If it finds nothing, it passes the message on without any intervention.

Most of the Jeelabs sketches provided for use with Jeenodes, emit a basic signature on the serial port when they start. This signature is generally produced during the sketches ‘setup()’ process, and is emitted in square brackets ‘[]’.

For the blink sketch ‘blink_demo.ino’ its something like “[blink_demo]”

For the RF12Demo sketch its “[RF12Demo.nn]” where ‘nn’ represents version information.

If this sketch name is obtained by SketchType, it strips away any data after the dot (.) to leave the base sketch name.

[blink_demo] becomes ‘blink_demo’
[RF12Demo.12] becomes ‘RF12Demo’

If a sketch name has been obtained, the SketchType gadget emits this sketch name using a tagged message format with <dispatch> as the tag and sketch name as the message data, as discussed in previous tutorials.

Why <dispatch>?.

If you review the circuit again, you will find nothing specific within the circuit that understands how to communicate with a sketch. Sure, the description says # pipeline used for decoding RF12demo data and storing it in the database, but we could use the output from an IR library or a simple 433Mhz OOK library. It all hinges on the <dispatch> tag, which is handled by the next gadget in the circuit, the ‘Dispatcher’ that is given the ‘d1’ name.
You will notice within the ‘feeds’ section, that the Dispatcher(d1) is sent Sketch- to its .Prefix pin .

What happen now, is rather complicated, but I’ll try to summarize:

Dispatcher (d1) passes messages through until it sees a <dispatch> tag. When it sees this tag, it compares it’s message content with an internal state variable. If sketch name was ‘RF12Demo’ the Dispatcher(d1) would say “Am I currently handling any sketch name?”

If it IS handling a sketch of type ‘RF12Demo’ already, it ignores the dispatch tag, and sends any input messages through to the gadget it is handling messages for.

If it is NOT handling a sketch of type ‘RF12Demo’, it then attempts to begin handling sketches of this type. It does this by combining the ‘prefix’ it was provided via its feed during initialization, with the sketch name provided via the <dispatch> tag, and looks this gadget up in the Registry.
So, in the case of ‘RF12Demo’ sketch name, it would see if a gadget Sketch-RF12Demo exists in the Registry.

If this gadget (Sketch-RF12Demo) exists in the Registry it is loaded into the circuit that Dispatcher(d1) exists within, and the gadgets .In and .Out pins are wired into the Dispatcher(d1)*.
If the gadget cannot be found/loaded, the message is ‘rejected’.

The end result is that the Sketch-[sketch name] ends up being loaded into this circuit with its .In and .Out pins linked into the Dispatcher(d1) where it can get at them for future message routing.

We now have a circuit that routes all its inbound messages into a gadget that knows how to handle the data it receives (we hope – you can still mis-wire a software circuit just as in real life electronics).

In the case of a Jeenode running RF12Demo, the circuit has just become self-aware, and knows that any serial data should be expected to be “compliant” with the RF12Demo sketch/protocol.

But…a Jeenode running RF12Demo, just reports the packets it receives using OK\n??

How do we know what this data means?

Gadget Sketch-RF12Demo had been loaded just at the moment SketchType gadget saw ‘[RF12Demo.12]’. It was loaded ahead of the message itself, and the message (and subsequent data) is passed through it.

Sketch-RF12Demo immediately attempts to gather other information it knows should be emitted from a Jeenode running RF12Demo. The main ones being the sketches RF configuration (Frequency, Band, Node ID etc).

Once it has this information it emits a tagged message using the tag <RF12Demo>, then listens for the familiar OK \n sequences, and for each one it sees, it emits a tagged message <node>, “data”

The next gadget in the circuit is ‘NodeMap’.

This gadget is all about ‘configuration’. It is simply a glorified lookup tool, but critical to the next stage in the circuit. Its Feed data is a database containing a mapping between an RF group/node and the name of a decoder that should be used in order to decode the data.

If the Sketch-RF12Demo gadget sends a tagged message <RF12Demo> containing band of 868 and group of 5, then NodeMap is going to remember this data. When it then sees a <node> message containing node of 2 it will use this data to attempt to make a mapping. Using the example snippet below, we can see the ‘feed’ with ‘RFg5i2’ present, that will be used to make the match described above.
You’ll note that the Frequency (868 in my example) is not accounted for in this example, we will deal with this later.

# the node mapping for nodes at JeeLabs, as pre-configured circuit
circuits.nodesJeeLabs =
  gadgets: [
    { name: "nm", type: "NodeMap" }
  feeds: [
    { data: "RFg5i2,roomNode,boekenkast JC",  to: "nm.Info" }

OK, so our <RF12Demo> tag and our <node> tag messages are used within NodeMap to match on the record we have used in the example above.

The ‘data:’ field comprises the “lookup” RFg5i2, the “decoder name” roomNode, and the “location” boekenkast JC.

After ‘NodeMap’ in this circuit, is another ‘Dispatcher’ (d2). You will see that it is initialized by sending ‘Node-‘ to its .Prefix pin. After nodeMap has made a match on ‘RFg5i2’ it will send out a tagged <dispatch> message containing the ‘decoder’ from the matched record above. In this case <dispatch> “roomNode”.

The Dispatcher (d2) now goes through an identical process to (d1), but using the prefix ‘Node-‘, so again using the example, the <dispatch> tag will cause the Dispatcher(d2) to see if a gadget ‘Node-roomNode’ is present in the Registry. If so, it is loaded into the circuit (if not already loaded) and subsequent matched messages will be routed to this gadget.

Subsequent matches within NodeMap will cause additional <dispatch> tags to be emitted, causing Dispatcher(d2) to load additional decoders which then process the relevant messages they are supplied.

So, we have reached a stage where the circuit has been able to determine that it is an RF12Demo circuit, and that the user has supplied a lookup that allows the circuit to find and load specific decoders for each matching OK\n message.

What does a decoder do?

In brief, in the case of an RF12Demo packet decoder, it is provided the RF12Demo state data (via <RF12Demo> tag messages and the <node> message itself) and is simply tasked with using this data to emit a tagged <reading> message containing from 1 to n string:int tuples.

In the case of a roomNode, a <reading> looks something like this:
“temp”: 221,
“humi”: 80,

Imortant: You will note that the decoders used in this circuit MUST emit integers. How this data is interpreted and represented outside of the decoder, is dependent upon the rest of the circuitry as we will see later. Just remember that you must emit integers, you cannot emit string values like so:

“LAMP”:”ON” //not allowed in ‘this’ circuit.

After our decoders have decoded some data, they emit the data as a tagged <reading> message. This message comes out of the decoder and back into the Dispatcher(d2), which then emits the message back out into the main circuit.

Imagine the Dispatcher as a type of message ‘incubator’. It gobbles up messages, routes them internally to the gadgets it needs to create, and when a gadget spits something out, the message is sent back into the world for onward consumption.

Reviewing our original circuit, we see that Dispatcher(d2) sends its output to a ‘Readings’ gadget, where ‘Readings’ is simply interested in gathering passing data to construct ‘state’, as in <RF12Demo> data, etc etc, until it finally sees a tagged <reading> message. At this point it sends one huge message containing all the ‘state’ it has collected, and implants the current reading it has just seen (via the tagged <reading> message) into the message and emits it.

The ‘Readings’ gadget sends its huge message containing ‘state’ + ‘reading’ into the ‘PutReadings’ gadget, which sits in the circuit looking for these large composite messages. It breaks the large message into some constituent parts with the goal of constructing a message that looks like:

"ms": millisecond timestamp,
"val": {"temp": 221, "humi": 80, "light":200, "moved":1}
"loc": "boekenkast JC",
"typ": "roomNode",
"id": "RF12:5:2"

This message is then emitted from ‘PutReadings’ using a tagged message ‘/reading/<id>’, such as:

/reading/RF12:5:2, <data>

The output of ‘PutReadings’ is sent to the ‘FanOut’ gadget we covered in an earlier tutorial. This allows the same message to be routed to multiple targets. In this case, one copy of the message is sent to ‘LevelDB’ where its stored against the ‘/reading/RF12:5:2’ topic, and a second copy of the message is sent to the ‘SplitReadings’ gadget, which simply decomposes the ‘val’ field above in combination with the ‘loc’ field to construct a series of ‘sensor/<subkey>’ tagged messages, one for each “decoded field” of the decoder output.


‘sensor/boekenkast JC/temp/<millisendstimestamp>’, 271
‘sensor/boekenkast JC/humi/<millisendstimestamp>’, 80

Each of the above messages emitted from ‘SplitReadings’, are then sent back into the ‘LevelDB’ gadget via the defined ‘wire’.

I have not spoken about the reason behind decoders using ‘integer’ data, and neither how other parts of HouseMon can make use of this seemingly intractable limitation!!

I’ve run out of time now, so that will be covered in a post shortly.

Problems to overcome

In the next tutorial I will show you how we overcome the following problems:

Add ‘Frequency’ as an aid in allowing multi-frequency networks.
Add a more flexible ‘location’ lookup process.
Provide a suffix mechanism from decoders to allow us to generate multiple targets from a single node/decoder.
Provide a way to generate multiple ‘readings’ from a single ‘OK’ message.
Identify a few gotcha’s with the current baseline code, and suggest alternatives.

Additionally, We will discuss a custom decoder circuit, that is more of a game than anything practical in itself, but the experience obtained will be valuable in constructing your own non-RF based circuits.

Until next time…

Posted in Uncategorized
4 comments on “HouseMon – about the ‘default’ decoder circuit
  1. Dan says:

    Great explanation. When the follow up? ๐Ÿ˜‰

  2. Christian de Boor says:

    That was a great job!
    That was easy to understand and exciting!
    I learned a lot, even though my english is bad. (Google ๐Ÿ˜‰
    I would be very happy if you do it on.

    with kind regards

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: