Monitoring Prototype – Part 1

We were inspired by a YouTube video by Everlanders to monitor of solar production and battery state. We took a slightly different technical approach, but the end goal is the same. We wanted to create a complete monitoring solution, that will be able to monitor all sorts of things in Molly. More things than would seem reasonable.

The original thought was to monitor to just monitor the solar panels, so we could see how much electricity we make each day. But each day we think of new things to monitor. To make sure this monster does not grow out of control, one of the design principles is that none of the monitoring gadgetry should be visible. It all has to be hidden away quieting logging away, so we can access it when we want to.

The final monitoring system is made up of four basic components:

  • The things that produce measurements. These will be called the sensors, and range from simple things like a temperature sensors to more complex devices like the Engine Control Unit (ECU) on the FUSO truck.
  • The software that connects the sensors to the computer and processes the sensor information into a standardized form. The node-red software is a good choice to mediate between the sensors and the database.
  • The sensor measurements are stored in a database, and more specifically a time-series database. For our application we went with a lightweight database called influxDB.
  • Lastly the information in the database needs to be visualized onto a display. We wanted the display to look good and be very flexible, we settled on Grafana as our visualization software.

The first step in any project is making sure the idea will work and sort out any fundamental issues. So, just spent the morning installing all of the software and playing around. For this proof-of-concept test, our HAM radio was used as our sensor to generate messages, the HAM radio was connected to our Mac computer where the messages were stored in the database and ultimately visualized. The HAM radio will receive GPS messages from other HAM radios, extract the height above sea level and this height is what will be displayed. This prototype is not really related to monitoring solar generation or batteries or anything to do with Molly, but regardless, it will test the approach.

Before getting into all the details of how to install the software, just wanted to say the approach looks good and the initial testing worked out well. The combination of node-red, influxDB and Grafana is going to work for us. A few days after the prototype was complete, we decided to add a Raspberry Pi into the mix. The nice thing about the Raspberry Pi is all the four USB ports it has. Means we can plug sensors into the Raspberry Pi instead of the main computer. But more on that later.

The following is a blow-by-blow of the installation process for the prototype. This was all done on a Mac, but wanted to write it down, as it has to be repeated on the computer in Molly.

HAM Radio connected to the Mac.
This image has an empty alt attribute; its file name is Screen-Shot-2020-04-04-at-11.34.26-1024x812.png
Grafana with maps served from our local tile server with dots received from the HAM Radio,

This is a two part post, and this first part is about gathering all the needed software and setting up all up to see if it works. The prototyping phase. Part 2 is about the how we installed it all into Molly and how we will use the system.

Over time we will add more sensors, connect them to the computer and configure them correctly. The details are described separately in the following posts

  • GPS tracking
  • FUSO Engine vitals
  • Weather Station
  • Solar Charger (what started all of this)
  • Battery condition
  • Tire Pressure (working on this)

Guide to Install the Software

Installing Node-js

The Mac does not have node.js, and this is required for node-red. The node.js website advises to use the following command to install node-js.

curl "https://nodejs.org/dist/latest/node-${VERSION:-$(wget -qO- https://nodejs.org/dist/latest/ | sed -nE 's|.*>node-(.*)\.pkg</a>.*|\1|p')}.pkg" > "$HOME/Downloads/node-latest.pkg" && sudo installer -store -pkg "$HOME/Downloads/node-latest.pkg" -target "/"

Which produced the following successfully installation

$ curl "https://nodejs.org/dist/latest/node-${VERSION:-$(wget -qO- https://nodejs.org/dist/latest/ | sed -nE 's|.*>node-(.*)\.pkg</a>.*|\1|p')}.pkg" > "$HOME/Downloads/node-latest.pkg" && sudo installer -store -pkg "$HOME/Downloads/node-latest.pkg" -target "/"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30.2M  100 30.2M    0     0  7324k      0  0:00:04  0:00:04 --:--:-- 7324k
installer: Note: running installer as an admin user (instead of root) gives better Mac App Store fidelity
installer: Warning: node-latest.pkg was signed with a certificate that is not valid for store submission.
installer: Installation Check: Passed
installer: Volume Check: Passed
installer: Starting install
installer: Install 0.0% complete
installer: Install 18.8% complete
installer: Install 26.5% complete
installer: Install 31.7% complete
installer: Install 88.7% complete
installer: Install 100.0% complete
installer: Finished install
Installing node-red

The node-red getting started guide provide information on installing node-red on your local machine. I used the following comamnd

sudo npm install -g --unsafe-perm node-red

And a 15 seconds later had the following result

$ sudo npm install -g --unsafe-perm node-red
...
+ node-red@1.0.4
added 293 packages from 303 contributors in 15.371s

That looks successful. Let’s see if it runs with the command

node-red

This command was successful with the result below. As this is on the mac, no need to start-up automatically on boot-up.

29 Mar 10:00:24 - [info] 

Welcome to Node-RED
===================

29 Mar 10:00:24 - [info] Node-RED version: v1.0.4
29 Mar 10:00:24 - [info] Node.js  version: v13.12.0
29 Mar 10:00:24 - [info] Darwin 19.3.0 x64 LE
29 Mar 10:00:24 - [info] Loading palette nodes
29 Mar 10:00:25 - [info] Settings file  : /Users/carolinediericx/.node-red/settings.js
29 Mar 10:00:25 - [info] Context store  : 'default' [module=memory]
29 Mar 10:00:25 - [info] User directory : /Users/carolinediericx/.node-red
29 Mar 10:00:25 - [warn] Projects disabled : editorTheme.projects.enabled=false
29 Mar 10:00:25 - [info] Flows file     : /Users/carolinediericx/.node-red/flows_Our-Computer.local.json
29 Mar 10:00:25 - [info] Creating new flow file
29 Mar 10:00:25 - [warn] 

---------------------------------------------------------------------
Your flow credentials file is encrypted using a system-generated key.

If the system-generated key is lost for any reason, your credentials
file will not be recoverable, you will have to delete it and re-enter
your credentials.

You should set your own key using the 'credentialSecret' option in
your settings file. Node-RED will then re-encrypt your credentials
file using your chosen key the next time you deploy a change.
---------------------------------------------------------------------

29 Mar 10:00:25 - [info] Server now running at http://127.0.0.1:1880/
29 Mar 10:00:25 - [info] Starting flows
29 Mar 10:00:25 - [info] Started flows

On the computer in Molly, will likely start node-red as service so it starts up automatically on a reboot. The command to do this is sudo systemctl enable nodered.service

The interface can be accessed at localhost:1880, which was successful

This image has an empty alt attribute; its file name is Screen-Shot-2020-03-29-at-10.01.56-1024x680.png
Installing the Serial Port listener

Node-red does not come with a serial port listener as standard. After a few tries the following worked for me

$ cd ~/.node-red
$ npm i node-red-node-serialport

This just resulted in an error, as maybe I was missing the necessary tools to compile the plugin. So, I installed Xcode from Apple. And then running the command again was successful. But I had to also add via the palette manager in node-red.

This image has an empty alt attribute; its file name is Screen-Shot-2020-03-29-at-10.23.51.png

The buttons were added to node-red. Some mild success. I later worked out I needed to add a influxdb node. I did this through the interfave

This image has an empty alt attribute; its file name is Screen-Shot-2020-03-29-at-11.52.21.png
Installing InfluxDB

InfluxDB is the time-series database. This is installed with home-brew, with the following commands

$ brew update
$ brew install influxdb

Start the InfluxDB with

$ influxd

Testing the InfluxDB install as follows with some test data.

$ influx -precision rfc3339
Connected to http://localhost:8086 version v1.7.9
InfluxDB shell version: v1.7.9
> CREATE DATABASE solar
> SHOW DATABASES
name: databases
name
----
_internal
solar
> USE SOLAR
ERR: Database SOLAR doesn't exist. Run SHOW DATABASES for a list of existing databases.
DB does not exist!
> USE solar
Using database solar
> INSERT batTemp,host=Molly42,type=AGM value=45
> SELECT "host", "type", "value" FROM "batTemp"
name: batTemp
time                       host    type value
----                       ----    ---- -----
2020-03-28T23:40:02.68956Z Molly42 AGM  45
> INSERT houseTemp,country=USA,sensor=hobo external=37,internal=35
> SELECT * FROM "houseTemp"
name: houseTemp
time                        country external internal sensor
----                        ------- -------- -------- ------
2020-03-28T23:43:14.210488Z USA     37       35       hobo
> SELECT * FROM "houseTemp"
name: houseTemp
time                        country external internal sensor
----                        ------- -------- -------- ------
2020-03-28T23:43:14.210488Z USA     37       35       hobo
2020-03-28T23:47:30.606567Z Canada  47       45       hobo
> INSERT houseTemp,country=USA,sensor=hobo external=32,internal=32
> INSERT houseTemp,country=USA,sensor=hobo external=34,internal=37.5
> 

Later will need to work out how to automatically start influxDB, probably with

$ sudo systemctl enable influxdb.service
Installing Grafana

Grafana is the data visualization tool. There are installation instructions on their website, and I followed these. Again, home-brew was our friend and used to install this software.

$ brew update
$ brew install grafana

Grafana is started with the following command, which should survive a reboot.

$ brew services start grafana

The Grafana dashboard can be accessed at http://localhost:3000.

Installing MQTT

Most people add a message broker to ease sending of messages around. Mosquitto MQTT seems like a good choice for small computers. I took the installation instructions for from this site and used home brew to install. Not entirely sure if I need a message broker, but will use as part of testing.

$ brew install mosquitto

And to make sure it starts on reboot

$ brew services start mosquitto
Making it all Work

At the moment, neither node-red and influxdb do not automatically start on a reboot. So after a reboot, the following two commands are needed

$ node-red
$ influxd

Also, it all has to work when off-the-grid. So, isolated the computer from the Internet, and all looks good.

This image has an empty alt attribute; its file name is Screen-Shot-2020-04-03-at-13.45.27-1024x385.png
A simple test in node-red, to connect the Victron Energy smart Shunt to the database.
Adding a Worldmap Panel to Grafana

Grafana has this very nice plugin to display geo-tagged information onto a world map. while it is easy to install this plugin, unfortunately it pulls it’s map information from the Internet, which is not ideal for us, as we will not always be connected to the Internet. So, needed to make some small changes to the plugin.

The basemap was changes from CartoDB Dark to maps served from our local tile server, powered by open tile maps. The following line was added to the worldmap.ts function

url: 'http://localhost:32769/styles/klokantech-basic/{z}/{x}/{y}.png'

And the stylesheet was fixed with replacing the previous style sheet in worldmap.dark.css with

.worldmap-popup .leaflet-popup-content-wrapper, .worldmap-popup .leaflet-popup-tip {
  background-color: #ECECEC;
  color: #000 !important;
}

With the code changes made, it can be installed with

$ cd /usr/local/var/lib/grafana/plugins/grafana-worldmap-panel
$ brew install yarn
$ yarn install
$ sudo yarn build
$ brew services restart grafana

A very pleasing end result. Now ready to deploy all of this to Molly.

Adding Track Map Panel

The Worldmap Panel is nice, but does not lay down a track based on geo tagged points. The good news, a different plug-in, Track Map Panel, does this. so, with the following commands, got this panel

$ cd /usr/local/var/lib/grafana/plugins
$ git clone --recursive https://github.com/pR0Ps/grafana-trackmap-panel

Of course, this panel also pulls the maps from the imternet, so changed the trackmap_ctrl.js file like above. And then built the panel as per the instructions

$ sudo npm install
$ sudo npm run build
$ brew services restart grafana

I do not have very good test data, but it clearly works.

This image has an empty alt attribute; its file name is Screen-Shot-2020-04-12-at-15.24.31-1024x569.png
Trackmap Panel using local Tile Server
Raspberry Pi

On second thoughts, best to install node-red on a raspberry pi, freeing up the USB ports on the main computer. Used the guidelines from this website to install node-red. This was so easy I am not even going to describe it here.

This image has an empty alt attribute; its file name is IMG_5958.jpg
HAM Radio connected to the Raspberry Pi.
Connecting the Victron Energy Smart Shunt

The HAM Radio was a good test, but the solution should work with any serial based sensor, like the Victron Energy Smart Shunt. So, found a power supply and load, and wired it up to make sure everything worked fine. After a little programming, looking good.

This image has an empty alt attribute; its file name is IMG_5940-1024x768.jpg
Power Supply simulates the battery. The Victron Energy SmartShunt measures the voltage and current flow, along with many other things.
This image has an empty alt attribute; its file name is Screen-Shot-2020-04-03-at-13.57.04-1024x497.png
Mockup to test getting measurements from sensors