Monitoring Raspberry Pi Servers With Prometheus & Grafana

,
Prometheus + Grafana

Intro

Hosting servers at home may cost more than cloud providers but if you don’t need much computing power like me, you could use Raspberry Pi, and by any chance, if you have a solar-power you could plug it into the solar-power source. That way, your only expense is probably your ISP bill, which you are already paying at your home.

I have two raspberry pis, one of them is Pi Zero W and the other one is Pi 3 B+. I use my Pi Zero as my home gateway, it has privacy focused DNS server and it can also be switched to a VPN gateway. I use my Pi 3 as a storage server, it runs containerized services such as Traefik, PostgreSQL, and NextCloud. Two of my Pis are only accessible from the home network or my VPN.

One day I was taking backup my phone remotely, it crashed and Pi 3 went offline. It had some thermal issues therefore I installed a heatsink on top of its CPU but I worried about if the problem is going to repeat and decided to monitor my raspberries.

Prometheus

A quote from Prometheus’s GitHub[1] Repo:

Prometheus is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true.

Yea, it collects metrics let’s install it.

I am going to use the pre-compiled armv6 binary because I am installing Prometheus to Pi Zero. Poor Zero :(

Download the armv6 binary from its github releases page. After you extracted the archive, save the following yaml files into the extracted folder then you can start Prometheus by using ./prometheus --config.file=prometheus.yml command

prometheus.yml
1
2
3
4
5
6
7
8
9
global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'zero'
    file_sd_configs:
      - files:
        - targets.yml
targets.yml
1
2
3
4
- targets:
       - 127.0.0.1:9090
  labels:
       job: zeroProm

The targets.yml file dynamically loaded by Prometheus if we change it. After you started Prometheus, go visit your Pi’s IP address and you should be able to see your Pi’s state is UP. Keep in mind that the address should be in this format http://<IP-ADDRESS>:9090/targets.

Prometheus targets

Prometheus collects the metrics of itself but I also want to install Node exporter. If you don’t want to install Node exporter you may skip this part.

Node exporter:

Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors.

Download pre-compiled armv6 binary version of Node exporter from its github releases page and extract the archive to a folder. After that, you can start Node exporter by using ./node_exporter --web.listen-address=:8088 command. If you neglect web.listen-address flag it will start listening on port 9100 by default.

Until here we did basically the same steps as in the docs of Prometheus[4].

I also wrote basic systemd service files for both Prometheus and Node exporter. So that you can control them via systemctl.

/lib/systemd/system/prometheus.service
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[Unit]
Description=Prometheus service
After=node_exporter.service
Wants=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5s
Type=simple
ExecStart=/root/prometheus/prometheus --config.file=/root/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target
/lib/systemd/system/node_exporter.service
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[Unit]
Description=Node exporter service
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=500
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=5s
Type=simple
ExecStart=/root/node_exporter/node_exporter --web.listen-address=:8088

[Install]
WantedBy=multi-user.target

So let’s stop previously opened prometheus and node_exporter processes and start them via systemctl enable --now prometheus.service and systemctl enable --now node_exporter.service. Since Prometheus on Pi Zero can collect Pi 3’s metrics there is no need for Prometheus to install to Pi 3. Therefore, I am going to install only Node exporter to Pi 3. Installing Node exporter to Pi 3 is pretty much the same as Pi Zero, I’ll just download the pre-compiled arm64 version and extract it to a folder after that I’ll just use systemctl to start it.

Now, I need to update targets.yml file. I’ll add Node exporter‘s endpoints as in the following file

targets.yml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
- targets:
       - 127.0.0.1:8088 # Node exporter that runs on my Pi Zero
  labels:
       job: zero # Job name

- targets:
       - <IP-ADDRESS>:8088 # IP address of my Pi 3 that runs Node exporter
  labels:
       job: three

- targets:
       - 127.0.0.1:9090 # Prometheus that runs on my Pi Zero
  labels:
       job: zeroProm

After changing the targets.yml file Prometheus will reload it automatically. At this point, my raspberries are started being monitored and I didn’t actually need Grafana because Prometheus has the ability to show some graphs about collected metrics. But Grafana is much cooler.

Prometheus graph

Above is an example of showing the CPU temperature of my Pi Zero.

Grafana

A quote from Grafana’s GitHub[3] Repo:

The tool for beautiful monitoring and metric analytics & dashboards for Graphite, InfluxDB & Prometheus & More

Yea, it displays collected metrics in a beautiful way. Let’s install it.

I use it with docker[5] because it felt so easy because of this command docker run --name="grafana" -p 127.0.0.1:3000:3000 grafana/grafana :D I executed that command on my computer, I don’t need to install Grafana to my Raspberries.

Open your web browser and go to this address http://127.0.0.1:3000, it will ask you to log in and the default credentials are admin:admin.

Once you log-in, add Prometheus as a data source. After that go to Dashboards tab and import Prometheus dashboards. Importing Prometheus dashboards is not required but I think it’s inspirational.

Grafana add data source

You may want to create a dashboard from scratch as I did. I googled things a lot and I am still in the learning process.

Grafana add data source

Conclusion

Since I have some future-plans regarding clustering Raspberries I thought learning basic monitoring techniques is worth-to-try. Monitoring different subjects are limited by your imagination. For example, in the non-far-away-future I may want to monitor the electricity consumption of my house by using raspberry’s GPIO pins with some sensor devices, to do that I would need to learn how to write custom exporter code for Prometheus which is a good thing because I’ll learn something new.

I wanted to share this story-like blog post nevertheless. Mustafa is out.

Reference:

  1. 1.Prometheus ↩
  2. 2.Node exporter ↩
  3. 3.Grafana ↩
  4. 4.Getting started ↩
  5. 5.Grafana docker install ↩
  6. 6.Prometheus + Grafana image at the beginning is from blog.sakuragawa.moe/deploy-granafa-with-prometheus-and-oauth2-on-openshift this blog post. ↩