Match different server successful response, Fixed error log, Added date and time on logs

.gitignore

@@ -1 +1,2 @@
 *.cfg
+healthcheck/healthcheck-virtualenv

README.md

@@ -1,17 +1,37 @@
 # Selfhost utilities
+
 A collection of utilities for self hosters.
 Every utility is in a folder with its relevant configuration and is completely separated from the other, so you can install only the ones you need.
 
-## HEALTHCHECK
+## 🚨 HEALTHCHECK
+
 A simple server health check.
+Allows to keep under control the machine vitals (cpu usage, raid status, thermals...) and alert the sysadmin in case of anomalies.
+
 Sends an email and/or executes a command in case of alarm (high temperature, RAID disk failed etc...).
 As an example, the command may be a ntfy call to obtain a notification on a mobile phone or desktop computer.
 Meant to be run with a cron (see healthcheck.cron.example).
 Tested on Debian 11, but should run on almost any standard linux box.
 
-![Email](images/healthcheck_email_notification.png)      ![Ntfy](images/healthcheck_ntfy_notification.png)
+![Email](images/healthcheck_email_notification.png)
+
+![Ntfy](images/healthcheck_ntfy_notification.png)
 
 Please see [healthcheck documentation](healthcheck/README.md)
 
+## 🖥 MDDCLIENT
+
+A DynDns2 client supporting multiple domains with individual API calls. Developed to allow updating multiple (sub)domains on Infomaniak dynamic DNS, that supports only one domain per request. Works with any provider supporting DynDns2 protocol.
+
+Please see [mddclient documentation](mddclient/README.md)
+
+## 📟 ESP32-LCD
+
+A status LCD for your homelab! Low cost (about 20€ in parts), simple to build (no circuit board, no components, only an LCD, a ESP32 and, if needed, a potentiometer). Connects to your local wifi and receives HTTP requests from your homelab machines, and shows them to the screen.
+
+![ESP32-LCD prototype](images/esp32-lcd.jpg)
+
+Please see [ESP32-LCD documentation](esp32-lcd/README.md)
+
 # License
 This whole repository is released under GNU General Public License version 3: see http://www.gnu.org/licenses/

esp32-lcd/README.md

@@ -0,0 +1,56 @@
+# ESP32-LCD
+
+A status LCD for your homelab.
+
+![ESP32-LCD prototype](../images/esp32-lcd.jpg)
+
+## BOM
+
+- Any HD44780-based character LCD display, up to 20x4 characters. The most common are 16x2 characters (16 characters per row, 2 rows).
+- An ESP-32 board
+- A potentiometer, usually a 10 or 100k one, for controlling contrast (a couple of resistances arranged as a voltage divider may also be fine)
+
+## Assembly
+
+Connect the LCD to the board:
+
+**LCD Pin  ->  ESP32 Pin**
+
+- PIN01-VSS -> GND
+- PIN02-VDD -> 5V
+- PIN03 V0 ->  10K Pot (Middle pin)
+- PIN04 RS ->  GPIO19
+- PIN05 RW ->  GND
+- PIN06  E ->  GPIO23
+- PIN07 D0 ->  NOT USED
+- PIN08 D1 ->  NOT USED
+- PIN09 D2 ->  NOT USED
+- PIN10 D3 ->  NOT USED
+- PIN11 D4 ->  GPIO18
+- PIN12 D5 ->  GPIO17
+- PIN13 D6 ->  GPIO16
+- PIN14 D7 ->  GPIO15
+- PIN15 A  ->  5V
+- PIN16 K  ->  GND
+
+Connect the potentiometer lateral pins to VCC and GND. Use the potentiometer to set the screen contrast.
+
+Open config.h file and set display size and your wifi access data.
+
+Flash the code to the ESP32. If you use the Arduino ide to do it, just open the esp32-lcd.ino file with the Arduino ide and follow [this instructions](https://randomnerdtutorials.com/getting-started-with-esp32/)
+
+Restart the ESP32. The display shows "Conn to wifi..." with the WIFI name in the second line (if using a two or more lines display) and then will show the IP address.
+
+## Use
+
+- Turn on the circuit, wait for connection and note down the IP address shown on the screen.
+- Make a GET request to the same IP address with a parameter "message" containing some text
+
+> Example: to make the request using CURL from command line, try something along this lines (replace the IP addr with the one shown in the display):
+> `curl -G http://192.168.1.78 --data-urlencode "message=Something interesting happened!"`
+
+## Troubleshooting
+
+The ESP32 logs are written in the serial monitor at 115200 baud. Just open the Arduino ide Serial Monitor from Tools menu and look at the logs.
+
+If the screen is supplied with power but not initialized (maybe due to bad contacts or non working esp32 firmware), it should show some black blocks on the first line. If you canot see those (nor any other text), first of all check the contrast using the potentiometer.

esp32-lcd/esp32-lcd/.gitignore

@@ -0,0 +1,2 @@
+config.h
+

esp32-lcd/esp32-lcd/config.h.example

@@ -0,0 +1,15 @@
+/**
+	CONFIGURATION FILE
+	Change the values in this file and rename it "config.h" before uploading the code to the board.
+*/
+
+#ifndef CONFIG_H
+#define CONFIG_H
+
+const char* WIFI_SSID = "Squicky";
+const char* WIFI_PASSWORD = "SediaChinita@Terrazzo2017";
+const unsigned int DISPLAY_WIDTH = 16;
+const unsigned int DISPLAY_HEIGHT = 2;
+
+#endif
+

esp32-lcd/esp32-lcd/esp32-lcd.ino

@@ -0,0 +1,129 @@
+#include <WiFi.h>
+#include <WiFiClient.h>
+#include <WebServer.h>
+#include <ESPmDNS.h>
+#include <LiquidCrystal.h>
+#include "config.h"
+
+// ------- Configuration is in config.h file -------
+
+const int WEBSERVER_PORT = 80;
+const char* WEBSERVER_MESSAGE_PARAM = "message";
+
+/*
+ 
+LCD Pin –>ESP32 Pins
+
+    PIN01-VSS -> GND
+    PIN02-VDD -> 5V
+    PIN03 V0-> 10K Pot (Middle pin)
+    PIN04 RS->  GPIO19
+    PIN05 RW-> GND
+    PIN06  E  ->  GPIO23
+    PIN07 D0-> NOT USED
+    PIN08 D1-> NOT USED
+    PIN09 D2-> NOT USED
+    PIN10 D3-> NOT USED
+    PIN11 D4->  GPIO18
+    PIN12 D5->  GPIO17
+    PIN13 D6->  GPIO16
+    PIN14 D7->  GPIO15
+    PIN15 A-> 5V
+    PIN16 K-> GND
+
+ */
+
+WebServer server(WEBSERVER_PORT); // Server on port 80
+LiquidCrystal lcd(19, 23, 18, 17, 16, 15);
+const int led = 13;
+
+void lcdPrintMultilineMessage(String message) {
+  lcd.clear();
+  int startFrom = 0;
+  for (int i=0; i<DISPLAY_HEIGHT; i++) {
+    lcd.setCursor(0,i);
+    lcd.print(
+      message.substring(startFrom, startFrom + DISPLAY_WIDTH)
+    );
+    startFrom += DISPLAY_WIDTH;
+  }
+}
+
+void handleRoot() {
+  digitalWrite(led, 1);
+  lcdPrintMultilineMessage(server.arg(WEBSERVER_MESSAGE_PARAM));
+  server.send(200, "text/plain", "ok");
+  digitalWrite(led, 0);
+}
+
+void handleNotFound() {
+  digitalWrite(led, 1);
+  String message = "File Not Found\n\n";
+  message += "URI: ";
+  message += server.uri();
+  message += "\nMethod: ";
+  message += (server.method() == HTTP_GET) ? "GET" : "POST";
+  message += "\nArguments: ";
+  message += server.args();
+  message += "\n";
+  for (uint8_t i = 0; i < server.args(); i++) {
+    message += " " + server.argName(i) + ": " + server.arg(i) + "\n";
+  }
+  server.send(404, "text/plain", message);
+  digitalWrite(led, 0);
+}
+
+void setup(void) {
+  // LCD: set up
+  lcd.begin(DISPLAY_WIDTH, DISPLAY_HEIGHT);
+
+  // SERIAL: set up
+  Serial.begin(115200);
+  Serial.println("");
+
+  // STATUS LED: set up
+  pinMode(led, OUTPUT);
+  digitalWrite(led, 0);
+  
+  // LCD: Show SSID
+  lcd.print("Conn to wifi...");
+  lcd.setCursor(0,1);
+  lcd.print(WIFI_SSID);
+
+  // Connect to wifi
+  WiFi.mode(WIFI_STA);
+  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
+
+  // Wait for connection
+  while (WiFi.status() != WL_CONNECTED) {
+    delay(500);
+    Serial.print(".");
+  }
+  Serial.println("");
+  Serial.print("Connected to ");
+  Serial.println(WIFI_SSID);
+  Serial.print("IP address: ");
+  Serial.println(WiFi.localIP());
+
+  // Print IP addr to LCD
+  lcd.clear();
+  lcd.print("Connected! IP:");
+  lcd.setCursor(0,1);
+  lcd.print(WiFi.localIP());
+
+  if (MDNS.begin("esp32")) {
+    Serial.println("MDNS responder started");
+  }
+
+  server.on("/", handleRoot);
+  server.onNotFound(handleNotFound);
+
+  server.begin();
+  Serial.println("HTTP server started");
+
+}
+
+void loop(void) {
+  server.handleClient();
+  delay(2);//allow the cpu to switch to other tasks
+}

healthcheck/README.md

@@ -1,11 +1,15 @@
-# HEALTHCHECK
+# 🚨 HEALTHCHECK
 A simple server health check.
+Allows to keep under control the machine vitals (cpu usage, raid status, thermals...) and alert the sysadmin in case of anomalies.
+
 Sends an email and/or executes a command in case of alarm.
 As an example, the command may be a ntfy call to obtain a notification on a mobile phone or desktop computer.
 Meant to be run with a cron (see healthcheck.cron.example).
 Tested on Debian 11, but should run on almost any standard linux box.
 
-![Email](../images/healthcheck_email_notification.png)      ![Ntfy](../images/healthcheck_ntfy_notification.png)
+![Email](../images/healthcheck_email_notification.png)
+
+![Ntfy](../images/healthcheck_ntfy_notification.png)
 
 ## Alarms
 Provided ready-to-use alarms in config file:
@@ -40,9 +44,13 @@ Copy the script and the config file into the system to check:
 cp healthcheck.py /usr/local/bin/healthcheck.py
 cp healthcheck.cfg.example /usr/local/etc/healthcheck.cfg
 ```
+Make the script executable:
+```
+chmod +x /usr/local/bin/healthcheck.py
+```
 Edit `/usr/local/etc/healthcheck.cfg` enabling the checks you need and configuring email settings.
 Run `/usr/local/bin/healthcheck.py /usr/local/etc/healthcheck.cfg` to check it is working. If needed, change the config to make a check fail and see if the notification mail is delivered. If you need to do some testing without spamming emails, run with the parameter `--dry-run`.
-Now copy the cron file:
+Now copy the cron file (it runs healthcheck every minute):
 ```
 cp healthcheck.cron.example /etc/cron.d/healthcheck
 ```

healthcheck/healthcheck.cfg.example

@@ -1,3 +1,5 @@
+# The DEFAULT section contains the global configuration applied to all checks.
+# You can re-define this variables in a check to override the global one.
 [DEFAULT]
 
 #### EMAIL NOTIFICATIONS ####
@@ -39,10 +41,27 @@ MAILTO=root@localhost, user@localhost
 #ALARM_COMMAND=curl -H "%%CHECKNAME%% alarm on %%HOSTNAME%%" -d "%%ERROR%% on %%DATETIME%%" ntfy.sh/my-unique-topic-name
 
 
+#### NOTIFICATION POLICY ###
+# Defines when to send the email and/or execute ALARM_COMMAND. Useful to avoid email flooding.
+# Possible values:
+# EVERY_RUN In case of alarm, sends a mail every time the script is run
+# START Sends a mail only when an alarm starts
+# ONCE_IN_MINUTES In case of alarm, resends a mail only if NOTIFY_MINUTES has passed
+NOTIFY=EVERY_RUN
+
+# Used only if NOTIFY=ONCE_IN_MINUTES. A mail is sent only if NOTIFY_MINUTES has passed from the previous one
+NOTIFY_MINUTES=60
+
+# Sends a mail when the alarm has ended
+NOTIFY_ALARM_END=TRUE
+
+
 #### HEALTH CHECKS ####
 # Every health check is based on a command being executed, its result being parsed with a regexp
 # to extract (as a single group) the numeric or string value, and the value being compared with
 # a configured value. This checks are ready to be used, just enable the ones you need.
+#
+# CUSTOM CHECKS:
 # You can add your own custom check declaring another section like this:
 #
 # [my_custom_check_name]
@@ -55,28 +74,37 @@ MAILTO=root@localhost, user@localhost
 # ALARM_VALUE_LESS_THAN=12
 # COMMAND=/my/custom/binary --with parameters
 # REGEXP=my regex to parse (awesome|disappointing) command output
+#
+# First test your custom command executing it in the command line
+# Take the text output and write a regex to match it. Check every case:
+# success result, error result, command failure. Then paste the command
+# and regex in this config, enable the check and run to verify is working.
+
 
 [system_load_1min]
 # The system load average in the last minute
-DISABLED=True
+DISABLED=False
 ALARM_VALUE_MORE_THAN=1.0
 COMMAND=uptime
 REGEXP=.*load average: (\d+[,.]\d+), \d+[,.]\d+, \d+[,.]\d+
 
+
 [system_load_5min]
 # The system load average in the last 5 minutes
-DISABLED=True
+DISABLED=False
 ALARM_VALUE_MORE_THAN=1.0
 COMMAND=uptime
 REGEXP=.*load average: \d+[,.]\d+, (\d+[,.]\d+), \d+[,.]\d+
 
+
 [system_load_15min]
 # The system load average in the last 15 minutes
-DISABLED=True
+DISABLED=False
 ALARM_VALUE_MORE_THAN=1.0
 COMMAND=uptime
 REGEXP=.*load average: \d+[,.]\d+, \d+[,.]\d+, (\d+[,.]\d+)
 
+
 [used_disk_space]
 # Used disk space (in percent, i.e. ALARM_VALUE_MORE_THAN=75 -> alarm if disk is more than 75% full)
 DISABLED=True
@@ -84,6 +112,7 @@ ALARM_VALUE_MORE_THAN=75
 COMMAND=df -h /dev/sda1
 REGEXP=(\d{1,3})%
 
+
 [raid_status]
 # Issues an alarm when the raid is corrupted
 # Checks this part of the /proc/mdstat file:
@@ -95,6 +124,7 @@ ALARM_STRING_NOT_EQUAL=UU
 COMMAND=cat /proc/mdstat
 REGEXP=.*\] \[([U_]+)\]\n
 
+
 [battery_level]
 # Issues an alarm when battery is discharging below a certain level (long blackout, pulled power cord...)
 # For laptops used as servers, apparently common among the self hosters. Requires acpi package installed.
@@ -104,6 +134,7 @@ COMMAND=acpi -b
 REGEXP=Battery \d: .*, (\d{1,3})%
 ALARM_VALUE_LESS_THAN=90
 
+
 [laptop_charger_disconnected]
 # Issues an alarm when laptop charger is disconnected
 # For laptops used as servers, apparently common among the self hosters. Requires acpi package installed.
@@ -112,20 +143,29 @@ COMMAND=acpi -a
 REGEXP=Adapter \d: (.+)
 ALARM_STRING_EQUAL=off-line
 
-[free_ram]
-# Free ram in %
-# Shows another approach: does all the computation in the command and picks up
-# all the output (by not declaring a regexp).
+
+[shutdown_on_battery_low]
+# For laptops used as a a server. Requires acpi package installed.
+# When the battery is low, shuts down cleanly the system instead of waiting for it
+# to shut down itself leaving all filesystems dirty.
+# ALARM_COMMAND is the command executed when this check fails. Shuts down the system in
+# 15 mins to allow for logging in and cancel the command. If you want to shut down
+# immediately, replace the ALARM_COMMAND with "shutdown now".
+# To cancel the shutdown, log in and "shutdown -c".
 DISABLED=True
-COMMAND=free | grep Mem | awk '{print int($4/$2 * 100.0)}'
-ALARM_VALUE_LESS_THAN=20
+COMMAND=acpi -b
+REGEXP=REGEXP=Battery \d: Discharging, (\d{1,3})%
+ALARM_VALUE_LESS_THAN=50
+ALARM_COMMAND=shutdown +15 "Shutdown in 15 mins due to battery low!"
+
 
 [available_ram]
-# Like Free ram, but shows available instead of free. You may want to use this if you use a memcache.
-DISABLED=True
+# Shows available ram in %.
+DISABLED=False
 COMMAND=free | grep Mem | awk '{print int($7/$2 * 100.0)}'
 ALARM_VALUE_LESS_THAN=20
 
+
 [cpu_temperature]
 # CPU Temperature alarm: requires lm-sensors installed and configured (check your distribution's guide)
 # The regexp must be adapted to your configuration: run `sensors` in the command line
@@ -136,6 +176,7 @@ ALARM_VALUE_MORE_THAN=80
 COMMAND=sensors
 REGEXP=Core 0: +\+?(-?\d{1,3}).\d°[CF]
 
+
 [fan_speed]
 # Fan speed alarm: requires lm-sensors installed and configured (check your distribution's guide)
 # The regexp must be adapted to your configuration: run `sensors` in the command line
@@ -143,4 +184,80 @@ REGEXP=Core 0: +\+?(-?\d{1,3}).\d°[CF]
 DISABLED=True
 ALARM_VALUE_LESS_THAN=300
 COMMAND=sensors
-REGEXP=cpu_fan: +(\d) RPM
+REGEXP=cpu_fan: +(\d+) RPM
+
+
+[host_reachability]
+# Check if a remote host is alive with Ping. You can replace the ip with a domain name (e.g. COMMAND=ping debian.org -c 1)
+#
+# Shows another approach: uses the return value to print a string. Leverages ping's ability to return different error codes:
+# 0 = success
+# 1 = the host is unreachable
+# 2 = an error has occurred (and will be logged to stderr)
+# We are throwing away stdout and replacing it with a custom text.
+# If there is a different text (the stderr), something bad happened, and it will be reported in the mail.
+DISABLED=True
+ALARM_STRING_NOT_EQUAL=Online
+COMMAND=ping 192.168.1.123 -c 1 > /dev/null && echo "Online" || echo "Offline"
+
+
+[service_webserver]
+# Check if a webserver is running on port 80. You can replace the ip with a domain name.
+# You can check different services changing the port number. Some examples:
+# 80 HTTP Webserver
+# 443 HTTPS Webserver
+# 21 FTP
+# 22 SSH
+# 5900 VNC (Linux remote desktop)
+# 3389 RDP (Windows remote desktop)
+DISABLED=True
+ALARM_STRING_NOT_EQUAL=Online
+COMMAND=nc -z -w 3 192.168.1.123 80 > /dev/null && echo "Online" || echo "Offline"
+
+
+[dummy_always_alarm]
+# A dummy check that is always in alarm. Useful for testing notifications.
+DISABLED=True
+ALARM_STRING_EQUAL=Core meltdown!
+COMMAND=echo "Core meltdown!"
+
+
+[security_updates_available]
+# Checks for security updates via apt (works on Debian and derivatives, like Ubuntu).
+# Needs the repositories to be updated with `apt update`, but is an heavy command, so it may
+# be configured to be executed daily in a command in the same cron of healthcheck.
+# E.g.: place this string in /etc/cron.d/healthcheck, before the healthcheck command:
+# 	1 1 * * *       root    apt update
+DISABLED=True
+ALARM_STRING_EQUAL=security updates available
+REGEXP=(security updates available|NO security updates available)
+COMMAND=apt list --upgradable 2>/dev/null | grep -e "-security" && echo "security updates available" || echo "NO security updates available"
+NOTIFY=START
+
+
+[ups_power]
+# Raises an alarm when UPS runs on battery.
+# Requires NUT installed and configured on the system
+# See complete documentation and support lists: https://networkupstools.org
+# See simple start-up guide for Debian: https://wiki.debian.org/nut
+# This config is for usbhid-ups driver. If you use a different driver, you may need
+# to change the REGEXP to fit your output.
+DISABLED=True
+ALARM_STRING_NOT_EQUAL=OL
+COMMAND=upsc eaton1600 2> /dev/null
+REGEXP=^ups\.status: (OL|OB)$
+NOTIFY=START
+
+[ups_battery]
+# Raises an alarm when UPS battery is discharged below 50%.
+# Requires NUT installed and configured on the system
+# See complete documentation and support lists: https://networkupstools.org
+# See simple start-up guide for Debian: https://wiki.debian.org/nut
+# This config is for usbhid-ups driver. If you use a different driver, you may need
+# to change the REGEXP to fit your output.
+DISABLED=True
+ALARM_VALUE_LESS_THAN=50
+COMMAND=upsc eaton1600 2> /dev/null
+REGEXP=^battery\.charge: (\d{1,3})$
+NOTIFY=ONCE_IN_MINUTES
+NOTIFY_MINUTES=15

healthcheck/healthcheck.py

@@ -46,13 +46,18 @@ import socket
 import getpass
 import re
 import locale
+import json
 
 
 NAME = 'healthcheck'
 VERSION = '0.1'
 DESCRIPTION = 'A simple server monitoring software'
-EMAIL_SUBJECT_TPL = 'Host {} failed health check for {}'
-EMAIL_MESSAGE_TPL = 'Alarm for sensor {} on host {} on {}: {}'
+EMAIL_START_SUBJECT_TPL = '\U0001F6A8 {}: {} ALARM!'
+EMAIL_START_MESSAGE_TPL = 'Alarm for sensor {} on host {} on {}: {}'
+EMAIL_END_SUBJECT_TPL = '\u2705 {}: {} OK'
+EMAIL_END_MESSAGE_TPL = 'Alarm ceased for sensor {} on host {} on {}'
+# Healthcheck saves the current status (alarms triggered, last run... in this file)
+STATUS_FILE = '/tmp/healthcheck.tmp'
 
 class Main:
 
@@ -78,6 +83,10 @@ class Main:
 	def run(self, dryRun):
 		''' Runs the health checks '''
 
+		# Load status
+		status = Status()
+
+		# Run checks based o the config
 		for section in self.config:
 			if section == 'DEFAULT':
 				continue
@@ -85,6 +94,7 @@ class Main:
 			s = Settings(section, self.config)
 			if s.disabled:
 				self._log.info('Ignoring disabled check "{}"'.format(section))
+				status.unsetAlarm(section)
 				continue
 
 			self._log.info('Checking "{}"'.format(section))
@@ -93,11 +103,37 @@ class Main:
 			if error:
 				# Alarm!
 				logging.warning('Alarm for {}: {}!'.format(section, error))
+				if self.shouldNotify(section, s, status):
+					status.setAlarm(section)
 					if not dryRun:
 						if s.mailto:
-						self.sendMail(s, error)
+							self.sendAlmStartMail(s, error)
 						if s.alarmCommand:
 							self.executeAlarmCommand(s, error)
+			elif status.getAlarmTriggeredTimestamp(section) is not None:
+				logging.info('Alarm ceased for {}: OK!'.format(section))
+				if s.notify_alarm_end and not dryRun and s.mailto:
+					self.sendAlmEndMail(s)
+				status.unsetAlarm(section)
+
+		# Save updated status
+		status.save()
+
+	def shouldNotify(self, section, settings, status):
+		almTriggeredTime = status.getAlarmTriggeredTimestamp(section)
+		# Notify if alarm just started
+		if almTriggeredTime is None:
+			return True
+
+		# Notify if NOTIFY=EVERY_RUN
+		if settings.notify == 'EVERY_RUN':
+			return True
+
+		# Notify if time elapsed
+		if settings.notify == 'ONCE_IN_MINUTES' and (time.time() - almTriggeredTime) > (settings.notify_minutes * 60):
+			return True
+
+		return False
 
 	# Calls the provided command, checks the value parsing it with the provided regexp
 	# and returns an error string, or null if the value is within its limits
@@ -112,12 +148,15 @@ class Main:
 		stdout = ""
 		ret = subprocess.run(config.command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
 		if ret.stderr:
-			self._log.info('{} subprocess stderr:\n{}', config.command, ret.stderr.decode())
+			self._log.info('{} subprocess stderr:\n{}'.format(config.command, ret.stderr.decode()))
 		if ret.stdout:
 			stdout = ret.stdout.decode()
-			self._log.debug('{} subprocess stdout:\n{}', config.command, stdout)
+			self._log.debug('{} subprocess stdout:\n{}'.format(config.command, stdout))
 		if ret.returncode != 0:
-			return 'subprocess {} exited with error code {}'.format(config.command, ret.returncode)
+			return 'the command exited with error code {} {}'.format(
+				ret.returncode,
+				'and error message "{}"'.format(ret.stderr.decode().strip()) if ret.stderr else ''
+			)
 
 		# Parse result with regex
 		match = re.search(config.regexp, stdout, re.MULTILINE)
@@ -143,15 +182,34 @@ class Main:
 		if config.alarm_value_less_than and locale.atof(detectedValue) < float(config.alarm_value_less_than):
 			return 'value is {}, but should be greater than {}'.format(locale.atof(detectedValue), config.alarm_value_less_than)
 
-	def sendMail(self, s, error):
+	def sendAlmStartMail(self, s, error):
+		subject = EMAIL_START_SUBJECT_TPL.format(self.hostname, s.name)
+		body = EMAIL_START_MESSAGE_TPL.format(
+			s.name,
+			self.hostname,
+			time.strftime("%a, %d %b %Y %H:%M:%S"),
+			error
+		)
+		self.sendMail(s, subject, body)
+
+	def sendAlmEndMail(self, s):
+		subject = EMAIL_END_SUBJECT_TPL.format(self.hostname, s.name)
+		body = EMAIL_END_MESSAGE_TPL.format(
+			s.name,
+			self.hostname,
+			time.strftime("%a, %d %b %Y %H:%M:%S")
+		)
+		self.sendMail(s, subject, body)
+
+	def sendMail(self, s, subject, body):
 		if s.smtphost:
-			logging.info("Sending alarm email to %s via %s", s.mailto, s.smtphost)
+			logging.info("Sending email to %s via %s", s.mailto, s.smtphost)
 		else:
-			logging.info("Sending alarm email to %s using local smtp", s.mailto)
+			logging.info("Sending email to %s using local smtp", s.mailto)
 
 		# Create main message
 		msg = MIMEMultipart()
-		msg['Subject'] = EMAIL_SUBJECT_TPL.format(self.hostname, s.name)
+		msg['Subject'] = subject
 		if s.mailfrom:
 			m_from = s.mailfrom
 		else:
@@ -161,12 +219,6 @@ class Main:
 		msg.preamble = 'This is a multi-part message in MIME format.'
 
 		# Add base text
-		body = EMAIL_MESSAGE_TPL.format(
-			s.name,
-			self.hostname,
-			time.strftime("%a, %d %b %Y %H:%M:%S"),
-			error
-		)
 		txt = MIMEText(body)
 		msg.attach(txt)
 
@@ -204,6 +256,34 @@ class Main:
 			self._log.error('subprocess {} exited with error code {}'.format(cmdToRun, ret.returncode))
 
 
+class Status:
+	''' Represents the current status (alarms triggered, last run...) '''
+
+	def __init__(self):
+		try:
+			with open(STATUS_FILE, 'r') as openfile:
+				self.status = json.load(openfile)
+		except FileNotFoundError:
+			self.status =  {
+				'lastRun': 0,	# unix time in seconds
+				'alarms': {},	# key-value, alarmName : alarmTriggeredTimestamp
+			}
+
+	def save(self):
+		self.status['lastRun'] = time.time()
+		jo = json.dumps(self.status)
+		with open(STATUS_FILE, "w") as outfile:
+			outfile.write(jo)
+
+	def setAlarm(self, almName):
+		self.status['alarms'][almName] = time.time()
+
+	def unsetAlarm(self, almName):
+		self.status['alarms'].pop(almName, None)
+
+	def getAlarmTriggeredTimestamp(self, almName):
+		return self.status['alarms'].get(almName, None)
+
 
 class Settings:
 	''' Represents settings for a check '''
@@ -241,6 +321,10 @@ class Settings:
 		self.alarm_value_not_equal = self.getStr(name, 'ALARM_VALUE_NOT_EQUAL', None)
 		self.alarm_value_more_than = self.getStr(name, 'ALARM_VALUE_MORE_THAN', None)
 		self.alarm_value_less_than = self.getStr(name, 'ALARM_VALUE_LESS_THAN', None)
+		## Notification policy
+		self.notify = self.getEnum(name, 'NOTIFY', 'EVERY_RUN', ['EVERY_RUN', 'START', 'ONCE_IN_MINUTES'])
+		self.notify_minutes = self.getInt(name, 'NOTIFY_MINUTES', 0)
+		self.notify_alarm_end = self.getBoolean(name, 'NOTIFY_ALARM_END', True)
 		## Command to obtain the value for comparation
 		self.command = self.getStr(name, 'COMMAND', None)
 		## Regexp to extract value from command output (default to match full string)
@@ -252,12 +336,20 @@ class Settings:
 		except configparser.NoOptionError:
 			return defaultValue
 
+	def getInt(self, name, key, defaultValue):
+		return int(self.getStr(name, key, defaultValue))
+
 	def getBoolean(self, name, key, defaultValue):
 		try:
 			return self.config.getboolean(name, key)
 		except configparser.NoOptionError:
 			return defaultValue
 
+	def getEnum(self, name, key, defaultValue, values):
+		val = self.getStr(name, key, defaultValue)
+		if not val in values:
+			raise ValueError("Invalid value {} for configuration {}: expected one of {}".format(val, key, ', '.join(values)))
+		return val
 
 
 if __name__ == '__main__':

mddclient/README.md

@@ -0,0 +1,53 @@
+# 🖥 MDDCLIENT
+It's a client for `dyndns2` protocol: it is used to interact with dynamic DNS services like Dyn or Infomaniak's.
+Updates the IP assigned for a domain, allows the use of dynamic ip internet access (commin in residential contexts like DSL, Fiber, 5G...) for hosting services.
+Like [ddclient](https://github.com/ddclient/ddclient), but updates multiple domaing with multiple REST calls.
+
+I developed this because [Infomaniak's DynDns APIS](https://www.infomaniak.com/en/support/faq/2376/dyndns-updating-a-dynamic-dns-via-the-api) doesn't support multiple domains updates on the same call. So, even if ddclient was supported for a single domain, in case of multiple (sub)domains it was needed to setup multiple instances to update them all on the same machine.
+
+## Compatibility
+Works for any provider supporting `dyndns2` protocol. Was implemented following (this documentation)[https://help.dyn.com/remote-access-api/].
+I wrote this to solve the multiple domain update problem on Infomainak.
+
+## Use case
+Let's say we have our Nextcloud instance on `https://mysite.cloud`. As self hosters, we run this instance on our server at home, behind our fiber or DSL provider with a dynamic IP. We need to use ddclient to keep the dynamic DNS updated, so the requests to `https://mysite.cloud` are sent to our current IP, that may change in any moment.
+Now we decide to host a Matrix chat instance on `https://matrix.mysite.cloud` and a Mastodon social instance on `https://mastodon.mysite.cloud`. We configure ddclient adding the new subdomains in `/etc/ddclient.conf`, but the requests begin to fail with a 400 http code. This happens because our Dynamic DNS provider doesn't support multiple updates on the same request.
+
+So we need a script that issues an update request for every (sub)domain. This is mddclient.
+
+## Setup
+Copy the script and the config file into the system to check:
+```
+cp mddclient.py /usr/local/bin/mddclient.py
+cp mddclient.cfg.example /usr/local/etc/mddclient.cfg
+```
+Make the script executable:
+```
+chmod +x /usr/local/bin/mddclient.py
+```
+Edit `/usr/local/etc/mddclient.cfg` setting up the server url, username, password and the domains to update. If you have multiple domains or subdomains running on the same fiber/dsl, configure them, one for subsection as shown in the config example.
+Run `/usr/local/bin/mddclient.py /usr/local/etc/mddclient.cfg` to check it is working.
+Now copy the cron file (it runs mddclient every 5 minutes):
+```
+cp mddclient.cron.example /etc/cron.d/mddclient
+```
+For increased safety, edit the cron file placing your email address in MAILTO var to be notified in case of mddclient.py catastrophic failure.
+
+Setup is now complete: the cron runs the script every five minutes and updates the dns.
+
+## Check status
+Some status informations are available with the -s flag:
+```
+root@myhost# /usr/local/bin/mddclient.py /usr/local/etc/mddclient.cfg -s
+
+lastRun: 2022-05-31 10:40:17.006371
+lastRunSuccess: True
+lastUpdate: 2022-05-31 10:23:38.510386
+lastIpAddr: 151.41.52.133
+```
+
+## Optimization
+As the update requests are subject to rate limit, the script checks the current IP against Dyn's checkip tool and updates only when necessary. To force an update, use the -f flag.
+
+## Thanks
+Thanks to `dyndns.org` for the (checkip)[https://help.dyn.com/remote-access-api/checkip-tool/] tool returning current public IP address.

mddclient/mddclient.cfg.example

@@ -0,0 +1,28 @@
+# The DEFAULT section contains the config common to all subsections.
+# Redefining a value in a subsection overrides the DEFAULT's one.
+[DEFAULT]
+
+# Dynamic DNS Server
+SERVER=infomaniak.com
+LOGIN=myUserName
+PASSWORD=mySuperSecretPassword
+
+[mysite]
+# Main domain
+DOMAIN=mysite.cloud
+
+[matrix]
+# The matrix.mysite.cloud subdomain
+DOMAIN=matrix.mysite.cloud
+
+[mastodon]
+# The mastodon.mysite.cloud subdomain
+DOMAIN=mastodon.mysite.cloud
+
+[mysite2]
+# Another domain, but this is provided by another dynamicdns service
+# so we re-define server, login and password to override the DEFAULT ones
+SERVER=anotherdyndns.com
+LOGIN=anotherUserName
+PASSWORD=anotherSuperSecretPassword
+DOMAIN=mysite2.io

mddclient/mddclient.cron.example

@@ -0,0 +1,6 @@
+# Cron to execute mddclient
+# As a security measure, the address in MAILTO will be notified
+# if the mddclient.py script crashes.
+
+MAILTO="your-email-address"
+*/5 * * * *	root	/usr/local/bin/mddclient.py /usr/local/etc/mddclient.cfg -q >> /var/log/mddclient.log 2>&1

mddclient/mddclient.py

@@ -0,0 +1,282 @@
+#!/usr/bin/env python3
+
+""" @package docstring
+Mddclient
+
+A DynamicDns client like ddclient, but supporting multiple (sub)domains,
+every one in its autonomous request.
+
+Installation:
+- Copy mddclient.cfg in /usr/local/etc/mddclient.cfg and customize it
+- Copy mddclient.py in /usr/local/bin/mddclient.py
+
+Usage:
+Place a cron entry like this one:
+
+* * * * *     root    python3 /usr/local/bin/mddclient.py /usr/local/etc/mddclient.cfg
+
+Exit codes:
+0: success
+1: unmanaged error (crash)
+2: managed error (could not update due to bad config/server error)
+
+For more informations on dyndns2 protocol: https://help.dyn.com/remote-access-api/
+
+@author Daniele Verducci <daniele.verducci@ichibi.eu>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.
+"""
+
+import os
+import sys
+import logging
+import configparser
+import traceback
+import requests
+import re
+import datetime
+import json
+
+
+NAME = 'mddclient'
+VERSION = '0.2'
+DESCRIPTION = 'A DynamicDns client like ddclient, but supporting multiple (sub)domains'
+STATUS_FILE = '/tmp/mddclient.tmp'
+CHECKIP_REQUEST_ADDR = 'http://checkip.dyndns.org'
+CHECKIP_RESPONSE_PARSER = '<body>Current IP Address: (\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})</body>'
+DDCLIENT2_REQUEST_ADDR = "https://{}/nic/update?system=dyndns&hostname={}&myip={}"
+DDCLIENT2_RESPONSE_PARSER = '^(nochg|no_change|good) (\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})$'
+USER_AGENT = 'Selfhost Utils Mddclient ' + VERSION
+
+class Main:
+
+	def __init__(self, configPath):
+		''' Reads the config '''
+		self._log = logging.getLogger('main')
+
+		if not os.path.exists(configPath) or not os.path.isfile(configPath):
+			raise ValueError('configPath must be a file')
+
+		self.config = configparser.ConfigParser()
+		self.config.read(configPath)
+
+	def run(self, force, printStatusAndExit):
+		''' Makes the update requests '''
+
+		# Load status
+		status = Status()
+
+		if printStatusAndExit:
+			status.print()
+			return True
+
+		# Check current ip
+		currentIp = self.getCurrentIp()
+		if (currentIp == None):
+			return False
+
+		self._log.info('Current ip is {}'.format(currentIp))
+
+		if currentIp == status.getIp():
+			self._log.info('Ip is up-to-date.')
+			if force:
+				self._log.info('User requested forced refresh.')
+			else:
+				self._log.info('Nothing to do.')
+				status.save(True, False)
+				return
+
+		success = True
+		updated = False
+		for section in self.config:
+			s = Settings(section, self.config)
+			if s.name == 'DEFAULT':
+				continue
+
+			self._log.info('Updating "{}"'.format(section))
+			try:
+				newIpAddr = self.update(s.ddserver, s.dduser, s.ddpass, s.domain, currentIp)
+				self._log.info('Success update {} to addr {}'.format(s.domain, newIpAddr))
+				updated = True
+			except Exception as e:
+				self._log.error('Error while updating {}: {}'.format(s.domain, e))
+				success = False
+
+		# Save current ip
+		if success:
+			status.setIp(currentIp)
+
+		status.save(success, updated)
+
+		return success
+
+	def getCurrentIp(self):
+		'''Obtains current IP from checkip.dyndns.org'''
+		response = requests.get(CHECKIP_REQUEST_ADDR)
+
+		match = re.search(CHECKIP_RESPONSE_PARSER, response.text, re.MULTILINE)
+		if not match:
+			self._log.error('Unable to obtain new IP addr: Response format not valid: {}'.format(response.text))
+			return
+		groups = match.groups()
+		if len(groups) != 1:
+			self._log.error('Unable to obtain new IP addr: Unable to parse response: {}'.format(response.text))
+			return
+
+		return groups[0]
+
+	def update(self, server, user, password, domain, ip):
+		apiUrl = DDCLIENT2_REQUEST_ADDR.format(server, domain, ip)
+		try:
+			response = requests.get(apiUrl, auth=(user, password), headers={"User-Agent": USER_AGENT})
+		except requests.ConnectionError:
+			raise Exception('Server {} is unreachable'.format(server))
+
+		match = re.search(DDCLIENT2_RESPONSE_PARSER, response.text)
+		if not match:
+			raise Exception('Response format not valid: {}'.format(response.text))
+		groups = match.groups()
+		if len(groups) != 2:
+			raise Exception('Unable to parse response: {}'.format(response.text))
+
+		operationResult = groups[0]
+		ipAddr = groups[1]
+
+		# Check operation result and return appropriate errors
+		if operationResult == 'good':
+			# Success!
+			return ipAddr
+		elif operationResult == 'nochg' or operationResult == 'no_change':
+			# Should not happen: IP didn't need update
+			self._log.warning('Ip addres didn\'t need update: this should happen only at first run')
+			return ipAddr
+		elif operationResult == 'badauth':
+			raise Exception('The username and password pair do not match a real user')
+		elif operationResult == '!donator':
+			raise Exception('Option available only to credited users, but the user is not a credited user')
+		elif operationResult == 'notfqdn':
+			raise Exception('The hostname specified is not a fully-qualified domain name (not in the form hostname.dyndns.org or domain.com).')
+		elif operationResult == 'nohost':
+			raise Exception('The hostname specified does not exist in this user account')
+		elif operationResult == 'numhost':
+			raise Exception('Too many hosts specified in an update')
+		elif operationResult == 'abuse':
+			raise Exception('The hostname specified is blocked for update abuse')
+		elif operationResult == 'badagent':
+			raise Exception('The user agent was not sent or HTTP method is not permitted')
+		elif operationResult == 'dnserr':
+			raise Exception('DNS error encountered')
+		elif operationResult == '911':
+			raise Exception('There is a problem or scheduled maintenance on server side')
+		else:
+			raise Exception('Server returned an unknown result code: {}'.format(operationResult))
+
+
+class Status:
+	''' Represents the current status '''
+
+	def __init__(self):
+		try:
+			with open(STATUS_FILE, 'r') as openfile:
+				self.status = json.load(openfile)
+		except FileNotFoundError:
+			self.status =  {
+				'lastRun': None,
+				'lastRunSuccess': None,
+				'lastUpdate': None,
+				'lastIpAddr': None,
+			}
+
+	def save(self, success, updated):
+		self.status['lastRun'] = str(datetime.datetime.now())
+		self.status['lastRunSuccess'] = success
+		if updated:
+			self.status['lastUpdate'] = str(datetime.datetime.now())
+		jo = json.dumps(self.status)
+		with open(STATUS_FILE, "w") as outfile:
+			outfile.write(jo)
+
+	def setIp(self, ip):
+		self.status['lastIpAddr'] = ip
+
+	def getIp(self):
+		return self.status['lastIpAddr']
+
+	def print(self):
+		for k in self.status:
+			print('{}: {}'.format(k, self.status[k]))
+
+
+class Settings:
+	''' Represents settings for a domain '''
+
+	def __init__(self, name, config):
+		self.config = config
+
+		## Section name
+		self.name = name
+
+		## DynDNS server data
+		self.ddserver = self.getStr(name, 'SERVER', None)
+		self.dduser = self.getStr(name, 'LOGIN', None)
+		self.ddpass = self.getStr(name, 'PASSWORD', None)
+
+		## Domain to update
+		self.domain = self.getStr(name, 'DOMAIN', False)
+
+	def getStr(self, name, key, defaultValue):
+		try:
+			return self.config.get(name, key)
+		except configparser.NoOptionError:
+			return defaultValue
+
+	def getBoolean(self, name, key, defaultValue):
+		try:
+			return self.config.getboolean(name, key)
+		except configparser.NoOptionError:
+			return defaultValue
+
+
+if __name__ == '__main__':
+	import argparse
+
+	parser = argparse.ArgumentParser(
+		prog = NAME + '.py',
+		description = NAME + ' ' + VERSION + '\n' + DESCRIPTION,
+		formatter_class = argparse.RawTextHelpFormatter
+	)
+	parser.add_argument('configFile', help="configuration file path")
+	parser.add_argument('-q', '--quiet', action='store_true', help="suppress non-essential output")
+	parser.add_argument('-f', '--force', action='store_true', help="force update")
+	parser.add_argument('-s', '--status', action='store_true', help="print current status and exit doing nothing")
+	args = parser.parse_args()
+
+	if args.quiet:
+		level = logging.WARNING
+	else:
+		level = logging.INFO
+	logging.basicConfig(
+		format='%(asctime)s %(levelname)-8s %(message)s',
+		level=level,
+		datefmt='%Y-%m-%d %H:%M:%S'
+	)
+
+	try:
+		main = Main(args.configFile)
+		if not main.run(args.force, args.status):
+			sys.exit(2)
+	except Exception as e:
+		logging.critical(traceback.format_exc())
+		print('FATAL ERROR: {}'.format(e))
+		sys.exit(1)
+
+	sys.exit(0)