Munin or other monitoring

[sobcz]

I'm replacing years old local Nominatim instance. There were some big changes like whole PHP frontend gone, but everything works. Nice.

The only thing I'm missing is monitoring. Old instance uses Munin with the scripts that are still present in repository. The one for import lag works, but other two depend on now removed new_query_log table.

Is there any replacement that I perhaps missed? It's not critical, but it was nice to have.

[mtmail]

The *_query_log scripts in https://git.openstreetmap.org/nominatim.git/tree/refs/heads/master:/munin should work.

OpenStreetMap devops team has a prometheus exporter https://github.com/openstreetmap/prometheus-exporters/

[sobcz]

Thank you, I didn't know about that other repository.

Quick test says that it works, but there's small problem with timezones. Nominatim logs in utc, but these scripts use local time (utc+1 in my case) and don't find any data because of it. I'm not familiar with Python, but I added timezone.utc to created datetime() objects and it seems that it fixed it.

Any tips how to optimize things when I'm not really interested in log? I only need it for pretty graphs, i.e. just last five minutes of it. Previously I simply deleted old records from table, but with file it's not that easy. If I had only one Munin script, I could rotate it from there and drop everything that was processed, but when I have two, it becomes problematic, unless I come up with some synchronization mechanism (which might not be too difficult...). Rotating the log asynchronously means that I will lose some data for that 5min interval when it happens. Probably not a big deal if I do it once a day. But I wonder if anyone already invented something more elegant.

[lonvia]

We simply use logrotate to cut the logs to something like an hour or so. You need to restart the nominatim server after you have done that. See https://github.com/openstreetmap/chef/blob/master/cookbooks/nominatim/templates/default/logrotate.nominatim.erb for an example. This works fine with the munin script as it will simply keep reading the rotated log when it was already opened. There is the occasional glitch with the numbers but it is not enough of an issue to make me want to rework the script to take into account the previously rotated file as well (which would be the prefect solution here).

[lonvia]

The munin scripts really should be cleaned up.

I was thinking that we should go with the times and provide an optional Prometheus endpoint for monitoring. I would be open for a PR on that front.

[nilsnolde]

somehow I just stumbled on this:) and wanted to share our recent monitoring setup with statsd (we already had support for that, so I stuck with it), exporting to prometheus (with statsd_exporter service) & grafana: valhalla/valhalla#6004. that was almost a delightful experience on the monitoring setup, AI really excels at config and in the end most of config is pretty declarative and understandable. can only recommend:)

[sobcz]

My current cunning plan is to use cron job that runs every 5 minutes, renames log, makes Nominatim open new one with original name, and then Munin scripts will work with the renamed one. Advantage is that I can keep only the smallest part of log that I otherwise don't need for anything, and also Munin scripts can be simplified, no more seeking and date parsing, simply take all lines.

Disadvantage is that unless I missed something, making Nominatim reopen the log means reloading service and restarting all workers, which is unnecessary and also not fast at all, it takes a while before old ones stop logging in original file. I found that gunicorn is supposed to reopen logs on USR1 signal, but it probably applies only to some own logs, it doesn't do anything with Nominatim.

I'm currently experimenting with following change to Nominatim, which seems to do the trick. But I've never done anything like this before, so I'm not very confident about it, it probably needs some error handling and who knows what else.

--- server.py.orig      2026-03-27 14:15:58.193170405 +0100
+++ server.py   2026-03-30 22:16:14.465459153 +0200
@@ -11,6 +11,7 @@
 from pathlib import Path
 import asyncio
 import datetime as dt
+import signal

 from falcon.asgi import App, Request, Response

@@ -123,9 +124,14 @@
     """

     def __init__(self, file_name: str, logstr: str):
+        self.filename = file_name
         self.logstr = logstr + '\n'
         self.fd = open(file_name, 'a', buffering=1, encoding='utf8')

+    def reopen_log(self):
+        self.fd.close()
+        self.fd = open(self.filename, 'a', buffering=1, encoding='utf8')
+
     async def process_request(self, req: Request, _: Response) -> None:
         """ Callback before the request starts timing.
         """
@@ -201,7 +207,11 @@
     middleware: List[Any] = [apimw]
     log_file = apimw.config.LOG_FILE
     if log_file:
-        middleware.append(FileLoggingMiddleware(log_file, apimw.config.LOG_FORMAT))
+        file_logger = FileLoggingMiddleware(log_file, apimw.config.LOG_FORMAT)
+        middleware.append(file_logger)
+        def reopen_log(signum, frame):
+            file_logger.reopen_log()
+        signal.signal(signal.SIGUSR1, reopen_log)

     app = App(cors_enable=apimw.config.get_bool('CORS_NOACCESSCONTROL'),
               middleware=middleware)