Dockerized Blackbox Prometheus Monitoring Stack

I recently have to set up a Prometheus stack to perform black-box monitoring. As it is widely known, Prometheus excels at white-box monitoring of internal system components, now with the help of “blackbox exporter”, we can add synthetic monitoring (sort of) to that list.

Most of the tutorials online are focused on having the stack (Prometheus + Blackbox Exporter + Grafana) set up as native binary apps running on the same host, nothing wrong there but nowadays most of the apps are no longer running locally and the trend of moving to cloud apps requires we run the stack as docker containers.

After following the tutorials online (but instead of using the native binary app, I set up dockerized Prometheus/blackbox exporter separately) I can’t get the set up working, target page in Prometheus always reports “Connection refused” when trying to scrape from Blackbox exporter, through 0.0.0.0:9115/localhost:9115/127.0.0.1:9115. Because I used a dockerized version, of course, things will change slightly but that makes all the difference.

After pulling hairs and killing millions of brain cells, I suddenly realized it makes perfect sense for the connection to be refused! Since Prometheus docker container and Blackbox exporter container are not in the same docker network, so they can’t communicate with each other! So the solution is simple, to put both containers (and any other exporters you want to use) in the same network, but the process to that realization is not that simple.

Note: in “Prometheus.yml” configuration file:

You have to provide the actual blackbox service name (or any service container name)

So here’s the complete solution (with node exporter and Grafana as well):

docker-compose.yml

blackbox.yml

prometheus.yml