I’m hoping to find something that:

  • has a nice dashboard
  • is quick and simple to install
  • is very lightweight and unobtrusive
  • can send alerts via http request
  • RegalPotoo@lemmy.world
    link
    fedilink
    English
    arrow-up
    10
    ·
    1 day ago
    • Base ansible role installs Prometheus node exporter, configured with the text file collector
    • VM automations push DNS records so that the Prometheus dns-sd automatically discovers them
    • Ansible roles for add Cron jobs that generate metrics for specific systems and dump them for the text file collector
    • Grafana for dashboards
    • Karma as a UI in front of Prometheus alert manager
    • tetris11
      link
      fedilink
      arrow-up
      1
      ·
      1 day ago

      Cron jobs that generate metrics for specific systems and dump them for the text file collector

      Details please

      • RegalPotoo@lemmy.world
        link
        fedilink
        English
        arrow-up
        2
        ·
        17 hours ago
        • https://github.com/prometheus/node_exporter?tab=readme-ov-file#textfile-collector - which makes node exporter watch a specific directory for files that contain metrics, then re-export them back to the central Prometheus server
        • Some systems have their own metrics endpoints - instead of getting Prometheus to scrape these directly I set up a Cron job to curl these into files for node exporter - this means I don’t need extra config in Prometheus to find the endpoints, and don’t need to mess with firewall rules
        • Other systems don’t directly expose metrics in a format Prometheus can use - in this case I will write/find a script that can do the conversation, then either set it up to write the metrics file directly and run it on a Cron, or run it as a service and another Cron job to do the scrape