Thanks!
I took a lot of inspiration from here https://github.com/onedr0p/home-ops, there is also a discord server on there where you can ask people for help, they are quite helpful.
I mostly shopped around and found parts and implementations I liked and coppied that and made it my own
Preemptively responding to this, because I just know there’s going to be a comment about this.
Yes Yes I know, GitHub, screw GitHub, Screw Microslop. I am trying to move away from Big Tech as much as possible, but it’s a marathon not a sprint, and every win should be celebrated.
Grafana dashboard
You can find more details and the source code here: https://erasmus.works/
I have basically taken a power strip, and made each plug a module.
The inside of a power strip, like in this image, is in series.
mains → yellow → red → blue → green → blue(end)
If a join between yellow and red is broken, it knocks out red and everything after it.
If what is plugged into yellow is plugged out, or disconnects everything else keeps working.
The same is true for my setup, the joins between Yellow, red etc. are just on the outside of the case, so you can disconnect them if you want, which would kill the power to what is after it. But you don’t have to do that.
My setup is as resilient as this would be:
I haven’t added any failure-modes
Thanks yea I made sure to add this Safety section to my write-up on how to build this yourself, and I reference back to it a lot, so it’s painfully clear.
I printed this in PETG which can take the heat easily, but ideally you use UL 94 V-0 which is also flame retardant.
I did make sure everything is very well insulated, and pulled hard on every connection to make sure it’s held down properly.
**Safety**
This part carries live mains. Mains wiring kills people who get it wrong, and a fault in a printed enclosure can start a fire.
Do not build it unless you are competent to wire mains and permitted to do so where you live. Sleeve or shroud all live terminals. PLA softens around 60 °C, PETG around 80 °C. Neither is flame retardant. Only a UL 94 V-0 filament is.
Wire the protective earth if anything you plug in needs it.
Build it at your own risk. I take no responsibility for injury, death, fire, or damage resulting from anything in this repository.
See the “monitoring” namespace
kube-prometheus-stack plus VictoriaLogs for logs, about 100 pods on 2 nodes.
The whole stack uses about 2.2 GB RAM. Prometheus takes about 1.1 GB of that and Grafana about 700 MB. CPU is basically idle, around 0.2 cores.
Prometheus has about 150k series and uses about 7 GB for 15 days of metrics, so roughly 14 GB a month. Logs are about 3 GB a month. Grafana itself is under 1 GB.
It’s mostly the default exporters, so a smaller lab would need less. A 20 GB volume would be plenty.
Prometheus is set to retention: 15d and retentionSize: 18GB. Anything older than 15 days is dropped, and so is the oldest data if the total goes over 18 GB, whichever happens first. It’s sitting at a steady ~7 GB now that it has a full 15 days stored.