• 0 Posts
  • 12 Comments
Joined 1 year ago
cake
Cake day: June 23rd, 2023

help-circle




  • Ceph is excellent as a distributed storage solution - but should really have 4 machines with 2 or more drives each to reach a good level of redundancy - which is a bit much for most people on this sub.

    One nice feature is it deals with heterogeneous drives well, like if you need to buy a bunch of used ones on eBay for cheap.

    Probably not a good solution for your case because of the footprint - but good to be aware of it.






  • I have 2 pi4 4GB boards and was waiting forever to get a third to run RAFT based services across.

    I gave up last year and bought 3 chinese boards at $60/ea with 2x 2.5Gb Ethernet each, emmc, and a m.2 slot - and they run at half the temp of the pi4 boards.

    I never needed the wifi/bt and form-factor the pi boards offered anyway - really no reason to stay as long as you can find software that boots on other boards.



  • Ceph (rbd,s3) on 4 poweredges.

    Nomad, Consul servers running in a 3 node raft on some ARM SBCs.

    Nomad clients on 2 poweredges and 3 arm SBCs running:

    • s3 CSI (media and large file storage is all on s3)
    • rbd CSI (all the stateful jobs have rbd block devices)
    • NATS cluster with Jetstream to enable MQTT support
    • mosquito mqtt (had to run mosquito for mqtt v5)
    • mosquito<->nats bridge (via benthos)
    • nodered (just for easy Google home integration)
    • zwavejs2mqtt
    • zigbee2mqtt
    • frigate (can only talk mqtt v5 so had to run mosquito just for this)
    • grafana
    • gotify
    • gitea
    • drone CI
    • postgresql
    • BitTorrent client
    • ceph rgw s3 gateways
    • NATS based home automation lambdas - I wrote these in go
    • Adguard home
    • traefik as main ingress
    • Prometheus
    • prom node exporter
    • jellyfin
    • jackett
    • a program I wrote that manages torznab->acquisiton->s3 lifecycle
    • a website
    • wireguard servers

    And that is just in the server room - I also have more like the 3d printer and CNC machine controllers etc.