Please, see the linked thread on the community for official communication.
Feel free to rant and vent in this lemmy-thread, but don’t speculate or “give good advice” to the operations-team; they know what they are doing.
We have dual redundant links via separate physical hardware from our side to our Tier 1 ISP. We unexpectedly discovered their equipment is a single point failure.
Our Tier 1 ISP has just confirmed hardware failure of the upstream router (after 16 hours!) and will provision new replacement device once it has been shipped to site.
I wonder why the OSM team isn’t calling the ISP out by name. Not having on-site spares at a Tier 1 is just, insane.
Amazingly, this isn’t the first time I’ve heard of this happening.
Charging for a redundant line, then discovering that it actually wasn’t one day when it goes down.
“We paid for a redundant line but it wasn’t redundant!”
‘Sure it was. They just went to the same router.’
Can you imagine the straight face that tech must have had to keep?
I wonder why the OSM team isn’t calling the ISP out by name.
Because this would open up OSM for defamation (and hinders a commercial resolve). It doesn’t solve anything right now.
TL;DR
The OSM team is very small and constrained heavily by financials. They had a single ISP which acted as a single point of failure. They are moving to a new ISP but they expect the outage to last until Wednesday
SL;SR
The OSM team had redundant lines to the Tier 1 ISP, but the ISP bridged them into the same hardware (not a normal or expected thing).
That sounds like a very silly thing to do.
It sounds like a normal ISP
Updates: It should be operational as of tonight. OSM are switching to their new ISP.
You heard it here first…
We should have all service back by the European evening tomorrow (Tuesday). We are switching to the new ISP.
(Minor matter of me having to get new ISP setup working on our routers with some virtual networking required for simultaneous upstream ISPs, and some special VRRP setup required for the new ISP redundancy setup)
