Tweet
A story of woeful management.
Due to a overloaded power substation power to a certain University was cut off meaning all the network and core services went off-line.
After months of meetings this shouldn't be a problem as a disaster recovery plan was laid out.
Sadly the People charged with implementing this were running around like headless chickens and decided to ignore the plan.
A more senseable lower ranking manager asked why weren't they following the plan as they should be following it or had they an updated plan available.
The following problems then came to light. None of the assistant directors had a copy.
A copy was on the University departmental Wiki but the web server was down so that was unavailable.
Another copy was on usb stick but that had been lost.
So systems had to brought back online in a haphazard method.
The Server room ups wasn't working and hadn't been for three months due to a dispute over the service level agreements with the company that supplied it.
The backup Generator was brought online but then the low ranking Manager asked a simple question.
How much fuel has it got. Unfortunately none of the Directors had checked whether the Gen had any method of checking the fuel level and it hadn't. So there was a possibility that the power could go off again at any time.
Luckly it lasted long enough for the local power to be partly restored.
So systems were brought on line. Certain parts of University were on low power and the buildings had to closed for safety reasons and to prevent further overloading of the temporary circuit.
However the farce wasn't over. Another director decided to try and fix the UPS which had a faulty switch. He fixed this and then decided to do a "Non Critical test in the middle of the day"
The Power was switched over but the UPS didn't start and in theory the Gen should have then kicked back in but it didn't. All the services were brought down again.
Now everything is back but I wonder if anyone has ordered any more fuel for the Gen.
Due to a overloaded power substation power to a certain University was cut off meaning all the network and core services went off-line.
After months of meetings this shouldn't be a problem as a disaster recovery plan was laid out.
Sadly the People charged with implementing this were running around like headless chickens and decided to ignore the plan.
A more senseable lower ranking manager asked why weren't they following the plan as they should be following it or had they an updated plan available.
The following problems then came to light. None of the assistant directors had a copy.
A copy was on the University departmental Wiki but the web server was down so that was unavailable.
Another copy was on usb stick but that had been lost.
So systems had to brought back online in a haphazard method.
The Server room ups wasn't working and hadn't been for three months due to a dispute over the service level agreements with the company that supplied it.
The backup Generator was brought online but then the low ranking Manager asked a simple question.
How much fuel has it got. Unfortunately none of the Directors had checked whether the Gen had any method of checking the fuel level and it hadn't. So there was a possibility that the power could go off again at any time.
Luckly it lasted long enough for the local power to be partly restored.
So systems were brought on line. Certain parts of University were on low power and the buildings had to closed for safety reasons and to prevent further overloading of the temporary circuit.
However the farce wasn't over. Another director decided to try and fix the UPS which had a faulty switch. He fixed this and then decided to do a "Non Critical test in the middle of the day"
The Power was switched over but the UPS didn't start and in theory the Gen should have then kicked back in but it didn't. All the services were brought down again.
Now everything is back but I wonder if anyone has ordered any more fuel for the Gen.
Comment