Telco Network Failure: Optus 2023 & More

Since earlier this morning

Still a problem looking for resolution at 6:26 EST (7:26 if on daylight saving time in the eastern states).

Note the 2 bars for Telstra and nil for the second sim on the OPTUS network. Why many in the bush or nearby remain welded on to the Telstra network.

3 Likes

And still a problem at ~09:10 AEDT, so must be something pretty serious. So far it’s been out for ~5 hours.

https://www.optus.com.au/living-network/service-status#mobile__45g_internet

Our engineers are currently investigating a network fault that is impacting Optus Mobile & Fixed customers, we apologise for any inconvenience caused

I have one Optus-backed SIM, status currently “Emergency access only”. Fortunately, I don’t depend on that one. My everyday phone access is via Telstra.

2 Likes

There have been cases of turnabout.

Coverage, Telstra wins. Superior reliability? A roll of the dice.

5 Likes

The moral of the story is: don’t put all your eggs in one basket.

Always assuming you have a choice of baskets 
 :confused:

6 Likes

Nah, in that scenario, where available, you have both (or all three) networks.

All technologies and all operators have their outages.

As far as I read, Optus is still down at midday (AEDT). :open_mouth:

Coming on top of “the data breach”, it has been a bad year and a bit for Optus. Time to give their customers some love? :wink:

2 Likes

Looks like coming back online/NBN services are returning -

4 Likes

Internet via my modem-router (still very slow and intermittent) back on here after 10 hours- it went off just after 3am according to my weather & PV monitoring laptop. No phone or data via the phone though.

3 Likes

I had to turn not phone off and back on to get it to reconnect to the Optus Network. Worth a try?
My partners mobile reconnected without prompting. Could be my dual sim mobile, or could be Optus.

3 Likes

My phone service came back on a little while ago with no restart, but my wife’s didn’t, I suggested she do the usual off and on again method.

4 Likes

Where available, only if one chooses to do so.
I know Aussies whose economy choice of connectivity is a single device served by a single network. Any second device, a tablet or laptop uses the shared data of the mobile. Urban and rural citizens.

There’s a long list of explanations (reasons/excuses) for those hanging on to ‘the one and only’.

1 Like

A business contingency plan is having a backup in case my network connection fails. Access to another network? Offline transactions? Cash?

If your business plan involves hoping that natural business rivals will get together and agree to provide enough capacity to handle each others usage in case of failure, then you are a business that deserves to fail. You have NO plan.

Many newer terminals run on both 4G and wifi. If a business had a different RSP for their mobiles and internet connection, they would have been able to continue yesterday without any issue. This is the setup we have in the event that one is down (say local 4G network)
the other (wifi through the NBN or hotspot using a phone from another mobile provider) can be used as a backup. We have a third backup with ability to do MOTO payments through the POS terminal (this is dependent on the POS terminal provider). MOTO payments can use used in the event that nothing is working (or a extended power loss where the terminal battery backup is exhausted).

Our own terminal came with an option for a Optus or Telstra SIM. This allows a business to select a different provider to their business mobile (which could be used for hotspotting if necessary).

I am surprised a business would not have at least two of the above as an option. If they don’t, they mustn’t have looked at their risks or read some of the information widely available/got advice on how to maintain payment services. Hopefully yesterday’s event if they didn’t have a alternative option (everything was on Optus), they will now be exploring one.

Yesterday’s event was not dissimilar to a electricity outage. In the event of a power outage, generally there isn’t any alternatives like those above to provide backup. In a power outage, depending on the POS device, a business may have been worse off by not being able to process any form of payments (nor supply goods/services to a consumer).

1 Like

In August of this year I asked the question: “How Bad is OPTUS”
I think the answer was then (and is now)
VERY BAD INDEED.
My take at the time was that their problems were systemic and that they had a culture of handling the truth very carelessly. I think now that my take was correct and that in time those practices would lead to the inevitable. I wrote to the CEO and told her as much. She now has even more work to do.

1 Like

Industry expert Mark Gregory told 7.30 the catastrophic outage was likely due to a “single point of failure” and a lack of backup systems that could have prevented the extended length of the outage.

“I think we can already conclude that the Optus network is not fit for purpose,” Mr Gregory said, adding that unless the systems are improved the network remains vulnerable to more outages.

"We don’t have the reliability and redundancies that many of us would expect our technology companies to have.

“We know it was more than a human error or infrastructure error, what we know is that it’s a decision made by the company in the way that their network is designed and built.”

1 Like


 but that criticism is not limited to Optus. The others suffer from the same problems, in this expert’s opinion.

“The telcos, like all technology companies want to minimise their engineering spend, they’re trying to maximise profit and they’re trading off the possibility there will be outages or failures in their efforts to make a larger profit,” he said.

1 Like

A more technical analysis of what probably happened:

I find it surprising that Optus had not kept their internal network separate from their external services, to make sure they could still get access to the infrastructure when things like this happen.

As well as bringing down Optus’ landline, mobile phone and internet services, the outage also appears to have brought down the internal network used by Optus to manage its network, forcing technicians to travel to affected locations and attempt to fix problems in person.

Industry insiders said it was “unusual” for a telecommunications company not to keep that management network “out of band”, or completely separate from its main backbone, so devices can still be managed remotely when the main network is down.

Sadly Fin Review is paywalled. But I assume the article says pretty much what I said in an earlier post. Been in big company IT support long enough and done recoveries many hundreds of times.

Paywalls are strange at times. I could get to that article without even a warning that it’d let me in just this once. A couple of days ago I tried to read a Fin Review article discovered via Google but couldn’t get into it because of the paywall. So I thought this one might be publicly accessible. :confused: And now, having let me see it once, it won’t let me read this one again.

The cause of the network outage is said to have been a BGP routing change, probably made by one of Optus’s overseas stablemates - if so, not actually Optus’s fault. However, fixing it would have been far more complicated than it should have been because of not having separated the internal from the external networks. :worried:

2 Likes

Found it and read in ingognito mode. Which does not save prior visits in cookies.

Seems possible. A failure in the routing protocol rules that link the networks together into ‘the internet’ would cause nasties.

BUT, any serious network company would NOT implement management controls dependent on the internet. The network would be internal.

1 Like

Having deprived me of phone, SMS and data services for most of a day last week, Optus have magnanimously granted me 200G of additional data to use over Summer. Optus knows that I never use as much as 5% of the data allowance on this device and, even if I did, it would cost them nothing to provide the offered data.

Do they seriously believe that people are going to fall for this?

1 Like