Telco Network Failure: Optus 2023 & More

I think you won’t be the only one thinking the Optus offer is ‘lame’ and unsuitable based on the mobile network usage. Optus would have granted this additional data knowing that it costs them next to nothing and also many users won’t see any benefit (like you have indicated as it you are a low data user). It is another example of seen to be doing something, even if the something has no effect/benefit.

I hope that that their users pressure them into a better offer to offset the inconvenience they caused.

2 Likes

I get grumpy that the media is prepared to print this. Unless “Mark Gregory” has discussed this with Optus insiders then this is at best informed speculation.

There is anecdotal evidence that the outage had little to do with a “single point of failure” and therefore little to do with “lack of backup systems” and more to do with “human error” - but unless and until Optus makes public a detailed analysis of the root cause, speculation is of limited value.

It is not uncommon to run the network-management network as a VLAN or similar (e.g. MPLS) i.e. over the same physical network as the data network - which in turn can be split into “internal data” and “external data”. This does make the network-management network vulnerable to go down when the data network goes down but of course you are supposed to have redundancy so that the failure of one piece of equipment does not take out any network (it just gets “routed” around).

Didn’t come up for me either - and as far as I know I haven’t visited the site any time in the last several months.

If it had never been privatised, you can imagine that we would all be using mobile services provided by the Postmaster General and when the PMG suffers a lengthy, widespread outage such as the one just experienced, whom would we be force ported to?

I thought the same on hearing that but bear in mind that your situation is not necessarily anyone else’s situation. I will probably be able to use a fair amount of the 200 GB because I am on a low quota plan normally. I would prefer though if they gave me a bill discount. :wink:

Some good points. You may not be aware, but the NBN backbone has redundancy built in. The most expensive components of the NBN are the laying of fibre; and the terminating equipment (muxes and switching components) that aggregate and route the traffic. They do this for load balancing and redundancy. When they lay fibre, they usually run much more fibre than they currently need, leaving large quantities of dark fibre (i.e. not connected at either end), allowing for future expansion simply by upgrading the terminating equipment.

That said, that redundancy doesn’t mean that the network has no single points of failure.

If you’re in an area with only one mobile service, such as your cousin is, you may need UHF backup to a landline connection - if you can justify the cost, or a satellite phone service, or sat sms service (which will probably cost more than the mobile service). Or you may simply accept the risk.

When Optus set up in Australia, they hired experienced local engineers - practically all of them came out of Telstra (or its predecessor). They brought the same cultural issues that Telstra had.

I don’t know what caused the Optus outage. I’m not convinced that there is an easy solution. I would hope that the Telecommunications Industry Ombudsman has a good, hard look at the Optus incident report, preventative measures taken to prevent recurrence, and the compensation plans offered to customers.

While the backbone (day to day operational load) is close to 100% NBN. However in the event of a failure, where possible data is rerouted through other fibre connections, such as OPGW across the electricity network. This has occurred a number of times in the past when cables have been damaged and network sections have been lost. This provides an alternative pathway and backup to the NBN so that

does not occur.

This however doesn’t prevent localised outages where alternative fibre doesn’t exist, but won’t cause total network failure. There could be greater network outages (say regions or a state) through loss of both the backbone and backup fibres. I am only aware of this occurring twice (in Tassie) a year or so ago when both cables were damaged through digging and once on Qld when switching for rerouting failed causing one mobile network to have an outage.

That currently applies in most of Australia where there is no coverage… and handheld technology can’t connect to satellite phone network.

The issue with the Optus outage wasn’t emergency calls on the mobile network, but in ability to make emergency calls on a Optus landline. Those in the community without mobiles were at risk (esp. older Australians).

The other lesser issue with emergency calls on mobiles using an alternative network is the calls are unidirectional - from the person making the call to the 000 call centre. If the call is lost or terminated for some reason, there is no way for the call centre to ring the caller back. This may cause negative outcomes in some circumstances.

That might apply to backbones but locally our down time is 99% NBN failure, be it a power outrage either planned or unplanned, or an equipment failure, or an ‘oops’.

Are you on FTTP; FTTN; or Mobile broadband? There’s no redundancy at all in the last leg for any of them. Power outage will affect all of them, because they probably don’t have UPS power protecting the network attachment points. FTTP is much less power dependent, although it still needs some power once it gets to concentrator points…just not as much as wires do. My knowledge is out of date, so I only understand generalities these days, and you probably won’t get a genuine explanation from NBN, just CYA and other vague-waffle from a script.

HFC. From NBN (not RSP) router failures to NBN not knowing the local grid was doing scheduled maintenance so NBN lost power (!!!) for the day, to many ‘reasons’… my fav was when their index tables got scrambled and our internet services worked but VOIP did not. It turned out we were not connected to our RSP and neither NBN nor our RSP had any idea who was servicing our internet or why it was working. It finally stopped working after a few hours only to be restored to normal operation hours later.

So is mine having retired a decade ago after 40+ years in the business.

HFC will do better than FTTN, unless you’re within 200 metres of the node, but equally dependent on electricity close to the user end (e.g. node equivalent).

And the crash was because of … drum roll…

Seems their processes and procedures might be somewhat short of sufficient.

4 Likes

Gee didn’t I say that a few days ago?

Stumbling forward? The details will dictate whether this has anything beyond PR value.

5 Likes

Looks like it could be another case of ‘seen to be doing something’ even if it may not serve any additional beneficial purpose. The Telcos already do risk assessments, as these would be needed for a wide range of reasons including insurance and financial reporting. They would also report these broadly as part of their financial reporting obligations as it would be classed as information which must be disclosed to the market as it impacts on a companies financial situation.

If one looks at the Telstra Annual Report for example, it is littered with statements about company risks and resilience.

Or it may. It would be negligent to walk away knowing that inadequate action could lead to worse outcomes for the nation in the future. Although the same can has been kicked around by successive governments for many years. We have landed where we have landed. Although not where many would want to be.

For those looking to compensation the AFR with its business focus offers a broad view on Optus’s current circumstances. One suggestion is the greatest impact could be from customers leaving Optus. Considering the AFR pointed out,

“The Telecommunications Industry Ombudsman has the power to deal with complaints from individual consumers and small businesses regarding service faults, but they must try and resolve the complaint with Optus first,” Mr North said.
“It is important to note that when the TIO assesses a compensation claim, it will consider what steps the customer took to protect their own interests and minimise their losses.”

Assume similar would apply for consumers?

Exactly what is covered by the insured and which costs customers can legally recover from Optus or Telstra? Prudence is to limit one’s liability through the T&C’s of the customer agreements while insuring against loss of income plus cost of recovery/resumption.

Risk assessments are done for insurance but for a lot of other reasons. In the network provider I worked for in the past, they were used to support business planning and investment in/augmentation of the network. Risk assessments of different operational scenarios under different conditions, including planned and unplanned outages are also undertaken. Resilience modelling is also regularly undertaken. Telcos would also do this in relation to the ongoing operation of their networks.

Reading some more on the ‘change’, it appears to be targeted possibly at Optus since isn’t an Australian public company and as such, has no obligations to report risk information to the financial markets like Telstra or TPG (which are Australian listed public companies).

The other change is semantics calling the mobile network ‘critical infrastructure’. It has l been critical infrastructure for about 25 years, and it is possibly a catch up exercise as part of seen to be doing something.

Compensation is another issue.

Optus points the finger at Singtel, their parent.

1 Like

You get the gong. An article about Telstra getting some Optus customers closes with

It resulted in routers disconnecting from the Optus IP Core network to protect themselves, so they had to be physically reconnected or rebooted, which required “the dispatch of people across a number of sites in Australia”.

2 Likes

So perhaps to create a fairer sharing of tower and connection to a POI infrastructure (because that seems to be the major bottleneck) , we need to nationalise (by paying a fair market price to the current owners) the entire tower networks of all our providers and further roll out more towers in the areas that have a dearth. Similar to our NBN Co owned connection to the homes and the backbone and only after POI ( Point of Interconnect) are the networks owned and operated by the various RSPs (Retail Service Providers). Then in the case of a RSP Telco fail, the service could be shared across all RSPs perhaps similar (only as an example of the ability to share infrastructure) to how ATM devices are shared by various financial institutions even though owned by a single entity at that point. At the end of the day/period the funds required to repay a RSP for the use of their mobile network infrastructure could be balanced. While Telstra at this point would probably get the lion’s share of the buyout funds, into the future this would perhaps increase access fairness to all users in Australia regardless of whom they buy their time/data/usage with.

4 Likes

It is noted though that in a natural disaster situation, particularly bushfire, towers are quite vulnerable. The tower may lose power and will only have a limited amount of time on battery - and the tower may be unreachable for someone to visit the tower and top up the diesel where there is a backup generator.

Likewise the tower may lose its backhaul if the backhaul is wired.

There is a reasonable likelihood that either of this misfortune for a tower, particularly if nationalised, would take out all three mobile networks equally. (And this assumes that the tower is not just completely wiped out.)

I understand that your focus may be telco stuff-up rather than natural disaster. The government however was interested in resilience in the face of natural disasters (mostly fire and flood) long before the Optus stuff-up.

For this reason, I continue to be interested in satellite as a way of getting much greater resilience.

Are you suggesting a sat dish and infrastructure doesn’t need power, gen set or batteries similarly to a fixed tower installation, assuming fixed towers get mandated to have their connections underground? Seems deuce or pretty close to deuce at the end of the day?

There is much talk here of the technology that may provide the holy grail, cheap portable devices to enable instant communication 24/365 that is 100% reliable. This will satisfy the burning desire to be able to transact business or to remain in contact for any purpose, anywhere, all our waking hours.

Like all such grails in can never be found because each time you think you have found it your desire grows in another direction. Not to mention that the common man or woman will not be able to afford to pay for such a thing.

We also have much talk of the injustice of employers who expect you to answer your phone whenever they desire, the cost of doomscrolling and machine mediated information, FOMO, distortion of social connections from lack of human contact and frustration building because people demand instant gratification always and lose their temper when they can’t have it. The list of issues stemming from the perceived need for constant and immediate connection grows apace.

I have a better idea - don’t bother. Any time one of the phone systems goes down declare a public holiday, stay home and enjoy being disconnected for a while, meditate on what life was like before humans had such devices grafted to the ear and what it will be like again when we discover that is not a sensible way to live.

1 Like