Amazon Web Services storm outages serve as a warning of cloud risk to businesses

Australian businesses have been warned they need to spread the risk in their cloud computing operations across different regions after the Sydney storms on Sunday knocked out the operations of numerous Amazon Web Services customers.

The ferocious storms that hit NSW, left AWS clients including Domino’s Pizza, Foxtel, The Iconic, Stan and Domain without websites or key systems for hours.

It served as a warning that sending systems to the cloud, rather than hosting them on-premise did not remove the risk of costly failures.

Amazon Web Services declined to discuss whether it would compensate customers who lost business during the outage. AP

The failure represents a major embarrassment for the company, which generated $US2.57 billion revenue in the latest quarter, based largely on the fact that it is perceived as being hugely reliable.

It meant that customers that had committed all of their systems to its care were unable to trade from mid-afternoon until as late as Monday morning.

On Monday AWS refused to discuss the reasons behind the outage, instead referring inquiries to an online site, which showed the status of its data centres.

A spokesman also declined to comment on whether affected businesses would be entitled to compensation, and what damage the outages had caused for its own business locally.

Dan Nolan, co-founder of Proxima said his company was saved from the outage because it had decided to take a multi-availability zone failover approach. Anthony Johnson

It is understood that power was lost in a number of systems in its data centre, before being restored about 90 minutes later. It then took hours for some systems to be rebooted and brought back online.

Experts said that customers who insisted all their systems and data remained in Australia had been impacted, as other customers simply had their systems clicked over to Singapore and continued to trade as normal.

IBRS analyst Joe Sweeney said many would use the outage as an example of why businesses should not adopt cloud computing.

However it was more of a wake-up call for businesses to structure their technology in such a way that one data centre failing would not be fatal.

AirService co-founder Dominic Bressan said he understood things could go wrong with cloud computing services. Supplied

“In short, I think the outage is damaging for the entire market – not just AWS,” Mr Sweeney said.

“I think this event is a great opportunity for organisations to rethink what they were trying to get from moving to the cloud. It is also a cautionary tail – ensure your critical applications are built to survive failures of any data centre.”

The outages came as more than 226,000 homes and businesses lost power during the weekend storms. Roads, bridges and public transport were also affected.

The bodies of three men were also discovered in cars caught in floods in separate incidents in the ACT, the NSW Southern Highlands and Sydney’s south-west.

So far, $30 million worth of insurance claims have been made.

Co-founder of event software company Proxima, Dan Nolan, said his company was an AWS client, but was saved from the outage because it had decided to take a multi-availability zone failover approach.

“When we were setting up our infrastructure a bit over a year ago, we took a punt on multi-availability zone failover,” he said.

“It was literally for if this situation ever occurred … the cost otherwise though would have been quite substantial to the business. Having multi-availability zone is a fraction of the cost to having a site go down.”

Mr Nolan said the company had taken heed of lessons learnt in the United States where businesses such as Instagram had suffered major outages in the past.

Proxima was not the only company with a backup strategy in place.

Mobile ordering and payment technology company AirService uses AWS as its main provider, but had another ready to go in case of an outage.

“As amazing as AWS is, and we’ve been using it for a long time, we do understand that things can sometimes go wrong,” AirService chief executive Dominic Bressan said.

Telsyte analyst Rodney Gedda said Sunday’s storm showed that cloud computing wasn’t infallible, and that organisations needed to factor in risks when weighing up their strategy. Despite the failure on Sunday, AWS systems are likely to be much more robust than those run by individual organisations.

“No outage is ‘good’, but it’s unlikely this event will damage AWS’s brand too much. All the main cloud players have experienced some form of downtime so Amazon is not alone and events like these are generally accepted by customers,” Mr Gedda said.

“AWS will no doubt perform a post mortem on this event and do what it can to prevent it from happening again … Cloud services won’t magically protect your business from downtime or data loss. You still need to be proactive.”

 

Advertisements

Talking innovation and disruption with Telstra’s CRO

Kate Hughes talks to StrategicRISK about how risk management is helping Australia’s largest telecommunications provider become a global technology player

It’s 6.20am and Kate Hughes’s phone goes off. The chief risk officer for Australia’s largest telecommunications provider, Telstra, has been called to activate the crisis management team to deal with a major outage affecting thousands of customers.

By 7am, an action plan is in place and Hughes can begin her day. But an hour later she receives a report from a whistleblower alleging bad behaviour of a senior executive, which sees her launch an immediate internal investigation through her fraud team. Then, a few hours later, Hughes is alerted to a customer privacy breach, so it’s then a call to the regulators to alert them of the incident.

It’s not even lunchtime, and Hughes has already fielded more incidents than most chief risk officers would see in a month.

Hughes has agreed to an interview with StrategicRISK to discuss how risk management is helping Telstra navigate a strategic business model change from a traditional domestic telecommunications provider to a global technology company.

But first, a history lesson.

Telstra is one of Australia’s most well-known companies. The country’s largest telecommunications provider builds and operates networks around Australia and markets mobile, internet access, pay television and other entertainment products and services.

But the pace of digital change has not been kind to traditional telcos, forcing Telstra, and most of its competitors, to pivot from their historic business model.

Today the company has its sights on being a global technology company.

Last year the company invested almost $1.2b in acquisitions, including a controlling stake in 15 new businesses. It also expanded its reach in Asia through acquiring Pacnet in Singapore and launching TelkomTelstra in Indonesia, and activated new business units such as Telstra Health.

This pace of change, coupled with the profound shift in the way people connect and communicate, means Telstra faces a challenging set of business risks that threaten it achieving its growth ambitions and financial targets.

This is where Hughes comes in.

“Most people say to me, I’ve got one of the most interesting jobs in the company and I would agree that I do. There’s very little that I’m not across, or not involved in, or not able to add value to,” Hughes says. “I get to make decisions about the kind of ladders we use in the field, I get to talk about the risks of having handbrake alarms in some of our cars, and I also get to talk about the risks of technology disruption as it will impact on our strategy to be a world-class technology company.”

The risk function at Telstra has evolved significantly over the past three-and-a-half years under Hughes’s leadership. The 160-strong risk office now looks after the group’s risk management, compliance and privacy functions, as well as its law enforcement capabilities, fraud investigations, enterprise resilience, security, and health, safety and environment arms.

Hughes, who reports into chief financial officer Warwick Bray, admits she is lucky to work for an executive team who take risk management seriously.

“It’s a privilege to be involved in something that helps our executives make better decisions,” she says.

And with the pace of change that Telstra is facing, that decision-making needs to happen quickly.

“We can be disruptive or we can be disrupted and we’ll probably be both. That’s not necessarily a bad thing. I think disruption creates solid incentive to be more innovative and that’s good,” she says.

Telstra is undergoing a major internal simplification process, driven by the risk of not being able to keep up with younger, more agile, tech start-ups.

“I’m in a meeting every Tuesday morning on this to see what am I doing to help us get there,” Hughes says, adding that she sees the company’s simplification and disruption impetus as an opportunity to show the benefits of risk-based decision making.

“Everything we do requires us to do a risk assessment and that shouldn’t be seen as an onerous, bureaucratic thing, but actually built in to our processes every day.

“Part of the business case is doing a risk management assessment. You don’t tack it on the end, it’s not done at five minutes to midnight, it’s not done once we’ve agreed to everything else … it’s part of the process.

“That is the evolution of risk management – to take it out of the academic, out of the process, and make it much more part of the business conversation so that it actually adds value to the commercial decision-making challenge that your leader has,” she says.

Hughes cites an example with the head of Telstra property, who had to decide how to allocate his spending when it came to upgrade work on the group’s exchange sites. By applying a safety rating to every exchange, Hughes team was able to prioritise which sites should be worked on first.

Back to where it started

In some ways Hughes has come full circle to her role at Telstra.

After graduating with a commerce degree with majors in economics and finance, she took up a role at the NSW Treasury. One of the first companies she audited was Telstra, sitting in the very same Melbourne offices that she does today.

She then moved to the Sydney Futures Exchange where she was responsible for surveying the open trading floor for rouge or illegal trades during its final year of operation.

“I was one of about four women in a room of 400 men that had some pretty bad behaviours,” Hughes recalls.

From there, she moved to the Australian Securities and Investments Commission (ASIC), the country’s corporate, markets and financial services regulator. And it’s this insider experience which has proved invaluable to Hughes at Telstra – one of the country’s most highly regulated companies.

“One of our big risks is going to be a rapidly changing regulatory environment,” she says. “It will go to things like how we regulate data ownership and data sovereignty in the long term.”

Regulators around the world are struggling to keep up with the implications of new technology – and most are doing so at different paces, not to mention with vastly different strengths of legislative iron fists.

For a company with global expansion plans, this adds a huge layer of complexity.

“How do you grow in those countries where your company’s cloud strategies aren’t going to fit with theirs, for example,” she says.

“[Regulation] has the potential to certainly change how we develop and market products. It’s one of the material risks that we talk to the board about. What you have to get very good at doing is staring over the horizon beyond your normal two to three-year period, out to five to eight years and start to think about what regulation will matter then.”

In a disruptive environment, Hughes also sees the potential for corporates to challenge existing regulation.

“If you look at Uber and Airbnb as two business model challenges, everybody talks about those as being challenging at a business model level, but what for me was most interesting is that they challenged existing regulator models as well. Uber drivers never stopped and said ‘I need a taxi license’. So what would happen to us if we fundamentally changed [current] regulation? We do a lot of black swan thinking about some of those risks,” she says.

Cyber and security challenges

In the nearer term, Australia is set to bring in data loss notification laws which will force companies to advise customers when their details have been unlawfully accessed.

“It’s not going to be a huge issue for us because we’ve always thought long and hard about who we should tell when we’ve had a breach of some kind,” Hughes says.

This stance was put the test last year. Just two weeks before Telstra’s $697m acquisition of Pacnet was finalised, the Asian telecommunications business was hacked by an unknown third party which gained complete access to the company’s network including emails and other administrative systems.

Telstra said it wasn’t told about the breach until after the deal’s completion on 16 April.

In that instance, Hughes says Telstra voluntarily went to eight different regulators about the breach.

“Each one had different expectations about whether or not we would or should tell them,” she says. “We’ve always felt better to be upfront and honest. The worst thing you can do is look like you’re hiding it.”

But Hughes fears that the new breach notification laws could result in consumers getting “notification fatigue”, where they fail to act on important data breaches because they are being alerted of them so frequently.

Instead, when it comes to cyber security, Hughes is turning the lens to the company’s employees, which are often considered the weakest link in any cyber security programme.

“We run drills to see if we can trick our employees into doing something that they shouldn’t have,” she says, such as clicking on a link or opening a suspect attachment.

In the first drill, 30% of employees failed. That dropped to 18% in the second round.

What’s in a name?

Managing major reputation crises is also something that Hughes is well versed in.

In 2005, she was asked to join a company in the midst of a major corruption scandal that saw it on the front page of the papers for more than 400 consecutive days, and its shareholder value slashed by almost $1bn overnight. That company was the Australian Wheat Board (AWB), which was accused of paying millions of dollars in bribes to Saddam Hussein’s regime in Iran in exchange for lucrative wheat contracts.

“Part of my job was to build the right internal controls, the right risk processes and the right compliance controls to ensure we never ever did that again,” she says.

For four years, Hughes worked with a new management board to help turn the business around.

“Leadership in good times is always a pleasure. The hardest job you will ever do is lead in tough times when there’s bad news on the front page of the paper and your employees feel embarrassed to work for you,” she says.

Hughes believes reputation isn’t a risk as such, but an “outcome of other things you didn’t do very well”.

Regardless, when you’re an organisation the size of Telstra, reputation is incredibly important.

“This year we have put in place much more formal metrics to measure the impact of our resilience on reputation,” Hughes says.

For example, during network outages, Telstra can map social media mentions against the network issues to give an indication on the importance of resilience to its customers.

“It’s also a really good predictor of consumer behaviour, so how many of these [incidents] does it take before a consumer, one, rings up and complains, two, gives us a negative rating, or three, possibly changes services. That’s critical insightful data that we work with marketing, media and communications teams on,” she says.

Hughes is one of the most passionate advocates for strategic risk management that you will meet. But she’s far from traditional.

“The one thing I rarely say to people is that I’m the chief risk officer; what I often say is I’m an executive at Telstra, because part of my job is not just talking about the risks, but talking about the opportunities. At the end of the day my real job is to make sure that our executives know how to make decisions.

“Helping people consciously choose to take risks is good because it means that they’re doing it utterly informed.”

Hughes says that risk managers must move from talking about the “what” – the list of risks and risk registers – to talking about the “now what”.

“Being the person who forces people to sit through three-hour long risk workshops so we can satisfy ourselves that we’ve got 25 pages of risk registers is an academic exercise that has never sat well with me,” she says.

“Doing [risk management] for the sake of governance, whilst necessarily, is not necessarily always valuable. Doing it because it helps [the company] make a better decision, save money, spend it more wisely … and potentially be a disruptor yourself because you’ve found a hole in the market that no one else has, that’s where the real value comes from.

Scandal in fantasy sports underscores the importance of internal process and controls

Details of the scandal engulfing the online fantasy sports company DraftKings should be common knowledge by now – A DraftKings employee admitted to the early release of data not generally available to the public and won US$350,000 on a rival site, FanDuel.

The comparisons to insider trading quickly – and logically – followed.

Questions abound that why a major player in the largely unregulated, multibillion-dollar (US$3.7B annually) fantasy sports industry didn’t have stronger controls in place to restrict access to protected information or ban its employees from participating in fantasy games elsewhere.

Without the structure and processes, the mess was totally predictable and only a matter of time. While DraftKings and FanDuel announced permanent bans on employees participating in fantasy leagues within days of the scandal breaking, the damage had already been done to their brands and reputations:

  • ESPN initially announced it would end DraftKings’ sponsorship and later said it would stop its ads.
  • The New York attorney general’s office announced an investigation
  • A Kentucky man is seeking class-action status that accuses DraftKings and FanDuel of negligence, fraud, and false advertising.

The lesson from this latest corporate blunder should be crystal clear: A well-designed system of internal controls is fundamental to reducing business risks.

Should DraftKings executives be accountable for not anticipating such problems? Considering the industry is largely unregulated, has seen remarkably rapid growth, and handles huge sums of capital on a weekly basis, the answer is an unequivocal “yes.”

While DraftKings is not currently publicly traded, it is a textbook example of how limited or poorly designed internal controls can quickly be overwhelmed by the pressures of rapid business success. One of the criticisms of mandatory internal process and controls regime is that start-ups lack the resources to support them. But without such an investment, organizations are at greater risk of making much costlier mistakes in the future.

It’s all about the old expression, “Pay me now, or pay me later.”

Online gambling review should not ignore the problems in our own backyard – Charles Livingstone

As those who watch sport will attest, online gambling is seemingly ubiquitous. Certainly advertising for it is.

In Australia, the regulation of gambling services is a matter for state governments. However, the federal government has responsibility for telecommunications, which includes the internet. So, there is some division of responsibility for online gambling. This has arguably left the area less well regulated than it might be.

This is one ostensible reason the federal government has announced a review of the online gambling industry.

Background

The current federal legislation is the Interactive Gambling Act. It allows Australian operators to offer online betting. It also seeks to prohibit the provision of casino-style gambling – roulette, slot machines – to Australian residents, but doesn’t prohibit Australians from using such services.

This means that Australian-registered services are not allowed to offer some gambling services, but are permitted to take online bets.

The most recent review of the act reported in 2012. It concluded that it would be useful to consider a trial of some online gambling – suggesting online poker, which is thought to be a less harmful form of gambling than slots or other casino-style gambling.

The review also recommended a host of harm-minimisation measures be introduced into the online gambling arena. These included a pre-commitment system, an effective self-exclusion system and much-improved practices among bookies. The review recommended that better enforcement of offshore providers be implemented, although effective regulation of extra-jurisdictional gambling providers is likely to be futile.

Nonetheless, the review suggested that banking institutions should be rewarded for blocking transactions between Australians and nominated unlawful gambling providers. This may have some effect, although mainstream banking institutions provide only some of the plethora of ways of moving money around the world.

A recent Financial Counselling Australia report highlighted a number of what can only be regarded as very dubious practices among prominent bookmakers operating under Australian regulation. These include extending unsolicited lines of credit, failure to pay winnings on request and repeated inducements to gamble.

These practices are not caught by current consumer protections under credit law or gambling regulation. Bookies also appear to regularly share data on their customers, which is likely to breach privacy legislation.

What this review will focus on

Media reports early this month – when Social Services Minister Scott Morrison confirmed that a review would be held – appeared to focus on a range of the issues highlighted by the 2013 review, including consumer protection.

However, the terms of reference headlined this new review as being into the:

Impact of Illegal Offshore Wagering.

In fairness, one of the terms of reference of the review is concerned with increasing consumer protection.

It will be a quick review. The final report must be with Morrison by late December. Submissions will be sought from industry and the public.

Those concerned with the growing harms of online gambling – and particularly sports betting – will be disappointed with the terms of this review. There are a number of pressing concerns that, from a consumer protection perspective, might have ranked higher in both the terms of reference and Morrison’s messaging.

Online bookies are competing for market share in Australia, where the operators now include global giants such as the British bookies Ladbrokes and William Hill. Their practices have attracted considerable criticism as the scramble for revenue escalates.

Troubling practices include the continuing provision of credit, the pushing of boundaries on such issues as the prohibition of online in-play betting, and blanket advertising of their wares – including to children during sporting events – and the aggressive branding of sporting teams with gambling providers.

What is Australia’s real gambling problem?

Sports betting in Australia is likely to generate revenue – that is, player losses – of around A$750 million in 2015-16. It is the fastest-growing gambling sector and is likely to produce a new wave of gambling problems among the young men to whom these products are marketed.

Although modest in comparison to poker machines – which generated around $11 billion in losses in 2014-15 – it needs to be effectively regulated if Australia is to avoid adding to the already significant burden of gambling harm. The good news is that preventing this harm is actually quite straightforward.

Unfortunately, substantial and powerful segments of the Australian body politic are now closely affected by the fortunes of the bookies. These include Packer interests via CrownBet, the AFL’s official wagering partner. State and territory treasuries are also abundantly interested in maintaining the flow of money.

It is worth asking if the offshore online gambling sector is Australia’s most pressing gambling problem. Undoubtedly, some Australians get into a lot of trouble gambling online. Most of them will fall prey to bookies already licensed in Australia and offering services lawfully. Some will end up in trouble because of offshore sites offering unobtainable services such as online slots or roulette.

Overall, the market going to such offshore providers is estimated at around $1 billion, although there is no way of verifying this under current circumstances.

But, at least 75% of those with a gambling problem have it because of poker machines in clubs or pubs. We see little concern from the government about this group.

And, even in the online gambling environment, there appears to be little concern about first cleaning up our own backyard. The 2013 review made some very sensible recommendations about harm minimisation, including restricting or prohibiting credit betting. This is clearly a source of considerable harm to many. And prohibiting credit betting is in fact current federal government policy.

The Financial Counselling Australia report provided ample evidence of the excesses of the Australian online gambling industry. A recent whistleblower article from within the industry confirmed these concerns. These need to be a major focus of any review of the Interactive Gambling Act and other relevant federal legislation, including the regulation of advertising and banking services.

But if the renewed urgency behind this review is to highlight the “dangers” of offshore online gambling providers, then the bookies will be arguing as hard as they can that the solution is to allow them to offer the same services from Australia. After all, the internet is notoriously difficult to regulate and service providers licensed in Australia would be expected to observe Australian regulation.

It is important to ensure gambling is properly regulated. But it is probably better to address the main game first, or at least simultaneously. That involves making sure that current providers are adhering to the best possible harm-minimisation practice.

The 2013 review set up a clear set of goals for that. We don’t need another review to know what needs to be done, or to do it.

The changes Australia must make in the digitally disrupted world

If we are to keep up in the new world, I think Australia needs to change. We are besotted with the “corridor of comfort” – we don’t like tall poppies and distrust bankrupts – we like the middle too much.

We need to celebrate our successes and support those who fail. In the words of Winston Churchill: “Success is stumbling from failure to failure with no loss of enthusiasm.”

http://www.afr.com/technology/the-changes-australia-must-make-in-the-digitally-disrupted-world-20150904-gjeym9