Just over a decade after its formation bet365 has grown into a global operation that employs over 1,400 people and is live in 17 languages. It has four datacentres located around the UK that deliver its web presence, enables data flow and bet reconciliation and provides disaster recovery and business continuity functions.
The initial operation was very small with the bare minimum of physical infrastructure; switch, router, firewall web app server and a database, housed in a third party collocation facility. It was Denise’s vision to focus on In-Play betting that saw player numbers soar and with it the complexity of the network needed to sustain the company’s growth.
THE CHALLENGE
The DNA of In-Play gambling is such that data exchange and transaction reconciliation has to be handled extremely quickly. With In-Play users have the ability to bet on a range of events that take place whilst a sport is in progress from the next corner in a football match to the next point in a tennis game. They need information quickly to make the right wager and want the best odds for that bet.
As such In-Play requires the delivery of a continuous stream of real-time information, whilst simultaneously receiving and processing a huge amount of incoming customer data. It’s a game of high availability and low latency on a massive scale. If your service does not meet your customer’s sometimes impatient demands, you run the risk of losing them.
“Compare it to buying a sofa,” says Neil Selby, Head of Networks and Security, bet365. “It doesn’t really matter if the webpage you want is unavailable for a while; the sofa will likely still be there a day later and at the same price. If you’re watching the Grand Prix and the drivers are on lap 40 it’s a different matter altogether. You don’t have time to wait 15 minutes for the odds information or to place your bet. Any delay or glitch can negatively impact the player’s enjoyment and as a result the retention of that customer.”
Availability of the system and the odds data is therefore paramount both in terms of ensuring that the user experience is smooth and glitch free but also in terms of having the infrastructure in place, so that everyone that wants to play can do so.
“We will accept nothing less than being the best in our industry. That means offering the best customer experience possible,” said Selby.
There are 4 major factors that drive bet365’s pursuit of perfect availability:
Player Numbers
Consider that bet365 can have many 10s of thousands of concurrent users on the main In-Play system and over 1,000,000 concurrent users on its affiliate banner system - all tracking odds. This is a lot of users for any system to handle and with the business doubling in size every 18 months or so it’s imperative that bet365 infrastructure is prepared to meet the increase in players.
Data Exchange
With so many players it’s not surprising that bet365 clocks around 1,000,000 high velocity packets in and out of its network every second. It’s vital that the data reaches the recipient as quickly as possible so that they are betting with the current odds not those from before the last goal, wicket or try. Reducing latency to as close to real-time as possible is a key objective.
Peaky Traffic
In other industries companies attempt to shape demand - off-peak fares on trains, off-peak tariffs on phone networks, lower fares out of high season on airlines - but betting on a events is typically when the event is imminent and the luxury of shaping is not realistic. bet365 cannot influence when a goal is scored or a race is won. The result of each event is that there are 1,000s of bets that need to be reconciled at the same time. What would be considered special events for most, happen on a daily basis for bet365. They can’t streamline the load and so they have to dimension for the peak.
Packet Behaviour
With so many packets moving through the network the likelihood of issues such as congestion or packet loss are much higher. Any anomalies from the system running slow to broken links and even missing images must be identified in moments. A retrospective view is not good enough; but with gigabytes of information written every 30 seconds for monitoring and forensic analysis it is difficult to get a real-time view.
“There are a number of readily available alternatives to bet365 and unlike banking or the mobile phone market, there are minimal barriers to migration, so it’s imperative to please customers, every time they bet – in quiet periods or at the peak of the busiest days/events,” said Selby.
THE SOLUTION
bet365 has overcome its challenges by always having one eye on present challenges and the other on potential future issues. Denise understands the vital role that technology plays in delivering a best-in-breed Internet company and has worked hard to create a culture where the Technology teams have the freedom to do what’s needed to deliver a first class service.
IT investment runs into the tens of millions every year to ensure that the network infrastructure and considerable bandwidth needed is in place to ensure the availability of bet365’s products. The technology challenges faced by bet365 are considerable and in many cases the company is dealing with issues that few others have had to deal with before. The solutions therefore require people who are not afraid to tread new ground.
Distributed Computing
Traditional technologies can only take bet365 so far. With player numbers set to grow far into the future, the company is coming away from SQL database technologies and looking to software solutions to achieve better scalability and flexibility. The middleware team is helping to solve the scalability issues through the development of sophisticated distributed software systems. They are taking the load off the backend database systems by storing data in memory clouds. Not only does this make the system faster and more scalable but also makes it more cost effective to run as the software can be housed on commodity servers across multiple data centres.
Data Push
In conjunction with the cloud development, innovation in push technology has enabled bet365 to develop a stateful relationship with its connected customers. Rather than have the user repeated pull the data, which can be frustrating, information is “pushed” to their screen in close to real-time and in response to real-time events (goals, wickets). Odds data appears to refresh instantaneously with latency under two seconds.
Complex Event Processing
Traditional monitoring techniques like looking at the CPUs on the server or studying the network ports looking for congestion only provide a very limited view of what is happening. For bet365 there is no substitute to watching the flow of client requests coming in from the Internet and monitoring the response back from the server to the client.
Instead of storing information in a static database and running queries on it, bet365 has integrated a CEP system that feeds event streams from the network through a continuous query stream. The events are then aggregated in memory with very low latency. The CEP system has the intelligence. It takes the data, looks for patterns that suggest an error and then escalates the issue to the appropriate team.
This allows bet365 to construct performance analysis systems that run on a single CPU core. The result is a completely passive monitoring system that puts little additional load on the network but sees everything that is occurring and can spot anomalous behaviour.
THE RESULT
The hard work has certainly paid off. In 2010 bet365 was recently voted number 1 in eGaming Review’s Power 50, which rates the leading online gambling businesses in the world.
Key Statistics
§ £60,000,000 IT spend in 2010
§ 10s of thousands concurrent players on the main In-Play system
§ Over 1000,000 concurrent users on the affiliate banner system
§ Around 1,000,000 high velocity packets clocked per second
§ 100,000s of markets covered at any one time
§ Millions of members worldwide
CONCLUSION
While perfect availability remains a Holy Grail, bet365 is doing everything it can to ensure that its customers get as good an experience as possible. Whether it requires increasing bandwidth and the physical infrastructure, developing innovative new software solutions or creating a 360 degree view of the network, bet365 is dedicated to giving its network teams the tools needed to pursue perfection.
“Our IT teams are consistently smashing through barriers to overcome our challenges and bet365 is always on the lookout for people that are not afraid of breaking new ground,” concluded Selby.
 ShareThis
Tags: Data Centres |
