Speed matters. That’s why you’re willing to pay a dollar more for milk at the convenience store — it’s quicker than waiting in line at the supermarket. That’s why Facebook spent nine months developing Instant Articles; it saves users seven seconds of load time. And that’s why you will lose business if your database slows down the customer experience.
This is especially true of e-commerce websites, where databases are the source of truth for items, stock, prices, orders — everything. Forty percent of people won’t wait more than three seconds for a mobile shopping site to load. Amazon estimates it would lose $1.6 billion a year if its web pages took one second longer to load. If consumers won’t wait one second for Amazon, they won’t wait for your database to come back online.
And that’s just the direct bottom-line impact. Consider the other effects of poor database performance:
1. Lost time: Database incidents ripple throughout companies. The operations department is first to respond, but developers and other information technology workers are impacted, too, along with employees in seemingly unrelated departments.
In some companies, outages are board-level issues because they are so costly. The time spent resolving an outage is time not spent on other company initiatives such as product development, marketing, or recruiting.2
2. Reduced productivity: Repetition is the bane of efficiency experts everywhere. When databases go down, more work needs to be redone.
Canceled meetings have to be rescheduled, aborted software deployments have to be restarted, and sales proposals have to be sent back to legal to add service level agreements. And workers who were interrupted must return to their tasks and reestablish flow.
3. A damaged reputation: Aside from leading to a slow website and distracted workers, a sluggish database can mar a company’s reputation for quality service. And this means it will lose customers and partners.
Fighting this reputation not only requires an intense technical battle, but also a public relations race to appease disgruntled customers and respond to social media complaints. Some customers can be saved, but others will be lost forever.
4. Shattered confidence: Failing in basic competencies such as keeping a database up and running will sap your team’s confidence — a necessary ingredient for high-achieving teams that move fast, act decisively, and take calculated risks to innovate.
5. Work-life disruption: Database downtime often occurs after work hours and involves late-night heroics to fix. Employees who are caregivers, have a long commute, or must attend to other priorities in life can quickly burn out if they are constantly on call. If you regularly disrupt employees’ work-life balance, you risk losing those team members and having to find and train replacements.
Companies that have their act together, however, can be havens for these burnout victims.
Having a fast, reliable database does more than prevent these pitfalls. Workers who don’t have to solve 2 a.m. database emergencies are more rested, productive, and creative. And when a database doesn’t make completing routine tasks a slog, developers will test innovative ideas that result in new features and functions.
Imagine being the first company to build a feature — such as a related products suggestion engine for e-commerce websites. If you had to struggle with database performance just to build the product page, you probably wouldn’t try such an idea at all. So what precautions can you take to keep your database up and running as often as possible?
Strategies to Improve Database Reliability
Given the mission-critical nature of database performance, you need to:
1. Monitor uptime and performance. This is your starting point. You can’t improve what you don’t measure, so you need to monitor your database on a granular level. Your customers should not be the first ones to notice a problem.
2. Reduce database diversity and specialization. Having too many technologies in your platform leads to brittle systems that are difficult for one person to understand as a whole. As a result, only certain employees can manage and operate certain portions of the system.
You want many people to be able to manage and operate as much of the system as possible. Resist the temptation to hire people to specialize in the more complex portions of your platform. Cross-training your team members is a vital way to build a flexible team that can respond to any crisis.
3. Align responsibilities and authority. Don’t hold others accountable for tasks they have no control over. This only breeds an atmosphere of disempowerment, distrust, and disengagement. When there’s an outage, managers who create this situation typically are the ones who find a scapegoat instead of fixing the systemic problems, guaranteeing a “Groundhog Day” scenario the next time there’s trouble.
People should only be responsible for what they can control. If you appoint team members to monitor and preserve database uptime and performance, give them what they need to get the job done and get out of their way.
4 Make performance a top-down priority. Establishing a mandate from above — even from the board level — to measure and improve performance will rally everyone around a shared goal. Etsy’s quarterly performance reports showcase the brand’s company-wide commitment to performance.
5. Invest the necessary resources. This means money to hire staff and purchase equipment and space. But you’ll also have to dedicate time to writing a strategic plan that outlines goals, schedules, tactics, and metrics, including a process for revising the plan itself.
By applying these methods, as well as seeking and maintaining the right team and vendors to implement them, your company won’t have to suffer from the obvious and hidden costs associated with poor database performance. Don’t be a tortoise — free your entire team to capitalize on other important initiatives and innovate into the future with a fast, glitch-free system.
Baron Schwartz, founder of VividCortex, is one of the world’s leading experts on MySQL, and he has helped build and scale some of the largest web, social, gaming, and mobile properties. His award-winning tools are used by tens of thousands of large MySQL deployments, including Facebook and Twitter. His book “High Performance MySQL” is widely regarded as the definitive reference for MySQL.