Welcome to ExtraQuest’s new ezine, DataBuzz. It is designed to give you tools to leverage your success. In this monthly ezine, we share our knowledge and expertise about linking database administration with business strategy to improve operational maturity.
ExtraQuest’s vision is built around three Ps – product, people, and process. At the heart is our Product, which standardizes, automates, is proactive, and provides a 360 degree view of your operations. This heart is nurtured by our People, who are expert DBAs with an average of 10 years’ experience. Industry standards are the platform for our Process, which keeps you at the leading edge of business.
If your DBAs are constantly fighting fires rather than helping your business grow, ExtraQuest’s managed services solution could be your answer. Our VP of Managed Services, Matt Wilkinson, has provided a diagram that encapsulates all that we do so that you can see how we integrate and automate to create powerful efficiencies for companies like yours. In his article crash, bang, wallop he shows how our basic services methodology works in practice.
If you have a burning question, then our technical expert and co-founder Venkat Devraj is here for you. This month he answers your ever-pressing question: “What is the one thing you can give me to get me the biggest cost savings?” You can ask Venkat a question by emailing us.
Our other co-founder, Rainier Luistro, had a vision and that vision is now reality through ExtraQuest. He tells all in this issue.
And as for the market: where is it going? Our VP of Marketing, Brian Staff, talks about the latest trends in database administration and how they can provide you with operational and resource advantages. You can also tune into B-Eye Network’s podcast to hear Brian being interviewed.
For this issue, we’ve also profiled a typical customer whose database administration function we have taken over, allowing them to focus on driving their business forward. If you want to be showcased in future issues then just let us know by contacting our editor Alison J. Macmillan.
I am proud to be ExtraQuest’s CEO leading a world-class team of DBAs, professionals and executives. We want to serve you well today and tomorrow, so we need to get to know each other. This ezine is 2-way. Click on the links to contact a particular person, ask a question, or find out a bit more. Together we can improve your database administration AND cut your costs.
Thor Culverhouse
CEO
You are in the middle of a restore. Your database crashes. Your adrenalin rises and your heart starts pumping. You apply the fix you did last week and everything calms down. The user stops complaining …. until next time. You have not learned, you have not automated, and you have simply reacted.
For the ExtraQuest user, they get their issues fixed fast and the support system automatically learns from what has happened so that it does not recur. Our services methodology is based on three simple principles:
1 - When something breaks, fix it and fix it fast.
2 - Learn what caused the problem and make sure it does not happen again.
3 - Look for other potential issues and fix them before they become a problem.
It is as simple as that. Of course, the more you drill down into detail the more complex it becomes. Thankfully, we have some solid industry standards in place such as Six Sigma processes based on the ITIL (the IT Infrastructure Library –http://www.itil-itsm-world.com/) service framework that help us enforce best practices at all levels. At ExtraQuest we have another weapon in our armory called the Operational Maturity Model, which simply means improving the operational efficiency of our clients’ database administration functions. In future articles I will go into these standards in more detail. But for now I would just like to focus on two particular areas that improve your database operations.
The first is standardization. Standardization around best practices means that we always apply agreed-upon and optimal procedures for the appropriate circumstances. As an example, in your Los Angeles IT Department your DBAs may do an Oracle refresh on Linux in one way, whereas in your Chicago IT department your DBAs do the same type of database refresh another way. For an ExtraQuest customer, on the other hand, these tasks are performed identically for each platform according to agreed-upon procedures. What this means for you is that you get a much more efficient and auditable operation.
The second area is automation. Up to 85% of a DBA’s work consists of repetitive, mundane tasks and responding to crises. At ExtraQuest we automate as much of the client’s workload as possible and in this way improve the efficiency of the operation. Nowadays, smart DBAs are engaged in capacity planning, architectural reviews, talking to the business units and helping drive the business forward, not just keeping things going. Enabling DBAs to participate in the strategy of the business adds major value.
Aligned with standardization, automation of common processes is a winning formula. In next month’s article I will talk in detail about how our Managed Services are structured to deliver you a higher quality service at lower cost. You do not need to wait, though. If you have questions right now just email me.
Matt Wilkinson
VP Worldwide Services
ps This diagram encapsulates our services.
Venkat, "what is the one thing you can give me to get me the biggest cost savings?"
"Hah, that’s easy, my friend - automation. Next question, please!"
OK, I’ll explain.
Automate and you’ll realize the biggest tangible cost savings in IT today. Everything else is a temporary panacea.
You want substantial cost savings without degradation in service quality. IT services are delivered using three components: people, process, and technology. The cost of delivering and managing all three components has to be controlled effectively.
Organizations are finding out that cost savings are not sustainable when they rely only on personnel reduction and process tweaks to control them. The costs of human capital in places such as India and Eastern Europe are rising. Additionally, as increasing numbers of bodies in far-off places are leveraged, higher inconsistencies in quality of service develop. Furthermore, the practice of leveraging bodies just does not scale – there are just not enough senior technicians on the planet! Less seasoned technicians take longer to do the work and often cause more errors. This compromises the organizational service level agreement and degrades the end-user experience with IT.
The only real solution is automation. Your senior technicians need to categorize their daily work tasks into mundane and non-mundane/creative. The mundane need to be automated and run with little to no human intervention (lights-out execution). Your technicians can then take on more tasks, especially the non-mundane/creative ones, without you having to hunt constantly for more and more of these scarce bodies at a lower and lower price to scale their operations.
Unfortunately, over the last decade automation has acquired somewhat of a bad reputation in database administration due to sub-optimal design and implementation by the early practitioners. Traditional automation routines were implemented via scripts, which were written for one application or database, without considering generic and mass deployment. For instance, these scripts had specific environmental meta-data such as database names, software and data-file locations hard-coded in them; they made several assumptions about the target system they were expected to run on. As a result, while they were reliable for the single job and server they were built for, they couldn’t automate the same task across a multitude of servers and database platforms. When the target server environments changed, they couldn’t keep up, so they would often break, requiring the DBAs to spend hours manually troubleshooting and cleaning up the mess.
Also, DBAs would tend to accumulate and carry around a tool-bag of 20 or so scripts to handle routine monitoring and alerting. Each of these scripts would need to reside on the database servers and be scheduled to run via cron or a comparable scheduler. As environmental complexities and the number of servers began to go up, maintenance of these scripts turned into a nightmare. Even changing one line of code would require the DBA to log into tens, or even hundreds, of servers to make the change manually, potentially causing human errors, and overworked and frustrated DBAs.
Some DBAs are partial to certain scripts, whereas other DBAs prefer other scripts. As a result, there is little uniformity and a significant difference in tribal knowledge. A single individual may know more about a certain environment and a set of scripts than her peers. If that individual parts ways with the company, they leave a huge gap that others can’t fill, causing rising costs and effort, and a decreasing bottom line.
The intelligent solution is automation, driven from a central/shared knowledge-base that evolves with the changes in the underlying environment. It uses generic and centrally located automation routines capable of running on a variety of target platforms. The organization is injected with higher efficiencies, and the disadvantages of the old script-based automation are kept at bay. Your people can then channel their intelligence and energies into strategic tasks.
The good news is opportunities for automation are staring us in the face every day. My rule of thumb is: if you have to perform any fixed sequence of steps more than twice, document the task and share it with everyone in the team. Now, everyone can be on the “same page” and perform the task in a consistent manner. Subsequently, that documentation serves as a great blueprint for any downstream automation using a central mechanism, like the one described above.
Venkat S. Devraj
Chief Architect
I felt I was watching Bill Murray in Groundhog Day. He gets a plea for help from the Support Desk to fix Joe Blow’s application because it’s running slow. He heaves himself out of his chair, picks up a coffee and lumbers on round to the DBA room “Hey, guys and gals, that guy from yesterday says his system is still going slower than a tortoise after a heavy night out.” And the roomful of DBAs gives him no end of solutions. All of them different. All of them highly recommended.
“Thanks, guys, I like Jinx’s solution, I’ll go phone Joe and get him off my back.” Whistling to Shirley Bassey's Heh, Big Spender. The minute you walked in the joint…, Bill jives back down the corridor, and passes the lunchroom where he meets five more DBAs. They give him five more solutions that are different. Bill’s not whistling now, he’s scratching his head and thinking “wouldn’t it be better if these people marched to the same band?”
As I saw this scenario played out again and again, I had a compelling desire to help the DBAs improve their lives. If they could stop fighting fires, their companies could reduce their DBA budgets and serve their clients better.
Wouldn’t it be better if we got the best and most efficient way of resolving the issue, documented it, and put it in a best practice bag that goes with the DBAs wherever they go? We can gather all the best practices in the DBA world put them on a repository and share it with everyone. Even more, DBAs can instead do the more interesting fun part of DBA work like database architecture, performing trend analysis, checking and testing out new DB products and features to benefit the organization, and much more.
The seeds of this vision gave birth to ExtraQuest Corporation on January 10, 2001. We vowed to commit ourselves to reshaping the DBA world through standardization and process automation. This coupling produced RoboDOC, the tool you can use to help document your Standard Operating Procedures, and the framework that automates your processes. ExtraQuest knows it is not the tool alone that will ultimately solve your problems. ExtraQuest also supports your people and nurtures a mindset that believes in standardization and automation.
If your DBAs’ groundhog days are pager calls in the wee hours of the night, upset vacations and family time, panic attacks when they can’t contact the on-call person to fix something, and heavily pounding heartbeats when they miss the SLA deadline, then your temperature is likely to rise when you hear about it the next day. My vision is now a reality through ExtraQuest and I am proud to stamp out the groundhog days of the companies we serve. How’s your day?
Rainier L. Luistro
Co-Founder
The database administration market is like the restaurant market. Both are huge and growing fast. There are about as many DBAs in the country as there are chefs. The US Department of Labor states that there were 104,000 DBAs in the USA in 2004. The profession is the sixth fastest growing in the country and is expected to shoot up by 38% by 2014, which means that there will be close to 145,000 DBAs in the USA by that time. To put it in perspective, there are about as many DBAs in the country as there are chefs, and although I don’t mean to imply that DBAs in any way cook the data, they are as essential as the people who feed us. And so they should be, data nourishes an organization and without it the company starves.
But it’s an expensive meal. In 2005 the cost of an entry level DBA in the USA was between $65,000 and $95,000 a year, which points to an average salary of around $100,000+ a year for an experienced professional. Add benefits, overhead, and the tools they use to do their jobs, multiply by 104,000 and you’ve got an awfully big number, a tens of billions of dollars a year number, and a number that’s growing at one of the fastest rates that the US Department of Labor tracks.
This is, of course, great news for DBAs, who are part of a fast-expanding job market, and if demand outgrows supply, as it very likely will, DBAs are going to be very popular folks.
But for companies chasing a bourgeoning rare resource, this growth is going to be something of a headache. Job slots will be expensive to fill, and increasingly they will go unfilled as DBAs get to pick and choose. And the DBAs themselves will be confronted with ever-growing workloads. They may land plum jobs, but they’ll be expected to work all hours of the day and night to meet the mountainous tasks that are dumped on their plates.
One solution to this impending lack of DBAs is outsourcing to countries where resources are more plentiful, and many organizations are already using that route. But outsourcing, as Venkat says in this month’s Ask the Expert, isn’t a universal panacea. Costs of labor in countries like India are rising, and DBAs there are also in demand. And some organizations have policies that prohibit them from using outsourcing, either for security reasons, or because of contractual agreements with unions that prohibit them from sending jobs overseas.
So there needs to be another solution. According to leading analysts (such as IDC and Forrester) and other industry experts, pain relief for the DBA headache could come from database administration tools that deliver in three key areas of functionality. So here’s the punch line of this article and the tip for the week: When looking for database administration solutions, focus on products that enable you to work towards these three goals:
Standardization – applying best practices across an organization’s entire data asset, no matter the quantity or type of database. Most companies have a variety of ways of performing a single task, depending on who’s performing it and what platform they’re performing it on. Finding the best way, documenting it and applying it rigorously and universally is the way to reach operational perfection.
Automation – automating as many of the repetitive tasks that DBAs perform (and up to 85% of a DBA’s workload can be spent on tasks that can be partly or fully automated). Needless to say, automating the standardized best practices identified in the previous bullet is the way to go.
Autonomics – or proactive automation, which means finding problems before they occur and fixing them before they affect operations. Also, correlating the practices or events that lead to the best outcomes provides the ability to automatically tune the system for optimal performance.
To go back to our chef analogy, the most efficient restaurant has one standard way of making a cake, they automate as much of the process as possible, and they heat up the oven before they start mixing the ingredients!
Dr. Brian Staff
VP Marketing
Digital Reliance, a premium provider of wireless account management services, needed a better way to optimally manage its mission-critical Oracle 8i database in a highly secure environment. They chose ExtraQuest. In return, they realized savings of $190,000 (more than 75 percent).
How did they do it? Well, by automating data management practices and minimizing complexity they were able to focus on business issues rather than having to deal with technology-related challenges. ExtraQuest helped Digital Reliance to adopt the philosophy that Forrester Research, a leading market research firm, says is going to be standard by 2009. By that year, most of the database administration tasks such as migrations, upgrades, patch deployments, replication, data loading and performance tuning will be completely automated. ExtraQuest’s clients achieve this level of automation today.
Mission Critical Data
Being a leader in wireless account management services, Digital Reliance process millions of mission-critical data elements. The company’s data is highly confidential and needs to be maintained and managed in a highly secure environment. The database is subject to both near real-time transactions, as well as large-scale batch and reporting operations. In addition, the database is subject to high growth and usage and requires a robust, scalable, optimally performing and highly available database environment.
Due to usage constraints and customer presentations, Digital Reliance could not tolerate any substantial downtime or slow performance and suffer the consequential direct loss in revenue and reputation. The database had to be up and running ceaselessly and had to offer lightening-fast response times. In addition, the company’s database administration costs had to be kept to the bare minimum, without sacrificing quality of service or compromising security of data.
In conjunction with Digital Reliance personnel, ExtraQuest helped stabilize and “wire” Digital Reliance’s Oracle 8i database environment. The “wiring” enables the company to perform proactive diagnosis, optimal configuration, rigorous monitoring and specific repairs, as needed.
"ExtraQuest’s expertise and professionalism, combined with the full-time coverage and cost savings associated with the outsourcing of this IT function to a team of professionals, made it an easy choice to go with ExtraQuest," said Digital Reliance CEO Brick Thompson.
ExtraQuest is proud to have served an innovative market leader like Digital Reliance. In future issues of DataBuzz, ExtraQuest will be featuring other client successes and tips. If you have an ExtraQuest success story to share, please contact our editor, Alison J. Macmillan, at info@extraquest.com and we would be honored to work with you to showcase your story. We look forward to hearing from you.
* Digital Reliance has been acquired by Vercuity
 |
 |
 |
 |
 |
 |
 |