Publication: INFOSECURITY PROFESSIONAL INSIGHTS
Latest Issue: 2016 Volume 6, Issue 6
Publisher: ISC2
By Raj Goel
“When I say that the company’s prosperity rests on such things as our sixty-six-steps-to-clean-a-room manual, I’m not exaggerating.”
— J.W. Marriott Jr.
As I look into my crystal ball, 2017 looks a lot like 2016. We’ll still be short-staffed, and making new hires will be a challenge in spite of, or because of, information security’s near 100 percent employment. Breaches will get bigger and attack vectors will expand. On a more positive note, we’ll still have jobs: an unending queue of tickets, upgrades, projects and meetings.
So, how do you deliver better quality of service, reduce response times, and make your users, management and board happier next year? By increasing staff productivity.
First, let’s consider the cost when there’s a lack or loss of productivity in the workplace, based on my own experiences and research:
1) If performing a task costs you X hours and Y dollars, fixing mistakes and having to re-do that work adds 150 to 300 percent more hours and dollars to the original costs.
2) Those mistakes and do-overs also cost you user trust, client trust, sleepless nights, and make life miserable for everyone involved.
3) Removing productivity loss by standardizing staff training, ticket-handling procedures and customer service processes (yes, we’re all in the customer service business!) has increased our team productivity by 40 percent over the past three years.
Increasing Productivity without Working More Hours
The secret to increasing productivity, good service, and making staff, users, management and stakeholders happy is delivering consistent service. What does consistent service mean? Things are done using standards, processes and checklists so that you, your staff and your clients get the same or similar results every time.
Once you look at your existing tickets or open issues, you’ll realize that approximately 70 to 90 percent of what we do is a repeatable, reproducible process:
• Checking firewall logs for anomalies? Systematize it.
• Checking backups? Systematize.
• Setting up desktops, phones, servers, networks, widgets? Systematize.
• Training new hires? Give them runbooks.
Most of us are familiar, at least in theory, with standard operating procedures, or SOPs. A runbook is a collection of organized SOPs. As business expert T. Harv Eker once said, “How you do anything is how you do everything.”
In our world, consistently creating and, importantly, using SOPs and runbooks shows how to do something repeatedly, without deviation or failure. It also means any changes or updates are reflected in these “living documents” that are routinely reviewed instead of gathering dust on a shelf or in a shared folder.
Need another reason? How about this: No great brand thrives without building SOPs.
Another way to think of it is to consider an SOP as a recipe:
• It shows how to configure tools.
• It shows how to respond to alerts.
• It provides specific steps to investigate an event.
• It outlines how to set up that desktop.
A runbook is more like a cookbook, a collection of all the recipes organized thematically.
Proof of Concept for a Stronger SOP Culture
My company is a managed services provider and managed security service provider for New York City-based hedge fund, private equity and construction companies. These are highly regulated, highly process-driven industries with exceptionally high demands for uptime, reliability and service excellence.
Here are several ways that adopting a formal SOP culture has led to major productivity gains, which also translates into cost savings that make our clients happy.
Client Onboarding
Before: It used to take us 20 hours per server to onboard a client. If a client had, say, six servers, that translated to 120 hours or three to four weeks of onboarding time, with frequent errors, dropped balls and delays.
After: Soon after we began developing SOPs, project plans and checklists, we reduced the per-server onboarding time to six hours per server, which translated to two people working together for 18 hours each, or two calendar days.
Firewall Deployments
Before: Firewall installations used to take 60 hours, including time spent on errors, missed configuration items, unasked questions, etc.
After: Firewall deployments were reduced to 20 hours without any “dropped balls.”
Synology SAN Setup
Before: Synology SAN setups with ISCSI and round-robin ISCSI setups used to take 12 hours of our senior systems administrator’s time.
After: The same setup now takes four hours, and we are using newer, lower-cost staff to implement them.
Daily Backups and Security Reviews
Before: Checking backups daily was reserved for “backups technicians,” while daily security reviews were reserved for the security staff.
After: By systematizing the Daily Backups Review (DBR) and the Daily Security Review (DSR) processes, we are able to train our entire technical team in performing DBRs and DSRs. If the primary DBR/DSR person is unavailable, anyone else on the team can complete these mission-critical processes daily. It also means we get a fresh pair of eyes on the processes on a regular basis as we rotate the responsibility, and that has made our DBR and DSR processes significantly stronger.
Semi-annual Disaster Recovery Tests
Before: Until four years ago, we couldn’t fathom performing annual, much less semi-annual, disaster recovery tests for every one of our clients. The amount of planning, time, manpower and vendors involved seemed insurmountable. If we were lucky, we got one or two of these tests done each year.
After: After adopting an SOP Culture, we mandated semi-annual DR tests for all clients. We have performed 68 DR tests in the past three years, with 66 being completely error-free, and two exposing gaps in documentation, processes and systemic issues that were subsequently resolved.
So, as you can see, since adopting a culture of standardization that’s developed documents and been deployed throughout our organization, we’ve lowered our internal expenses, delivered better, more consistent service, and/or increased staff training by implementing cross-training.
How We Became Better at our Jobs
As a 22-year-old IT services company, I thought we had processes and documentation nailed. After all, we do this stuff for a living, so we must be good, right?
WRONG!
Despite our best efforts, most of our documentation was scattered. It was held in Word docs, Excel files, project plans, emails, Post-it notes, and critical information buried in people’s heads. Scattered or inconsistent documentation and processes is no way to run a business or department, and it causes an incredible amount of stress. It also leads to working long days, staff burnout and high turnover.
How did we go from working 80-hour, thankless weeks to 50-hour, highly productive ones? (Yeah, I wish I could say we work 40-hour weeks or four-hour weeks like that Tim Ferriss guy, but that’s just not our reality.) By embedding SOPs and runbooks into what we call SOPCULTURE, we reduced staff turnover, reduced staff burnout, and increased revenues and profits.
Today, we are a much healthier company with a happier workforce because:
• We create SOPs for everything, from the simplest task (setting up signatures in Outlook) to the most complex.
• We live by the SOPCULTURE mantra in which you are either:
a. using a SOP unmodified
b. updating an existing SOP
c. creating a brand new SOP
d. working for some other firm (yeah, we fire people for not following/using SOPs)
• After we had developed 1,000 SOPs, we identified the 70 most critical SOPs that our technical team needs to be competent in the field. We compiled that into our “New Hire Technical Training Runbook,” which clocks in at 1,672 legal pages, outlining detailed, step-by-step recipes on what we expect everyone on the technical team to be able to accomplish.
• We identified the 15 SOPs our marketing teams need to execute on a regular basis.
• We bake our SOP Culture into our hiring process. Once we identify promising candidates, we interview them, check their qualifications, and require them to submit sample or existing SOPs.
Like any culture shift, creating a SOP mentality takes time and enormous effort. But I can tell you from personal experience, it is worth it — for you, your company and your clients.
Raj Goel is CEO of Brainlink and can be reached at raj@brainlink.com to learn more about SOPs.
Some additional resources to get you on the way to developing your SOP Culture:
RECOMMENDED READING & VIEWING:
https://www.amazon.com/Checklist-Manifesto-How-Things-Right/dp/0312430000
https://www.brainlink.com/creating-obvious-sops/
https://www.brainlink.com/the-joy-of-sops/
http://www.rajgoel.com/2014/12/mspdojo-client-runbooks/
https://attendee.gotowebinar.com/recording/5011078477814407682
https://attendee.gotowebinar.com/register/7237392723543632641
SOP Word template:
https://www.brainlink.com/wp-content/pdf/SOP_Template.doc.docx
