<p>The CIO’s Guide to DevOps Contents Software enables every business 3 The old ways don’t work anymore 7 You want digital transformation? Look to DevOps. 9 Infrastructure as code: a cornerstone of DevOps 12 The metrics of DevOps: proof that it works 17 Don’t fear the DevOps: security and compliance 21 DevOps is great for talent recruitment and retention 23 DevOps and your change fitness level 26 Introducing and enabling DevOps culture: it’s a journey 28 Puppet and DevOps 40 Resources 44 Software enables every business… 3 …so every company must behave like a software company. Stop wondering if your business will be disrupted. It’s happening. Every aspect of our economy and society is being changed by digital transformation. Countless globally scalable startups are trying out and discarding new business models faster than most companies can launch a new application. 4 That’s because software drives every business now. It’s creating disjointed — and sometimes incompatible — initiatives inside companies. It’s increasingly common to find the chief digital officer (or as McKinsey says, “transformer in chief”) leading digital integration across the company, forcing the traditional CIO to play catch up — or worse, sit on the sidelines. Whether you’re selling high-speed commodity trading services or high-thread-count organic sheets, the customer experience is driven by software. The speed with which your team can deploy new software has become inextricably linked to customer acquisition and retention. If your software gets hung up in the testing stages; if you can't scale effectively from test to full production; if your services keep breaking — you're going to lose customers. And when that happens, a competitor who's already overcome old patterns and learned to operate like a fast-moving software company is going to jump in and give those customers what they want. 5 These torrents of change are pushing CIOs to embrace digital transformation — to transform IT into an innovative, fast-moving software organization. It's the CIO's responsibility to empower the business with technology that will propel it past competitors and beyond their reach. As CIO, you have a choice: ignore digital transformation and supporting movements like DevOps, and get disrupted into irrelevance — or clearly understand what's at stake, and lead your organization through its digital transformation. “Someone is moving faster than you are, and probably much faster. That someone could be a competitor, and if they iterate this much faster than you do, you’re toast.” Matt Asay of TechRepublic, “Why the DevOps faithful keep pulling away from their competitors” 6 The old ways don’t work anymore Traditional ways of managing software and the infrastructure that runs it — the endless meetings, layers of approvals, change ticket backlogs and accumulating cruft — won't cut it in a world where companies run in the cloud, quickly adopting new technologies such as Docker containers, or container-cluster managers like Kubernetes and Mesos. 7 Your IT organization needs to respond quickly to market changes by making more experiments faster, learning from frequent customer feedback, and then using that learning to rapidly deliver new features. To do that, you need to build your change fitness: your organization's ability to transition gracefully from current practices to new practices, leaving behind old tools and processes as you adopt new ones to serve your business better. Once you boost your change fitness, IT becomes a streamlined, standardized and forward-thinking organization. Instead of being the last one invited to the meeting, you’re leading and driving the business. IT gets to influence the customer experience in a more powerful and direct way than any marketing campaign ever could. Ultimately, there won’t be a single project at the company that doesn’t rely on IT as a strategic enabler or leader. You’ll be prepared to meet digital disruption head-on, and perfectly positioned to create new opportunities for your company, faster. How do you transition from the old bureaucratic ways to the new agility? It's called DevOps. 8 You want digital transformation? Look to DevOps. Traditional IT organizations aren't built for this new world. They are organized around functions, not around business value, so they're siloed, with each group using different tools and processes. DevOps isn’t a fixed methodology or process; it’s a community of practice and a set of principles. 9 DevOps arose originally among practitioners frustrated by the old siloed ways. But plenty of IT managers and C-level executives also recognized early on that DevOps — a natural outgrowth of Agile methodologies and lean principles — could help the business move faster by cutting time-wasting processes and creating shorter feedback loops with the customer. “Of course, if you’re slow to act, unresponsive and focused on just keeping the lights on, then your organization will find a different kind of CIO.” Mark Shapland, CIO.com, March 2015 10 You'll find quite a few principles and methods from Agile and lean thinking in DevOps — for example, limiting work in progress and working in small batches; pair programming; continuous delivery; including quality measures early in the development cycle; test-driven development; automation of just about anything that can be automated; and monitoring, coupled with making metrics visible to everyone. Continuous learning and improvement are also a core part of DevOps, so you'll see practices like developers and IT operations people attending each other's standups and planning meetings, and regular retrospectives. Getting the right processes and tools in place vastly improves software quality. But they deliver other important benefits, too. The DevOps toolchain and practices improve how teams work by promoting greater collaboration and faster, more frequent feedback loops. Faster feedback lets teams learn much more quickly from operational results and from customers, so they can build technical and process innovation into the software release cycle — and delight customers, too. 11 Infrastructure as code: a cornerstone of DevOps DevOps is about more than simply adopting the right tools and improving processes. It's also about shifting the culture of the organization away from siloed responsibilities to a pattern of ongoing communication, sharing and trust between individuals and teams. 12 That's why the practice of managing infrastructure as code is so fundamental to DevOps. When you describe all infrastructure configurations and changes in code, you can version that code, review, integrate, test and deploy it with the same tools your organization uses for software. This makes it much easier for your infrastructure and development engineers to collaborate. And IT can shift away from its traditional role of gatekeeper to the new role of enabler, by creating self-service portals for developers and testers. “Automation is the best documentation. Gone are the complex and out-of-date documents explaining how to create and install systems. They have been replaced by code which is used to automatically and repeatably create our critical IT systems.” Geoff Clitheroe, systems development manager for GeoNet, New Zealand's national geological event alerting service 13 Infrastructure as code enables all of these practices: • Version control and peer code review. This enables more teammates to review and collaborate on code earlier in the cycle. These practices build trust in the quality of the code, and the process of creating it. Version control assures that code can be rolled back to its last known good version. Commonly used tools include Git, Mercurial, Perforce, Subversion, and Team Foundation Server • Configuration management. Managing configurations across all environments assures that everyone is developing and testing code in environments that match production, building trust in code quality and confidence that deployments will be successful. Configuration management through code also ensures that all infrastructure changes are reviewable, and that you can see the differences caused by changes. Puppet is the most-used tool for managing configurations as code. 14 • Continuous integration. When you develop in small batches and continually integrate these changes, you ensure that every change works within the code base. It's also much easier to roll back to the last known good version when necessary. Like other tools listed above, continuous integration also builds confidence in deployments. Commonly used tools for continuous integration include Jenkins, Bamboo, Hudson, Go, TeamCity, Travis CI, and Visual Studio Team Services. • Peer code review. It's been shown that lengthy and bureaucratic review-by-committee slows down development, and that it doesn't produce better code. Peer code review, on the other hand, does result in better code, and it's faster. Common tools include Git, Stash and Gerrit. • Monitoring. Along with displaying the results on dashboards where everyone can see them, monitoring exposes everyone to the results of the work. Making data public and visible allows the team to hold itself accountable, to immediately see the results of experiments, and also allows management to understand what's going on. Monitoring enhances communication and trust. Popular tools include New Relic, Nagios, Splunk, AppDynamics, Loggly, and Elastic. 15 • Automated testing and deployment. For consistent, repeatable code, you need automation. Automated testing makes the process both faster and more reliable; automated deployment is much faster than manual, and less error-prone. Together, automated test and deployment deliver better release velocity, while reducing big fires. That frees your engineers, allowing them to focus on improvement and innovation. The tools for automated testing are legion, and depend on what you're testing and which phase of testing you're talking about. Tools for automated deployment include Capistrano, MCollective (part of Puppet Enterprise), Octopus Deploy, Visual Studio and Nomad. By now you're probably thinking, "Show me the numbers." If you're going to consider DevOps as the path to digital transformation for your organization, you need to know how likely it is to deliver success. “If we have the package already developed, what took days before now takes literally minutes. For a new capability — say a new application server container we need to install — what took several weeks is now down to a week.” Jeff Quaintance, senior cloud & automation engineer, Staples 16 The metrics of DevOps: proof that it works Over the past five years, we’ve surveyed more than 25,000 technical professionals worldwide to better understand how the technical practices, cultural norms, and lean management practices we associate with DevOps affect IT performance and organizational performance. 17 We have discovered that DevOps culture and practices can deliver significant improvements in IT processes and business outcomes. The 2016 DevOps survey analysis, for example, showed that high-performing organizations — which employ DevOps practices and live DevOps culture to a high degree — far outperform others, as measured by throughput and reliability. • High performers deploy 200 times more frequently than lower-performing peers. That gives the DevOps organizations opportunity to experiment more, learn more and get new features out to customers more rapidly. Organizations that develop the muscle and habits of frequent deployment have a distinct competitive edge: In a race to get a new service or capability to market, the team that's good at frequent deployment is likely to be first. 18 • Their change failure rate is three times lower, and they recover from failures 24 times faster than lower-performing peers. High performers reported a mean time to recover of less than one hour, while respondents from other organizations reported it takes as much as a day to recover. Being able to make changes so much more reliably lets high-performing orgs move faster, and with a high degree of confidence — confidence that inspires even greater willingness to perform experiments and learn. • They create more shareholder value. We analyzed 2014 survey results from 1,000-plus respondents who volunteered the names of companies where they worked. We found that publicly traded companies with high-performing IT teams had 50 percent higher growth in market capitalization over three years than companies with low-performing IT teams. Respondents from high performers also reported their organizations were nearly twice as likely to exceed their own targets for profitability, market share and productivity. 19 While these findings are dramatic, they should not surprise anyone who understands that DevOps enables tighter customer feedback loops, enabling a company to learn more quickly what's working for customers and what isn't. DevOps also enables organizations to release higher-quality code, eliminating a lot of pullback and rework, because quality is built in from the beginning. At another level, DevOps makes it much easier for organizations to quickly cycle old technology out and new technology in — a critical competitive capability. With technology research firms like Gartner and Forrester issuing reports on the advantages of DevOps, it's not surprising that leading companies no longer consider DevOps a fad. Instead, DevOps is increasingly seen as a highly effective and desirable set of skills and behaviors — a must for companies wanting to gain and hold a leading position in their markets and industries. “We’ve had some great results from our DevOps initiatives so far. We reduced our cost per release on one application by 97 percent ... By automating our testing, we’ve reduced multiple man days of effort down to an overnight hands-free process.” Jonathan Fletcher, enterprise architect and lead for technology, platform and DevOps at Hiscox 20 Don’t fear the DevOps: security and compliance Most of us who have built careers in IT are cautious by nature — even a tad conservative. We know just how badly things can mess up. And it seems like every week there are more headlines to justify our caution. Emails between entertainment executives get hacked. An airline loses millions when flights are cancelled due to the failure of a single router. A system failure causes a huge telecom to send wildly inaccurate bills to customers, triggering a deluge of customer calls, service cancellations and all-too-public complaints. 21 So it's natural for some to view DevOps with skepticism, wondering if faster code throughput will cause more errors or compromise security. In fact, the opposite is true. Organizations that adopt DevOps reduce downtime; release cleaner code because they find (and correct) flaws earlier; and improve both their regulatory compliance and their ability to report to auditors, by improving transparency and trackability. The 2016 State of DevOps Report showed that high-performing IT teams — which make extensive use of DevOps practices — spend 50 percent less time remediating security issues, compared to lower-performing IT organizations. That's because they build security in from the beginning by including security colleagues in the planning process, and by building security policies and tests into software development and infrastructure architecture. less time remediating security issues 50% High performers spend 22 DevOps is great for talent recruitment and retention Smart developers, operations people, network and database admins need and want to work with the latest tools, and to work on challenging problems. Younger employees, in particular, expect to see the results of their work quickly, and the tighter customer feedback loops that DevOps enables give them that satisfaction. If employees don't get what they want at your company, there's another one down the road where they will get it — and that's where they'll go. 23 Most executives understand that highly engaged employees contribute more value. In fact, research has shown that companies with highly engaged workers grow their revenues 2.5 times as much as companies with low employee engagement. Publicly traded companies with a high-trust work environment (a characteristic of DevOps organizations) outperformed market indices by a factor of three over a 14-year period. Today's engineers want and expect DevOps, with its emphasis on close collaboration and continuous learning. DevOps practitioners want to stay up to date, so they question the status quo, look for better ways to do things, experiment with tools, and alert you to when you're at risk of falling behind. These are the kind of engaged employees you need to stay competitive. 24 “It’s great that AON cares about the cultural change of DevOps, and invests in the technology to support that change. We have a mature, modern delivery system that most enterprises don’t have. I can use that for recruitment.” Glenn Mason, solution architect at AON Employees in high-performing teams were more likely to recommend their organization as a great place to work. 2.2x 25 The 2016 State of DevOps Report shows that employees in high-performing teams are twice as likely to recommend both their team and their organization to a friend as a great place to work. So DevOps can also help you recruit more smart people to grow your team. DevOps and your change fitness level You can never really know what changes will come at you from the market, from your company leadership, or the overall economy. All you really know is that change will come, and you need a tech organization that's ready to meet it. 26 Companies that adopt DevOps culture, processes and tools get very good at managing change. They become both disciplined and flexible: able to evaluate new conditions, brainstorm ideas, come up with a plan, and execute on that plan. These teams experiment, evaluate results, and move forward based on their learning. DevOps builds the muscle and flexibility you need to cycle new technology in and old technology out as business needs evolve. That's why DevOps is no longer a cult movement, but a proven way to succeed in fast- moving markets. “You’re more reluctant to pull something back that took six months to release rather than just a couple of weeks to release, regardless of its actual business value.” Juan Rivera, manager of storage and platform engineering at Getty Images 27 Introducing and enabling DevOps culture: it’s a journey Adopting new tools and processes can look daunting. But as it turns out, the hardest part of digital transformation turns out to be changing people’s behavior. Change is scary. No one wants to fail. So adopting a mindset that tolerates failure and encourages experimentation is exactly what you need to transform your company. 28 That mindset is the culture of DevOps. Beyond introducing automation, version control and other technology changes, it's critical to shift your organization's culture to one of continual experimentation and learning, backed by the accountability that comes from exposing performance metrics to everyone. In DevOps, the practices support the culture, and the culture supports the practices. You can't have one without the other, and you can't transform your company without both. Companies that successfully implement DevOps don't do it all at once — they embark on a journey of organizational transformation. This journey is, naturally, different for every company, but there are still distinct stages: 1. Discovery 2. Standardization of tools and processes 3. Continuous iteration and improvement 4. Implementing business strategy through technology innovation 29 1. Discovery Before you make any changes, you need to discover the state of your technology organizations, including their tools, processes and practices. This is the time to ask penetrating questions. • How do we release code? Who's involved, and what do they do? • How long does it take to for developers to get the environments they need? • Do we use version control? Which system do we use, and what's kept in version control? • Do we use continuous integration, and to what extent? • How often do we build? • How often do we deploy? • Which elements of our software cycle are automated, and which are manual? • At what point in the software cycle are our security and compliance policies and processes involved? 30 The answers to these questions should help you understand what to tackle first. If not, ask your technical managers where they think the greatest obstacles are to faster, easier deployments. 2. Standardization of tools and processes Now it's time to eliminate ad-hoc processes, and standardize the tools and processes used in your technical teams. What to change first really depends on the answers you got to your questions about what your teams are doing and where the greatest blockages exist. Here are the top changes to make at this stage. • Automate. Technical teams are often so buried in manual work, they can't even think about how to improve efficiency. DevOps experts often recommend that you start by automating something small but painful. This can improve morale and free people up to think about what's next. Make sure environment configuration gets automated so that development environments and test environments match production. 31 • Get everything that's needed to produce a build under version control. This includes application code, infrastructure code, deployment scripts, test scripts, etc. • Implement continuous integration if it's not already in use. • Make metrics visible. Insist on displaying in workplace areas your organization's key quality and productivity metrics, plus the state of work in progress. • Build self-service platforms. Developers should be able to provision themselves with correctly configured environments. Same for test and QA. And other departments, such as finance and HR, should also be able to safely provision themselves with tools in a way that's consistent with your organization's security policies. This is a later stage for many organizations, but some are able to begin building these platforms pretty quickly. 32 Cultural changes must also take place at this stage. Teams that have successfully adopted DevOps recommend a number of practices that help improve understanding and communication between technical teams, and that will help your teams see the how and why of standardizing processes and tools. Encourage or mandate these behaviors: • Development teams should invite IT operations people to software planning meetings. • Operations should invite developers to planning meetings. • Have operations hold blameless postmortems after incidents, and invite developers to these meetings. • Encourage developers and IT operations people to shadow the other team. • Institute peer review of changes, rather than having a change approval board review changes. 33 • Have managers publish progress reports. This will surface any potential blockers early, and also encourage teams to share successes with each other and with the company. • Bring your security people into software planning meetings and infrastructure planning meetings. The earlier security is involved, the better you can build policy into your software and infrastructure. You can get more suggestions for implementing and supporting the cultural changes of DevOps in Get Started with DevOps: A Guide for IT Managers. 34 3. Continuous iteration and improvement Many companies start their DevOps journey with a specific initiative, or by creating a new team to implement and prove a new set of tools and processes. Once that initial DevOps project has shown success, it's time to spread the goodness further: to other projects, to other teams, or both. How you do that depends on how you started and what your business needs most. Hiscox, a global speciality insurance company based in the United Kingdom, started with a newly created DevOps team and a new application. After this team proved that the process worked — they reduced the cost of release by 97 percent — Hiscox has spread its DevOps learnings to its VMware virtualized infrastructure and cloud infrastructure. Eventually, DevOps will just be how Hiscox does everything, including the use of Puppet to deploy every type of software Hiscox runs. 35 Some companies start with one type of environment and then expand to another. Walmart first deployed Puppet to its in-store servers. After successfully deploying Puppet to more than 49,000 Linux machines, the retail giant now in the process of deploying to its 50,000-plus Windows servers. FINRA's DevOps initiative started with the regulatory agency's desire to move its entire application portfolio from its on- premises data centers to Amazon Web Services. FINRA knew it would need to treat infrastructure as code in order to make that transition and to manage applications successfully in AWS; hence its decision to adopt Puppet. The agency was able to move two applications to AWS within six months, and then develop self-service infrastructure provisioning for developers in AWS. Using the same infrastructure code for all of its infrastructure, whether on-premises or in AWS, has allowed FINRA to port the strict security standards it applies in on-prem data centers to its AWS infrastructure, and to take new server tagging methods learned in AWS and apply them to the agency's on-prem servers. 36 In your own organization, you'll want to ask some questions at this stage: • Which pieces of deployment and steps in testing are still being done manually? • Get these automated, starting with the pieces that cause the most blockage. • How successful are our deployments, as measured by error rates, service outages, unplanned work and other important quality metrics? • Here's where you may discover some gaps in monitoring. Fill them. Add more automation where needed to correct deployment issues. • Which tools in our toolchain can be integrated? • Examples: Integrate deployment and chat tools so the teams involved with deployment can work together more efficiently. And make sure to integrate monitoring and configuration management. 37 4. Implementing business strategy through technology innovation Finally, your team gets to work on new, high-value projects! By now, you should have a list of ideas for what to improve — both your own ideas and ideas from your technical teams — tied to your organization's growth strategy. These might include any of the following: • Migrating workloads to the public cloud. • Building self-service portals for developers and other teams (such as accounting and HR). • Expanding automation to legacy environments or other functional areas in the organization. • Replacing monolithic management platforms with your newly established DevOps toolchain. • Adopting containers, microservices or other technologies to increase your velocity. 38 There may already be pockets of DevOps culture and practice in your organization. After all, the movement was originally driven by practitioners facing exponentially increasing workloads and static or shrinking resources. You should seek out those parts of your company where DevOps is happening, learn what people are doing, and socialize these efforts to the rest of your organization. DevOps can grow much more quickly when executives champion the transformation. “The business value of self-service is to allow development teams to rapidly respond to emerging business needs. Puppet allows us to delegate a lot of these infrastructure tasks — traditional sysadmin tasks — to development teams, so that they can stand up their own servers, configure their own servers, and deploy their application code, all through the magic of Puppet.” Peter Magnaye, director of systems engineering at FINRA 39 Puppet and DevOps. Puppet isn't the only thing you need to adopt DevOps. It is, however, the cornerstone of any successful DevOps initiative, providing both powerful automation and the ability to define your infrastructure as code. Automation is the foundation of DevOps, and Puppet is the industry standard for automation in every part of your technology platform — across operating systems; from laptops and workstations to physical and cloud servers; from switches and routers to load balancers and storage devices; from big on-prem data centers to public cloud services, and private clouds, too. 40 Today's IT leaders use Puppet to manage and optimize the entire software delivery pipeline — from initial provisioning of dev environments to version control and continuous integration, through every stage of automated testing, to automated deployment. When it's time to scale from hundreds to thousands to tens of thousands of machines, Puppet assures you'll scale consistently, reliably and verifiably. Unlike other IT automation software, Puppet was built from the ground up to support infrastructure as code: the practice of managing IT infrastructure with the same tools and practices used in software development. With Puppet, you can version-control all aspects of your IT infrastructure, employing the human-readable Puppet language. Puppet is easy to learn, and it works across a huge range of operating systems, major equipment vendors and cloud providers. Puppet lets all your technical staff collaborate on their work, across the data center and throughout the entire software delivery cycle. That collaboration is the heart of DevOps. 41 Shifting to DevOps can seem daunting — but it doesn’t need to be. 42 With Puppet as your foundation, your team can start with one initiative, adopt new practices, and expand from there. The same language, methods and even code modules you use at the outset can be applied again and again to bigger projects. Your team will have the power and flexibility to surf the next new technology wave — and the ones that keep rolling in over the years to come. 43 Resources 2016 State of DevOps Report The fifth annual industry-leading survey of IT professionals discusses the state of deployment, security, stability and employee loyalty at organizations that have (or have not) adopted DevOps. The report digests findings from the most recent survey of more than 4,600 professionals, plus learnings distilled from more than 25,000 respondents worldwide over the past five years. Cutting-Edge IT: Moving from Nodes to Applications at Wells Fargo Read about the DevOps journey that led one of the world's largest banks to shift its IT organization from a server-centric view to an application-centric view. Now Wells Fargo can use code to model its applications and business processes in alignment with business goals. Hiscox Reduces the Cost of Release by 97% with DevOps and Puppet A global specialty insurance company details its DevOps journey, including initial steps, how it overcame obstacles, and its plans for spreading DevOps throughout the organization. Get Started with DevOps: A Guide for IT Managers Practical advice for managers of IT teams on how to work effectively and empathetically with their own teams, development teams and upper management. 44 About Puppet, Inc. Puppet is driving the movement to a world of unconstrained software change. Its revolutionary platform is the industry standard for automating the delivery and operation of the software that powers everything around us. More than 32,000 companies — including more than two thirds of the Fortune 100 — use Puppet’s open source and commercial solutions to adopt DevOps practices, achieve situational awareness and drive software change with confidence. Based in Portland, Oregon, Puppet is a privately held company with more than 400 employees around the world. Learn more at puppet.com. 45 </p>