Adventures in Enterprise

Lessons Learned
Promoting DevOps
in a Legacy Organization

Steve Desmond

@stevedesmond_ca
#devopsdays

DevOpsDays Fued

What is Ithaca known for?

Gorges	54
Cornell	39
Road Trip	7

DevOps

Culture
Automation
Measurement
Sharing

Enterprise

Software used to satisfy the needs of an organization rather than individual users

-Wikipedia

Payroll / HR systems
Classroom / lab computers
On-Campus LAN / Wi-Fi

Legacy

On-premises datacenters
Physical servers
Weekend downtime
Project-centric planning
Traditional org structure

Story Time

Opportunity

org struggling to keep up w/user expectations
self-described "backwater", grad app quote
potential to make everything better
first-hand experience
amped up, turn ship around, guns blazing

I saw an organization struggling to keep up with its users' expectations. Self-described as "backwater", with an Online Application for Graduate Admission that was, and I quote, "so bad, I almost didn't apply." And I saw the potential of being able to help make everything better. After all, I had first-hand experience in how to improve software quality and team performance. And as my start date approached, I got myself more and more amped up. I was all like "Yeah! I'm gonna turn this ship around! I'm going in with guns blazing!"

Part 1

Do Not Go In
With Guns Blazing

Big Orgs Move Slow

https://channel9.msdn.com/Events/Build/2017/T6006-R1

Why?

Stability over innovation
Control change vs embrace it
Physical scaling limitations
Project-centric planning

but why though?
legacy enterprise orgs (define) value stability over innovation
one way = controlling change (rather than embracing)
payday example
research = can have both, but hard, so S wins + I languishes
budgeting + provisioning hardware limits rollout
not capital P project = doesn't have priority

But why though? Well, with legacy enterprise organizations, and by that I mean enterprise organizations who haven't really embraced "the new way" yet, it's because they value stability over innovation. One of the ways they do this is by controlling change, rather than embracing it. In some cases, that's with good cause, right? How many of you would be upset if you didn't get your paycheck because the payroll system crashed on payday after an upgrade? Research is starting to show that you can have both stability and innovation, but it's hard to get there, and so stability wins and innovation languishes. Similarly, when setting up a new system means budgeting for new hardware, you're limited in how fast you can roll new things out. I also mentioned the project-centric nature of legacy organizations, and this is another contributor: if it's not a capital P project, it doesn't have the priority.

Software is not a Building

Don't construct them the same way

electical wiring, lights, insulation, resilient channel, A/V wiring, drywall, taping, primer, paint, crown molding
miss a step, difficult and expensive to remedy
planning up front to get ducks in row
waterfall development, shrinkwrapped software, physically shipped
not that time anymore, processes need to adjust
moved in, added floor, painted when time, no drywall dust?
how different process?

This is my home office. Things had to be done in a very specific order. For the ceiling alone, that meant electical wiring, followed by lights, then insulation, resilient channel, A/V wiring, drywall, taping, primer, paint, crown molding. If I forgot a step and moved on, it would be very difficult and expensive to remedy. And so you do the planning up-front to make sure you have all your ducks in a row and do things right the first time. This is waterfall software development, and in the days of shrinkwrapped software, that's how it was done, because the cost to ship -- physically ship -- was so high, and once you shipped, there was no turning back. But we're not in that time anymore, and so we need our processes to adjust accordingly. What if I could have moved in right away, been able to work, slid the floor in under my desk, painted the walls when I had time, not had to deal with any drywall dust? How different would my process have been?

PL/SQL

Filesystem I/O
LDAP management
HTTP connections

Be Prepared
For More Resistance
Than You Can Imagine

Assume Awesomeness

https://github.com/BeardedCoder/RealWorldRetros

Lead By Example

"Be the CAMS model!"

Don't Rock The Boat

(too much)

(or do)

also be careful
early on, w/in 1st yr, being moved teams
mostly thrilled, part of me knew why
young hotshot

Also be careful. One day pretty early on, definitely within my first year there, I found out I was being moved to a different team. While most of me was thrilled to be working with coworkers I was closer with, and on technology I was more familiar with, a part of me knew that I had rocked the boat, maybe stepped on some toes, and that this was probably a contributing factor to the decision. While I continued to wear my "young hotshot" badge with pride -- yes, I actually got called that; I was the youngest on the team by at least a decade -- I realize in hindsight that I maybe could have tread a little more lightly.

Team Dynamics Are Paramount

Part 2

Celebrate Your Small Wins

Recognize Your Victories

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Small Controlled Experiments

http://verraes.net/2014/03/small-controlled-experiments/

Go.CD, 1st gen Intel Power Mac
spare machine, lying around, brought back to life as prototype CI
less cost and effort than procuring "pre-scaled" server
eventually upgraded to Mac Mini as grew, finally real server once value shown

This is Go, not the language but the continuous delivery tool, running on a first-generation Intel Power Mac under my desk. Literally a spare machine, lying around, brought back to life to live again as a prototype CI server, because it was less effort than procuring a new "pre-scaled" server for the physical datacenter. We later found a Mac Mini that was more powerful, which was good because we continued adding more pipelines, and eventually, when we were able to show the value, we got a real server for it.

Manual

ChatOps

with
Todd

example exchange: real deploy, slightly redacted
don't see everything happened behind scenes
release branch pushed to QA, browser tests run
release tagged, deployed to staging
previously manual, automated 1 by 1
experimenting how, while maint auditing reqs
sowing seeds for CD

Here's an example exchange. This was a real deploy, slightly redacted. I simply ask Todd to update an app to a given tag and it gets deployed. What you don't see is everything that already happened that led up to this point. All of these steps -- the release branch being pushed to QA by the CI server after unit tests pass, automated browser testing against QA, tagging the release for staging, deploying to a staging environment identical to production -- previously done manually, starting to be automated one by one, by experimenting how to best do it while still maintaining things like auditing requirements, at the same time sowing the seeds for continuous delivery.

Decouple from your Enterprise Systems

Extend via Configuration,
Don't Fork Vendor Code

Look for Low-Hanging Fruit

Lead Times

ERP: 2-7 days
Web: 5-20 minutes

By doing all these things (list) + compound interest earned
achieved lead times orders of magnitude faster than ERP (over the wall)
Also able to deploy multiple times per day, compared to couple times a week
things looking good in web team bubble

By doing all of these things -- decoupling our apps, continuous integration, constant communication with ops -- and the compound interest we earned when combining them, we achieved lead times that were orders of magnitude faster than the process for ERP changes, which were historically more of the traditional "over-the-wall" approach. In doing so, we were also able to deploy multiple times per day, compared to a couple times a week. Things were looking pretty good in our little web team bubble!

Re-Org

dreaded R word
despite victories, clear to me, everyone, org needed big changes
not what I was expecting
ops got pulled further away from dev
less dev, more off-the-shelf
absolutely valid direction, just not what a dev wants to hear

And then came the dreaded R word. Despite our victories on the web team, it was clear to me, and everyone I think, that the organization needed some big changes. Unfortunately they were not what I was expecting. The ops team got pulled further away from the dev team, and it became clear that the focus was shifting from software development to off-the-shelf solutions. Absolutely a valid direction for the department to take, just not what a developer like myself wants to hear.

You (as an individual)
Can't Change Culture

Part 3

Maybe DevOps Isn't Right
For Every Organization

Part 3

Maybe DevOps Isn't Right
For Every Organization

Part 3

Maybe It's Not The Right Time
For DevOps Yet

Part 3

Maybe It's Not The Right Time
For DevOps Yet

You (as an individual)
Can't Change Culture

Spark change in others
Lead by example (be the CAMS model!)
Be patient

Experiment

Low-cost
Low-risk
Goal = Kaizen

Decouple, Decouple, Decouple

UI from Database
Front-End from Back-Office
Custom Features from Vendor Software

lucky
trying to sum up
sometimes great (examples) others not (examples)
can't imagine effort involved
made connection

I'm currently lucky enough to be able to take my kids on weekly hikes, and a couple weeks ago, right around when I took this picture, I was trying to figure out how to sum all this up into a single thought. Now, sometimes, like on this day, things are going great: everyone's in a good mood, we making good progress, things feel easy. Other times, I feel like I can't get through to anyone, the whole thing is a struggle, it's just not working. But this was a good day. And I obviously made the connection, as I was thinking about this talk. If I had one final piece of advice to instill today, it would be this:

Guide Your Team
On A Nature Hike

Don't Just Drag Them
Through The Woods

High Fives

https://stevedesmond.ca

stevedesmond-ca

stevedesmond_ca

hi@stevedesmond.ca

Adventures in Enterprise

Lessons Learned Promoting DevOps in a Legacy Organization

Steve Desmond

@stevedesmond_ca #devopsdays

DevOpsDays Fued

What is Ithaca known for?

DevOps

Enterprise

Legacy

Story Time

Opportunity

Part 1

Do Not Go In With Guns Blazing

Big Orgs Move Slow

Why?

Software is not a Building

Don't construct them the same way

PL/SQL

Be Prepared For More Resistance Than You Can Imagine

Assume Awesomeness

Lead By Example

"Be the CAMS model!"

Don't Rock The Boat

(too much)

(or do)

Team Dynamics Are Paramount

Part 2

Celebrate Your Small Wins

Recognize Your Victories

Manifesto for Agile Software Development

Small Controlled Experiments

Manual

ChatOps

with Todd

Decouple from your Enterprise Systems

Extend via Configuration, Don't Fork Vendor Code

Look for Low-Hanging Fruit

Lead Times

Re-Org

You (as an individual) Can't Change Culture

Part 3

Maybe DevOps Isn't Right For Every Organization

Part 3

Maybe DevOps Isn't Right For Every Organization

Part 3

Maybe It's Not The Right Time For DevOps Yet

Part 3

Maybe It's Not The Right Time For DevOps Yet

You (as an individual) Can't Change Culture

Experiment

Decouple, Decouple, Decouple

Guide Your Team On A Nature Hike

Don't Just Drag Them Through The Woods

High Fives

Lessons Learned
Promoting DevOps
in a Legacy Organization

@stevedesmond_ca
#devopsdays

Do Not Go In
With Guns Blazing

Be Prepared
For More Resistance
Than You Can Imagine

with
Todd

Extend via Configuration,
Don't Fork Vendor Code

You (as an individual)
Can't Change Culture

Maybe DevOps Isn't Right
For Every Organization

Maybe DevOps Isn't Right
For Every Organization

Maybe It's Not The Right Time
For DevOps Yet

Maybe It's Not The Right Time
For DevOps Yet

You (as an individual)
Can't Change Culture

Guide Your Team
On A Nature Hike

Don't Just Drag Them
Through The Woods