ZNDP 030 – Designing for DevOps with Nick Russo

Demystifying The Roles, Network Community, Network Design, Podcast, ZNDP

Podcast: Play in new window | Download

Designing for DevOps

In Today’s Zigbits Network Design Podcast (ZNDP) episode we invite back an up an comming icon within this industry, my good friend Mr. Nick Russo to have a conversation around Designing for DevOps. If you’ve heard of DevOps in any form and even if you haven’t, you will want to tune in to this episode.  Nick and I get into a lot of real world examples of utilizing DevOps to meet business outcomes and requirements.  We also define a number of concepts and techniques that can help you when Designing for DevOps. Strap in for a great 90 minute show!

Designing for DevOps – Purpose, Process, People, Tools

Purpose, Process, People, and Tools should be considered in sequence and its very important when we are discussing designing for DevOps.

Purpose:

What are we trying to solve? Why does our organization even exist? What is the goal? Jim Womack: ”What do your customers want that you are not currently able to supply?” From the customer’s perspective, how do we add value?

Process:

Set of dependent events to achieve the goal. Waste is any action taken that doesn’t add value (and for which customers would not be willing to pay)

People:

People must understand the purpose and design/improve processes to achieve it

Tools:

Tools are the mechanisms by which we implement steps in a process to add value. We in IT focus far too much on this. Automation solutions are just tools in our toolbox, just like OSPF, EIGRP, Inter-AS options, and CSC.

Designing for DevOps – Analogies

  1. A race car with me driving
  2. A Kia Optima with a professional driver
  3. A race car with a professional driver but only allowed to drive up to second gear

None of these combinations are likely to win the race.

Holistic look at process improvement through an entire system.

Designing for DevOps – Types

Purposes vs Business Drivers:

  • Growth:
    • Take risks and show a decisive competitive edge without cost concerns. think Amazon, very few profitably quarters, almost 500B market cap.
  • Profitability:
    • Focus on new revenue streams (external) and cost reduction (internal). external like generating reports from web servers to marketing faster (lower lead time). internal like faster resolution of tickets with better quality, less rework (lower operating expense).
  • Stability:
    • Typically not true for companies, but definitely for govt. start small and be conservative

Enterprise vs Service Provider:

  • Enterprise:
    • Internal IT service team would be focused on bug fixes, request fulfillment, WAN expansion/migration, etc. IT as a service provider for the business
    • External: Ecommerce websites, marketing analytics, etc. IT as a business partner
    • Similar for military. one group of people focuses on providing access to users, the other designs the comms plan for military operations
  • Service Provider: been automated for a long time
    • Service provisioning: edge services to deliver customer connectivity. short lead time and customer flexibility, sometimes even customer control
    • Core routing: TE optimization, HA/FRR, cost reduction to avoid expensive DWDM equipment upgrades (capex) or new fiber runs

Brownfield vs Greenfield:

  • Brownfield:
    • Less tolerance for risk, slower migration, backwards compatibility with older stuff (unusable via SSH, not some API-based mechanism). start small and focus on business value. processes tend to be crystalized
  • Greenfield:
    • With proper planning, easier to do right the first time, which is generally less expensive over the long term. example: NETCONF serializing XML over SSH based on YANG models. if the network is new, more than likely the processes used to manage the network are also new. be sure to map out the procedural steps in advance and target your automation towards the bottleneck pieces

Designing for DevOps with Scalability in mind:

With automation, scale often means scaling up/out the automation system to manage the large device count. once you define the golden config and have it in version control, ensuring it is applied to all devices (IAC model) doesn’t take much more logical effort. of course this is product specific. its similar to code. copy/pasting code and lack of abstractions (like functions) is less acceptable as the code grows. example: Ansible static inventory gets less attractive as system grows. but the playbooks don’t really change. also consider the load on your process steps (bottleneck)

Designing for DevOps – Our Story 

Background:

Government is bureaucratic. Its the race car with professional drivers but constrained to second gear example above. Broken processes drive the craze over panacea tools to solve all problems.

Our purpose: improve product quality without increasing cost or lead time

Challenges:

  1. Customers regularly change their orders just hours before they are due to be shipped
  2. When the product is delivered, the customer may procrastinate performing a quality check

Our Process:

Side note: most of our processes are configure to order, so our bottleneck is our first processing step. Our fabrication activities that feed assembly are fast!

1. Time the release of materials to some fixed time before they are due for shipment. eg, customer places order 30 days in advance, but production only takes 30 minutes. release materials 2 days prior to delivery a few days early. ideally, materials should be released as close to 30 minutes prior as possible. The longer the gap between material release and shipment, the more likely rework is needed.

2. Before fabrication begins, a series of integrated checks on operator input is conducted.  This allows for “fail fast” to minimize NVA time.

3. Track the product as WIP until it is delivered to the customer. Use the Kanban methodology to enforce WIP limits per product type (level by mix).

4. Fabrication activities (configs, documents, checksums, etc) and the assembly of these files into a winzip bundle is fully automated. There is no need to maintain finished goods inventory as products are shipped immediately (we keep a copy, but comes at insignificant carrying cost)

Tool Itself

The main purpose is to create files (take from readme).  Suppose you have a branch site with a router, switch, and firewall. you have 8 branches. You will end up with 8 folders with 3 configs each. You’d make 3 templates (r, s, f) and specify 8 entities with their unique inputs, like IP ranges, etc

Some extra features:

1. Infra: supporting infrastructure for the sites. update the SNMP map, AAA server, BGP route-reflectors, or DMVPN headend

2. Checksum: SHA256 computed for each file to ensure integrity when delivering to customer

3. Docs: each entity can have an auto-generated document, like a site diagram, to accompany it. Jinja2+LaTeX

On the surface, doesn’t seem impressive. no device logins and still a lot of manual application. That’s just how it is right now. Removing waste never happens in one pass. New waste is revealed each time old waste is removed. This is progress!!

Tips for Starting out with Designing for DevOps

Start with: what is your purpose?

Risk-averse managers are not going to fall for “infra as code” on day 1. Most will be afraid to even let automated tools make changes. Start small, with baby steps that lead to an end goal and an end destination.

– Instead of doing configs manually, auto generate files. We are here!!.
– Then, write them direct to devices (as if copy/pasted)
– Then, make it idempotent, adding in only the things that change
– Then, stop using CLI, and migrate to structured data serialization through APIs

We already know automation can:

1. Improve quality by reducing variation (more consistent results)
2. Reduce lead time by increasing throughput for the same WIP (less time spent in fab)
3. Reduce cost by decreasing rework (corollary of increased quality) via direct labor

Kind of flies in the face of quality, speed, cost (QSC) but I know for a fact it works!

The code is here on Github, and the specific tool we discussed is called “mkfd”

Term of the Show:

  • What is the definition of DevOps?

Call to Action:

  • What topics would you like us to spotlight on our next Design episode?

Guest Expert: Nicholas (Nick) Russo

Today’s guest Expert has become an icon within this industry, he has been featured on the community show, The Network Collective, he has been on our show episode 5 (ZNDP 005 – Carrier Supporting Carrier (CSC) with Nick Russo) where we talked about CSC for 90 minutes. He holds Cisco Certified Design Expert (CCDE), and two Cisco Certified Internetwork Expert (CCIE), one in Routing & Switching and the second in Service Provider.  He is just coming off an amazing week at Cisco Live US 2018 in Orlando, Florida where he shattered the Cisco Live presentation bar with his Troubleshoot OSPF session! 

How to stay engaged with Nick:

Related Resources:

 

Mentoring and Coaching with Zig:

Through your participation in a healthy mentoring and coaching relationship, you will benefit greatly from the education, the experiences, the influences, leadership and even the resources provided. Learn how you can accomplish more, in one year, than you could accomplish in your career…in your business…and in your life.

Accomplish More Now!!

Ask Zig:

Ask Zig episodes feature answers to the questions that you provide. Yes You! The questions can be technical, business, certification, or personal related.  I can help out in all of these areas and much more.  If you would like your question spotlighted and answered on the next #AskZig episode submit them now!

Submit Your #AskZig Question Now!!

Provide Feedback

Engage with Zigbits further:

Engage with me further:

Transparency:

This post may contain affiliate links to products or services were I may receive a level of compensation from your actions by following those links. This is seamless to you and does not add any additional cost to the products or services in question. In addition, I do not let any affiliate relationship cloud my judgement or my recommendation of a product or service. My recommendations will always be above reproach.  This is my commitment to you Ziglets!