Automating The Entire Parliament Precinct Network with John Capobianco – ZNDP 078

ZNDP 078: Automating The Entire Parliament Precinct Network with John Capobianco

Zig Zsiga: [0:00] Automating the entire Parliament Precinct network with John Capobianco episode 78 in this episode we're going to highlight, Automation in the real world today's topic is a real automation use case with all the good,

the Bad and the Ugly that goes into goes with a production environment.

[0:23] I'm joined once again by my good friend John Capobianco and he is going to dive into how automating the entire Canadian Parliament Precinct Network happened from.

[0:34] To finish this is a part 2 from Zigbits Network design podcast episode 75 where John highlighted the network design transition he went through from a legacy Nortel Network to a world-class enterprise-wide Cisco Network that ensures business and customer success hey if you haven't had a chance to listen to that episode I would highly recommend it truly a great design use case and if you want to go listen to that it's https://zigbits.tech/75

In this automation case study with John John is going to talk about Python Ansible CI/CD Git GitHub and so many more automation tools you must listen to this episode I promise you if you're in automation you want to hear a real world use case listen to this episode this is real my friends and it starts right now so stick around and here we go.

[1:26] Welcome back my friends nerds geeks and ziglets out there we have another episode of the zigbits network design podcast where the zigabytes are faster than those gigabytes we strive to provide real-world context around technology,

what's up everybody I hope everyone is doing great zsiga here and welcome to episode 78 of the zigbits network design podcast once again my name is zsiga and I'm here to help you with network engineering Network design and network architecture,

we have our very first Dev net devops,

automation programmability real-world use case / case study today and I'm super excited about this show and I hope you enjoyed,

John welcome to the zigbits network design podcast once again thank you so much for being here today how are you doing.

John Capobianco: [2:14] I'm doing great it's I'm really excited about today's discussion pick up where we left off so thanks for having me again.

Zig Zsiga: [2:19] Yeah so real quick so that people are listening where did we left off so I think would be beneficial hear that

you know everyone we did an episode a few weeks ago zigbits Network design podcast episode 75 zigbits dot text / 75 are the show notes but could you give a quick two minute kind of overview of what we talked about in that episode just to kind of set the stage for everyone.

John Capobianco: [2:41] Yeah so I we recap the journey from taking a Brownfield Legacy aging poorly Design Network and transforming it into a modern Next Generation,

infrastructure for the parliament of Canada House of Commons under a program called the Next Generation parliamentary Precinct Network program and gpn as we call it internally and,

we talked about replacing the infrastructure from top to bottom from the perimeters to the data center to the campus which is about 50 buildings and the wide area network which is about 450 buildings.

And how we did that,

over about four years and that's sort of where we left off and we talked about all of the new services that that that next Generation Network then became the foundation and platform for the digitization of,

higher level functions at the.

Committees and

digitally onboarding new members of parliament when they're elected and having Wireless capabilities and enhance security threat of to an X and a more secure robust high speed

Network for the for the country's government.

Zig Zsiga: [3:55] That that's that's a lot right so we talked about a lot there and I guess I would summarize it as well as a true Enterprise architecture Next Generation environment that really supports the business now again it's under traditional business but you're literally changing

workflows and governance and structure on how the parliament works.

John Capobianco: [4:15] Yeah it's a good way to wrap it up so the funny thing is today we're going to talk about Automation and the so it's built it's great and you're done right

and a job you walk away the Network's built we all know it doesn't really work that way in the real world there's new demands new buildings changes to Services New tenants if there's the ongoing care and maintenance of this large

monster right so that that is what led me to automation was the complexity of the network,

the scale of the network and the agility and the bottleneck that the network had rapidly become for the business so those three things were my driving factors to automation,

and we had done some automation when we,

for example plug-and-play if anyone's used plug and play with where you set up a server and it uses you plug in your hardware and it picks up a DHCP address and then pulls a configuration,

we did do a lot of that to scale all of these floors at the axis layer but we hadn't really done full configuration Management on going automation.

[5:23] It was very much a waterfall approach where you know and this is going to sound may be familiar and maybe a sore spot for some people but an architect or designer comes up with the high-level design or workflows someone turns that into,

iOS or nx-os or a command line configurations and then that gets sent further down the chain to operations who schedule it and log into the CLI and copy and paste the commands line-by-line device by device,

does that sound familiar to you.

Zig Zsiga: [5:53] Yeah it does.

John Capobianco: [5:55] So at scale though and that complexity a lot can go wrong right every human touch point is an opportunity for a misstep,

I'm on the wrong device I pasted the wrong code I had the wrong thing in my copy paste buffer I hit space I typed the wrong vrf whatever could happen right.

Zig Zsiga: [6:11] Now we never make mistakes come on.

John Capobianco: [6:13] No and even if even if you're good like I consider myself a CLI Wizard and I can fly through the command line and go device to device to device but that's not you know you don't want.

To have to have heroic effort every time you want to make a change on the network right.

So I had a particular change and I'll talk about the design because it's a design focused discussion so we wanted to we were implementing the kind of fourth or fifth phase of our NG PPN which was the data center and,

what we had realized was we had made I don't want to say mistakes but we had made some decisions some design decisions upstream and these could you be simple things like the names of RV RFS,

they were little things like that so we thought when we're going to put the data center in we're going to have a bit of an outage anyway let's take the opportunity to clean up the campus in preparation for a more homogeneous well-named seamless.

Structure between the campus and the data center we work interconnecting now this meant changes at every vrf on every physical router on all 50 buildings at the distribution layer.

Zig Zsiga: [7:17] And how many routers was that because you said earlier there was like 50 buildings and then there's.

John Capobianco: [7:21] Yeah there's a bit so we're talking about 50 buildings so 50 physical routers the global routing table and then each building could have a variable number of virtual routers depending on the tenant of that building so up to.

We're talking about 300 routers if they were all physical right.

Zig Zsiga: [7:39] Now do you have any idea how long that would have taken before you did the automation just curious ball.

John Capobianco: [7:44] Yeah so our approach was to use some existing Cisco capabilities on the on the distribution layer that we have so the config replace option,

which is very handy now not every platform supports that but what you can do is put a Target State configuration in be Ram.

MN use this config replace command to seamlessly it's like doing a write copy start run except not destructive,

State fully replace the memory with the copy command it also has a nice rollback feature so that's was our approach using the Cisco automation they give you they do give you some tools to automate it's not all about Python and ansible right,

there are things like config replace but then so then we did the timing to your point right we're looking at,

call it half an hour a building times 50 building.

Not good right and and the thing is the way the sequent the way of sequence of events work.

That first building you touch is going to be offline until the very last building is done because the core happens last.

So that first building you're leaving dangling from eight in the morning when you start until whatever time in the evening when you're done your last building not good.

Zig Zsiga: [9:00] There's a huge impact right huge impact and I did the math real quick I took me a minute actually I was like how do I do half an hour and.

50/50 buildings and and all that so I mean if it takes half an hour per building right what's that math is it 25,

25 hours right half of it.

John Capobianco: [9:21] My guess yeah and then say you divide that into a couple of teams of operators and maybe tackle it,

you know this group will take on these 10 routers in this group of take on these you know you can slice and dice it a little bit with human scaling but at so this is around 2017 ice combed over my emails my first paying on the radar of ansible,

which which is you know if anyone who knows me I'm you know in love with answer.

But I found it in a through an article or something led me to find it and in April 2017 and then,

I was like maybe we could apply some of this ansible stuff to this config replace maybe I could mix and match and and orchestrated automating,

the sequence of events through this automation tool now this is where I believe I made my first mistake was to try to use something new like ansible in a production facing.

Very complicated very complex series of events right but you know you don't know what you don't know and necessity is the mother of invention and all that good stuff so I took it on and here's here's my recap.

[10:33] After about six weeks of development and I did say six weeks we had it working and we scheduled the change and a bunch of people came on site and we press the magic button,

and we waited and 45 or 48 seconds went by and then it stopped.

I apologize to the room I thought geez I'm really sorry I did this in the lab I swear this was going to work something my you know and then the operator says no no look.

All the new routes are in place all the new config no no we're using the new naming convention look look look so we start hopping the machines and and the traffic flow wasn't disrupted and it was like totally seamless and went totally off like perfectly right.

[11:15] And their celebration and everyone that you know everyone's Minds just get blown out like what could we what else could we do with this right but,

if there was in my in the back of my mind there was this nagging like,

that shouldn't have taken six weeks like there's no way this is what people mean by automation is a future that was toiling back and forth with and and here's why it's because I didn't have the right tools,

I use my trusty old Notepad,

and and winscp and a Linux box and I wrote my code and windows and lint notepad and I transferred it over to Linux and I did trial and error back and forth and my syntax is wrong on my Mo was wrong,

typos and had files called bike,

working V2 underscore good underscore not tested B3 right like it was just awful just awful so the processes were there I didn't really know how to work with these new tools I wasn't a software developer right.

So the success happens I'll be stopped there do you have any questions.

Zig Zsiga: [12:15] Yeah I didn't want to have some questions of course I didn't want to cut you off as always so.

John Capobianco: [12:19] Yeah no I think it's a good time so that's that's my first attempt my first Endeavor and network automation successful but painful.

And didn't didn't feel like I had hit the nail on the head regardless of the you know positive production impact right but now we did cut the time down from 25 hours,

248.

Zig Zsiga: [12:39] Yeah 48 seconds is what I heard I wanted to make sure that's what I heard so so legit you went from 24 25 hours

to 45 seconds 48 seconds let's say 50 seconds let's take a minute let's see 60 seconds right

now obviously the development process in that first attempt for you was six weeks right so there's an uplift here and you said there's things that you didn't know and you don't know what you don't know

right that that is a underlying theme when you're starting out something new you don't know what you don't know and it takes time to learn that and it takes lessons but the fact that you were able to learn this even if you didn't do it right,

the right way you're able to leverage it in production and cut down the deployment time for 25 hours to 45 seconds.

John Capobianco: [13:20] At scale.

Took it just went device to device and and issued the command now how you know certain people how did you really do it well we set up the orchestration so that one it distributed based on the hostname.

A file called you know.

Running config underscore whatever the file type no file type and we distribute those through the tasks to each host,

then we set the you know run the config replace command,

we had a validation check in there to run the rollback if we lost connectivity or something so there was some some safety nets built into this as well that didn't get invoked but they were there and then now the post,

and I would do things differently now and there's will talk about some tools that I could have made this even better,

was operators logging in like I said the operator called back to me no no I've logged into the device in the validation looks good so the validation post wasn't quite automated but again first attempt something new and it worked out very well.

I'm in the IT department and you know most organizations kind of have that split with it and is or the in the developers on one side of the house and the,

infrastructure people on the other side of the house it was a senior.

Director in the in the developer side who approached me everybody heard about this obviously this success went all the way up the chain.

Zig Zsiga: [14:43] You had some big impact right so everyone heard about it yet.

John Capobianco: [14:45] So he he thought it was funny that the network infrastructure guy was taking a stab at programming right and and he and I told him about the file names and the problems I went through and blah blah blah

and he pulled me aside and said listen what we're going to do is going to sit you down a few of our developers and we're going to get you a get git Repository,

and you're going to start using developer tools and have version and Source control and it's going to solve so much of your process so many of your problems and you can since you're trying to work programmatically and treating infrastructure as code,

there's you know there's a whole,

Suite of tools that are going to empower you and make that easier and make that better and make that more powerful and and wow so at the time it was team Foundation server is an on-prem,

so let me make a difference differential for anyone new here there's get which is the software the version controlling software and that's can that's included with Linux distributions and you can install it in the windows environment and then there's GitHub which people have heard of,

which is kind of this is the central repository for git repos but it's not the programming language like there is a distinction between GitHub repository or the repository system.

[16:03] Or a git repo or get the version control system so,

team Foundation server is kind of like GitHub but on-prem and private for your network or your organization and its been rebranded as your devops recently the 2020 flavor is as your devops which is in the cloud,

and on-prem they have both offerings so.

[16:25] We're going to get you a repository in this ecosystem and right Athena the first link I click there's a few tools I have to install I need get and then one of the other links in team Foundation server stew vs code.

Which has become like my best friend in the world I wish I could send the developers a Christmas card for vs code every year because it's,

becomes so much like I live and breathe and Bs code now three years later right no no pad is I'm not doing that anymore right so the vs code is the integrated development environment or IDE it's really like a text editor on,

on steroids and it has extensions that you can in a further enhance the power of the base editor so that's became my ecosystem was,

you know ansible still running in a Linux environment that had access to my infrastructure so your Linux host with ansible needs.

SSH conductivity of the management plane of the of the network you're trying to Target or HTTP access if you're using apis so the system Network firewall flows there required,

and then it also needs access to your git repository because you're going to develop how I do it is develop locally and windows envious code.

[17:36] Push that to my git repository in TFS or as your devops and then clone and pull that code down into my ansible world and run my plays now if the plays generate artifacts intended configs or markdown files or CSV files or whatnot,

I then push those back up into the central repo so we're working with software infrastructure as code and it also scales much better people aren't sending files back and forth,

people aren't you know working on the underscore V2 or whatever files everyone works with a local copy of the centrally distributed Version Control,

code so my colleague can add a vrf in a branch get uses a branching system independent of me developing a,

bit of code in a different branch and then get does the mechanism of merging all that code seamlessly back into the main branch which is where you run into Ploy your playbooks.

So that's a lot to unpack but that's that is how when people say what is devops or net devops or what is working like,

a developer mean that is what it means to me is to use those tools to begin with right.

Zig Zsiga: [18:48] A lot of tools right I mean it seems for me okay so I don't know anything really about devops I I can,

I know programming I know I have a computer science degree I've learned a lot about programming when I went through my degree plan but I have yet to jump on the automation

learning curve it is on my to-do list and I'm being fairly transparent as always so.

My first implication here is that there's a lot of tools that you have to learn theirs

but there's ansible right there's the I want to hear more about the vs code thing or you know program text editor,

get there's GitHub there's also Concepts like Ci CD and I hope we can get into all that right there's also Docker containers,

maybe I'm missing something so let's let's spend some time and do a kind of brief if you don't mind overview of each of these tools and maybe how to get started in them.

John Capobianco: [19:41] Yeah so I kind of gave you you know jumping,

two feet into the fire right into a production change and let's just automate it and go for it right and hit the Home Run well in retrospect and where I'm at and I'm doing this continuously right I'm still working on this here at the House of Commons.

Zig Zsiga: [19:57] This is the live right is.

John Capobianco: [19:58] We say when we say it but we automated right I'll go layers the building blocks and where you can start,

I believe it's a bit of a pyramid approach and the foundation is where we started and it's truly there's a deck if you go to my GitHub repository here speaking of GitHub,

if you go to my github.com / automate your network there's an ansible Fest deck in their my aunt's will Fest presentation,

don't start their read that but in there there's a particular image of a pyramid that I've put together and the foundation the first big heavy layer of any pyramid to me in this space is the tooling,

so you need the right tools get vs code it doesn't have to be vs code but an enhanced text editor yes Dockers one of my tools,

Azure devops is one of my tools and then even things like Json and how to read a Json file how to get Json data,

restful apis I would consider one of those tools Postman an API browser and development tool I would consider one of those tools I'm probably missing a few python,

yamo Ginger to ansible right that mix of tools.

[21:12] Spend some time and build up and now I'm not a get expert by any means High get by on the 8th get commands that I've memorized in the order that I need to run them to be successful in get.

I'm going to throw these out there I'm not affiliated with them but learn get branching all one word learn get branching dot Js dot-org.

Is an interactive online web page that will teach you everything you know about everything you need to know about how to get around get and branching and the terminology like git push and pull and commit.

[21:51] So that's all there and then there's another great website when you once you've learned get a little bit you're going to get in trouble with get oh shit get all one word is another really fun.

It's a fun website that takes problems that you commonly incur in get and explains how to get around them or get out of those problems so those two websites will get you going with get and there's more out there believe me that vs code.

[22:17] The base editors great but you want to spend some time looking through the extensions then again in my in my ansible Fest deck there's a slide there just about the extensions that I use,

so I use extensions like yeah Moe and ansible and python,

so that the editor can highlight and colors and underlying problems it's known as linting it can link to the code.

So that I don't have to run bad code and get an error I can fix it right in my editor because the editors helping me right.

You know well-formed code as it's known now,

you mentioned the computer science degree I happen to have a computer programming diploma three year diploma from Community College that's all I've needed and then with ansible and ginger and and where your logic lives.

You know it's if else and if logic and for end for Loop.

That's really about it right now you can get into things like filters and more advanced regex and things like that but if the foundation it's for loops and if statements.

Now now that you have the toolkit.

[23:27] You're going to need to establish connectivity to the devices and I mentioned that already you need SSH HTTP and credentials now credential handling I'm going to stop and mention it.

You can do it a couple of ways right security wants to make sure automation is secure you want to make sure automation is secure.

Now there's the problem with get is that it's essential repo that people can clone and get the code.

So you don't want to keep playing text enable passwords right admin my secret in plain text.

In a connection string for ansible right that's no good no way no you need to have either a prompted mechanism.

And I'll talk about the problem of prompts.

Which are secure but they don't lend themselves to full automation like a Docker container you want to run these things in containers down the road where they just run like software releases and there's no opportunity to answer a.

So ansible has a tool called Vault.

Any and as a nice web page out there that you can follow to change that my secret incription into an encrypted string of just garbage.

And then you use a voltage.

Key to unlock that file at runtime so then in your git repository your safely enable you know you can safely store your enable secret because it's just garbled text that you need a key to unlock at run time.

Is that you follow there.

Zig Zsiga: [24:55] Yeah that's really cool actually so that you don't actually have to have it all in clear text you create that that almost hashed Secret.

You know and then you have a key to leverage that hash secret so that no one else can use it so when you upload it to get it's there the hat that hashed value but they can't leverage it because you have the key,

and so it's like a it's a fully secure way of doing it right it's not there's no limitations to it it's a private key and then a public key kind of situation.

John Capobianco: [25:26] So that works very well and and so now okay I have my tools I can connect to a device where do I start like what could I do with this so here's my recommendation,

ansible has what are called facts modules.

Like just the facts from Dragnet.

These are safe one they're safe they're not changing deleting configuring they're not doing anything at all that could possibly disrupt Network flow.

Right nothing no management likes it it's just read operations there is C essentially running a bunch of show commands,

parsing the data and giving you back Json.

That's where I would start is go get the facts and there's iOS facts NSO X facts there's facts for VMware windows and Linux use a module called set up to get their facts,

Azure clouds AWS clouds storage you name it.

You can go get the facts now what are the facts well on a Cisco iOS device it's every neighbor CDP neighbor that's connected to the device it's every physical interface every logical interface every IP address,

on nx-os you get things like power supplies and fan information you get the modules.

Zig Zsiga: [26:44] You got a whole bunch of information.

John Capobianco: [26:46] All kinds of information in the thing is it's structured Json that might not mean a lot to a newbie who doesn't know you know who's just learning.

Once right but once you start to see Json and read Json and it's machine possible but it's also a human parsable.

What I love about it is ansible has techniques.

So if you like Yah Mo Better Than Json you can run that fax module through an ansible filter called to nice llamo,

and save it in a file now I have a Json file and a yam will file of all my facts at a point in time.

For audit information for problem management troubleshooting whatever you name it you have those facts at the point in time and it's a one line command and ansible and it only takes a few seconds to gather and create now I've gone a step further.

And you can manipulate that Json and create business ready documentation I call it things like CSV files Excel spreadsheets right those magical powerful Excel spreadsheets you can take those facts.

Get the Json manipulate the Json and create a CSV file.

Zig Zsiga: [27:59] Wow all right.

John Capobianco: [28:01] So now you have a CSV file of every CDP neighbor on your core or your building or your access which every wireless access point every phone they're all mapped in a CSV file that searchable searchable sortable filterable,

bar graphs pivot tables if you have that Excel weenie that's just amazing with Excel hand them one of these files and go give me some magic,

out of this data that's that that is my network right so,

that's one aspect that's where I would start any new person ansible run the facts modules they're very easy they're very well documented and they're valuable they actually create something you can use and get value out of,

now my next thing in the same K so.

Let's call this the hunt for Json right that's what we're looking for now can I get Json back,

from the network and then turn it into spreadsheets right that's the kind of my obsession right now.

Zig Zsiga: [28:58] Real quick I mean Jason's important then write that that is getting that back you can do a number of things with it as a my assumption based on how your mean it's more than passionate right you're looking to get that Json that's why you're calling it The Hunt for Json right.

John Capobianco: [29:12] Yeah because it's it's it's structured data so I can let so one I'm making CSV files out of it but to I could write python or other ansible or other programmatic tools,

language of choice.

And test against that data so maybe I run a python Loop that goes through each interface and looks for the keyword down.

[29:37] And now I get a list of every interface that's down programmatically right.

Or here's another example so I we if you have Cisco prime or Cisco ice or Aruba or vendor X they likely have a restful API on their.

Ansible has a module called URI URI that can do.

HTTP get against an API and what do you get back structured Json so I've written a play book that goes against my Prime API and creates the here's the output,

is a CSV file with all 3000 wireless access points and 75 pieces of information about its axis.

2G clients 5G clients kappahd tunnel status bandwidth throughput how long the tunnels been up you name it I get the information in a one line command that otherwise some operator would have to go through the prime GUI.

And click and click and click and find and sort and feel right you don't want to be in the GUI you can scrape that data and make your own reports using ansible and restful apis so you have ansible.

Zig Zsiga: [30:52] Sorry it sounds like the scrapings actually a lot easier to so I told you you told toiled what's that's not the word I played around with like

vbscript and JavaScript years ago and I was automating things back then and this was before we had all these tools this is before even like

I'm going to butcher what it's called Microsoft has a new automation item and I can't think of the name of it but this was like back when we had like batch scripting and and,

back then you're literally doing like he sends you sending keys right to a window and if you if you lose focus on that window it messes up everything and like we would do something like okay I have to go update the IP address of

the loopback address on a hundred devices so you'd script it in vbscript can be thousands of lines of code because for whatever reason and then.

You to run it but it's all sendkeys at the time and it's scraping out the output and putting it in a file so this is the I mean again I'm new to this right so.

John Capobianco: [31:49] So I've got some good news for you Zig if you could do that honestly you're going to take to this like a duck in water you're going to just love it is going to change your life and Empower you so much,

I mean it if you could if you have that back-in-the-day bash scripting shell.

I mean anybody even if you don't have that for listeners out there if you can make bgp neighbors or ospf neighbors or a static route work if you can figure out the OSI stack and make it functional Networks.

The automation is easy if statements it's for Loops it's Jama land Json it'll click in your head it might take time it took me it's I'm still learning I got a tip on the internet last week and I've been refactoring all my code.

Manically because I learned something new from the internet so it will come but but it's you should be excited about it don't be scared of it especially in a given your background so,

I mentioned two ways to get facts and here's my latest thing and this is going to appeal to.

[32:48] It's kind of like a to bridge the gap to give your legacy hardware and API capability that's the way I look at at this next thing so Cisco's released what are called Genie parsers.

If you don't know what parsing is regex is kind of like is parsing so you're going to run a command and then parse the output,

look over the string mathematically and extract values.

Appropriately that you're looking for now with regular expression that's challenging it's doable but it's very difficult.

[33:22] Genie parsers let you take a command show IP interface brief show interface status show.

Some key commands and you run them through the parser and what do you get back.

This structure Json version of that command yes that Json continues so,

it's like the mask it's like the killer with the hockey mask Jason right it's Jason though he's just Unstoppable.

Zig Zsiga: [33:49] I love the analogy.

John Capobianco: [33:50] He's ever present its there right so.

With the commands now I can take a show interface status command and turn it into a CSV file or show environment Commander show show X command,

turn it into a sortable filterable CSV file now at scale you've got a switch stack with eight switches in it with 48 ports per stack.

Do you really want to sit at the CLI and hit the spacebar on the show in status command and look line by line and try to find that inner face that says down no,

pump it into CSV hit the top row filter on interface status down hey look these for interfaces are down.

Like easily doable and and you know it's neat we've been talking about this for 10 minutes and always talking about is gathering information on what you can do with it I could stop here and this would be like a valuable discussion right.

Zig Zsiga: [34:43] Yeah oh yeah for sure I mean I'm on the hook.

John Capobianco: [34:44] We haven't even got to yeah but we haven't even got to the real power which is configuration management where we're going.

Turn our configs into structured data,

and push intended configs out automatically which is really powerful so I would move up the ladder from that information-gathering you're going to have lots of muscle memory,

in ansible and Json and all that by the time you've documented your network state.

And then you can move up to say tactical one time changes or full configuration management now are.

Zig Zsiga: [35:19] Through this sorry I'm going to kick you out for a minute right,

I have a couple questions and I want to ask these real quick and then get into an example that configuration management so we talked about the tools we talked about how to get started right in the facts modules and how to kind of get your feet wet I would like to know.

What was the driving factor for the parliament to go this route I mean for you right for your networking team is it to say

time is it to make things so that it's there's no fat fingering of configurations like how big is your staff when I don't know if it was a staffing thing what really was the main driver to go down this automation process for everything.

John Capobianco: [35:59] Yeah that's a good question and so a lot of its business value related and you touch on some of our reasoning scale and complexity and avoiding human errors were possible.

[36:13] Part of it is is delivery times obviously but I don't want to give the impression that automation is simply about going faster yes certainly we're going faster and were able to push things out.

[36:25] With much more agility and respond to business demands faster there's also pragmatic things right you kind of talked to operations go down into the weeds and say is there anything you're doing three or four times a day,

or weak or are there things that,

you find tedious that you have to log into you know what I mean that you're doing manually that an automated solution would help you with so it's not just about my vision and me driving it to say,

you know,

it's better because we're working with code and the tools are more advanced than Network Tools but there's you know I want to help the business and Achieve things so here's a practical example,

and it kind of touches on our business so there's certain committees and that's when the you know the members and the,

the government get together to establish laws or discuss sensitive materials or whatever business of parliament the doing certain committees are cut off from all Communications so they turn in their phones there's no tablets we knife,

meaning we have transceivers that we can power off to physically remove the fiber connections to the room.

But there's the the wireless is the challenge here so as you may know.

Wireless is is self-healing so if I turn off five access points around the room the 10 around them are going to try to crank up their power to try to fill that Gap.

[37:49] And that's a good thing on four ninety-nine out of a hundred times that's a good thing but when you want to black out when I want to create a physical space without any air waves without any radiation for Wi-Fi,

that becomes a challenge,

so we had to work with our partners and security to sweep the room for signals and then we shut off more APS or turned off different APS or until we got the mix right now for certain rooms this is like 30 or 40 a piece that have to be shut off.

And the business can't proceed right they can't they can't start doing business until it is a secure room.

So that means it's-- on the hook and we're now where the bottleneck how quickly can you go.

Shut those 35 interfaces on the right interface on the right device right it's a manual thing and you don't want to be doing that so.

Again automation right it's orchestrated it's cereal its on demand or scheduled,

and the Playbook doesn't make the mistake of the wrong Port because it's programmed and and we vet the program in the code is QA analyzed and more than one set of eyes is looking at it,

and it's quick an operator just I don't want to say it's not about the magic button right but the operator has the one line command we need to cut room 10 from the network.

Cut room 10 Playbook done taking care of turn room 10 back on run the turn 10 back on playbook right.

So it's pretty powerful stuff and that's just you know that's one example.

Zig Zsiga: [39:18] What's a good example

people do right like that's a good example so how does that work it's one Command right but how does it find out all the APS in that room and then finds the devices they're plugged into but I don't know if it's multiple switches that they're plugged into right it could be I don't know if rooms,

the one switch or how that works.

John Capobianco: [39:35] Yeah so we put together through the through the manual exercise with with the security partners,

once we establish the list of interfaces that became the list of interfaces we tested every once in a while to make sure that.

You know signal bleed isn't happening from another AP that's not in the list so it is tested occasionally for for you know relevance but but it's kind of it's not as dynamic as you would think.

And that's and actually that's you bring up a good point where to get started the CLI is not dead to me functionally.

[40:12] I still have to make,

that recipe work at the command line so what you know a lot of people wonder what my process is or how you do this practically you're going to make those those run commands or config commands or whatever it is you're trying to do you're going to have to figure out the recipe at the CLI,

and then and then I like to wrap it in the ansible Playbook and make that work and then we can we'll talk about CI CD in a little bit I want to get there but.

When I'm talking about full configuration management what I mean is.

To transform a running config so just a simple Show run your running State into two separate artifacts one is what I call a data model,

and you know you'll hear about data modeling or modeling data now that is a yellow file and in that llamo file it's made up of.

Key value Pairs and lists that's it it's a very simple and it should be simple so something like.

[41:11] Host underscore vlans.

Coleman right and that's the the key pair value and then under it would be a list of vlans VLAN 10,

: enter name blue VLAN so I want to turn that that construct that looks what it looks like in iOS or nx-os,

Stan's is on the CLI and turn them into human abstracted data models that are easy to read,

and they don't necessarily have to be networking lingo or jargon in your data model you should be able to give the data model to someone on the street,

and they could read it and kind of make sense of it right yeah okay this is switch it has 10 B lands and this vlans name blue and it has 4 V RF sand whatever right so that's your data model,

and then you're going to have a ginger to template that takes care of the stanzas of configuration now.

Sticking on vlans that is on a Cisco switch its VLAN 10 and then a line.

Two spaces the key name and then the name of the VLAN right so in that template you're going to have.

VLAN and then a variable.

One line two spaces name and then another variable those variables are the actual.

[42:34] Data in your data model so then you wrap that you do that one time in Jinja and make a for Loop out of it.

Seems safe or VLAN in host vlans generate this configuration stanza and four.

That creates a file and now I have 10 or 20 or 50 or a thousand vlans modeled and configured and templated appropriately.

Ansible has a module called the template module that I can push that template to my switch,

and that's that's how I handle full configuration management now you say what degree to what degree have you done this we have done it to every line every character,

in our running configuration in the data center on the campus in the wide area network every line of configuration is represented in a data model and a template.

[43:29] And.

We want what's called item potency so now item potency is a mathematic thing that means you can run the same result a billion times and it will always generate the same result right it it's a match.

So what we want to strive for is our intent.

And intended configure our git repository that is exactly the same as the running configuration and they're item potent and they match.

And then everything is handled through code if you have to add a VLAN or a route or a vrf or an access control list or whatever it is you have to add.

You add it to the data model now something new a new feature comes along let's say we want to add netflow when that flow has its own configuration stanzas.

So I'm going to make a branch in my git repo called you know netflow development I'm going to develop in my CLI the right mix of commands.

That I'm going to transform the commands into new Jinja template section stanzas and add the model data to the data model.

And that allows me to scale releasing say netflow to 300 routers in just a few minutes right.

Zig Zsiga: [44:48] Wow okay so my mind a little blown here right and

I'm taking a lot of this in right because I'm new to this and I'm sure everyone else is too but that was a great I think walks through how this actually functions and works with a lot of details

so the I want to kind of dive in just a piece of I'm interested right so that VLAN example that you just did right.

He pull out information how does it mean you have a structured list of the vlans,

with all the information for each VLAN that you now want to leverage and maybe push to devices is that just as simple as running some sort of ansible Playbook to then take those vlans and configure them on those devices.

John Capobianco: [45:29] Yeah yeah so so you're going to have your data model and template and here's here's how the way here's a way to look at it at an so we'll run time.

The two are compiled into the the intended config that you're going to then push to the device and.

[45:48] And ansible will only push things that are not present on the device so let's say.

I add a VLAN to my list and I rerun The Playbook it's not going to try to over right.

The previous 10 B lands it's only going to add the new two lines configuring the new VLAN,

so it has some intelligence there to say oh I see there's a diff push the diff right so what I really love about ansible and you've led me to a great point I'm glad you asked that question.

You can run ansible with check mode and it's one of the big highlights to me of ansible and why I loved it as a tool.

Is that you can do a dry run of your playbook that will go through all the steps of your play.

Like all the tasks in your playbook but it doesn't actually push anything it doesn't make any modifications it's like a dry run that you can validate your code.

And it will tell you at the end it will say Hey listen had you pushed me I would have added these four lines of config meaning I would have added VLAN 100 John's new VLAN.

Right oh that's what I want that's my intent oh no I don't want to do that right it gives you that break point so when a so when we move into operations like in the real world my steps are typically one.

[47:10] Generate the generate the intent to run the playbook in a documentation mode and all that does is it doesn't even touch the device it compiles the data model and the Jinja template.

To give me the file that would be pushed so operations generates the intent and we look at it offline yeah that looks like I what I want.

Run it in check mode a second validation step that will confirm to them here's the 10 lines or thousand lines or zero lines of changes that I would have made if you have executed me and execution mode.

They review that or they send that Upstream to a developer hang on this doesn't look like what you told me are you sure this is the change we're trying to make,

right there's a lot of QA capability here so then we resolved and say yep that's what we want to push then they run it in full execution mode.

[48:03] Now a little trick that we've learned is we run it a second time in check mode after we run it in execute mode.

So that we can see that it doesn't have anything left to push meaning,

run it in check mode I'm going to push five lines run in and execute mode it should have pushed five lines now when I run it in check mode again if everything worked in the above,

there should be zero lines to push and it should be item potent.

If it still has those five lines something went wrong and it didn't push the commands so there's you know you starting to work like the software developer yeah.

Zig Zsiga: [48:43] But you're inherently putting some help validation into that process right you're saying hey we're going to do these we're going to do this step to see what's going to change so we're going to go ahead and deploy that change in some fashion and then we're going to read check to see if that change actually got deployed you know and.

John Capobianco: [48:57] Yeah exactly and and they're all just steps in the orchestration and ansible can you know it will give you a nice summary at the bottom saying.

These you know here's what was pushed and these four changes were made it's very so I talked about why I picked danceable.

It being agentless is a big win you don't have to actually modify your environment to start using it you just start using it.

The barrier for entry is very low it's very very simple especially like if you gather facts and write a little plate as to make a Json file that of the facts like if you take that as homework,

it's only going to take you 10 or 15 minutes to figure out and then you're going to be like holy cow I've just documented my network in 10 minutes you know artifacts that most networks don't have the real state of the network.

In programmatic Json files it's just incredible.

Zig Zsiga: [49:50] So like if I had a network that had let's say I'm just gonna throw a random numbers out there a thousand devices maybe I had a lab

environment and avert like at home in my lab and I have a thousand devices and if I didn't know what I've done in the last six months because it's I keep my lab up all the time I don't have a thousand devices just so we're clear I might have like a hundred,

my service provider networks and stop

so let's say I didn't know where everything was is I could easily I don't have ansible installed today I don't have anything install don't have get I do have a GitHub repository for some stuff for Cisco live but other than that I don't have anything,

that that time for me to install everything get the facts configured and then run a fax check or whatever you want to call it

on all those devices what do you think roughly that would be for an average person.

John Capobianco: [50:38] You know I give yourself two hours two and a half hours you're probably need about half an hour to it depends on your Linux let's say dependent Linux Linux dependence skills,

but if you have a sentence bar here's here's how what I would do if you have Windows 10 install WSL to the windows subsystem for Linux it's in your control panel,

turn WSL to one go to the windows Store and install lubuntu,

now you can run Linux in your windows environment without separate Hardware or VMware or anything it just runs a boom tube now if you can ping.

That switch from your Windows machine you know you're going to need to be able to SSH into it there's no firewalls in the way right you're fine.

In a boon to install ansible and right now we're looking at your ear 10 minutes in the install Time download from the Internet you may be 10 minutes in ansible takes a minute to install.

The facts Playbook might take you five to 10 minutes to write depending on how quickly you take in the information from Doc's Dot ansible.com.

And then it takes 10 seconds to gather the facts from each machine and maybe two seconds to make each file right so like on a good day you might be able to do this in 45 minutes if you struggle a little bit you're looking at a couple of hours.

Zig Zsiga: [51:55] Yeah but me I'm just thinking like think of that the impact right there right two hours and you can get a full inventory of what's out there and where things are running the upstate downstate the protocols a neighbor ships and that fax gets all that information right.

Just that alone you're not going to changing anything not even.

John Capobianco: [52:11] Nothing nothing,

and so then you mentioned a couple other tools so here's the Pinnacle here's and I'm going to finish on this note we got about 15 minutes I should be able to cover it at the top is CI CD to me continuously integrated,

continuously delivered and so here's where Docker comes in so Azure devops has the ability to build,

and release software and in my terms software is a Docker container image so Docker when they say image and now I may be misspeaking I'm not the best at this doctor stuff it's very new to me so.

I don't want any angry emails that I didn't get this right.

[52:53] So when they say image it's like it's like a CD-ROM like a like an ISO like the ISO 9000 standards,

images like an immutable think of it as a CD-ROM that you build yourself that you can't change once it's built you can log into it and do stuff in it when you log out those changes are discarded.

So here's how I do it.

In my git repo I added a new file called the dockerfile and in this dockerfile it's like those old batch files are ini files I specify I want Alpine Linux I need python I need ansible I need get,

I need this repo and I need you to run this play.

I take all that in my Docker file then it as your devops I set up a build and in my build I say build from this git repo and specify this Docker file as your Docker build image.

[53:46] It builds a container image and registers it with a registry and that image is like those nesting dolls I got working at the CLI I made an Ansel Playbook make it work and now I'm rapping that whole thing in a Docker container image,

that I can run the container image without any humans.

So then in in a release I scheduled a release of that container meaning run that container somewhere,

so now it's fully automatic those facts that I'm talking about every four hours release the container which gathers the information and builds the CSV files and puts them back in the git repository just runs.

Zig Zsiga: [54:26] It just runs every four hours and then the docker does it does itself terminate like this itself go away.

John Capobianco: [54:32] It destroys itself at the end of the row.

Zig Zsiga: [54:34] Channing not terminate r8d provisioning.

John Capobianco: [54:35] And in the yeah and in the dashboards I can look at the interactive logs so I actually have the full Playbook.

Tasks and steps and all the logging information available to me and,

and it's just so incredibly powerful so the continuous part it's actually makes sense it's pretty more it's see I see these not as mysterious as you would think,

The Continuous integration but I mentioned get branching strategy that is the trigger for continuous integration so here I have a,

I have just my core facts now I want to do my building distribution so I make a branch I do my development when it's all ready to go I do what's called a pull request emerge that back into Maine,

if that's approved and happens that action triggers a rebuild,

oh there's a new information to be stored in the docker container rebuild the image that's the continuous integration every time there's a pull request rebuild the image,

never continuous delivery is oh there's a new release.

There's an right there's a new build I better release it automatically go ahead and Trigger the release so now it truly is code,

that's driving this right infrastructure is code is I add a vrf.

Do my poor request it triggers the build which pushes that brf out to the network.

Zig Zsiga: [56:04] Now this approval process right so if I have a branch and I say there's that right there.

John Capobianco: [56:06] Oh yeah I'm simplifying yeah.

Zig Zsiga: [56:09] Just so we're clear but like so if I create that.

John Capobianco: [56:11] Freddy any of my management any of my bosses who have tuned in to this there is definitely controls and change processes and testing and validation.

Zig Zsiga: [56:19] All right yeah all the good things.

John Capobianco: [56:21] Seriously I'm simplifying this quite a bit there's a lot more that goes into it but but that's the gist of see ICD yeah.

Zig Zsiga: [56:28] So it's a real quick and then we'll get into the rest so if I create a VR app I have a branch of whatever let's say have a branch of

that router that Network I don't know what the branch dictates what the main dictates right but like I have a branch I'm going to create a vrf for a new element a new secured you know

line of business for the parliament Network and let's say I'm going to call it zigbits cuz that's what comes to mind right so it's zig and I'm done I've done my testing it's good it's in the branch now if I want to commit that to Maine.

That is a process right there's an approval process once it's approved it commits it to Maine

right and then now the automation will rebuild the docker container with that new code in it from that main version and.

John Capobianco: [57:10] You nailed it you nailed it and our processes and and that pull request gives us an opportunity so we protect mean there's no direct commits,

you need a poor Quest and a poor Quest needs at least one reviewer maybe two reviewers so that's where I send it to a colleague and say,

right I've added this vrf could you please review my code look at the commit history look at the get artifacts they look over at all all you got an IP wrong or oh that looks good they approve it.

And then right and then the change management board dictates when that release can happen is we don't we're not Netflix we're not to microservices here we're like when I describe that oh a pull request triggers a build and they just push code out to the network,

I'm not doing that at two in the afternoon on a.

Zig Zsiga: [57:55] You're not doing it right now you know you got one going out right this minute.

John Capobianco: [57:56] No our we've decoupled our bills and our releases so that they're they're handled by you know we're still.

Zig Zsiga: [58:04] There's their governance right there's a governance process underneath the hood right and it's still easy right it's still simple it just you have to get it approved by that change control board totally understand right now that approval process is it like a week a month.

John Capobianco: [58:17] It depends on the impact to be honest we have to juggle.

You know yeah there's a lot of things for Juggle II that's a little bit of I just say my you know I'm constantly waiting when is it approve when's it going to prod right I'm at the mercy of cab quite a bit.

Zig Zsiga: [58:33] Yeah I get that I am too I totally understand.

John Capobianco: [58:35] You know,

it's quite the environment where you know we had talked about in a previous episode right all the security cameras all the security apparatus we don't have the luxury of just,

taking buildings down you know without serious considerations so it's very,

it's a balancing act right the CI CD is incredible and if you know maybe if I was in a vacuum,

without a business behind it if this was my own lab yeah I would have this stuff continuously delivering all the time.

Zig Zsiga: [59:05] Yeah that makes sense right if it's a lab environment right that makes.

John Capobianco: [59:07] But in a business you know.

Zig Zsiga: [59:09] This is a business impacting change right so I get it he wrote one more technical question for you

happens if something goes wrong like let's say you go through this process it's improved and then it does the automation it deploys it but let's say something wasn't known and there's a problem

what's the steps high-level to get out of that.

John Capobianco: [59:26] Yeah so we have a couple of options because of this build and release idea,

our Network kind of has like a version times their version stamp on it right so version 1.0 of the network is today's version I'm adding 1.1 I'm adding a new vrf.

1.1 is a bad release right that's a bad version quickly roll that 1.2 to fix it or rollback to 1.0.

So sort of like that that that bad software patch that goes out right and because it's container-based I can say just run the 1.0 container and release that code now.

That's it depends on the platform I can do that with restful platforms where I'm using the API in a post.

[1:00:15] But certain ansible things right it's just SSH serial SSH commands it's not that it's not like a Cisco 4500.

Unless I'm using config replace every time it's hard to roll back right,

so we've been fortunate and we've been very careful and cautious and I want to say prudent,

where we haven't actually had a lot of mishaps with the automation I can think of one where I had to come in and get onto the console and repair something,

and you know that's damages your reputation of Damages your progress and your momentum and you kind of live with it for six weeks and everyone forgets about the big problem that happened.

But I have had it happen you have to be very careful the blast radius.

Right is really what you need to consider lets you know in the old days that's the advantage of CLI is to see Ally has.

Okay here's your instructions operations go change all 50 routers by hand they do the first router and there's a problem in the instructions.

They only damaged one router right they know enough to back out because they're human.

If I get it wrong in the code it's happy to go and destroy all 50 routers immediately right it's not it doesn't have the smarts to take a step back and say hang on you got the IP wrong here.

[1:01:32] It's just going to blast it out and take down the whole thing so yes you're on a tightrope here with the automation you got to make sure you have appropriate safety nets under you and make sure that you know.

E

Zig Zsiga: [1:01:46] You to verify everything right it's still.

John Capobianco: [1:01:48] Yeah yeah I cheat as much as I can yeah you got to be very careful with the network.

Zig Zsiga: [1:01:53] Hey John nice

I really appreciate them and that's super helpful I thank you John this has been an amazing conversation it's amazing to hear the journey and the Automation and the impact it has truly had for you and the Parliament Network what advice would you give to finish off here to the person that's just starting out into automation maybe they are having a hard time

starting out what words of encouragement would you offer them at this time.

John Capobianco: [1:02:17] Yeah I want I want to be very positive and encouraging and I know it's a lot I know this is like a firehose blast of new terms and tools and techniques and methodologies,

it will make your life a lot easier I find I enjoy my work a lot more too because it's like little logic problems every day where you've got it right little Snippets of code to help you do your job so,

take that first step and really really go for it like,

this is the new way of doing things the CLI is you know it's got an expiry date on it and the old way of doing things or let's just say you know just knowing your network plus CCNA stuff,

I don't know that that's enough anymore I think you have to have some new skills and new tools,

to stay competitive in the job market but also because you can do a much better job using the new way.

Zig Zsiga: [1:03:08] That's awesome well you heard it here first John were thankful for you to being here today looking forward to see what else you come up with on the automation front and networking front last thing where should people be a coach sorry where should people find you

and learn more about your journey where can they reach out to you if they have questions.

John Capobianco: [1:03:25] Yeah so if I've tried to centralize everything at automate your network dot C A so we're on there there's a link to my book and on my podcast I'll be putting this up on there I'm on Twitter John underscore capobianco I love to,

interact and reach out I reach out to Ziggy reached out to me on Twitter so I am available if you have questions or if you need help or anything at all just let me know I'm here.

Zig Zsiga: [1:03:48] Truly amazing John thank you so much for coming today I appreciate you buddy.

John Capobianco: [1:03:52] Is it great to see you again man we have to do this again soon.

Zig Zsiga: [1:03:55] For sure hey friends nerds geeks and ziglets that's going to close out today's episode of the zigbits network design podcast where we highlighted a true a real

automation case study of the parliament Precinct network with my friend John capobianco hey make sure you go follow John on Twitter to see what else he is working on these days.

Today's Show notes will be at zigbits dot text / 78 hey we are doing another one

awesome I would standing giveaway zigbits themed giveaway this month and this month we are focused on Automation and programmability so our zigbits February 20 21 giveaway bundle includes Dev net,

devops python Dev ACS Dev Associates and so much more in fact today's guest John capobianco was book

titled automate your network is in this giveaway as well so to see all of these details

about this giveaway indicate yourself entered into it simply go to zigbits dot text / give away this will be our link for all of our new future zigbits giveaways moving forward.

[1:05:01] Hey if there's something you would like us to give away in the future let us know we are starting to plan out our March and April 20-21 giveaway items right now.

Hey if you didn't know we are creating a network design course I know it's weird right we're doing this a vendor agnostic certification agnostic Network design course and it doesn't matter where you are with your career we are creating a network design course so if you are just starting out

it'll work for you and if you have been doing this stuff working in networking and network engineering for the last 30 years it will still work for you the intent is to teach

everyone Network design and how important it is,

if you want to hear weekly status updates on where we are with this course join the email list at zigbits dot Text slash network design.

Hey if you liked today's episode if it inspired you resonated something within you or provided a level of real-world context let me know you can find zigbits on all the socials Twitter LinkedIn Facebook by searching for zigbits that's

zi gbit/s you can also send me an email to feedback at zigbits Dot.

As always I appreciate you and I thank you for listening,

remember don't forget to attack your goals attack the day attack your life and make progress my friends until next time bye for now.