I remember in my old days that our server farm had a separate network segment (sometimes even more than one) when I was using physical servers, (while I write this - I actually think it has been about 4 years since I actually touched a physical server, or plugged a cable/disk/device into a physical server) for our Domain controllers, Applications servers, and users had their own network segments that were dedicated only to laptops and desktops.

In the physical/on-prem world - this made sense - at the time - because what usually happened was the dedicated networking team that managed your infrastructure used access lists on the physical network switches to control which networks could go where.

Fast forward to the cloud.

There are people which equate VPC’s with Networks (even though it makes more sense to equate subnets to networks - but that is besides the point) - and think that segregating different kinds of workloads into multiple VPC’s will give you better security.

Let me give you a real scenario that I was presented with not too long ago (details of course have been changed to protect the innocent … )

A three tier application. Database, Application and a frontend. And the requirement that was laid down from the security team was that each of the layers must reside in the their own VPC. Think about that for a minute. Three VPC’s that would be peered to ensure connectivity between them (because of course the 2/3 layers needed to communicate with each other - Database - application and application to frontend). When I asked what was the reason for separating the three different layers in that way, the answer was, “Security. If for example one of the layer was compromised - it would be much harder to make a lateral move to another VPC and compromise the rest.”



So what is lateral movement? I know that there is no such a thing as a 100% secure environment. There will always be hackers, there will always be ways around any counter measures we try and put in place, and we can only protect against what we know and not against what we do not. The concept of lateral movement is one, of compromising a credential on one system and with that credential moving to another system. For example - compromising a Domain admin credential on an employees laptop - and with that credential moving into an elevated system (for example a domain controller) and compromising the system even further.

So how would this work out in the scenario above. If someone would compromise the frontend - the only thing they would be able to connect to would be the application layer - the frontend - does not have any direct interaction with the database layer at all, do your data would be safe. There would be a peering connection between the Frontend VPC an the Application VPC - with the appropriate routing in place to allow traffic flow between the relevant instances, and another peer between the Application VPC and the Database VPC - with the appropriate routing in place as well.

What they did not understand - is that if the application layer was compromised - then that layer does have direct connectivity with the data layer - and therefore could access all the data.

Segregating the layers into different VPC’s would not really help here.

And honestly - this is a risk that you take - which is why the attack surface you have - exposed on your frontend - should be as small as possible - and secure as possible.

But I came back to the infosec team and told them - what if I would provide the same security and segregation that you were trying to achieve but without the need of separate VPC’s ?

I would create a Single VPC - with three subnets and three security groups, Frontend, Application and Database. Instances in the frontend security group would only be allowed to communicate with the instances in the application security group on a specific port (and vice-versa) and the instances in the application security group would only be allow to communicate with the instances in the database security group (and vice-versa).


The traffic would be locked down to the specific flow of traffic and instances would not be able to communicate out of their security boundary.




As a side note - this could have also been accomplished by configuring very specific routes between the instances that needed to communicate between the VPC’s, but it does not scale to an environment larger than a handful of instances. Either you need to ensure that the IP addresses in a manual fashion, or keep on adding multiple routes in the route tables.

It goes without saying that if someone managed to compromise the frontend, and somehow managed through the application port to gain control into the application layer - they could gain access (in theory) to the data in the data layer.

Which is exactly what happened in the same scenario with 3 separate VPC’s. No less secure - no more.

But what changed??

I have come across two separate discussions this past week where it became clear that some people have no idea what DevOps is.

The first one was an Israeli company here in Israel - https://devopsexperts.co.il/. Here is the proposed syllabus:

They are offering this course - for a fee (of course), selling the hope that if someone would graduate the course - then they would be able to get a position as an DevOps engineer.

Someone asked on a channel - "Was this course worthwhile?".

I would like to share with you my answer.

I do not want to take away anyone's livelihood but there is no such a thing a "teaching/learning" DevOps. There is no single course that can encompass all the capabilities that one would need to become a successful DevOps professional. Above and beyond that - in each and every organization - the term DevOps will mean something completely different.

There are a number of basic topics that one can learn - and with them build up a strong foundation of skills in order help your specific company. If I would evaluate a potential candidate - and their education was based mainly on this course - I would not hire such a candidate.

The demand for talented professionals is high , everyone wants DevOps engineers - and there are not many people that have enough experience or the know how. Of course with a demand - people identify an opportunity to make money.

Looking at syllabus - it has so many flaws. The course was 45 hours (which means about 1 work week)

• Scripting - what language are they going to teach you? Python? But who says that the company you might work for - could be using something completely different.
• Version Control - so this is basically git.
• Linux fundamentals - basic Linux course
• Provisioning Resources - with what? Terraform? Ansible? something else?
• Build Automation - Building a pipeline - with which tools?
• Continuous Monitoring - is that even a concept?
• Working with containers - docker run, docker build, docker pull/push
• Configuration Management - use which technology - I can name at least 3 CM tools that you might use

As you can see, this is a 50,000 ft. view of what you might do in your day to day work as a DevOps engineer - but in no way or form can you learn any of these things in a course - and definitely not in 45 hours.

For me a good candidate would be someone that has the ability to learn, understands the big picture of how software is built, deployed and managed on a regular basis. There is no list of technologies that could be checked off a list that would qualify a candidate. Does someone know Jenkins? That might be great - but if we use something else - CircleCI, Electric Commander? What will the specific Jenkins knowledge help?

DevOps is not something that you can learn in school, or in a course. It is a collection of technologies that you collect during the years, it is a state of mind that you become accustomed to as you grow, it is a set of organizational practices that you pick up on your journey.

Not something you can learn in school.

Next one was Microsoft - who decided to rebrand VSTS into Azure DevOps. Again a shiny buzzword which Microsoft assumes will attract people to the product and their offering.

“Azure now has a new set of five DevOps services,” Jamie Cool, Microsoft’s newly retitled director of product management for Azure DevOps, told The New Stack. “They’re going to help developers be able to ship faster, [with] higher quality. Oftentimes when I have conversations, ‘DevOps’ can mean different things to different folks. So to us in this context, we really think of DevOps as the people, the process, and the products for delivering constant value to customers.”

Here in the statement above is the problem (the emphasis is mine). Products do not deliver DevOps, at least not what Azure is offering. I do agree with the part about the people and the process - but not the products. Maybe the tools - but not products.

If they would have branded the product Azure CI/CD  then I would have been all for it - but to me it seems that this is marketing play - trying to catch a goal that today everyone is trying to achieve.

In a group discussion last week – I heard a story (I could not find the origin – if you know where it comes from – please let me know) – which I would like to share with you.

John was floating out in the ocean, on his back, with his shades, just enjoying the sun, the quiet, the time to himself, not a care in the world.
When all of a sudden he got bumped on the head (not hard enough to cause any serious damage) with a small rowing boat.
John was pissed…. All sorts of thoughts running through his head.

• Who gave the driver their license?
• Why are they not more careful?
• I could have been killed?
• Why are they sailing out here – this is not even a place for boats.

And with all that anger and emotion he pulled himself over the side of the boat, ready to give the owner/driver one hell of a mouthful.
When he pulls himself over the side, he sees an empty boat. No–one there, no-one to scream at.
And at that moment all the anger and rage that was building up inside – slowly went away.
We encounter things every day – many of them we think are directly aimed at us – deliberately or not – but we immediately become all defensive, build up a bias against the other and are ready to go ballistic. Until we understand that there is no-one to direct all this emotion and energy at.
And then we understand that sometimes thing just happen, things beyond our control and we cannot or should not put our fate into some else’s hands.

That was the original story – which I really can relate to.

(Source: Flickr – Steenaire)
But before I heard the last part of the story – my mind took this to a totally different place – which is (of course) architecture related.
John was enjoying a great day in the sun – and all of a sudden he got hit in the head by a boat.
Where did that boat come from?
No-one knows.. I assume the owner had tied it up properly on the dock.

• Maybe the rope was cut.
• Maybe someone stole it and dumped it when they were done.
• Maybe there was a storm that set the boat loose.
• Or maybe there was a bloopers company that was following the boat all along to see who would get hit in the head.

There are endless options as to how the boat got there. But they all have something in common. The boat was never supposed to end up hitting John in the head.. John expected to be able to bake nicely in the sun and not be hit in the head by a boat
But what if John had taken additional precautionary measures?

• Set up a fence / guardrail around where he was floating
• Put someone as a lookout to warn him about floating boats
• Have a drone above his head hooked into a heads-up-display in his sunglasses that he can see what is going around him

There are endless possibilities and you can really let your imagination take you to where you want to go as to how John could have prevented this accident.
What does this have to do with Defensive Design?
When we design an application – we think that we are going to be ok – because we expect to be able to do what we want to do without interference.
For example.
My web server is suppose to serve web requests of a certain type. I did not plan for someone crafting a specific request that would crash my server or bombarding the webserver with such an influx of traffic that would bring the application to its knees.
But then something unexpected happens.
When you design your application you will never be able to predict every possibility of attack or some esoteric ways  people are going to use your software. There is always something new that comes up – or someone thinks of a different way to use your idea that you did not even think of.
What you can do, is put some basic guardrails into your software that will protect you from what you do know or think can happen.

• Throttling the number of connections or request – to prevent DDOS attacks.
• Introducing circuit breakers to prevent cascading failures
• Open only specific ports / sockets
• Sufficient authentication to verify that you should be doing what you are supposed to
• Monitoring for weird or suspicious behavior.

Again the options are practically endless. And you will not think of it all. You should address the issues as they happen, iterate, rinse, repeat.
That was a 4 minute read into thing that I think about during the day.
What kind of things do you think about when during your daily work? I would be interested in hearing. Please feel free to leave comments down below.

I am sure you are all familiar with those select few in your company who supposedly have super powers or hidden technological gifts.

Yes, I mean those co-workers who know exactly how to fix the most esoteric issues that no one has come across, ever. Perhaps you share a cubicle with the guy who’s able to conceive of breakthroughs time after time after time. Or maybe you were hired the same month as the woman who can code like no one has coded before, and it’s like reading poetry — smooth, with meaning, and plays on your most inner emotions. Or you’re the manager who hired the weird dude that sits in the basement, who can be woken up at 2:43 am on a Sunday morning after partying endless hours the night before, and still be able to drone off the precise sequence of events that you need in order to prevent your production NoSQL database from exploding … because someone forgot to run maintenance.

Continue reading the rest of the article

<RANT /ON>

I was rubbed the wrong way today - listening to a podcast.

This is a podcast that I hold in high regard, the hosts are people I have had multiple conversations with over the years and they are well renowned and public speakers.

This is a problem with society as a whole so I will not name the podcast as to not single them out - but I will use them as an example.

The podcast was about a certain configuration management product and what they are doing in the cloud space.

The total running time of the podcast 32:00

I collected some metrics from the episode.

Host spoke 09:08 (9 minutes and eight seconds).
During that time the host used the word “Like” 36 times and the word “y’know” 36 times

Guest spoke 22:52 (22 minutes and 52 seconds)
During that time the guest used the word “Like” 64 times and the word “y’know” 153 times

Assuming that for occurrence it takes approximately 1 second - that totals to 489 seconds of worthless content of the episode (15% of the duration of the episode was wasted these words)

I grew up in South Africa so I am accustomed to the Queens English - and therefore use the Oxford dictionary as my source of truth.

This is the Oxford Dictionary definition of like (and I would like to emphasize this entry)

ADVERB

1 informal Used in speech as a meaningless filler or to signify the speaker's uncertainty about an expression just used.
‘there was this funny smell—sort of dusty like’

2 informal Used to convey a person's reported attitude or feelings in the form of direct speech (whether or not representing an actual quotation)
‘so she comes into the room and she's like ‘Where is everybody?’’

3 (like as/to) archaic In the manner of.
‘like as a ship with dreadful storm long tossed’

This is the Oxford Dictionary definition of you know

PHRASE

informal
1 Used to indicate that what is being referred to is known to or understood by the listener.
‘when in Rome, you know’
1.1 Used as a filler in conversation.
‘oh well, you know, I was wondering if you had any jobs for me’

Some of our generation (and almost certainly our kids generation) has forgotten how to speak - I know this is the case in English - and unfortunately - this has become a norm in Hebrew as well.

If you want people to take you seriously - then speak proper English. It does not make a difference if it on  podcast, at a presentation or even a customer call.

When was the last time you heard Steve Jobs, Donald Trump or Queen Elizabeth use “like, y’know”?

Every time you use these as filler words, a kitten dies, a unicorn loses some of its color, and we all become the slightest bit more and more stupid.

If you don’t have anything to say, then don’t. If you need to collect your thoughts - then do so - in silence. I promise you, no one will think any less of you - quite the contrary.

<RANT /OFF>

This is third post in the The Three Pillars of DevOps series

Part 1 – The Three Pillars of DevOps Part 2 – Pillar #1 - Developers Part 3 – Pillar #2 - Operations Engineers Part 4 – Pillar #3 - Management Part 5 – Bringing it all together

In this post we will dive in to the third pillar – Management.

Being part of the Pillar

1. DevOps is a cultural change

   Most people do not like change. Personally, I do not have an easy time adapting to change, it makes me nervous, uncertain, unsure. Different people take to change in different ways. Adoption of a DevOps way of working is not a small thing.
   
   Boundaries are no longer clear.
   Who is in charge of what?
   Why do we have to take care of this?
   I have no idea of what this means.
   
   With any big change you should understand what this will do to your organization. See where you will have problems. Where people will need help. Expect that things are going to change. Over time.

2. Support both of the other pillars on the journey

   
   In continuation on the previous point. Help your developers with education. They are no longer responsible for a small part, but rather the big picture.
   
   Help your Operations Engineers with the world of code, again training, courses, books, seminars, pair programming there are a more than a number of ways.
   
   Bring in someone to help and coach your teams throughout the journey, going from waterfall to Agile is a huge change, even more so such a culture change. Help them in the beginning, through the ups and downs, and how they can continuously improve themselves and the way they work.
   

3. Have patience

   This is going to take time. Quite a lot of time. Give the teams the leeway to adapt and learn along the jurney. Let them learn from their mistakes, pave their own path. Your productivity will probably go down in the short term, which also mean that the bottom line might drop as well.
   
   Be prepared for this. Remember, if you are in it for the short term, the quick win, then this is the wrong reason. And it will most probably fail. Almost definitely. Your are here for the long term, even if it means losing in the short term.

Not being part of the Pillar (or being Samson)

1. We should start doing DevOps! Now!

   I went to this interesting seminar where they showed how they are delivering code 10-20 times a day. I want us to start do the same. Next week. Let’s just make some quick changes, merge some teams, get the Devs working together with the Ops, and we should be able to deliver even more often with better quality.
    

2. My employees can do double the work, in less time

   So now that we are ‘doing DevOps’ I don’t need so many people on my teams because they are doing the work on both sides of the fence. They can get their work done and also support the applications they are writing from start to finish with less people, and because they are more efficient probably in less time.

3. There is only one way to get this done.

   Agile, Extreme Programming, Kanban. Rally. We need one tool to monitor and rule it all. One process that everyone has to follow. Everyone has to fit into the box we create. That is the only way we can maintain control.
   
   The groups have to adapt to the way I want it to work. My way or the highway.

Next up we will look at Part 5 of this series – Bringing it all together.

This is second post in the The Three Pillars of DevOps series

Part 1 – The Three Pillars of DevOps Part 2 – Pillar #1 - Developers Part 3 – Pillar #2 - Operations Engineers Part 4 – Pillar #3 - Management Part 5 – Bringing it all together

In this post we will dive in to the second pillar – Operations Engineers.

Being part of the Pillar

1. Allow everyone to consume your infrastructure

   
   Infrastructure is there to be used. You are there to allow your business to create revenue, as much as possible, and in as short a time as possible.
   
   They will need resources in order to do that. You have probably been working with cloud and virtualization – long before they have, and have a decent amount of expertise and infrastructure already in place.
   
   In order to allow development teams to do their work, they will need resources, for a number of solutions, be it Continuous Integration, Continuous Delivery – or just plain old sandboxes for development purposes.
   
   Help them use what you have, help them build their own if they need it.
   

2. Become a trusted broker for your development teams

   
   Developers need your help. They have deadlines and problems that they are dealing with and will have a very difficult time learning all you know in a short time.
   
   Explain to them what the benefits are of using different kinds of infrastructure, when they should go to the cloud, and when they should stay in house. What are the security implications of choosing a cloud solution, what they need to be aware of. They are also on a journey and need to adapt to this new world.
   

3. Make it as easy as possible

   
   Again, infrastructure is there to be used. The same way that you take for granted that when you flip the light switch in your room you expect the lights to go on, that is what developers and the organization expect to happen when they need to turn on a server.
   
   Of course we all know that when you flip on a switch – there are so many things that happen, so much infrastructure is needed, from wiring to circuit boards, to light fixtures, to metering, to electric company.. (and so on..), but all of that is transparent to end user.
   
   You should aspire to make your infrastructure as easy to consume as the electricity in the building. No-one is saying that it is easy, but that should be your end goal.
   

4. Work with the development teams to help produce quality products

   
   The development teams have a way of doing things, not necessarily is this best way, and you probably no longer have a number of grey hairs that you pulled out over the years trying to solve problems created by your development teams.
   
   Explain to them what does work, what does not, and why. Work with them together to find a solution that acceptable and will work for all sides.
   
   Don’t expect that things will be perfect the first time around, because they won’t. Iterate and make improvements in stages, small steps until you get to Nirvana.
   
   Go through deployment models with them, explain to them what scaling is, how high availability is achievable in this new world, and what they need to change in order to get there.

Not being part of the Pillar (or being Samson)

1. Be an infrastructure hugger

   
   We paid for it. We installed it. I know more about this infrastructure than any of these developers think they know.
   
   They cannot use what we have already because:
   
   - We have no capacity
   - The environment is not suitable for their needs
   - They will make a huge mess
   
   Let them go and buy their own, learn what we have for the past 5-10 years and then we will talk.
   
   
   

2. Create workarounds to accommodate badly written software

   
   Software is not perfect, sometimes it is just really crappy. And over the years you have learned to deal with that. Creating cron jobs to restart processes on a regular basis due to a memory leak, or creating cluster mechanisms to to solve high availability issues.
   
   These workarounds make your life more livable, manageable, but do not solve the underlying issue, just work as a band-aid until they get their stuff together and fix the junk they wrote.
   
   And since the development teams don’t care about these things any way – you never relayed back to them that these issues exist.
   

3. Let them go and use AWS if they want to, and hang themselves..

   
   We cannot provide the developers the kind of cloud experience that the demand. It is too difficult for us to make these changes, due to budget constraints, manpower or perhaps time constraints as well.
   
   If they need these things – let them go the where it is available, and pay for it themselves, and let them worry about their own issues of security, administration etc..

Next up we will look at Pillar #3 – Management.

This is second post in the The Three Pillars of DevOps series

Part 1 – The Three Pillars of DevOps Part 2 – Pillar #1 - Developers Part 3 – Pillar #2 - Operations Engineers Part 4 – Pillar #3 - Management Part 5 – Bringing it all together

In this post we will dive in to the first pillar – Developers.

I apologize in advance – but I will be using a number of stereotypes in this series – on purpose. I will probably exaggerate – also on purpose – but this is in order to get a point across.
I do have the utmost respect for all people in all three pillars – and I promise you, I will dump on each and every single one of the pillars (hopefully equally).

Whenever I hear someone say developers – the first thing that comes to mind is the famous Steve Ballmer dance.

But these people are not as crazy as Steve on stage.
They are probably the people that work with you side by side. They create the things you use – daily.

For example – I can guarantee you that no Operations Engineer or management director was the one who wrote the applications and the functionality built into something that each and every one of us use every single day.

Be it your:

• Web browser
• Mail Client
• Facebook application
• Phone

All of these things need to be written in code. And I have the utmost respect for people who have the capability to create something from scratch and build it into something like the examples above.

Being part of the Pillar

1. Understand what you know how to do best (and learn what you don’t)

   
   I do not know everything, I don’t think anyone can, and if they say they do – then don’t believe them. You know how to write code, you know how create an interface and I could go on with examples all day. But there are also things you don’t understand, such as Operating system security, firewalls, resource usage, network traffic, databases – and again I can go on and on and on…
   
   There are people who have been doing this – just as long as you have been writing code – so use them, consult with them – learn from them.
   

2. Build with the broader picture in mind

   
   Applications do not run in a vacuum – they are part of a complete solution – they need to run in parallel or on top of other applications – so take that into account.
   
   Think about how you will interact with other parts of the solution – because it happen – you will interact with other pieces – all the time.
   

3. Take full responsibility for what you produce

   
   You created it, it is your baby. The same way you bring a child into this world – you worry and care for it. You send them to day care (where someone looks after them) but if there your kid has a fever – you will come and get them – and nurture them back to health.
   

4. Assume the worst will happen (eventually it will!)

   
   Nothing is perfect (and again if someone says it is – don’t believe them). Thinking out of the box and out of your comfort zone, not cutting corners and hoping for the best – but rather planning for the worst case scenario – will make the product you are creating a better one – and you a better developer as well (by the way).

Not being part of the Pillar (or being Samson)

1. The Operations Engineers job cant be that hard

   
   I mean how hard can it be to spin up a server, create a DNS record, run a Load balancer in front and have 5 million people hit the site at the same time. What could possibly go wrong?
   
   This is a quote I just saw yesterday on Twitter.
   
   

   "We don't need an ops person" ... shortly followed by ... "How does DNS work? What's a CIDR? Can't you do it for us?"

   — Colin Panisset (@nonspecialist) August 26, 2015

2. My application is the only one that counts

   
   I am writing something that will change the world. And I heard that this new fangled database will help me out – so that is what I will use, for a number of reasons. It is quicker for me to produce like this, the performance is very good, and I like trying the new stuff of course. I don’t care that every other application is using another database – because adapting to that standard is boring and more difficult. And of course I don’t care about what kind of resources my application needs – just give me whatever you have. I need it all!
   

3. It worked in Vagrant, in DevStack, on my laptop

   
   I tested it, I ran it and I got it to work. My unit tests passed. My acceptance tests passed. I got green on my dashboard. So obviously it is not something wrong with the application. It is the “other guy”.
   
   I also don’t care how the application is maintained over time, my job was to get it work, so obviously that when you upgrade – you will need to restart services – but hey – they was not part of the original specs and requirements.
   

4. My application works perfectly – and never fails

   
   I invested time in writing this baby, and I made sure that everything is perfect, runs like well oiled engine, purrs like a kitten, a masterpiece. I checked that everything works, and every possible scenario was covered (at least those that I thought of).
   
   And of course the assumptions that I took will always be true (such as – hardware never fails, networks are always available, I have an unlimited amount of resources available to me, all the time)

Next Up we will look at Pillar #2 – Operations Engineers.

I apologize for plagiarizing the holy concept of The Three Pillars, but I do think that a foundation needs to be laid down for a healthy DevOps culture to thrive and even survive. And I would like to share with you some of my thought on this.

To be true to the agile methodology it would only be appropriate that instead of talking about pillars I prefer to talk about personas.

These three personas are:

1. Developers
2. Operations Engineers
3. Management

If I was going to have to lay it out in a graphical form – I guess it would be something like this.

In order for a healthy DevOps culture ALL three of these need to exist – and work in sync.

In the upcoming posts – I will go through each of these pillars and how they should play a part
(but also more importantly – how they should not) in this highly abused and misused term…

DevOps.

Follow the rest of this series here:

Part 1 – The Three Pillars of DevOps Part 2 – Pillar #1 - Developers Part 3 – Pillar #2 - Operations Engineers Part 4 – Pillar #3 - Management Part 5 – Bringing it all together