Showing posts with label Architecture. Show all posts

2018-10-08

#AWS PrivateLink vs. NAT Gateway from a Pricing Perspective

A customer came to me with a request. They do not want to use a NAT gateway from their VPC to access the AWS API's. They had a number of security concerns regarding the use of a NAT gateway (no control, logs, auditing - but that is a for a different post) and they asked for a solution.

The AWS API's that they needed access to were:

S3
KMS
SSM
Cloudwatch
Cloudformation

Last year at re:Invent AWS announced the option to create VPC Interface endpoints using PrivateLink and have steadily been adding more endpoints over the past year.

With the use of these endpoints you can actually have a VPC with instances that will not have any internet access (at least not through AWS) and still be able to interact with all the AWS API's.

This is technically possible - and can easily be automated, but I wanted to look at the cost perspective.

The VPC in us-east-1 has 2 Availability Zones (you should always have at minimum 2).

That would mean deploying 2 NAT gateways in your VPC (Pricing)

I am going to assume that you have the same amount of data going through both options - so I will not factor this into the price.

Usually you have 730 hours in a month.

Each NAT gateway will cost you 0.045*730 = ~$33.

Total for 2 NAT Gateways would be $66 per month (not including traffic).

What does this look like for Interface Endpoints? (Pricing)

Each Endpoint will need to be deployed in both AZ's in pairs.

Each Interace Endpoint will cost 0.01*730*2 = ~15

Total for all the endpoints above (4 Interface Endpoints - KMS, SSM, CloudWatch and Cloudformation) would be $60 per month.
The S3 endpoint is a Gateway endpoint - and therefore does not cost you any extra.

As you can see - it is not that much cheaper.

Take into account the following scenario - you need API access to 15 out of the 21 possible interface Endpoints

This would run you the steep amount of $225 per month - which is a lot more than just a NAT Gateway.

Design decisions always have tradeoffs - sometimes you prefer security and other times it will be cost. I hope that this will enable you to make an informed decision in your VPC design.

2018-09-27

Replacing the AWS ELB - Final Thoughts

This is the last part in the Replacing the AWS ELB series.

If you haven't already read the previous posts in the series - please take the time to go through them.

So here are some additional thoughts and ideas about the whole journey.

First and foremost - none of this would have been possible without group effort of the team that worked on this.
Udi, Mark, and Mike - thank you all for your input, help and hard work that went into this.

Was it all worth it?

Yes, yes and hell yes!! The cost of having to refactor applications to work with the way that the AWS ELB works - was not financially viable and would take far to long . There was no way we could make our delivery dates and have all the applications modify the way they worked.

So not only was it worth it - it was a necessity, without this - the project was a non-starter.

What was the hardest part of the solution?

Definitely the automation. We had the solution white-boarded out after a an hour or two, brought up a PoC within another hour or two.

As I said somewhere else in the post - if this was a one-off then it would not have been worth while - but we needed about 10 pairs of haproxy instances in each deployment - and there were 10- 15 deployments - so manual was not going to work here. There was a learning curve that we needed to get over and that took some time.

This can't be all you were doing with haproxy..

Of course not.. The configurations in the examples are really basic and simple. The actual haproxy.cfg was a lot more complicated and was generated on the fly using Consul and consul-template. This allows for some very interesting and wonderful things that can be accomplished. The instances were what could be considered as pets, because they were hardly re-provisioned, but the configuration was constantly changing based on the environment.

So did you save money?

No! This was more expensive than provisioning an ELB from AWS. The constraints dictated that this was the chosen solution - not cost. Well in a way this was wasted resources, because there are instances that are sitting idle most of the time - without actually doing anything. The master-slave model is not a cost effective solution because you are spending money to address a scenario when (and if) you lose a node.

Does this scale? How?

We played around with this a bit and also created a prototype that provisioned an auto scaling group with that would work active-active-active with multiple haproxy's - but this required some changes in the way we did our service discovery. This happened a good number of months after we went live - as part of the optimization stage. Ideally - this would have been the way we would have chosen if we could do it over again.

For this example the only way to scale is to scale up the instances sizes - not to scale out.

So to answer the question above - in the published form - no it does not.

Any additional benefits to rolling your own solution?

This could be ported to any and every cloud - or deployment you would like. All you need to do it change the modules and the parts that interact directly with AWS with the cloud of your choice - and it would probably work. It is not a simple rip and replace - but the method would work - just would take a bit of extra time and coding.

What about external facing load balancers - will this work?

Yes, all you will need to do is replace the routes - with an elastic IP, and have the keepalived script switch the EIP from one instance to another. I should really post about that as well.

So why did you not use an EIP in the first place?

Because the this was internal traffic. If I was to use an external facing load balancer, the traffic would essentially go out to the internet and come back in - for two instances that were in the same subnet in the same AZ. This does not make sense neither from a financial nor a security perspective.

Can I contact you if I have any specific questions on the implementation?

Please feel free to do so. You can either leave a comment on any of the posts in the series, ping me on Twitter (@maishsk), or use the contact me on the top.

Replacing the AWS ELB - Automation

This is Part 5 in the Replacing the AWS ELB series.

It goes without saying that anything that I have described in the previous posts can be accomplished - it is just a really tedious work to go through all the stages when you are doing this manually.
Let's have a look at the stages

Create an IAM role with a specific policy that will allow you to execute commands from within the EC2 instances
Create a security group that will allow the traffic to flow between and to your haproxy instances
Deploy 2 EC2 instances - one in each availability zone
Install the haproxy and keepalived on each of the instances
Configure the correct scripts on each of the nodes (one for master and the other for slave) and setup the correct script for transferring ownership on each instance.

If you were to to all of this manually then this could probably take you a good 2-3 hours to set up a highly-available haproxy pair. And how long does it take to setup an AWS ELB? Less than 2 minutes? This of course is not viable - especially since it should be something that is automated and something that is easy to use.
This one will be a long post - so please bare with me - because I would like to explain in detail how this exactly works.
First and foremost - all the code for this post can be found here on GitHub - https://github.com/maishsk/replace-aws-elb (please feel free to contribute/raise issues/questions)

(Ansible was my tool of choice - because that is what I am currently working with - but this can also be done in any tool that you prefer).

The Ansible playbook is relatively simple

Part one has 3 roles.

Create the IAM role
Create the security group
Create the instances

The part two - set's up the correct routing that will send the traffic to the correct instance
The part three - goes into the instances themselves and sets up all the software.

Let's dive into each of these.

Part One

In order to allow the haproxy instances to modify the route they will need access to the AWS API - this is what you should use an IAM role for. The two policy files you will need are here. Essentially for this - the only permissions that the instance will need are:

I chose to create this IAM role as a managed policy and not as a inline policy for some reasons that will be explained in a future blog post - both of these work - so you can choose whatever works for you.

Next was the security group - and the ingress rule I used here - was far too permissive - it opens the SG to all ports within the VPC - the reason that this was done was because the haproxy here was used to proxy a number of applications - on a significant number of ports - so the decision was to open all the ports on the instances. You should evaluate the correct security posture for your applications.

Last but not least - deploying the EC2 instances - pretty straight forward - except for the last part where I preserve a few bits of instance details for future use.

Part Two

Here I get some information about all the rout tables in the VPC you are currently using. This is important because you will need to update the route table entries here for each of the entries. The reason that this is done through a shell script and not an Ansible module - was because the module does not support updates - only create or delete - which would made the process of collecting all the existing entries, storing them and them adding a new one to the list - was far too complicated. This is an Ansible limitation - and a simple way to get around it.

Part Three

So the instances themselves have been provisioned. The whole idea of VRRP presumes that one of the nodes is a master and the other is the slave. The critical question is how did I decide what should be the master and which one would be the slave?

This was done here. When the instances are provisioned - they are provisioned in a random order, but they have a sequence in which they were provisioned - and it is possible to access this sequence - from this fact. I then exposed it in a simpler form here - for easier re-use.

Using this fact - I can now run some logic during the software installation based on the identity of the instance. you can see how this was done here.

The other part of where the identity of the node is used is in the jinja templates. the IP address of the node is injected into the file based on the identity.

And of course the script that the instance uses to update the route table uses facts and variables collected from different places throughout the playbook.

One last thing of course. The instance I used was the Amazon Linux - which means that the AWS cli is pre-installed. If you are using something else - then you will need to install the CLI on your own. The instances of course get their credentials from the IAM role that is attached, but when running an AWS cli command - you also need to provide an AWS region - otherwise - the command will fail. This is done with jinja (again) here.

One last thing - in order for haproxy to expose the logs - a few short commands are necessary.
Here you have a fully provisioned haproxy pair that will serve traffic internally with a single virtual IP.

Here is asciinema recording of the process - takes just of 3 minutes

In the last post - I will go into some of the thoughts and lessons learned during this whole exercise.

2018-09-02

Replacing the AWS ELB - The Design

This is Part 3 in the Replacing the AWS ELB series.

So how do you go about using an IP address in a VPC and allow it to jump between availability zones?

The solution to this problem was mentioned briefly in a slide in a re:invent session - which for the life of me I could not find (when I do I will post the link).

The idea is to create an "overlay" network within the VPC - which allows you to manage IP addresses even though they don't really exist in the VPC.

A simple diagram of such a solution would look something like this:

Each instance would be configured with an additional virtual interface - with an IP address that was not part of the CIDR block of the VPC - that way it would not be a problem to move it from one subnet to another.

If the IP address does not actually exist inside the VPC - how do you get traffic to go to it?

That is actually a simple one to solve - by creating a specific route on each of the subnets - that routes traffic to a specific ENI (yes it is possible).

The process would be something like this:

An instance will try to access the virtual IP - it will go to the Route table on the Subnet and and because of the specific entry - it will be routed to a specific instance.

The last piece of the puzzle is how do you get the route to jump from one instance to the other instance of haproxy, this would be the initial state.

haproxya fails or the AZ goes down

haproxyb recognizes this failure

And then makes a call to the AWS API to move the route to a different ENI located on haproxyb

In the next post - we will go into a bit more detail on how the network is actually built and how the failover works.

2018-08-29

Replacing the AWS ELB - The Network Deep Dive

This is Part 4 in the Replacing the AWS ELB series.

Replacing the AWS ELB - The Problem

Replacing the AWS ELB - The Challenges

Replacing the AWS ELB - The Design

Replacing the AWS ELB - The Network Deep Dive (this post)

Replacing the AWS ELB - Automation

Replacing the AWS ELB - Final Thoughts

Why does this whole thing with the network actually work? Networking in AWS is not that complicated - (sometimes it can be - but it is usually pretty simple) so why do you need to add in an additional IP address into the loop - and one that is not even really part of he VPC?

To answer that question - we need to understand the basic construct of the route table in an AWS VPC. Think of the route table as a road sign - which tells you where you should go .

Maybe not such a clear sign after all
(Source: https://www.flickr.com/photos/nnecapa)

Here is what a standard route table (RT) would look like

The first line says that all traffic that is part of your VPC - stays local - i.e. it is routed in your VPC, and the second line says that all other traffic that does not belong in the VPC - will be sent another device (in this case a NAT Gateway).

You are the master of your RT - which means you can route traffic destined for any address you would like - to any destination you would like. Of course - you cannot have duplicate entries in the RT or you will receive an error.

And you cannot have a smaller subset the traffic routed to a different location - if a larger route already exists.

But otherwise you can really do what you would like.
So defining a additional interface on an instance is something that is straight forward.

For example on a Centos/RHEL instance you create a new file in /etc/sysconfig/network-scripts/
DEVICE="eth0:1"
BOOTPROTO="none"
MTU="1500"
ONBOOT="yes"
TYPE="Ethernet"
NETMASK=255.255.255.0
IPADDR=172.16.1.100
USERCTL=no

This will create a second interface on your instance.

Now of course the only entity in the subnet that knows that the IP exists on the network - except the instance itself.
That is why you can assign the same IP address to more than a single instance.

Transferring the VIP to another instance

In the previous post the last graphic showed in the case of a failure - haproxyb would send an API request that would transfer the route to the new instance.

keepalived has the option to run a script that execute when the it's pair fails - it is called a notify

vrrp_instance haproxy {
[...]
notify /etc/keepalived/master.sh
}

That is a regular bash script - and that bash script - can do whatever you would like, luckily that allows you to manipulate the routes through the AWS cli.

aws ec2 replace-route --route-table-id <ROUTE_TABLE> --destination-cidr-block <CIDR_BLOCK> --instance-id <INSTANCE_ID>

The parameters you would need to know are:

The ID of the route-table entry you need to change
The network of that you want to change
The ID of the instance that it should be

Now of course there are a lot of moving parts that need to come into place for all of this to work - and doing it manually would be a nightmare - especially at scale - that is why automation is crucial.

In the next post - I will explain how you can achieve this with a set of Ansible playbooks.

Replacing the AWS ELB - The Challenges

This is Part 2 in the Replacing the AWS ELB series.

Now that you know the history from the previous post - I would like to dive into the challenges that I faced during the design process and how they were solved.

High Availability

One of the critical requirements was "Must not be a single point of failure" - which means whatever solution that we went with - must have some kind of High availability.

Deploying a highly available haproxy cluster (well it is a master/slave deployment - it cannot really scale) is not the that hard of a task to accomplish.

Here is a simple diagram to explain what is going on.

Two instances, each one has the haproxy software installed - and they each have their own IP address.

A virtual IP is configured for the cluster and and with keepalived we maintain the state between the two instances. Each of them is configured with a priority (to determine which one of them is the master/slave) and there is a heartbeat between them vrrp is used to maintain a virtual a virtual router (or interface between them). If the master goes down - then the slave will take over. When the master comes back up - then the slave will relinquish control back to the master.
This works - flawlessly.

Both haproxy's have the same configuration - so if something falls over - then the second instance can (almost) instantly start serving traffic.

Problem #1 - VRRP

VRRP uses multicast - https://serverfault.com/questions/842357/keepalived-sends-both-unicast-and-multicast-vrrp-advertisements - but that was relatively simple to overcome - you can configure keepalived to use unicast - so that was one problem solved.

Problem #2 - Additional IP address

In order for this solution to work - we need an additional IP address - the VIP. How do you get an additional IP address in AWS - well that is well documented here - https://aws.amazon.com/premiumsupport/knowledge-center/secondary-private-ip-address/. Problem solved.

Problem #3 - High Availability

So we have the option to attach an additional ENI to the cluster - which would allow us to achieve something similar to what we have above - but this introduced a bigger problem.

All of this would only work in a single Availability Zone - which means that the AZ was a single point of failure - and therefore was in violation of requirement #2 - which would not work.

As it states clearly in the AWS documentation a subnet cannot span across multiple AZ's

Which means this will not work..

Let me explain why not.

A network cannot span multiple AZ's. That means if we want the solution deployed in multiple AZ's - then it needs to be deployed across multiple subnets (192.168.1.0/24 and 192.168.2.0/24) each in their on AZ. The idea of taking a additional ENI from one of the subnets and using it as the VIP - will work only in a single AZ - because you cannot move the ENI from one subnet in AZ1 - to another subnet in AZ2.

This means that the solution of having a VIP in one of the subnets would not work.

Another solution would have to explored - because having both haproxy nodes in a single AZ - was more or less the same as having a single node not exactly the same but still subject to a complete outage if the the entire AZ would go down).

Problem #4 - Creating a VIP and allow it to traverse AZ's

One of the biggest problems that I had to tackle was how do I get an IP address to traverse Availability zones?

The way this was done can be found in the next post.

Replacing the AWS ELB - The Problem

This topic has been long overdue.

This will be a series of posts on how you can replace the AWS ELB’s inside your VPC’s with a self managed load balancing solution. This will be too long for a single blog post so I decided it was best to split it up into parts.

Let me start at the beginning.

The product I was working with had a significant number of components that were communicating with each other. A long while back the product had decided to front all communication between components with a load balancer - for a good number of reasons such as:

scaling
high availability
smart routing

Here is a basic view of what the communication between component1 to component2 would like (and just to clarify - there were just over 50 components that were in this solution - not just one or two)

Simple load balancing example - something that you can see anywhere. The load balancers - were HAproxy.

There was some additional logic going on inside the traffic between the components which was based on HTTP headers which allowed us to route some to certain instances and versions of the application (you can think of it as blue/green or canary routing

A simple visualization of this would look something like this.

The team had already deployed this product and now it was time to move over to AWS.

The first part of the discovery was to identify how we could accomplish as much as possible of the solution while using the services that were provided by AWS - instead of deploying our own. one of the items for discussion that came was of course the Load balancers we were using - and if they could be replaced with AWS ELB's.

Here were the list of requirements (and the current solution that was used met all the requirements):

Must be able to serve as a load balancer

Define frontends
Define backends

Must not be a single point of failure
Provisioning will have no manual interaction
Must be able to route traffic based on specific custom HTTP headers

AWS met all the requirements except for #4.

There are options to route traffic based on HTTP headers in the AWS Application Load Balancer but they are limited (to say the least), you can only used a Host header or a path in the URL.

This was not an option, the engineering teams were not going to refactor the applications just to adapt to the AWS ELB. This caused me to go back to the drawing board an see how we could still use the know HAproxy solution inside of AWS - despite the well known problems.

2018-08-19

A Triangle is Not a Circle & Some Things Don’t Fit in the Cloud

We all started off as babies, and I am sure that not many of you remember that one of the first toys you played with (and if you do not remember - then I am sure those of you with kids have probably done the same with your children) was a plastic container with different shapes on the lid and blocks that were made of different shapes.

A triangle would only go into the triangle, a circle in the circle, a block in the block and so on.

This is a basic skill that teaches us that no matter how hard we try, there are some things that just do not work. Things can only work in a certain way (of course coordination, patience and whole lot of other educational things).

It is a skill that we acquire, it takes time, patience, everyone gets there in the end.

And why am I blogging about this – you may ask?

This analogy came up a few days ago in a discussion of a way to provide a highly available database in the cloud.

And it got me thinking….

There are certain things that are not meant to be deployed in a cloud environment because they were never meant to be there in the first place. The application needed an Oracle database and it was supposed to be deployed in a cloud environment.

What is the default way to deploy Oracle in highly available configuration? Oracle RAC. There are a number of basic requirements (simplified) you need for Oracle RAC.

Shared disk between the nodes.
That will not work in a cloud environment.
So we can try using dNFS – as the shared storage for the nodes – that might work..
But then you have to make an NFS mount available to the nodes – in the cloud.
So let’s deploy an NFS node as part of the solution.
But then we have to make that NFS node highly available.
Multicast between the nodes - that also does not work well in the cloud.
So maybe create a networking environment in the cloud that will support multicast?
Deploy a router appliance in the cloud.
Now connect all the instances in the cloud into the router.
But the router poses as a single point of failure.
Make the router highly available.

And if not Oracle RAC – then how about Data Guard – which does not require shared storage?

But it has a steep licensing fee.
And you have to find a way for managing the virtual IP address – that you not necessarily will have control over.
But that can be overcome by deploying a VRRP solution with IP addresses that are manually managed.

ENOUGH!!!

Trying to fit a triangle into a square – yes if you push hard enough (it will break the lid and fit).
If you cry hard enough – Mom/Dad will come over and put it in for you.

Or you come up with half-~~ass~~baked solution like the one below…

Some things will not fit. Trying to make them fit creates even more (and sometimes even bigger) problems.

In this case the solution should have been - change the code to use a NoSQL database that can be deployed easily and reliably in a cloud environment.

As always your thoughts and comments are welcome.

2018-07-05

Getting Hit by a Boat - Defensive Design

In a group discussion last week – I heard a story (I could not find the origin – if you know where it comes from – please let me know) – which I would like to share with you.

John was floating out in the ocean, on his back, with his shades, just enjoying the sun, the quiet, the time to himself, not a care in the world.
When all of a sudden he got bumped on the head (not hard enough to cause any serious damage) with a small rowing boat.
John was pissed…. All sorts of thoughts running through his head.

Who gave the driver their license?

Why are they not more careful?

I could have been killed?

Why are they sailing out here – this is not even a place for boats.

And with all that anger and emotion he pulled himself over the side of the boat, ready to give the owner/driver one hell of a mouthful.
When he pulls himself over the side, he sees an empty boat. No–one there, no-one to scream at.
And at that moment all the anger and rage that was building up inside – slowly went away.
We encounter things every day – many of them we think are directly aimed at us – deliberately or not – but we immediately become all defensive, build up a bias against the other and are ready to go ballistic. Until we understand that there is no-one to direct all this emotion and energy at.
And then we understand that sometimes thing just happen, things beyond our control and we cannot or should not put our fate into some else’s hands.

That was the original story – which I really can relate to.

(Source: Flickr – Steenaire)
But before I heard the last part of the story – my mind took this to a totally different place – which is (of course) architecture related.
John was enjoying a great day in the sun – and all of a sudden he got hit in the head by a boat.
Where did that boat come from?
No-one knows.. I assume the owner had tied it up properly on the dock.

Maybe the rope was cut.
Maybe someone stole it and dumped it when they were done.
Maybe there was a storm that set the boat loose.
Or maybe there was a bloopers company that was following the boat all along to see who would get hit in the head.

There are endless options as to how the boat got there. But they all have something in common. The boat was never supposed to end up hitting John in the head.. John expected to be able to bake nicely in the sun and not be hit in the head by a boat
But what if John had taken additional precautionary measures?

Set up a fence / guardrail around where he was floating
Put someone as a lookout to warn him about floating boats
Have a drone above his head hooked into a heads-up-display in his sunglasses that he can see what is going around him

There are endless possibilities and you can really let your imagination take you to where you want to go as to how John could have prevented this accident.
What does this have to do with Defensive Design?
When we design an application – we think that we are going to be ok – because we expect to be able to do what we want to do without interference.
For example.
My web server is suppose to serve web requests of a certain type. I did not plan for someone crafting a specific request that would crash my server or bombarding the webserver with such an influx of traffic that would bring the application to its knees.
But then something unexpected happens.
When you design your application you will never be able to predict every possibility of attack or some esoteric ways people are going to use your software. There is always something new that comes up – or someone thinks of a different way to use your idea that you did not even think of.
What you can do, is put some basic guardrails into your software that will protect you from what you do know or think can happen.

Throttling the number of connections or request – to prevent DDOS attacks.
Introducing circuit breakers to prevent cascading failures
Open only specific ports / sockets
Sufficient authentication to verify that you should be doing what you are supposed to
Monitoring for weird or suspicious behavior.

Again the options are practically endless. And you will not think of it all. You should address the issues as they happen, iterate, rinse, repeat.
That was a 4 minute read into thing that I think about during the day.
What kind of things do you think about when during your daily work? I would be interested in hearing. Please feel free to leave comments down below.

2018-01-30

5 #AWS Certifications in 237 days

Today I completed my 5th AWS Certification. Something which I had hoped to complete before the end of 2017, but life got in the way.

I started working dabbling with AWS a while ago - signed up for a free account at way back in the end of 2014 and started to play with it, but not too much.

It was not until the beginning of 2017 that I really went in full force. When I declared my goals for Q1 2017 one of them was to work on AWS. I am happy to say that I have accomplished this goal - this is what I do all day, every day.

So during the past year I decided to also pursue the AWS certification track, mostly to prove to myself that I could, but also to learn more about AWS, Their products and solutions

Here is my journey.

Solutions Architect - Associate

This was the first one I did. It took me almost six months since starting to use AWS to feel comfortable with my knowledge to go for the exam.

I used the A Cloud Guru course - which was great for this certification, Just the right amount of content - of course you need to know what you were doing in AWS (at least a little bit). Iwent over the lessons, did the quizzes, took the practice exam on the AWS certification site (and failed). I read the whitepapers (yes – all of the suggested whitepapers). I read the FAQ's for SNS, SQS, S3, EC2. Whatever I did not feel comfortable with – I went over in the actual AWS console – and learned how to use it.

Trying to remember all the options is not realistic – but you will be able to eliminate the really stupid options that are in the test.

Taking a test on a new technology is always scary - this was as well.

80 minutes. 60 questions.

Yep .. Certified #AWS pic.twitter.com/T8tKrmCPWt
— Maish Saidel-Keesing (@maishsk) June 5, 2017

Read the questions properly – even if you are not sure – there are some really obvious answers that blatantly incorrect – so that will help you eliminate the noise from the question.

This is an entry level exam - which expects you to understand that AWS concepts - and some cases - how to use them.

SysOps Administrator - Associate

Two weeks later. I sat the next exam.The amount of overlap with the previous exam is astounding. Again I used the A Cloud Guru course - but here I skimmed over the videos - as they were repetitive from the previous exam. There was a bit more emphasis on the Cloudwatch and logging aspect of the certification - but nothing too deep.

80 Minutes. less

#Achievement :) #Done #AWS #AWSSummit pic.twitter.com/xGKTv0igJN
— Maish Saidel-Keesing (@maishsk) June 21, 2017

Onwards and upwards.

Solutions Architect - Professional

Everything I had heard and read about this exam was the same - it is a beast. Damn, bloody hard. Not only do you need to have to understand the AWS services, how they work and when you should use them - but more importantly - you have to know when they will not be a good fit for the particular scenario.

It took me almost 6 weeks to prepare for this one.

The A Cloud Guru course was a waste of time. Not enough depth – the quizzes and practice questions are superficial – and it is mostly a re-hash of the previous Sysops Associate and Solutions Architect Associate – with maybe two-three lectures added in. I went through the whole thing until I realized that it was not enough.

I took a practice test available from AWS (everyone that passes a previous exam gets a free voucher – I found it a good practice (40 questions – with a time limit) although the questions on the practice exam were MUCH harder than the real thing and I failed the practice exam.

I received recommendations to do the CSA-Pro course from Linuxacademy.com – first 7 days are free – and then $29 / month – I used the subscription for one month to go through the whole course.

The lecturer on the course – speaks SO slowly – that it can be really annoying – but luckily – you can speed up the lectures to x1.5 which makes it a lot better. The course is long – the walkthroughs are excellent! And the labs are also really good and give you some hands on with the features discussed - if you have not used them before. They have a good practice exam – 3 hours 80 questions – you can take it multiple times – but the pool of questions or almost exactly the same.

The most important part of this is the option to practice sitting on your butt for 3 hours concentrating on the exam – it is crucial to prepare yourself mentally for an ordeal – this is one of biggest challenges in the exam.

The blueprint says you have to be an architect with at least 1-2 years experience on AWS (well we all know that we have not more than 6-8 months)

The test is a different level completely from the associate exam. A lot more detail. The questions are scenarios about how you combine multiple products within AWS and create a robust/cost effective/quick solution for each scenario. Not only do you need to know what each product can do – but more importantly – you need to know what each CANNOT do. The answers are similar enough to cause problems and you need to pay attention to the details.

So a little bit about the exam.

170 minutes (yes almost 3 hours)
77 questions

I went through all the questions within about 140 minutes.The two biggest hurdles in the (IMHO) are time and being able to concentrate for almost 3 hours straight.You do not have a lot of time to spend on each question (just above 2 minutes per questions) and some of them are long. So you have to be able to read the question – filter out the nonsense and noise in each question (and there is enough of it) and zoom in on what they are looking to find.

Most questions had at least 1-2 answers that I could disqualify off the bat – which makes it easier to focus on the ones left.

There were 3-4 questions where I had absolutely no idea what the correct answers were. I chose one of the answers and marked it so I could return to it later. If there were questions where I was uncertain between two of the choices – I marked them down as well – and jotted down on the paper provided by the testing center – which were my possibilities. Most questions were to choose a single answer – there were a few choose X answers in the test as well – but not many.

For the last 30 minutes – I went back to the questions I had marked as "no idea" – re-read them – and made an educated guess by eliminating the obvious wrong answers.The rest of the question that I marked as not sure about – re-checked the options I marked down – and confirmed my best choice.
Questions that I was sure I knew the answer to – I did not even go over.

I finished my review with 1 minute to spare… (169 minutes)

Oh Yeah…!! #AchievementUnlocked #AWS #booyah pic.twitter.com/3zDtAembdG
— Maish Saidel-Keesing (@maishsk) August 9, 2017

All in all – a fair but difficult exam – gruesome but fair… Glad to be past it

And then I went on a summer vacation with the family. Time to clear my head, chill and forget about AWS for a while.

Developer - Associate

I was a bit scared about this one - I must say. I am not a developer. Never have aspired to be. I dabble in code and can write a script with the best of them - but a development centric certification - I did not look forward to.

I used the A Cloud Guru course here. There was a lot of overlap with the previous 3 exams, but akso a lot of new stuff that I was not acquainted with - such as Cloudformation, DynamoDB, RDS, Elastic BeanStalk and such.

The exam was not difficult - it is an entry level exam.

One more to go. #achievement #4 #AWS Done! pic.twitter.com/pqiCB1Mskw
— Maish Saidel-Keesing (@maishsk) September 13, 2017

Then came the Jewish Holidays, Re:Invent and the first 3 chapters of my book The Cloud Walkabout.

DevOps Engineer - Professional

I must have re-scheduled this exam at least 6 times, really six times - because I felt I was not ready.

As a professional level certification - I was expecting hell like the Solutions Architect. Expecting that you have to know things in detail - a lot more detail than the Associate exam (and I was right).

The A Cloud Guru course was again to shallow. There are things in the blueprint that I do not use in my daily work, some of them - I have never even touched before, and I found the content to be too shallow, not deep enough for what I was expecting to see in a Pro exam.

The Linux Academy course was much better, again the instructor was a bit too slow for my taste (super speed helps though) and the practice exam was quite good, although 70% of the questions on my exam - i had never seen before.

You need to know how a developmentp pipeline works - I mean really works, blue/green deployments, rolling upgrades, CloudWatch, AutoScaling in and out, CloudFormation, Elastic Beanstalk, OpsWorks are some of the in-depth topics you need to know.

The exam was not completely scenario based - but more about technical details on some of the products. It is 80 questions in 3 hours - so you have to manage your time - but nowhere close to the pressure on the Solutions Architect - Professional exam.

And lo and behold…

Mission accomplished... 5/5 #AWS pic.twitter.com/OoAJZHK2NZ
— Maish Saidel-Keesing (@maishsk) January 29, 2018

My Closing Thoughts

There is a lot of information that is really outdated in the AWS exams, all the new shiny stuff, like Lambda, ECS, Kubernetes etc. is not there - maybe a small reference here and there - but no real knowledge of the new stuff. There is stuff in there - that no-one or hardly anyone uses.
I have learned a huge amount over these last 6 months, both by reading , listening watching videos, and lectures and by doing - mostly by doing than anything else.
These are not paper certs, you cannot pass with only reading or going over braindumps or cheatsheets. You need to actually use the products, understand how they work, and where they fit in the overall picture.
I did not take a single AWS course - I learned everything on my own. I have always been a self learner and prefer to play with the tech myself that spend the money on an official course.
The cost of the certification exams is not cheap, in total $1,050 for the 5 exams (PRO exams are $300 a pop) and luckily I had the costs covered by my employer - which made it easier - but also more pressure to pass - ROI you know.

What’s next - I don’t know.. Specialty exams (Networking, Big Data, Security)? Perhaps - I am not sure.

I do know that there are not many people that have the AWS Professional certifications in Israel, and I am pretty sure that I can count the the number of people in Israel with all 5 on one or two hands.

If you are looking to prove your knowledge and expertise in AWS - then go for it. It is possible - it takes time, commitment and support from your surroundings, but it can be done.

I am proud of my achievement and hope this post will give you the motivation to go out and learn something new.

As always, feel free to leave your thoughts and comments below.

2015-03-10

vCenter is Still a Single Point of Failure - An Update

My previous post vCenter is Still a Single Point of Failure caused a decent amount of discussion on Twitter and comments on the blog. I am actually glad about that because it raised a point that I think has been neglected for quite some time – and is finally being addressed.

Today I learned that the Availability Guide was recently updated – and there were a number of changes (the original version is no longer available – but are we all lucky that I have screenshots). The biggest one was a whole new section was added on the paper about clustering vCenter Services.

Firstly I would like to commend VMware on putting this out. Let’s go in and see what was added.

<sarcasm>
“vCenter can be a single point of failure” – I wonder where I have heard that before?
It is good to see that vCenter is now considered to be a critical component.
I wonder why VMware changed their tune as compared to the previous version? (below)
</sarcasm>

What is vCenter HA?

So what would such an architecture look like? This is described in the following diagram

You will need to setup a WSFC which will require a shared disk between the VM’s (RDM only is supported) and the document goes into some detail about how this should be set up. There are a large number of moving parts including DFS, How to perform upgrades, registry settings, certificates you name it. You will also need to protect the SQL / Oracle Database as well – this is not in the scope of this document.

You can see that a decent amount of work has gone into documenting the process. It is also apparent that this will be the same method used to provide the same availability features for vSphere 6.0 when it is released.

But one small caveat. I am sure you are not the only one who has gotten the feeling that VMware have been pushing us all towards the appliance – for a number of reasons, good and bad.

So where is the clustering solution for the vCenter Appliance? All of this is Windows Server vCenter only!

The only mention of such a solution is a tweet from my good friend Niran Even-Chen

@rnelson0 @maishsk details for the VCSA availability is still being finalized and documented

— Niran Even-Chen (@NiranEC) March 9, 2015

Feature parity – is a relative term.

The second update I would like to go into was a performance paper released last week regarding the failover time needed for vCenter 6.0 in the case of a Host failure. The original article can be found here.

Now we have been down this path before – but I would like to point out two things.

A VM protected by vSphere HA – will be restarted in approximately 30-50 seconds after failure – which is pretty fast and what we would expect from HA.
The amount of time required to restart a 6.0 vCenter is even more than that of a 5.5 vCenter. Quite a bit more.

From the original version of the paper:

5 minutes and 12 seconds.

From the performance document for vCenter 6.0

I would like to take the middle scenario – where we are dealing with 32 hosts and around 4,000 VM’s (I think that is safe to call that a standard environment).

The restart time has gone up to 7 minutes and 19 seconds – That is a 40% increase in downtime. That is a lot if you ask me – and probably due to the changes made in the architecture – hopefully for the better of us all.

Again I would like to thank the authors for there hard work, also VMware for recognizing that this is something that was lacking in the previous whitepaper – and something they were quick to rectify.

Last minute update
It seems there was an article published yesterday (after I had written this post) vCenter Server 6 Deployment Topologies and High Availability with some more information. It seems that even thought the vCenter Server has been consolidated down into two components – they taken even longer to start up.

It also seems that the Appliance will not be supported (at least not yet)

And one more question I had for VMware.

Why is Windows 2012 Datacenter Edition required? According to this document WSFC is included in Standard as well.

As always your thoughts and comments are welcome.

And don’t forget to vote!!!!

2014-12-11

vCenter is Still a Single Point of Failure

A few days ago VMware (Mike Brown, Anil Kapur and Justin King are the authors) announced the updated document for the vCenter Server 5.5 Availability guide.

I would like to make clear a few things from the start.

This is not a VMware bashing post. (Even it might be perceived as such)
I hold all three of the authors in very high regard.

Here goes.

When reading this document I was hoping to hear something new, something refreshing, something that VMware customers have been asking and verbally complaining about for a very long time.

Alas – this is not the case.

vCenter is a single point of failure. There I have said it. I have said it before, I will continue to say it in the future until this if fixed.

In the following article I will be taking statements directly from the text, providing my thoughts as I go along.

Great start – This document will discuss the requirements… After re-reading that statement – I understood what VMware did. VMware has not provided us with a method of providing HA for vCenter – but rather – have explained what they think should be defined as High Availability be for your vCenter server.

The authors then go into explaining all about MTBF and MTTR – they did a great job. I will not go into the details here – you should read the document

SLA’s are extremely important – and for and every environment – an SLA is something different – yours may differ from your neighbor, so it is important to understand what you need to achieve.

They then go into describing the tests that were run in order to measure the amount of time it would take for a vCenter server to recover. Fair enough.

Here is where it starts to get interesting. Let us look at this in a picture.

Bottom line is – that once a vCenter server has gone down – it will take a little over 5 minutes until it is fully functional.

This part of the document states that having vSphere HA – and having vCenter running as a virtual machine actually provides some level of protection.

A dedicated management cluster is of course advised – that way you have a dedicated environment to run your management components without having to worry that the client workloads will interfere.

Also putting the database in the same management cluster is recommended – seems logical.

I then noticed that the only SQL version that is supported for vCenter 5.5 is Enterprise and up – which was news to me. I gather this is a documentation bug – because the VMware Product Interoperability Matrix says that Standard is supported.

So how do you protect vCenter?

It would really be great if they would explain exactly how that would be possible and how that should be done. It still might be possible? How exactly? In order to protect vCenter – I will need another vCenter? Licensing? Implications?

Emergency Restore was a new one to me – ~~but it is only available in vSphere Data Protection Advanced Edition – that is something that was left out – which is approximately $1,500 (list price) / per socket~~. As a result of the feedback received in the comments – I have amended this. It seems that Emergency restore is also available in all editions of VDP – not only Advanced (more information here).

OK, enough copy and paste. This piece above is what set me off.

Essentially what VMware are saying the following:

Use a separate management cluster
Run vCenter in a VM
Run the Database in a VM
No matter what happens – if your vCenter crashes then it will be down for 5 minutes.
Your workloads are safe because they are running on your ESXi hosts are protected by HA and can continue running without a vCenter server.

Points 1-4 - I totally agree. With point #5 I also agree.

But there are environments that cannot afford to have a 5 minute outage. VMware might say that having vCenter go done and out for five minutes, is not really an outage per se, but I would very much like to disagree here.

If I cannot provision a new VM because my vCenter is not available – that is an outage.

Where would this be an issue?

VDI environments – If a user logs in and his desktop is not provisioned because vCenter is down? How about the whole 100 or 1000 employees?
Highly automated environments – ones that use products like vRealize Automation or vRealize Code Stream. Imagine having your code builds fail for 5 minutes because vCenter is not available? The whole continuous delivery process breaks down.

I might be exaggerating a bit – but I have voiced this more than once – I started more than 4 years ago - Troubleshooting Tools for vCenter.

vCenter is probably the most crucial part of your virtual infrastructure. And all that you can expect from from an availability perspective is that you have to accept as a given that vCenter might go down for 5 minutes at a time.

There are environments that will accept this - I would actually say that the large majority are fine with this – but what about those who are not? Those who cannot afford having this kind of outage? What do they do?

There used to be a product called vCenter Server Heartbeat – which was retired.

Where are those promised options? When will they be available? What do companies do in the interim? Pray that there vCenter does not crash?

Embedded below is the Twitter conversation that sparked this post.

The scenario on which VMware based their whole presumption was on the fact that the host on which vCenter was running would crash, HA would kick in and the VM would be restarted on another host within 5 minutes.

The whole scenario of having a problem with your database, or a vCenter service problem (and believe me it happens), that was not covered.

Take the following scenario. You have a vCenter appliance. For some reason the vCenter service stops responding on the VM. There is no automatic restart. Eventually you get a call, something is not right. You try and restart the service, nothing happens. You restart the VM, nothing happens.

Now what? Open a call with VMware? Deploy another vCenter appliance and hope that nothing goes wrong? I can guarantee you that will take a hell of a lot longer than 5 minutes.

Why does the document even go into providing a clustered solution for the MSSQL database? Because that might fail? Yes it could happen. But guess what – the whole system is only as strong as its single weakest link. So providing a clustered database solution might give some piece of mind – but it will not protect you from an outage – because there is no way to cluster a vCenter server.

In conclusion – yes there are considerations. I would definitely not say that VMware have a High Availability solution for vCenter, they have done their best to minimize the impact it will have when it vCenter crashes – but that is not HA!

What do you think? Am I making a mountain out of molehill? Or this a real and valid concern? Please feel free to leave your comments and thoughts below.