AWS Client VPN

So after leaking (or not really leaking) from some of the sessions from re:Invent it seems that AWS have finally released the Client VPN

AWS Client VPN is a managed client-based VPN service that enables you to securely access your AWS resources and resources in your on-premises network. With Client VPN, you can access your resources from any location using an OpenVPN-based VPN client.
So instead of you having to provision a EC2 instance on your own and configure your own OpenVPN server - you can use this service

But pricing is outrageous...

$0.05 per AWS Client VPN connection hour
$0.10 per AWS Client VPN endpoint association hour

Assuming I would like to bring up a EC2 instance that would handle a 5 VPN connections and I leave the server running 24/7 for a month users connect for approximately 8 hours a day - 5 days a week
LEaving this service provisioned for the entire month would cost

0.10 * 750(hours in a month) = $75
0.05 * 5(people) * 8(hours) * 5 (days) * 4 (weeks) = $40

Total cost for one month - $115

If I were to roll my own on EC2

Using a t3.small instance (2vCPU/2GB ram) should be more than sufficient.

0.02 * 750 (hours in a month) = $15

OK - it is not comparing apples to apples - not by a long shot

Client VPN offers the following features:

Secure — It provides a secure TLS connection from any location using the OpenVPN client.
Managed service — It is an AWS managed service, so it removes the operational burden of deploying and managing a third-party remote access VPN solution.
Highly available and elastic — It automatically scales to the number of users connecting to your AWS resources and on-premises resources.
Authentication — It supports client authentication using Active Directory and certificate-based authentication.
Granular control — It enables you to implement custom security controls by defining network-based access rules. These rules can be configured at the granularity of Active Directory groups. You can also implement access control using security groups.
Ease of use — It enables you to access your AWS resources and on-premises resources using a single VPN tunnel.
Manageability — It enables you to view connection logs, which provide details on client connection attempts. You can also manage active client connections, with the ability to terminate active client connections.
Deep integration — It integrates with existing AWS services, including AWS Directory Service and Amazon VPC.
Are all these extra features worth paying so much more for this managed service?
You are the only one that can answer this.

I am throwing the gauntlet out there - for someone to write the code that will enable the provisioning of a VPN Endpoint on demand - based on usage - which will make this service more cost effective.


#AWS Outposts - told you so..

I called it - to me it was obvious that this was going to happen. The signs were all there. This was the direction that the market has been pushing for, and AWS has a reputation of giving the customers what they ask for.

The last announcement that was Andy Jassey made on the keynote on Wednesday - was AWS Outposts.

Here was the announcement. Usually Jeff Barr (or as of late - someone else on the Technical Evangelist team) have a detailed blog post - on a new product that was just announced.

For AWS Outposts - nada… The only thing that is out there - is the announcement - and a “TBD” product page - https://aws.amazon.com/outposts/


Once the announcement was made - VMware went all out with as much information as they could describing the VMware variant of AWS outposts https://cloud.vmware.com/community/2018/11/28/vmware-cloud-aws-outposts-cloud-managed-sddc-data-center/

Blog posts, interviews, sessions you name it they went all in - for a very good reason - if you ask me. This expands their VMware Cloud on AWS in a substantial way.

And who was missing from this announcement ? AWS.

To me this is puzzling. The one sided coverage of something that is supposed to be a joint venture, means that either - this was a pure publicity announcement - and the product has not yet been finalized - or AWS dropped the ball on this one - big time!!

So what do we know about a this product? It will come in two flavors:

  • VMware Cloud on AWS Outposts allows you to use the same VMware control plane and APIs you use to run your infrastructure

  • A native variant of AWS Outposts allows you to use the same exact APIs and control plane you use to run in the AWS cloud, but on-premises. 

The AWS native variant of AWS Outposts allows you to use the same exact APIs and control plane you use in the AWS cloud, but on-premises. You will be able to run Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Elastic Block Store (Amazon EBS) on Outposts. At launch or in the months after, we plan to add services like RDS, ECS, EKS, SageMaker, EMR.

Not a word has been published since the announcement, of how this is going to work from the perspective of the  “AWS variant” Outposts.

I even went as far and asked Jeff Barr - what is the story here. The funny thing is - I actually met him at Starbucks about 15 minutes after I posted the tweet.

His answer (if my memory serves me correctly) was..

“The team had not yet had the opportunity to go into detail into the new offering, and would be publishing more details about it"

To me Outposts - is the biggest announcement of the whole of re:Invent - if played correctly - it will remove any and all competition that is hoping to provide a Hybrid cloud story - one that enterprises can understand.

You want AWS - you can have it - in the cloud - and also on prem - the same exact experience - this is something that customers have been asking for years for AWS to provide (and also something that AWS have consistently been completely against - because everything and anything should run in AWS - there is no need for on-prem… - until now :) )

And mark my words, once you have an Outpost in almost every single datacenter - the need for Edge locations in each and every country - will be no more...

I guess we will have to wait for the aftermath to die down - and wait to see exactly how this going to work….

And now some of my personal thoughts about this whole topic.

There are a lot of moving parts that AWS will now have to go into - especially regarding the logistics of providing the end service to the customer.

If you remember there was once another product - that provided you with a similar service - yep I am talking about the vBlock - a joint venture from VMware, Cisco and EMC. Which went the way of the dodo. The partnership fell apart for a number of reasons.

Customers loved the solution!! You had a single number to call - for anything and everything related to the deployment. Disk died? Called the support number. Network not working? Call the support number.  vSphere doing some crazy shit? Call the same support number. One neck to throttle, and customers loved it.

And now you have Amazon selling you hardware - or should I rather say leasing you the hardware. You will not own it - you will pay as you go.  I assume that there will be a commitment - of some kind - and you will not be able to order by the hour - the logistics on per hour would be too complicated.

But speaking of logistics - if there is a company that commit to having a 4 hour delivery time on a failed piece of hardware - it is Amazon - with their global presence. They have  the logistical capability to ensure delivery of practically anything in their inventory to anywhere in the world - in the shortest amount of time.

But there are still many unknowns... here are  a few that come to mind:

  • Will this come with a networking component? I assume it will - what will that network component be? Software? Hardware?
  • By providing you (the customer) with the same experience and AWS hardware - are they risking exposure of how AWS works getting out? I assume that this will be covered in TOS and NDA that you sign as part of the upcoming service.
  • I assume there will be redundant network connectivity requirements in order for this to work - I will also go out on a limb and say that a Direct Connect link will be a requirement as well. This means that it will be only be suitable for a certain piece of AWS's customers. Perhaps redundant VPN's might be suitable as well.
  • What happens if/when the AWS endpoints are not available?  How if at all can the instances and the workloads on the Outpost be managed?
  • How self-service  will the offering be? I assume it will only be a node-by-node expansion - or per 1/4 rack. you will not be able to add more disks on your own, more RAM on your own etc. This makes sense.

In short - since this was announced at re:Invent 10 days ago - and that AWS have already stated this will not be available before H2 2019 - I do not expect that we will see anything before October/November 2019 (but that is just my hunch).

At the moment - there is a lot more to this announcement than meets the eye....


My overall impression of re:Invent 2018 #reInvent #aws

I am now on a plane on my way back home, on a really long flight from SFO to TLV (13.5 hours) so now is a good time to re-cap and reflect on what happened last week at re:Invent.
I think that this will be a set of posts - because there are a number of topics that I would like to address - and some of them deserve their own dedicated insight.

The first and foremost post I would like to go into - is the overall impression about of the conference.


AWS made a significant number of changes as compared to last years event. And overall - I found the event to be amazing!!

If you ask me - last year’s event was not user friendly, for a number of reasons.

  • Tracks were located in a single venue. That meant going between topics was not really possible.
  • Transport - the shuttles had a route - along a number of of the venues. The shuttles took a great deal of time.
  • Lines in the sessions were bad - they were really bad - people were lining up for hours before, without any real indication if they were going to get into a session.
  • The mobile app - was pretty much useless - and was not at all helpful.
  • The amount of repeats were not enough, and overflow sessions were also scarce.

This year AWS fixed all of the above.

  • Tracks were not restricted to a single venue, you could get ML, serverless, Storage and networking - were not only in one venue - but in multiple venues, that meant you did not need to bounce around between the venues.
  • The shuttles were point to point. No more round trips. This was brililant to save time - but on the other hand - there were a number of times where there were 3-5 people on a shuttle at times, not really an efficient way to spend money - it was kind not elastic in any way - and not well utilized from a cost perspective.
  • The mobile app - was much better, still slow as hell - but there was more functionality. Such as when will the sessions be repeated, what sessions have open seats right now, how much time it will take to get from one venue to the other - in real-time.
  • There were many more overflows… The amount of repeats were by large - more than we had last year - which meant you had an option to choose..

The lines this year for sessions - were better - much, much better!! No more lines of 500 people wrapping round the whole of the Venetian to get into a session. No more disgruntled attendees - who were not able to get into a session after having waited for an hour in line.

Lines for buses were much shorter - no more “routes” - but point to point - which was very well managed and funneled throughout the event.

You were not allowed to line up for a session more than an hour in advance. Now this solved most of the long line problems, but was not always enforced (take the DeepRacer sessions for example)

For me the overall impression was amazing. I think that I was only turned away from a single session throughout the whole event - and that was a builder session - which I was not registered to.  I managed to get into any session I wanted, not only frontal sessions, but also workshops as well.

AWS pride themselves on being fanatical about their customers, they listen to what their customers want, they listen to their feedback and they want to make thing better, they want to solve our problems. The feedback that I heard from attendees from last year was that it was a in plain words - a train wreck - because of all the reasons above.

If you ask me - they addressed all of the feedback points, and fixed almost all of them.

And for that I take my hat off to the event team - and say Bravo, that is a job well done.

Next posts will go into some more details about the announcements and some of the sessions.


How I Get the Most Out of #AWS re:Invent 2018

I am not an expert, and I only went to re:Invent for the first time last year, but I have been to a quite a number of conferences over the years.

So here come my thoughts about making the most of the crazy week in Vegas.


The (regular) sessions

Contrary to what you might think, going to sessions where you have a speaker (or speakers) up on stage going through a slide deck, or a panel of speakers talking about a subject - is where you should be, is not a good use of your time.

There are currently 2358 sessions and activities listed on the portal (a good portion of them are repeats - but hell that is a lot of content)sessions

Almost all of the sessions (I will get back this in a few minutes) are recorded and therefore can be consumed after the event - in the car, on the bus or train - or even in the air during your travels.

Here is a podcast feed (http://aws-reinvent-audio.s3-website.us-east-2.amazonaws.com/2017/2017.html) for all 2017 sessions for your listening pleasure.

That is why you can spend your time better elsewhere.

The Builder / Chalk Talk / Workshop sessions

Here is where I would spend my time. The cost of re:Invent (if you paid the full price) is $1,800 for 4.5 days (Friday is a short day). These are the sessions that will not be recorded and where I will probably get the most benefit
(and here are some of my interests). The value I receive is from doing things that I learn from, not by being a passive listener, but by actively participating in a discussion or an activity.

Chalk talks

This is similar to getting a design session and time with an AWS expert in their field and diving deep into a specific subject. Most of the sessions are level 300/400 - which meant they are advanced and highly technical. The rooms are small - usually no more than 50-100 people and the participants there are usually people that are looking for a very specific answers about the journey they have embarked on - or are about to.

ARC210-R - SaaS Jumpstart: A Primer for Launching Your SaaS Journey
ARC213-R - Architecting for the Cloud
ARC216 - SaaS Operations: The Foundation of SaaS Agility
ARC301 - Cost Optimization Tooling
ARC306 - Breaking up the Monolith
ARC310-R - From One to Many: Diving Deeper into Evolving VPC Design
ARC317-R - Reliability of the Cloud: How AWS Achieves High Availability
ARC325 - SaaS Analytics and Metrics: Capturing and Surfacing the Data That's Fundamental to Your Success
ARC326-R1 - Migrating Single-Tenant Applications to Multi-Tenant SaaS
ARC408 - Under the Hood of Amazon Route 53

Builder Sessions

Looking for some personal time with an SA on a specific topic, and even better - you get to build the solution at hand with the guidance from the expert on-hand. Pure learning experience.

ANT402-R - Securing Your Amazon Elasticsearch Service Domain
ARC415-R - Building Multi-Region Persistence with MySQL


Again - a hands-on learning experience - 2-3 hours of sitting down on a specific topic getting my hands dirty...

ARC404 - Designing for Operability: Getting the Last Nines in Five-Nines Availability
ARC315-R1 - Hands-On: Building a Multi-Region Active-Active Solution
ARC327-R1 - Hands-on SaaS: Constructing a Multi-Tenant Solution on AWS
ARC403 - Resiliency Testing: Verify That Your System Is as Reliable as You Think
CMP403-R1 - Running Amazon EKS Workloads on Amazon EC2 Spot Instances
DEV303-R2 - Instrumenting Kubernetes for Observability Using AWS X-Ray and Amazon CloudWatch
DEV306-R - Monitoring for Operational Outcomes and Application Insights: Best Practices
GPSWS402 - Continuous Compliance for Modern Application Pipelines
GPSWS407 - Automated Solution for Deploying AWS Landing Zone
NET410 - Workshop: Deep Dive on Container Networking at Scale on Amazon EKS, Amazon ECS, & Amazon EC2
SEC331-R1 - Find All the Threats: AWS Threat Detection and Remediation
SEC337-R - Build a Vulnerability Management Program Using AWS for AWS


Want to geek out and build something, play a game or solve a whodunnit quest? This is where I will get my game on.  Some are for fun, some are for fame, and others just for plain doing some good.

Giving back

Being at re:Invent is something that is fun, and usually something that can involve consumption of many things. Food, alcohol, entertainment and even your hard earned cash. Me being me - I prioritize giving back to others  as part of my daily life. Spending a week at a conference only receiving is not something I am comfortable with.
So as a result I will be spending some of my time  here https://reinvent.awsevents.com/play/giving-back
The BackPack for Kids program provides bags of nutritious, single-serving, ready-to-eat food items each Friday to children who might otherwise go without during weekends and long breaks from school. Come by the Venetian Sands Foyer to get involved and help put together a backpack or two! Learn more about Three Square here.


Event though the keynotes can be consumed from a live stream - there is something about sitting in a room (or a huge hall) with a boatload of people - where Andy Jassy goes up on stage and bombards you with all the new features that are coming (some that will only be available sometime in the future). But still it is quite mesmerizing and if you have not been in one of these keynotes - I would suggest you go. It is quite an experience.

The Certification Lounge

As Corey Quinn just wrote a few days ago
it's a $100 lounge pass with a very odd entrance questionnaire
If you have an AWS certification - go to the lounge - it is a place to get away from the other 49,000 others in the hallways and the constant buzz around you.

The Expo

Do not under any circumstances miss going to the Expo floor. To really make proper use of the floor - I would say you will need a good 6-8 hours of your schedule (don't do it one shot though). Go to the vendors, especially the smaller ones that don't have the huge booths. Look at your competition, speak to people, make yourself known. Yes you will be bombarded after the show with sales calls - but all it takes is a simple "Sorry not interested anymore" and most vendors will leave you be.

Social media

I don't think I could get by without following what is going on in Twitter.
I have a search column dedicated for re:Invent (already for the past month)


I will also be checking the og-aws Slack channel to co-ordinate snark about the announcements and on-goings at the event and also some face to face meetings with some of the people that I only have met through their avatars.

(And as always the great set of posts at the Guide of Guides is invaluable.)

See you all in 2 weeks!


Events as a Service (EaaS)

Most vendors that perceive themselves as a market leader will have a major annual event (some will even have multiple events in different geographical locations).

Here are few of these major events that come to mind:

And every year we come around to the registration and scheduling of sessions to these events, and they almost always suck... 

(I am going to use re:invent as the victim here - but I am sure that the experience is probably the same with most conferences) 

There are more than enough things that one could find wrong with the way things go at a conference - and I am not diminishing the problems one little bit.

I would like us all to view it in a different perspective.

The companies that hold these events - are tech companies. They are great at selling technology, great at creating some amazing technology. An of course they also have people that are in charge of events and marketing - but it is not their core business. 

I do not underestimate the impact a good event can have on your product - or how a bad event can damage a company's brand - that is why companies like these spend many millions of dollars on events like this. But again that is not what they are trying to sell,  they are not trying to sell an event. They are not event planners, this is something we seem to forget from time to time especially when things are not optimal (another polite way of saying that they suck).

They outsource the events to an external company.

The signs, the transport, the advertising, the venue, website, the on-site services, scanners, the food - and yes - even the mobile app. All of these do not belong to any one of these companies they are all provided as part of the service that another company sells to these market leaders.

It does not make sense for any of the large vendors to bring up an event all by themselves. For an event that is sometimes no more than 5 days in a year - they will not maintain all the dedicated resources (physical, human and virtual) for just one event. 

So it make sense to outsource it all. And they do.

There are a few vendors out there that are capable of bringing up events on this scale - such as Cvent or Lanyon and if you ask me - they do a pretty good job.

There are always things that can be improved. The app could be better (this year there are significant improvements in the re:Invent app experience 😃 ) The registration could be better, the directing of human traffic at the conference could better, the list could go on and on.

Is IS the job of the tech vendor marketing teams to demand from these event companies to improve from one event to another and get better from year to year. To make sure the food is better, improve registration, make sure that the (also human) traffic flows. 

If I look at this from a technology perspective - it is a classic case of consuming something aaS (As a Service). AWS provides us with infrastructure, and they maintain software. but they do not employ all the people that put the chips on the motherboards of every server in their datacenters. They do have people that provide input into the design of the servers - in order for them to operate more efficiently, and in turn provide a better service to their customers (you and me). 

I would not expect them to have chip designers or assembly plants on the payroll to allow them to run their business. They outsource / contract that work from a 3rd party. 

They contract / outsource their event management. All the big companies do - it makes perfect financial sense. 

Does that mean we should stop bitching about the food, the lines, the app? Hell no! By providing constructive criticism (or complaining) we make things better, because that is what we the customer demand. And these event management companies - will hopefully improve.

Some food (pun intended) for thought - when you are your next conference. 


The #AWS Visio Stencils

It seems like only yesterday, but it was actually almost 10 years ago when I gave something awesome to the VMware community - the first version of the VMware stencils.

The reason I did this was because at the time - there was no decent set of VMware stencils out there - so I took the initiative and created a set. And I subsequently set out to update them over the years.
I have a small confession. The VMware Visio stencils have been the biggest driver of traffic to my blog over the years. Even till this day - I have a minimum of 5000 monthly views (and this on a post that is more than 5 years old).

And I already hear you say - there is already a number of architectural Visio icon sets available - and even an official one from AWS - you can find them here. (I assume that the graphics are going to be updated just before / after re:Invent - with the new design that they have released a few weeks ago, in the meantime - only the current one is available)

There are other tools that have online graphics as well. LucidChart, Cacoo, Creately, draw.io, Cloudcraft (the only vendor who has original graphics - the rest are all the standard AWS icons). 
If you are following some of the AWS community work - will probably have heard of Jerry Hargrove (better know as @awsgeek). He actually works at Lucidchart and is famous for his unbelievable sketch notes on AWS and their products. Not only are they beautiful, clear and sometimes even really funny, they are also very informative and extremely useful.

Jerry was also recently awarded the honor of AWS community Hero.
I mean really - these are a real work of art!!


So without further ado I present to you version 1.0 of the AWS Community Visio Stencils.
Jerry was kind enough to allow me to use his graphics and provide the AWS community with a set of graphics - that (in my opinion) are not only more appealing to the eye - but are just plain fun!!

40 icons of AWS services.
  • API Gateway
  • AppStream
  • Athena
  • Cloudfront
  • CloudTrail
  • CloudWatch
  • Code Build
  • Code Pipeline
  • Comprehend
  • Directory Service
  • EBS
  • EC2
  • EFS
  • Elastic Beanstalk
  • ElasticCache
  • ELB
  • GuardDuty
  • IAM
  • Kinesis
  • KMS
  • Lambda
  • Lambda Edge
  • Machine Learning
  • Neptune
  • RDS
  • Redshift
  • Rekognition
  • Route53
  • S3
  • SES
  • SNS
  • SQS
  • Step Functions
  • Storage Gateway
  • VMware on AWS
  • VPC
  • VPC Endpoint
  • WAF
  • WorkSpaces
  • xRay
All of the graphics are from Jerry's artwork.
Each of the Icons is resizable
If you so please, the blue background can be removed.
You can modify the text on each icon.
Each icon has 9 possible anchor points.
All yours to use for free, to modify, and diagram to your hearts content.

And yes - this is only the beginning...  There are over another 200 graphics and icons that I will be taking out of these sketches and converting them into usable icons for your diagramming pleasure.

v1.0 is available for download here

I would love to hear your feedback!

Update Dec. 13, 2018

Today I have released version 1.1 of the Stencils. Here is what changed.

  1. New AWS Product icon - DynamoDB
  2. I decided to add a few additional stencils

    a. People


    b. Icons

    c. Shapes and Banners


Enjoy!! There is still more to come.

v1.0 is available for download here
v1.1 is available for download here:


Keeping Kosher at re:Invent 2018

So we are a little more than a month away from the yearly ascent to all things AWS - re:Invent 2018.

Last year one of my most useful posts was the Kosher perspective on the event Keeping Kosher at re:Invent 2017.

So this year - nothing much has changed - there is still no kosher food.. #boo

No Kosher Food #boo

(This is not last years graphic but taken from the current site)

So again - no Kosher food throughout the day.

Last year, I went on Sunday to one of the Kosher supermarkets, and did some shopping. Every morning I made myself lunch for every day. Better than standing in lines or going out looking for food.

Hot and cold drinks throughout the day are available at the various venues, sometimes there are fresh fruit - and some snacks here and there that have a Kosher certification (OU, OK, Star-K, and many more - depending on what are comfortable with eating)

The Supermarkets were great and they had a wonderful selection of Kosher food

There is The Jewish Visitor's Guide to Las Vegas guide (downloadable here) which has accurate information as of June 2018.

Here is a list of the Kosher restaurants as of today (please check the sites for up to date information before you go)
  • Ace of Steaks (5825 W Sahara Ave Unit M. - +1 702-899-4223). Open till 23.00 Sun.-Thurs.
    It is about a 17 minute drive from the Venetian
  • Anise Tapas and Grill (3100 S Durango Dr. - +1 702-586-4088). Open till 22.00 Sun-.Thurs.
    It is about a 20 minute drive from the Venetian

    Last year a group of us sat down for a late dinner - the restaurant was empty - but the food was good

  • King Solomon’s Table (4561 W Flamingo Rd. - +1 725-244-4034). Open till 22.00 Sun.-Thurs.
    It is about a 10 minute drive from the Venetian
  • Haifa Restaurant (900 E Karen Ave # H102 - +1 702-940-8000). Open till 21.00 Sun.-Thurs.
    It is about a 11 minute drive from the Venetian

    Place is in the middle of nowhere - was practically empty - and the food was nothing special

  • Jerusalem Grill & Bar (4825 W Flamingo Rd. Suite 10 - +1 702-341-5555).
    Open till 22.30 Sun.-Thurs. It is about a 11 minute drive from the Venetian.

    I had dinner there (twice) - and the food was really good!

  • Sababa Grille & Restaurant (3220 South Durango Dr. - +1 702-547-5556).
    It is about a 20 minute drive from the Venetian
  • Shawarma Vegas (2521 S Fort Apache Rd. - +1 702-703-7700). Open till 21.00 Sun.-Thurs.
    It is about a 25 minute drive from the Venetian

    Shawarma Place - Fast food - was great for a quick meal

  • Simon & Joe’s (3720 W Tropicana Ave. - +1 702-759-0333). Open till 21.30 Sun.-Thurs.
    It is about a 10 minute drive from the Venetian.

If you are looking for a list of Kosher products - the list from the Ahavas Torah Center has a substantial amount of information.


The conference ends on Friday at around 12:00 which means for most of us that are visiting from outside of the States - that you either leave early - to get back home on time, or fly to family / friends somewhere else in America, or you stay in Vegas for Shabbat.

The Strip is of course not a Shabbat-friendly atmosphere - and there are a number of Jewish Orthodox (I am sure there are other denominations as well - I will only list the ones that I would go to) communities in the area.

If you so wish - many of them have some option of Shabbat hospitality as well

(I have personally spent a Shabbat in at the Young Israel community a good number of years ago) There is a hotel that is quite Shabbat friendly - literally 200 meters from the shul La Quinta Inn - not the best of hotels - but OK for Shabbat, and the community was very nice to invite me for meals.

Last but not least - there is also an Eiruv - http://www.lasvegaskollel.org/las-vegas-west-side-eruv

As we did last year - we have a WhatsApp group with those who are interested in meeting up for meals after a long day, or perhaps organizing a Minyan for Mincha - or just even to say hello.



Currently there are about 20 people (mostly Israelis - open to all!)

Looking forward to see some old faces and new ones as well next month!


How Long Until you Get the New Shiny Toys from re:Invent?

re:Invent is coming - and the frenzy of releases that will build up to the event is just around the corner.

I have always had in the back of my mind that all the products announced at re:Invent are great for the press releases and the small digs at other vendors, but sometimes it takes a while until we actually get what was announced on stage in front of ~20,000 people and the rest of the world.

And I went out to look for some data. It is obvious that not everything that we heard about on stage was baked and ready for production use.

Andy Jassy - re:invent 2017 keynote

Here are some examples from last years re:Invent

re:Invent 2017

EKS (188 days)

https://aws.amazon.com/blogs/aws/amazon-eks-now-generally-available/ (June 5, 2018)

Bare Metal (170 days)

https://aws.amazon.com/about-aws/whats-new/2018/05/announcing-general-availability-of-amazon-ec2-bare-metal-instances/ (May 17, 2018)

Serverless App repo (83 days)

https://aws.amazon.com/blogs/aws/now-available-aws-serverless-application-repository/ (Feb 21, 2018)

Neptune (183 days)

https://aws.amazon.com/blogs/aws/amazon-neptune-generally-available/ (May 30, 2018)

Aurora Multi-master (Still not released)

Yet to be released (Oct 14, 2018)

Aurora Serverless (254 days)

https://aws.amazon.com/blogs/aws/aurora-serverless-ga/ (Aug 9, 2018)

IOT 1-click (169 days)

https://aws.amazon.com/about-aws/whats-new/2018/05/aws-iot-1-click-generally-available/ (May 16, 2018)

Translate (127 days)

https://aws.amazon.com/blogs/aws/amazon-translate-now-generally-available/ (Apr 4, 2018)

Transcribe (127 days)

https://aws.amazon.com/blogs/aws/amazon-transcribe-now-generally-available/ (Apr 4, 2018)

Appsync (137 days)

https://aws.amazon.com/about-aws/whats-new/2018/04/aws-appsync-now-ga/ (Apr 13, 2018)

S3 Select (126 days)

https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-s3-select-is-now-generally-available/ (Apr 3, 2018)

re:Invent 2016

Lex (141 days)

https://aws.amazon.com/blogs/aws/amazon-lex-build-conversational-voice-text-interfaces/ https://aws.amazon.com/blogs/aws/amazon-lex-now-generally-available/ (Apr 19, 2017)

PostgreSQL for Aurora (329 days)

https://aws.amazon.com/blogs/aws/now-available-amazon-aurora-with-postgresql-compatibility/ (Oct 24, 2017)

GreenGrass (190 days)

https://aws.amazon.com/blogs/aws/aws-greengrass-run-aws-lambda-functions-on-connected-devices/ (Jun 07, 2017)

X-Ray (140 days)

https://aws.amazon.com/blogs/aws/aws-x-ray-update-general-availability-including-lambda-integration/ (Apr 19, 2017)

Batch (36 days)

https://aws.amazon.com/about-aws/whats-new/2017/01/aws-batch-now-generally-available/ (Jan 5, 2017)

Lambda Edge (229 days)

https://aws.amazon.com/about-aws/whats-new/2017/07/lambda-at-edge-now-generally-available/ (Jul 17, 2017)

At a glance it looks like the average amount of time from the list above was about 5 months.

Now don’t get me wrong. For all of the above items that were not actually available at re:Invent - I would estimate that there were the same number of products (if not more) that were available (at least in a limited number of regions) the same day they were announced. Above and beyond - the problems that AWS is trying solve and really complex - and a almost all of them have never been done before - so please AWS take your time in developing the game changing technology that you have been giving to the world.

So when Andy Jassy and Werner Vogels get up on stage at the end of November, and announce whatever wonderful stuff they are going to announce - we should all take into account that it could take anything from 1 day to almost a year until we can actually use it in all the AWS regions that we are consuming today.

Werner Vogels - re:invent 2017 keynote

How can this / does this affect you? I can give an example from the EKS announcement. We were actively looking at a kubernetes deployment on AWS and were contemplating whether we should deploy our own or wait for the managed solution that was announced at re:Invent.

Since we did not have an official release date - we decided to roll our own - and not wait for some some unknown time in the future.

It is nice to know what is coming. You will need to evaluate how long you can wait - are you ready to go with a version one product (that could / will probably have a good number of limitations) or come up with a contingency plan to solve your issues.


#AWS PrivateLink vs. NAT Gateway from a Pricing Perspective

A customer came to me with a request. They do not want to use a NAT gateway from their VPC to access the AWS API's. They had a number of security concerns regarding the use of a NAT gateway (no control, logs, auditing - but that is a for a different post) and they asked for a solution.

The AWS API's that they needed access to were:Endpoints

  • S3
  • KMS
  • SSM
  • Cloudwatch
  • Cloudformation

Last year at re:Invent AWS announced the option to create VPC Interface endpoints using PrivateLink and have steadily been adding more endpoints over the past year.

With the use of these endpoints you can actually have a VPC with instances that will not have any internet access (at least not through AWS) and still be able to interact with all the AWS API's.

This is technically possible - and can easily be automated, but I wanted to look at the cost perspective.

The VPC in us-east-1 has 2 Availability Zones (you should always have at minimum 2).

That would mean deploying 2 NAT gateways in your VPC (Pricing)

I am going to assume that you have the same amount of data going through both options - so I will not factor this into the price.

Usually you have 730 hours in a month.

Each NAT gateway will cost you 0.045*730 = ~$33.

Total for 2 NAT Gateways would be $66 per month (not including traffic).

What does this look like for Interface Endpoints? (Pricing)

Each Endpoint will need to be deployed in both AZ's in pairs.

Each Interace Endpoint will cost 0.01*730*2 = ~15

Total for all the endpoints above (4 Interface Endpoints - KMS, SSM, CloudWatch and Cloudformation) would be $60 per month.
The S3 endpoint is a Gateway endpoint - and therefore does not cost you any extra.

As you can see - it is not that much cheaper.

Take into account the following scenario - you need API access to 15 out of the 21 possible interface Endpoints

This would run you the steep amount of $225 per month - which is a lot more than just a NAT Gateway.

Design decisions always have tradeoffs - sometimes you prefer security and other times it will be cost. I hope that this will enable you to make an informed decision in your VPC design.


Bastardizing #DevOps

I have come across two separate discussions this past week where it became clear that some people have no idea what DevOps is.

The first one was an Israeli company here in Israel - https://devopsexperts.co.il/. Here is the proposed syllabus:


They are offering this course - for a fee (of course), selling the hope that if someone would graduate the course - then they would be able to get a position as an DevOps engineer.

Someone asked on a channel - "Was this course worthwhile?".

I would like to share with you my answer.

I do not want to take away anyone's livelihood but there is no such a thing a "teaching/learning" DevOps. There is no single course that can encompass all the capabilities that one would need to become a successful DevOps professional. Above and beyond that - in each and every organization - the term DevOps will mean something completely different.

There are a number of basic topics that one can learn - and with them build up a strong foundation of skills in order help your specific company. If I would evaluate a potential candidate - and their education was based mainly on this course - I would not hire such a candidate.

The demand for talented professionals is high , everyone wants DevOps engineers - and there are not many people that have enough experience or the know how. Of course with a demand - people identify an opportunity to make money.

Looking at syllabus - it has so many flaws. The course was 45 hours (which means about 1 work week)

  • Scripting - what language are they going to teach you? Python? But who says that the company you might work for - could be using something completely different.
  • Version Control - so this is basically git.
  • Linux fundamentals - basic Linux course
  • Provisioning Resources - with what? Terraform? Ansible? something else?
  • Build Automation - Building a pipeline - with which tools?
  • Continuous Monitoring - is that even a concept?
  • Working with containers - docker run, docker build, docker pull/push
  • Configuration Management - use which technology - I can name at least 3 CM tools that you might use

As you can see, this is a 50,000 ft. view of what you might do in your day to day work as a DevOps engineer - but in no way or form can you learn any of these things in a course - and definitely not in 45 hours.

For me a good candidate would be someone that has the ability to learn, understands the big picture of how software is built, deployed and managed on a regular basis. There is no list of technologies that could be checked off a list that would qualify a candidate. Does someone know Jenkins? That might be great - but if we use something else - CircleCI, Electric Commander? What will the specific Jenkins knowledge help?

DevOps is not something that you can learn in school, or in a course. It is a collection of technologies that you collect during the years, it is a state of mind that you become accustomed to as you grow, it is a set of organizational practices that you pick up on your journey.

Not something you can learn in school.

Next one was Microsoft - who decided to rebrand VSTS into Azure DevOps. Again a shiny buzzword which Microsoft assumes will attract people to the product and their offering.

“Azure now has a new set of five DevOps services,” Jamie Cool, Microsoft’s newly retitled director of product management for Azure DevOps, told The New Stack. “They’re going to help developers be able to ship faster, [with] higher quality. Oftentimes when I have conversations, ‘DevOps’ can mean different things to different folks. So to us in this context, we really think of DevOps as the people, the process, and the products for delivering constant value to customers.”

Here in the statement above is the problem (the emphasis is mine). Products do not deliver DevOps, at least not what Azure is offering. I do agree with the part about the people and the process - but not the products. Maybe the tools - but not products.

If they would have branded the product Azure CI/CD  then I would have been all for it - but to me it seems that this is marketing play - trying to catch a goal that today everyone is trying to achieve.


Replacing the AWS ELB - Final Thoughts

This is the last part in the Replacing the AWS ELB series.
  1. Replacing the AWS ELB - The Problem
  2. Replacing the AWS ELB - The Challenges 
    1. Replacing the AWS ELB - The Design
    2. Replacing the AWS ELB - The Network Deep Dive
    3. Replacing the AWS ELB - Automation
    4. Replacing the AWS ELB - Final Thoughts (this post)

    If you haven't already read the previous posts in the series - please take the time to go through them.

    So here are some additional thoughts and ideas about the whole journey.

    First and foremost - none of this would have been possible without group effort of the team that worked on this.
    Udi, Mark, and Mike - thank you all for your input, help and hard work that went into this.

    Was it all worth it?

    Yes, yes and hell yes!! The cost of having to refactor applications to work with the way that the AWS ELB works - was not financially viable and would take far to long . There was no way we could make our delivery dates and have all the applications modify the way they worked.

    So not only was it worth it - it was a necessity, without this - the project was a non-starter.

    What was the hardest part of the solution?

    Definitely the automation. We had the solution white-boarded out after a an hour or two, brought up a PoC within another hour or two.

    As I said somewhere else in the post - if this was a one-off then it would not have been worth while - but we needed about 10 pairs of haproxy instances in each deployment - and there were 10- 15 deployments - so manual was not going to work here. There was a learning curve that we needed to get over and that took some time.

    This can't be all you were doing with haproxy..

    Of course not.. The configurations in the examples are really basic and simple. The actual haproxy.cfg was a lot more complicated and was generated on the fly using Consul and consul-template. This allows for some very interesting and wonderful things that can be accomplished. The instances were what could be considered as pets, because they were hardly re-provisioned, but the configuration was constantly changing based on the environment.

    So did you save money?

    No! This was more expensive than provisioning an ELB from AWS. The constraints dictated that this was the chosen solution - not cost. Well in a way this was wasted resources, because there are instances that are sitting idle most of the time - without actually doing anything. The master-slave model is not a cost effective solution because you are spending money to address a scenario when (and if)  you lose a node.

    Does this scale? How?

    We played around with this a bit and also created a prototype that provisioned an auto scaling group with that would work active-active-active with multiple haproxy's - but this required some changes in the way we did our service discovery. This happened a good number of months after we went live - as part of the optimization stage.  Ideally - this would have been the way we would have chosen if we could do it over again.

    For this example the only way to scale is to scale up the instances sizes - not to scale out.

    So to answer the question above - in the published form - no it does not.

    Any additional benefits to rolling your own solution?

    This could be ported to any and every cloud - or deployment you would like. All you need to do it change the modules and the parts that interact directly with AWS with the cloud of your choice - and it would probably work. It is not a simple rip and replace - but the method would work - just would take a bit of extra time and coding.

    What about external facing load balancers - will this work?

    Yes, all you will need to do is replace the routes - with an elastic IP, and have the keepalived script switch the EIP from one instance to another. I should really post about that as well.

    So why did you not use an EIP in the first place?

    Because the this was internal traffic. If I was to use an external facing load balancer, the traffic would essentially go out to the internet and come back in - for two instances that were in the same subnet in the same AZ. This does not make sense neither from a financial nor a security perspective. 

    Can I contact you if I have any specific questions on the implementation?

    Please feel free to do so. You can either leave a comment on any of the posts in the series, ping me on Twitter (@maishsk), or use the contact me on the top.

    Replacing the AWS ELB - Automation

    This is Part 5 in the Replacing the AWS ELB series.
    1. Replacing the AWS ELB - The Problem
    2. Replacing the AWS ELB - The Challenges
      1. Replacing the AWS ELB - The Design
      2. Replacing the AWS ELB - The Network Deep Dive
      3. Replacing the AWS ELB - Automation (this post)
      4. Replacing the AWS ELB - Final Thoughts
      It goes without saying that anything that I have described in the previous posts can be accomplished - it is just a really tedious work to go through all the stages when you are doing this manually.
      Let's have a look at the stages
      1. Create an IAM role with a specific policy that will allow you to execute commands from within the EC2 instances
      2. Create a security group that will allow the traffic to flow between and to your haproxy instances
      3. Deploy 2 EC2 instances - one in each availability zone
      4. Install the haproxy and keepalived on each of the instances
      5. Configure the correct scripts on each of the nodes (one for master and the other for slave) and setup the correct script for transferring ownership on each instance.

      If you were to to all of this manually then this could probably take you a good 2-3 hours to set up a highly-available haproxy pair. And how long does it take to setup an AWS ELB? Less than 2 minutes? This of course is not viable - especially since it should be something that is automated and something that is easy to use.
      This one will be a long post - so please bare with me - because I would like to explain in detail how this exactly works.
      First and foremost - all the code for this post can be found here on GitHub - https://github.com/maishsk/replace-aws-elb (please feel free to contribute/raise issues/questions)

      (Ansible was my tool of choice - because that is what I am currently working with - but this can also be done in any tool that you prefer).

      The Ansible playbook is relatively simple

      Part one has 3 roles.

      1. Create the IAM role
      2. Create the security group
      3. Create the instances

      The part two - set's up the correct routing that will send the traffic to the correct instance
      The part three -  goes into the instances themselves and sets up all the software.

      Let's dive into each of these.

      Part One

      In order to allow the haproxy instances to modify the route they will need access to the AWS API - this is what you should use an IAM role for. The two policy files you will need are here. Essentially for this - the only permissions that the instance will need are:

      I chose to create this IAM role as a managed policy and not as a inline policy for some reasons that will be explained in a future blog post - both of these work - so you can choose whatever works for you.

      Next was the security group - and the ingress rule I used here - was far too permissive - it opens the SG to all ports within the VPC - the reason that this was done was because the haproxy here was used to proxy a number of applications - on a significant number of ports - so the decision was to open all the ports on the instances. You should evaluate the correct security posture for your applications.

      Last but not least - deploying the EC2 instances - pretty straight forward - except for the last part where I preserve a few bits of instance details for future use.

      Part Two

      Here I get some information about all the rout tables in the VPC you are currently using. This is important because you will need to update the route table entries here for each of the entries. The reason that this is done through a shell script and not an Ansible module - was because the module does not support updates - only create or delete - which would made the process of collecting all the existing entries, storing them and them adding a new one to the list - was far too complicated. This is an Ansible limitation - and a simple way to get around it.

      Part Three

      So the instances themselves have been provisioned. The whole idea of VRRP presumes that one of the nodes is a master and the other is  the slave. The critical question is how did I decide what should be the master and which one would be the slave?

      This was done here. When the instances are provisioned - they are provisioned in a random order, but they have a sequence in which they were provisioned - and it is possible to access this sequence - from this fact. I then exposed it in a simpler form here - for easier re-use.


      Using this fact - I can now run some logic during the software installation based on the identity of the instance. you can see how this was done here.


      The other part of where the identity of the node is used is in the jinja templates. the IP address of the node is injected into the file based on the identity.

      And of course the script that the instance uses to update the route table uses facts and variables collected from different places throughout the playbook.


      One last thing of course. The instance I used was the Amazon Linux - which means that the AWS cli is pre-installed. If you are using something else - then you will need to install the CLI on your own.  The instances of course get their credentials from the IAM role that is attached, but when running an AWS cli command - you also need to provide an AWS region - otherwise - the command will fail. This is done with jinja (again) here.

      One last thing - in order for haproxy to expose the logs - a few short commands are necessary.
      Here you have a fully provisioned haproxy pair that will serve traffic internally with a single virtual IP.

      Here is asciinema recording of the process - takes just of 3 minutes

      In the last post - I will go into some of the thoughts and lessons learned during this whole exercise.


      Replacing the AWS ELB - The Design

      This is Part 3 in the Replacing the AWS ELB series.
      1. Replacing the AWS ELB - The Problem
      2. Replacing the AWS ELB - The Challenges
        1. Replacing the AWS ELB - The Design (this post)
        2. Replacing the AWS ELB - The Network Deep Dive
        3. Replacing the AWS ELB - Automation
        4. Replacing the AWS ELB - Final Thoughts

        So how do you go about using an IP address in a VPC and allow it to jump between availability zones?

        The solution to this problem was mentioned briefly in a slide in a re:invent session - which for the life of me I could not find (when I do I will post the link).

        The idea is to create an "overlay" network within the VPC - which allows you to manage IP addresses even though they don't really exist in the VPC.

        A simple diagram of such a solution would look something like this:


        Each instance would be configured with an additional virtual interface - with an IP address that was not part of the CIDR block of the VPC - that way it would not be a problem to move it from one subnet to another.

        If the IP address does not actually exist inside the VPC - how do you get traffic to go to it?

        That is actually a simple one to solve - by creating a specific route on each of the subnets - that routes traffic to a specific ENI (yes it is possible).


        The process would be something like this:


        An instance will try to access the virtual IP - it will go to the Route table on the Subnet and and because of the specific entry - it will be routed to a specific instance.

        The last piece of the puzzle is how do you get the route to jump from one instance to the other instance of haproxy, this would be the initial state.


        haproxya fails or the AZ goes down

        haproxyb recognizes this failure

        And then makes a call to the AWS API to move the route to a different ENI located on haproxyb


        In the next post - we will go into a bit more detail on how the network is actually built and how the failover works.


        Replacing the AWS ELB - The Network Deep Dive

        This is Part 4 in the Replacing the AWS ELB series.

      3. Replacing the AWS ELB - The Problem
      4. Replacing the AWS ELB - The Challenges
      5. Replacing the AWS ELB - The Design
      6. Replacing the AWS ELB - The Network Deep Dive  (this post)
      7. Replacing the AWS ELB - Automation
      8. Replacing the AWS ELB - Final Thoughts

      9. Why does this whole thing with the network actually work? Networking in AWS is not that complicated - (sometimes it can be - but it is usually pretty simple) so why do you need to add in an additional IP address into the loop - and one that is not even really part of he VPC?

        To answer that question - we need to understand the basic construct of the route table in an AWS VPC. Think of the route table as a road sign - which tells you where you should go .

        Maybe not such a clear sign after all
        (Source: https://www.flickr.com/photos/nnecapa)

        Here is what a standard route table (RT) would look like


        The first line says that all traffic that is part of your VPC - stays local - i.e. it is routed in your VPC, and the second line says that all other traffic that does not belong in the VPC - will be sent another device (in this case a NAT Gateway).

        You are the master of your RT - which means you can route traffic destined for any address you would like - to any destination you would like. Of course - you cannot have duplicate entries in the RT or you will receive an error.


        And you cannot have a smaller subset the traffic routed to a different location - if a larger route already exists.


        But otherwise you can really do what you would like.
        So defining a additional interface on an instance is something that is straight forward.

        For example on a Centos/RHEL instance you create a new file in /etc/sysconfig/network-scripts/

        This will create a second interface on your instance.


        Now of course the only entity in the subnet that knows that the IP exists on the network - except the instance itself.
        That is why you can assign the same IP address to more than a single instance.


        Transferring the VIP to another instance

        In the previous post the last graphic showed in the case of a failure - haproxyb would send an API request that would transfer the route to the new instance.

        keepalived has the option to run a script that execute when the it's pair fails - it is called a notify

        vrrp_instance haproxy {
          notify /etc/keepalived/master.sh

        That is a regular bash script - and that bash script - can do whatever you would like, luckily that allows you to manipulate the routes through the AWS cli.

        aws ec2 replace-route --route-table-id <ROUTE_TABLE> --destination-cidr-block <CIDR_BLOCK> --instance-id <INSTANCE_ID>

        The parameters you would need to know are:

        • The ID of the route-table entry you need to change
        • The network of that you want to change
        • The ID of the instance that it should be

        Now of course there are a lot of moving parts that need to come into place for all of this to work - and doing it manually would be a nightmare - especially at scale - that is why automation is crucial.

        In the next post - I will explain how you can achieve this with a set of Ansible playbooks.