Operations Lock-in vs. Development Lock-in

Tue, Sep 1, 2015

I’m not a large enterprise developer. My experiences are at the unique environments of Google and Microsoft1. Over my career I’ve helped build platforms and I’ve learned that to build a great developer platform (or any good product at all) requires you to talk to many customers and put yourself in their mindset. This extends from the day to day usability of the product to the decision of a customer to bet a part of his/her business and career on the product in the first place.

Lock-in is obviously one of the critical things that many customers take into account. But lock-in isn’t absolute. Different types of lock-in are defined by the friction and cost incurred in switching.

One interesting way to view lock-in is by looking at who will bear the brunt that switching cost. You can look at this from the point of view of Ops vs. Dev2.

  • With Operations Lock-in the operations team is primarily involved in a migration from one place to another. The operations playbooks, policies, and procedures may have to change but the code will be substantially unchanged. Ideally, nothing needs to be recompiled or rebuilt.
  • With Developer Lock-in someone has to actually go in and crack the code. This could be something minor (swapping in a new library with a standard language interface) to more architectural changes (changing your database).

Different organizations have different skill sets and one type of lock-in might be more serious than another. While someplace like Google or Microsoft may have high quality software engineers to spare, most large enterprises are severly constrained here. Often projects were written by developers that are long gone – they were independent contractors, left the company, or have moved on to other projects. In this case developer lock-in can be much more costly than operations lock-in.

Think about it this way – a user has a large system built on open software packages running on on-prem hardware. Switching to a VM/IaaS offering is a relatively cheap project as it can be an operations focused project with little to no developer involvement. As IaaS offerings evolved to be more and more similar to real hardware (persistent network block stores, familiar networking) this trend really accelerated.

On the other hand, if a user builds a software product around a very proprietary set of APIs (either language APIs or service APIs) switching can be much more expensive. Many PaaS offerings would be in this category. For example, moving a large application away from AWS Lambda would require a rewrite. Google App Engine has similar issues if the native datastore is being used.

This is all critical to keep in mind as we create new ways to build and run systems. Many times betting on open APIs3 and open source systems helps to fight lock-in by making switching an operations project vs. a development project. This is why, despite the eye-rolls, I’m excited about the emergance of the Open Container Initiative and the Cloud Native Computing Foundation.

What other ways do you think about lock-in? Tweet at me (@jbeda) and let’s start a discussion.

  1. I started my career on Internet Explorer and moved on to create graphics APIs (including XAML) for Windows Presentation Foundation (WPF). At Google I started on Google Talk building out XMPP specifications for call signaling and eventually started Google Compute Engine and helped to start Kubernetes. [return]
  2. There are, of course, other types of lock-in. One that is really interesting is data locality lock-in. As data gets bigger and bigger, simply moving that data around can be complicated and costly. This gets even more complicated as that data continues to grow while it is being moved. [return]
  3. With the advent of cloud APIs, we’ve actually taken a bit of a step back in terms of open APIs. While there are some companies (such as Slack) that encourage alternate implementations of their APIs, many companies are silent or hostile to others implementing those APIs. I’d love to see a pattern of companies releasing their APIs with an open license or donating the IP to a foundation. This is made only more critical in light of the Oracle API copyright decision. [return]

Container Native Package System

Wed, Jul 1, 2015

A lot of exciting things happened at Dockercon 2015 last week. For me the most exciting was the announcement of the Open Container Project foundation. Not only is it great to see the community come together under one banner but also as a chance to entertain new ideas in this space.

These are some thoughts on how to improve what we consider a “container image.” I’m looking at both the container format itself and what goes on inside of it. This obviously builds on ideas in other systems and I’ve tried to call those out. These thoughts are still early so I’m hoping to find others of like mind and start a good discussion.

History and Context

The thing I’ve been exploring most recently is the intersection between the container image format and package management. While there has been plenty of attention on base OSs to host the container (CoreOS, RancherOS, Project Atomic, Snappy Ubuntu) and efforts to coordinate a cluster of hosts (Kubernetes, Mesosphere, Docker Swarm) we haven’t paid as much attention as we could to what is going on inside the container.

Docker Images are great. Images are pretty efficient to push and pull and, with new focus on security, it is getting easier and easier to be sure that what you want in the image is actually what you are running.

Dockerfiles are also great. They are a purpose built makefile analog that are super easy to understand and logically build on the layered nature of Docker images. Like most of the Docker project, they are much more approachable than other efforts in this area and solve real customer needs. When constructed appropriately, they allow for an efficient dev flow where many of the time consuming steps can be reused.

One of the best innovations of Docker is actually a bit of an awesome hack. It leverages the package managers for existing Linux distributions. Reusing the package manager means that users can read any number of guides on how to get software installed and easily translate it into a Dockerfile.

Think of it this way: a typical Linux distribution is 2 things. First is a bunch of stuff to get the box booted. Second is a package manager to install and manage software on that box. Docker images typically only need the second one. The first one is along for the ride even if the user never needs it. There are package managers out there that are cleanly factored from the underlying OS (Homebrew, Nix) but they aren’t typically used in Docker images.


This all mostly works okay. There is some cruft in the images that can easily be ignored and is “cheap” as the download and storage cost is amortized because of layer sharing for Docker images.

But we can do better.

Problems with the current state of the world:

  • No package introspection. When the next security issue comes along it is difficult to easily see which images are vulnerable. Furthermore, it is hard to write automated policy to prevent those images from running.
  • No easy sharing of packages. If 2 images install the same package, the bits for that package are downloaded twice. It isn’t uncommon for users to construct complicated “inheritence” chains to help work around this issue1.
  • No surgical package updating. Updating a package requires recreating an image and re-running all downstream actions in the Dockerfile. If users are good about tracking which sources go into which image2, it should be possible to just update the package but that is difficult and error prone.
  • Order dependent image builds. Order matters in a Dockerfile — even when it doesn’t have to. Often times two actions have zero interaction with each other. But Docker has no way of knowing that so must assume that every action depends on all preceding actions.
  • Package manager cruft. Most well built Dockerfiles have something like:

    RUN apt-get update \
      && apt-get install -y --no-install-recommends \
        build-essential \
      && rm -rf /var/lib/apt/lists/*

    This helps to minimize the size of the layer on disk. This is confusing boilerplate that is likely just cargo-culted by many users.

Ideas for Solutions

While I don’t have a fully formed solution to all of these problems, I do have a bunch of ideas.Imagine that we take the idea of a container image and break it down a little.

The first thing we define is a FileSet. A FileSet is a named, versioned and verified set of files. Google has a system internally called the “Midas Package Manager” (MPM) that does this3. Dinah McNutt gave a great talk on MPM at a 2013 USENIX conference. A further tweak would allow a FileSet to import other FileSets into the file namespace of the host. This allows for a FileSet to have multiple “parents” – unlike the current Docker layered image format.

Second, we define a Package as a type of FileSet. It would have a standard directory structure and include metadata on other packages required along with simple instructions to “install” the package4. Ideally, these packages would be built from verified sources and a verified tool chain to enable true provenance for every bit. This would be optional.

Finally, we would redefine a ContainerImage also as a type of FileSet that has metadata necessary to make it runnable. The definition of this metadata is a big part of what the Docker Image format and the ACI format are.

A ContainerImage that is using this container native package system would define a set of read-only imports (using the FileSet import feature described above) of all required packages FileSets. Image construction tools would verify that all package dependencies are satisfied. Furthermore, the install steps would be run to symlink5 all of the packages into the appropriate places6.

User code could either be packed up as a Package or just inserted directly into the ContainerImage.


By creating a container friendly packaging system and expanding the idea of what an container image is, we can solve most of the issues outlined again.

  • The list of FileSets imported into, say, /packages would be the list of all packages versions that are included in that image.
  • Individual FileSets could be cached by hosts and easily and safely shared between disparate images.
  • A package could be updated in a straightforward way. The toolset would have to make sure that all dependencies are satisfied and that the install steps are run as necessary.
  • Image build tools would list the packages necessary and order wouldn’t matter. Because there are multiple “parents” to an image, order cannot matter.
  • The package install cruft (archived version of the package) would be handled on the host side similar to images themselves. The only thing the container would see would be the actual files – and they would be symlinked in.

There are some missing and underdefined parts to this story.

  • How are packages created? I’m thinking that we could do that by running a container with build time packages that produces output file into a specific directory. Files in that directory are then used to create a package. As part of this the inputs into the build container could be included in the package metadata and signed.
  • What does a package distribution look like? I imagine we’d have curated sets of packages that are known to work well together. For instance, xyz.com could create xyz.com/apache that depends on xyz.com/openssl.
  • How do users override packages? Perhaps abc.com/openssl could specify that it can be used in place of xyz.com/openssl. Any guarantees by xyz.com would be void but it would be a way to do custom versions and carry patches.
  • Opportunity: Kernel and capability requirements. Packages could specify their requirements in a way that would be visible to the host. This would provide a more direct requirement chain between the host and the code running in the container.

This solution obviously borrows from both Homebrew and Nix. What I think is new here is the idea of expanding the definition of an container image with FileSets and making this be fundamentally decentralized. We also need to ensure that the easy to approach spirit of Dockerfile isn’t lost. If we do this right we can make images much easier to efficiently create, verify, update and manage.

Ping back to me on twitter (@jbeda) or we can talk over on Hacker News

(Thanks to Kelsey Hightower for reviewing an earlier version of this post.)

  1. The standard golang image is a great example of this. golang:1.4.2buildpack-deps:jessie-scmbuildpack-deps:jessie-curldebian:jessie. Most of this is done to enable efficient sharing of installed packages. [return]
  2. Best practices should be to track every single input into your docker file. That means that if you are pushing sources you should know which git commit, for example, those sources come from. My guess is that this is rarely done. [return]
  3. Actually, we need our system to be decentralized. MPM, like may package management systems (including Homebrew and Nix) has a single central repository/database of all packages. Whatever is used here must be distributed — probably in a namespace rooted with DNS. Something like Docker Notary would play a role in signing and verifying packages. Something like the Nix archive format (NAR) will help make this more stable and predictable. [return]
  4. Package install should consist of simply symlinking files into some common directories (/bin, /lib). This would all be done via a declarative manifest. There are probably going to be cases where an “install” is a little bit more complex and a script is necessary. I’d love to see how far we could get before that becomes absolutely necessary. It is also assumed that the package directories themselves are only ever mounted read only. [return]
  5. There is a choice on when the package install happens. It could happen early as the container is created. Or it could happen late as part of the container start process. I’d prefer late binding as it makes surgical package updating simpler. The directories that store the symlinks could be tmpfs directories to keep this all very speedy. [return]

A New Beginning...

Wed, Jun 10, 2015

Welcome to the new 80%!

I’ve rewritten my ancient blog on top of Hugo using Bootstrap and Google Fonts. The whole thing is hosted on GitHub if you want to check out the source.

For headlines, I’m using Economica. I picked it because it is narrow and I tend to write long headlines. The body text is Lora. I like a nice Serif font for the body text and it has a bit of style while still being very readable. Code is in Roboto Mono and the Logo and Navigation is in Coda.

I’m not in love with the color theme I have now (dark gray background with red and blue highlights) but it’ll do for now. I played with some CSS gradients to show a preview on the main index page. I’m still not sure – it may be a little too cute.

I’ve done the work of importing my old blog posts and making Hugo create compatible URLs. My original system used XML files for each day so I wrote a short Go program to convert from the XML to something Hugo can consume (markdown files with a JSON header). I had to introduce a verbatim HTML shortcode to make Hugo pass through the HTML from the old blog directly.

The other tricky thing was to generate a page per day instead of a page per post for these old posts. Rolling things up on a daily basis was the way things were done back when I started the blog in January 2003. Doing this daily roll up used the taxonomy feature of Hugo where the taxonomy was “archive” and the term was the day. This was easy to generate from the Go conversion program.

I’m hosting this on GCE under Docker and NGINX. I’m going to write up a post on how I’m doing that and automatically syncing it to the git repro on each submit.

Now that I have it all set up, we’ll see if I actually have anything to say…

HOWTO: Installing Highpoint Rocketraid 222x on Ubuntu Dapper (6.06 LTS)

Tue, Sep 18, 2007

I haven't updated my blog in forever and I'm probably going to abandon my homebrew static client generated site for something like Mephisto at some point but I haven't had time to make the transition.

In the meantime, I'd like to save people some pain and document the steps I have go through to upgrade my Highpoint RocketRaid 2220 on Linux.  I installed the driver a while ago and don't remember the exact steps for that, so this is just what I do to upgrade.  I wrote a little shell script:

#! /bin/sh -v

# Update this version every time you upgrade.

# To update the highpoint driver:
# 1) Download latest highpoint driver from: 
# http://www.highpoint-tech.com/BIOS_Driver/rr222x/Linux/
# 2) Patch driver by changing wrong kernel #ifdefs in osm/linux/os_linux.c  
#   KERNEL_VERSION(2,6,15) -> KERNEL_VERSION(2,6,16)

# Disable sata_mv.ko by moving it to a new directory.  This driver
# conflicts with highpoint driver.  I don't know if this is the
# "right" way to do this, but it works.
sudo mkdir -p /lib/modules/${VER}.disabled
sudo mv /lib/modules/${VER}/kernel/drivers/scsi/sata_mv.ko /lib/modules/${VER}.disabled/

# Make sure kernel headers are installed
sudo apt-get install linux-headers-${VER}

# Make new hpt driver:
cd ~jbeda/sources/rr222x-linux-src-1.07/product/rr2220/linux/
make KERNELDIR=/lib/modules/${VER}/build
sudo make install KERNELDIR=/lib/modules/${VER}/build

# make a new ramfs
# mkinitramfs -o /boot/initrd.img-2.6.15-27-686 /lib/modules/2.6.15-27-686
sudo dpkg-reconfigure linux-image-${VER}

Good luck. I hope this helps people out there that are stuck with this thing. I'm still looking for a good cheap solution to host lots of SATA drives on Linux.  Port multipliers are there and aren't as cheap as they should be.  The driver situation is pretty dire and there aren't that many non RAID (fake or not) >4 ports out there.  I haven't tried any SAS cards though -- perhaps the situation there is better.  I'm also running an LSI MegaRAID SATA 300-8XLP with the megaraid driver.  It wasn't as cheap but at least it works with a true open driver. 

"Avalon marks the end of the American Dream"

Thu, Aug 3, 2006

Miguel de Icaza says "Avalon marks the end of the American Dream."  He also compares it to J2EE -- apparently implying that it is overly complex and overarchitected.


While I wouldn't put it that way, I can't disagree.  I left Microsoft almost two years ago and Avalon still hasn't shipped.  A 5+ year ship cycle for a project can't be seen as anything but a sign that something is horribly wrong.  When I was on Avalon we kept talking about building an API for the next 10 years.  Apparently, when Avalon ships there will be 5 years left on that clock.

I take partial responsibility for this.  When we were first starting Avalon, I was all about "Go big or go home" and "We should build something only Microsoft can build."  In retrospect, the project and the company might have been better served by starting with a much smaller team, aiming lower to start and shipping 5 times over those 5 years.  Version 1 might not have been that impressive, but relentless improvement would have built something better factored, simpler, and more in tune with what users actually need.

I named this blog "eightypercent" in honor of the 80% rule.  It just so happens there there are lots of 80% rules to apply.  In this case, a simpler system that only solved 80% of the problem would have been good enough and would have shipped multiple times already.

It looks like the WPF/E project is an effort to strip Avalon down to something much more approachable.  Cross platform, no full CLR, lower memory footprint -- sounds a lot like Flash/Flex.  I know some of the guys working on the project and I have high hopes that it will be something interesting.  The only question, when will it ship?

Seattle's Homeless Alcoholics on NPR

Wed, Jul 19, 2006

Coming in this morning, I heard a segment on NPR covering a unique program that King Country is running to provide rooms for homeless alcoholics in Seattle.  The unique and controversial part of this program is that the residents can continue to drink.

My wife, Rachel, has first hand experience with this problem from her work at the Harborview ER.  Some of these "frequent fliers" are indeed part of the community at the ER. In fact, one of the patients that Rachel had lots of interactions with (ever since she was a med student!) recently died and it really shook her up.  Anecdotally, Rachel has seen this program provide a positive impact.

In any case, this novel program seems to me to be a unique way to approach a very difficult problem.  It reduces the cost to the taxpayers and provides a safe place for these individuals.  Obviously we would all like to see these problems solved, but, failing that, at least the county is trying to manage it.

1 to 30

Fri, Jul 14, 2006

Check this out.  Get to 30.  Your day is now shot :)

(I just finished.)

Custom Weighted Vests

Tue, Jul 4, 2006

Happy fourth!

I just wanted to post a note bragging about my sister, Jill.  The Chicago Tribune just ran an article about her and her daughter Ellie.  Ellie has what they are calling "Sensory Integration Disorder."  She is basically really hyper and needs to jump, spin, rock, swing, etc.  I think I probably had a touch of something similar when I was small.  One of the things that helps Ellie has been a weighted vest.  Jill couldn't find any she and Ellie liked and so made one herself.  It turned out so well, she decided to start making them for others. 

I've helped her get a web site up and running to show off what she has done.  Check it out at www.customweightedvests.com. You can also check out some other custom sewing she does at www.stitchessosweet.com

Link-Backup v 0.6

Mon, Jun 19, 2006

Scott Ludwig and I have released a new version of his python based backup script, Link-Backup.  This new version ignores broken symlinks and has an option to ensure that only one backup at a time is going on.

Also with this release is a cgi script (viewlb.cgi) for exploring the backups.  This makes it easy to keep tabs on what is going on.  Scott wrote this script a while ago and I updated and improved it.

Details here.

Annie loves flowers

Sat, Jun 3, 2006

I haven't been posting anything lately because I've been so busy with work and Annie.  It is funny how life gets in the way.

In any case, she really loves flowers.  We try and go on a walk in the Washington Park Arboretum every day.  She loves to hold and pick little flowers.  Here is a quick photo I snapped of her today:

Speaking of the Arboretum, almost every time I go there, I see the official bird of Seattle, the Great Blue Heron.  Here is a snap of one of those guys from a couple of weeks ago.  He has just caught himself some dinner.