There's treasure everywhere: a devops perspective on the Port of Long Beach

Posted Tue 02 November 2021 by dillon niederhut

The CEO of Flexport posted a thread on Twitter last week about supply chain shortages that ended up getting a lot of attention.

The basic gist of the thread is that one of the causes of supply chain shortages in the United States is cargo getting stuck at ports, waiting to be unloaded and shipped further inland. These supply chain shortages have led to disruptions in everything from smart phones to lotion, and is a concern for inflation and for retail earnings for the upcoming holiday season.

The thread goes on to talk specifically about the Port at Long Beach -- one of the busiest in the country -- and how a rule about maximum stack height for containers is causing a major operational slow down.

There is already a great blog post talking about how the CEO convinced the City of Long Beach to address the issue, and the ramifications that has for other things in life. I'd like to take a different perspective on this news story: a perspective from devops and process improvement.

To start, let's think about the value chain of the port. Their added value comes from the ability to quickly and cheaply transfer cargo from boats onto terrestrial vehicles -- maybe a train, maybe a truck. The ultimate goal then is to send the container to another distribution network.

The process that implements this value chain might look something like this (maybe, I've never worked at a port):

  1. truck arrives at port
  2. truck unloads container into yard
  3. boat docks at port
  4. crane removes container from boat
  5. crane puts container on truck
  6. container leaves port

There is no value added until a truck (or train) leaves the port with one or more containers, so this is the step where we measure our success. However, there are a lot of steps earlier in the process that it depends on, and those all need to succeed as well. This brings us to step one, which is:

1. Flow

If no trucks arrive at the port, no containers can leave the port. Likewise, if no trucks can unload the cargo they already have, no containers can leave the port. When a single part of this process fails, the whole process will fail. This is flow.

Our goal is to ensure that the flow that creates value proceeds without disruption. That is to say, we want to maximize containers transited per unit time. To state this another way, we want to minimze the time it takes for a single container to move all the way through this process.

We can accomplish this by looking for stoppages or slow downs at any individual step, and rally any available resources to remove those issues. In the specific case of Long Beach, the problem was step 2: unloading existing containers.

Flexport's CEO identifies this, and proposes different ways to swarm the slowdown, which include removing the height restriction on container stacking, and using federal land to remove empty containers from the port to a location further inland.

Now, according to that same Twitter thread, "everyone" knew that the piles of containers were the issue. Presumably, this means that everyone working at the port knew, but no one with the authority to dedicate more resources to the problem was aware. This brings us to step two, which is:

2. Feedback

The people closest to the work are usually in the best position to identify nascent issues and slowdowns. The Toyota factories famously include a ripcord that, when pulled by any employee, stops the entire assembly line. When this happens, everyone else on the line pitches in to resolve the issue, so that everyone's work can continue.

This, of course, requires that employees are not only empowered to voice concerns, but also to trigger fast and decisive action. No one at the port who knew about the slow down had the authority to change the stacking rule, or to allocate more land for storing containers.

Empowering employees is very important, but it isn't the only way to measure when things are working well (or not!) and step in. There may be issues which are not easily visible to any employee -- maybe a single container that has been stuck at the port for several months. We want to be have automated performance measurements to help us uncover problems which are hard to see. In this specific case, we may want to measure how long our cargo spends in each individual step.

The reasoning looks like this:

  1. what we care about is the rate at which containers leave the port; but,
  2. if something goes awry early in the process, it might take a long time before it gets reflected in the end result that we care about; and,
  3. that intervening time could have been spent correcting the issue; so,
  4. we want to measure our performance at all of those earlier steps.

There are a lot of measurements that we could apply to this specific case. For example, Long Beach could track the total number of containers in staging yards, or the remaining available room for new containers. They might also track the rate at which containers are entering or leaving. They could track on a per-container basis how long it has been stuck in the yard. If they put alerts on these numbers (only 5% of staging capacity remaining!), then they have the information they need to solve a problem in the process before it affects the end result, where their business is providing value.

This gets us to a place where we could have successfully handled the port slow down this time. But there will always be new challenges in the future, which brings us to step three, learning and experimentation, or as I like to call it:

3. Always Be Learning (ABL)

Actually -- we're doing this one right now! We've taken a specific instance of a problem we faced (too many containers at the port), and we're drawing general lessons and process improvements that will help us face the next challenge. This is important because there will always be events that disrupt your value chain -- maybe stormy weather, maybe high gasoline prices, maybe a crane needing urgent repairs.

If your business only works when everything is perfect, your business will always be failing.

One way to get ahead of this is to do scenario planning. What happens if there is a major road closure in Long Beach? There should be a contingency plan. Some companies take scenario planning out of the planning stage, and actually implement major service disruptions to ensure that their contingency plans work as expected (Netflix is famous for this).

Another way to get ahead of this is to experiment with new processes and new technology. What if Long Beach built a short rail that could, with latency but also high throughput, move containers from boats out to a separate staging yard? I'm not an expert in shipping or transportation, so I won't have great ideas here; but, there are thousands of people working at ports across the United States who will.

That kind of on-the-ground expertise is only going to help if there is also top-down support -- a company culture -- that empowers and rewards these employees for flagging issues and making the port successful. In this case, because the blocking issue was a local ordinance, the notion of top-down might need to extend to the Long Beach government as well. Are they in regular contact with port management? Are they open to new ideas about improving the shipping operations within their city?

4. Addendum

The intention here is not to lay blame and point fingers at the management or city government for not doing The Right Thing™️. The goal is to learn (Always Be Learning) about process improvements we can put in place to help our businesses be the best they can be.

In that spirit, if you've been reading this post and thinking "wow, I should be doing this", you aren't alone! We're all learning (ABL) how to be better at the stuff we care about. If you're interested in learning more (ABL!), check out The DevOps Handbook.

I'd like to close with a selective quote / mangled paraphrase from this earlier blog post about the container issue, about just how much low-hanging fruit there is in business process improvement:

There's treasure everywhere! The people need to know.

devops value chain process improvement