Category: Agile software development

Scaling Agile Delivery – Revisiting the Spotify Model

I was thinking about some of the challenges with scaling Agile software delivery recently and I took another look at the Spotify Agile engineering model which I hadn’t looked at for some time.

The YouTube video further down this article is a few years old now but it is excellent and still relevant to today’s challenges. I outline some of the key points in this article.

Many of the Agile teams that I have worked with have adopted many of these practices and it is worth taking another look and seeing if your teams can improve their performance by adopting some of them too. It is often the case that what wouldn’t work 6 months ago might work now as people move around between teams, learn more and change their minds.

Before we jump into the scaling Agile part let’s have a quick look at Spotify the company.

Spotify overview

Spotify launched in Sweden in 2008. Today, Spotify is the most popular global audio streaming subscription service with 271m users, including 124m subscribers, across 79 markets. They have 50 million tracks, including 700,000 podcast titles, available to discover, manage and share.

Their development teams are distributed globally so they know a thing or two about managing Agile at scale.

This is what they discovered…

Challenges of scaling Agile delivery

Scaling Agile delivery is challenging. It is quite straight forward to make big strides over the first few years, setup Agile delivery well, and get it functioning effectively. But then what? Where do you go? How far can you go with it? What are the benefits of further improvements and more agility?

Starting with Scrum delivery

Spotify started out using Scrum delivery and this worked very well for a number of years. This gave a nice team-based culture but as the company grew with multiple teams, they found that some of the Scrum practices were getting in the way. So, they decided to make some of them optional.

Agile matters more than Scrum

They discovered that Agile principles mattered more than specific Scrum practices. This was more of a change in mindset, so they renamed some of the elements:

  • The “Scrum Master” became an “Agile Coach” to focus on working with teams to improve Agile delivery
  • They became “Servant Leaders” rather “Process Masters” there to serve the team rather than stick dogmatically to a process

Squads and autonomy

Spotify replaced the terms “Scrum team” with “Squad”. The key driver became autonomy, and this meant:

  • Small cross functional teams of less than 8 people that sit close to each other
  • The Squad have end to end responsibility for one or more products
  • They do design, build, testing, deployment and operations
  • Each Squad has a long-term vision

This means they own the whole thing – so if they build it, they own it and if they break it, they fix it.

Autonomy with boundaries

Autonomy means the Squad decides:

  • What to build
  • How to build it
  • How to work together when building

There are some boundaries that the Squad needs to operate within:

  • The Squad’s mission
  • The overall product strategy

Agile workspace

You can tell a lot about how Agile teams are by walking around their workspace – this is so important. The workspace setup increases the collaboration for teams:

  • Squad members sit close together
  • They have easy access to each other
  • There is nearby space for retrospectives, planning and problem solving
  • All walls are whiteboards

How this works in the changing workplace where more diverse teams are often located in different places, offices and even countries also needs careful consideration (the subject of another video).

The importance of autonomy

Autonomy is important because it motivates people. They are motivated to build better products faster. Autonomy means that teams can make decisions locally and minimise hand-offs between teams so that they don’t get bogged down in dependencies and coordination.

Squad alignment

Although each Squad has a mission they need to be aligned with:

  • Product Strategy
  • Company priorities
  • Other Squads

So, this means:

  • They need to be a good citizen in the Spotify ecosystem
  • Spotify’s overall mission is more important than one individual Squad
  • Be autonomous but don’t over optimise
  • Like a jazz band – everyone has their key role
  • Loosely coupled but tightly aligned

The Squads experiment a lot with different ways of working now and for the future.

Alignment with other Squads enables autonomy.

Squads, Tribes, Chapters and Guilds

These are some of the other organization and community elements:

  • A Squad is a product team owning one or more products
  • Squads are grouped into Tribes
  • A Tribe is a lightweight matrix
  • Chapter is a competency eg. Developer, Agile coaching, DevOps, Testing
  • Chapter lead is the line manager of engineers

A Guild is a lightweight community of interest where people share knowledge eg. Leadership, Web Development, Continuous Delivery. Anyone can join or leave a Guild. A Guild has dedicated communication channels and regular meet-ups.

Spotify preference is for community over organisation structure

Role of the company leadership

The company leader’s roles are different too.

Their job is to communicate what problem needs to be solved and why.

The Squad’s job is to collaborate with each other to find the best solution.

Architecture and internal Open Sourcing

Spotify’s aim is to have a systems architecture where components are small and decoupled. They use an internal Open Source model and there is a culture of sharing rather than owning.

Anyone can edit any code even if another Squad technically owns the codebase. So, if a Squad needs some changes to a product or system component and the owning Squad don’t have time to do the change themselves; then the requesting Squad can make the change instead. The owning Squad will simply do a peer review of the code before it can be released.

Small frequent releases

The focus at Spotify is on doing small software releases often. This led, over time, to the reorganization of the architecture so that it is de-coupled to enable independent releases that can be done separately to reduce impact radius of any failures.

Architecture and Squads evolve to enable easier releases

Because the architecture has evolved to enable decoupled releases, then so has the structure of the Squads. So, the Squads evolved to include:

  • Client Squads – eg. iOS, Android
  • Feature Squads – eg. Search
  • Infrastructure Squads – that provide tools to make it easy for other Squads eg. AB testing, CI etc.

Spotify strive for a self-service model and they aim for no handoffs between teams.

Synchronisation of releases

Some coordination of Squad releases is needed. This is achieved with Release trains and Feature toggles.  

Release trains are releases that are done at a specific regular frequency or date for example: Every 2 weeks on a Thursday. The release happens every time whether there is much ready for the release or not.

If features or functions are not ready for the release then they miss the release or if they are nearly ready then they can be released as a feature but toggled off. When the code is fully ready for release the release is done and the code is toggled on.

Built on trust

The Spotify model is based on trust so that the development teams have enough autonomy to make good decisions within their boundaries and work across teams to deliver great products.

This is a very brief summary of the Spotify experience of running Agile delivery at scale so have a look at the video too.

Scaling Agile delivery – the Spotify way

We Are Agile So We Don’t Need Project Managers Any More

In this article I look at how Agile software development works with Project Management. I struggled with this for a long time during my gradual conversion to Agile ways of working so hopefully this article can help you to get a better understanding.

The subject came up in conversation recently when someone said to me “We are Agile, so we don’t need Project Managers anymore”.

So how does this work, is it true and what does it mean?

Well the answer to this very much depends on how mature Agile software delivery is in your organisation.

Agile delivery should need fewer Project Managers

If you are running Agile software delivery, then you should be aiming to reduce the need for Project Managers in your organisation. How far you can go with this will depend on a number of factors including:

  • How mature your Agile delivery organisation is
  • Size and complexity of projects being delivered
  • Types of projects and the product being delivered
  • How many customers, suppliers and stakeholders you need to work with

Agile software maturity

Mature Agile software teams and organisations have these characteristics:

  • Strong Product Management and Product Ownership for defining product strategy, direction and requirements
  • Software teams building to coordinated Product Roadmaps
  • Minimal barriers between teams
  • Technologies, toolsets, processes and practices aligned with Agile delivery
  • Long term stable teams which are able to re-organise and change shape with minimal impact to the overall product delivery
  • High degree of buy-in for Agile delivery across the organisation
  • Long term department level budgeting rather than project by project basis

Agile delivery is focussed on the team and the product

Agile software teams are focussed on building products for users defined by a business expert in the shape of the Product Owner. It’s all about getting a product into the hands of real users as quickly as possible and iterating from there.

The Scrum Master’s role is to facilitate effective delivery of the team. They are focussed on ensuring that the team have everything they need to perform at their best velocity.

They will ensure the Agile ceremonies and processes; stand-up, retrospectives, Kanban board etc. are run effectively. They will work closely with the team and Product Owner to plan ahead and remove obstacles in front of the team. They may well work closely with external customers too within the context of the product delivery itself.

Project Management more focussed on managing stakeholders

The Project Manager has more of a dual role – firstly working with project teams to deliver the products into the project but also a large part of the role is to manage, coordinate and communicate with the various groups involved in the project.

The main activities for a Project Manager would include the following:

  • Managing stakeholders, customers, suppliers and delivery teams
  • Communicating across all of the groups involved in the project
  • Running the project finances and budget
  • Forecasting and planning the project timelines
  • Managing and communicating risks and issues
  • Coordinating all of the different groups involved in the project and managing dependencies

All of this activity can be really time consuming and as the size of the project increases the more effort this takes.

Scrum Master can pick up some Project Management activities

In some smaller projects the Scrum Master and / or Product Manager can pick up some of the above Project Management activities. This works fine for many projects but as soon as the project reaches a certain size then this becomes unmanageable for them.

The key issue in this scenario is that the Scrum Master needs to focus on the software development team and the Agile delivery process. If they become too focussed on Project Management activities, then the software delivery will start to slip.

Project Managers needed when project size and complexity increases

In the projects and programmes that I run we will typically bring in a Project Manager when we have these types of projects to manage:

  • Larger projects where there are several teams and departments working on a project
  • Projects where there are a mix of external suppliers and internal delivery teams
  • Projects where there are multiple stakeholder groups and delivery teams
  • Projects that include a significant change to business processes or the way users will use a product eg. that might require significant training effort and coordination
  • Projects where delivery teams are running both Agile and waterfall project delivery

Essentially as soon as the amount of project management stuff to do becomes significant then someone needs to dedicate their time to running that.

Let’s look at some examples to highlight these points.

Example Scenario 1 – Project with large number of stakeholders to manage

In this common scenario you may have a project to deliver to an external customer. You have three internal Agile software teams delivering their product enhancements to the project. You also have two new suppliers delivering their products that integrate with the products delivered by the internal Agile software teams. You are delivering on behalf of your Sales team to the customer.

Project Manager needed to coordinate different groups

This project will need a Project Manager to work with all of the parties involved. There are numerous groups to coordinate, manage and communicate with including the customer, the sales team, the suppliers, and the software teams.

The Project Manager will need to work with the suppliers to agree the delivery scope, costs, plan and approach and get this agreed within the company and then manage them through the project. Contracts will also likely need to be written and agreed.

Agile teams also deliver into the project

In terms of Agile delivery on this project – the three software teams would be delivering their products into the project and the Project Manager would be working with the team Product Owner and Scrum Master to coordinate priorities, dependencies, timelines and communications to all stakeholders.

Example Scenario 2 – In house software project across multiple teams

In this scenario a company wants to deliver a significant new set of product features on its large web-based business application. There are three Agile software teams working in the same department delivering a set of products for the web-based application. They are also working with two other Agile delivery teams in the same company, but different departments.

Mature Agile software teams with Roadmaps and Product Management

All of the software teams are working to Product Roadmaps, have good Product Management and Agile Product Ownership in place and have permanent standing teams that are budgeted as part of operations rather than on a project basis.

In this scenario the project could be delivered as a Product Management driven initiative without the need for a Project Manager. Because of the structure and maturity of this organisation the Project Management activities in this example are far fewer and could be picked up by the Product Management team. Planning and coordinating activities are done between teams and put into the individual team product roadmaps.

Some Project Management activities disappear

Most of the other Project Management activity has been removed – there are no external suppliers, no contracts to worry about, no external customers and no budget to manage. The barriers between teams should be low and dependencies can easily be managed between the Scrum Masters of the teams based on prioritisation by the Product Management team.

This is a mature Agile software organisation and it takes time and dedication to build this level of capability, but it is worth it as the productivity advantages and business agility it offers are great.

We are Agile but we do sometimes need Project Managers

So, we can see that in really mature Agile delivery organisations we really can reduce the need for Project Managers. However as soon as there are elements of complexity thrown into the mix such as suppliers, customers and larger delivery projects then the coordination, planning and communications activities required means we need a Project Manager to take on all of these activities.

%d bloggers like this: