Solving system Integration configuration problems

I am doing a lot of semi-automated application configuration management at the moment. I'm working on a system that has about 30 to 40 interdependent services, web sites, and applications. With an expanding team, splitting into several teams from one large team, the organisation needs to be able to easily replicate development, test and UAT environments on a team by team basis. We could use virtualisation to replicate a master environment set consisting of database and code servers, giving each team a clone of this, but we would still need to invest a lot of time configuring end points and database connection strings.

Whilst trying to solve this problem I came across a really interesting tool that almost does what I think is required.

Octopus Deploy

Octopus is an automated deployment tool developed by Paul Stovell for .NET that uses Nuget and configuration conventions to push out applications in a very effective way to multiple environments. 

Check out the feature set here.

Octopus has a central server that pushes packages out to tentacles that sit on servers. The tentacles look after the deployment server side meaning permission issues are never a problem.

One of the great things is the way you can store all your application or web configuration variables and easily assign different configuration sets to different environments. This is one of the key features for me.

This is pretty much what I want, except that it is heavily dependent on using Nuget packages. This is not a massive issue and if my proof of concept goes well, I will try and convince the teams that they need to be generating Nuget packages as part of their build process. It does actually link into tools like teamcity very well to do this so it may even be possible to leverage the generation of artefacts and use them as the packages.

Related Tools:

Chef - System integration configuration

Puppet - IT automation software

Engineering Support Teams (DevOps)

In agile environments, product owners that aren't in touch with development or who are not familiar with engineering practices always seem to push out of the backlog those technical stories that are required for a successful project.

Sometimes we need technical stories that allocate time to set up a CI project, or create a mocking framework, or create an environment, or some automated deployment scripts in order to continue developing and releasing a project. When we don’t have these mechanisms in place, we need to hope that we have a team that can pick up the functionality as we fire it out and position it into the deployment pipeline so that it will drop out happily into production.

Without these technical stories and engineering practices in place, the cycle time (time between conception and release of a story) climbs and we either never release, or if we do release, it’s a pressurised situation of manic updates done manually on live servers by technicians that may have no idea about the code they are deploying.

If you have multiple teams or programmes, and if you have the budget, an engineering support team is essential. This is a team that engages primarily in DevOps, but will also be domain experts capable of taking a holistic view of the project and identify any unforeseen issues.

You will often see this team type in large organisations, but in those organisations which may need them the most, i.e. those start ups that need to be capable of continuous delivery to keep ahead of the competition, it is usually never seen due to the difficulty in justifying a couple of heads working on technical stories instead of functionality. It takes a really technically savvy manager to understand and sell their worth to the business.

If you don’t have this team, and you are not afforded the necessary time to do technical user stories in your sprints or iterations, then you need to think outside of the box and form a group that can focus on these software engineering practices.

A software engineering group is one of these and is usually a selection of the most ambitious or creative engineers from your teams that are willing to get together on a regular basis to throw around current engineering problems and work on providing solutions between them. By utilising slack days, or lunch times, the team can fire through some of these completely necessary technical elements of the delivery pipeline and keep the cycle time down to a minimum.