Devops: multidisciplinary teams, not individuals

Most developers I know want to be more agile, and these days want to be more “devopsy”. This will be about what are the more attractive bits of Devops to me and how they relate to a problem I’m having on my current project.

My first experience with Devops was to read The Phoenix Project. Well, no, my first experience was to hear a whole load of rumour about what Devops is. I’ve come to the conclusion over time that there are many different definitions. However, the one that really piqued my interest at first was the one in The Phoenix Project.

My work problem

Lately we’ve started to notice that there’s some slowness in our legacy database accesses. We can’t explain it. We can see some contention in transactions but can’t see an obvious reason for that. The schema is old and horribly complicated, so it’s tough to know where to start in refactoring it, and tough to know ahead of time what will help most. I need a DBA.

The traditional solution

Back in the day there would be a team of DBAs, who did magical things with databases. Sometimes they designed your schema for you, other times not. All the same, if you had a problem with the database you’d go and ask them.

One place I worked, they sat nearby and we worked with them daily. If you wanted to change your schema at all or kill a session, they were the doers. They had many scripts for checking magical things we didn’t understand.

This was kind of slow and expensive. A lot of their day would be taken up with mundane things we devs could really have done ourselves if we were trusted.

The full stack solution (or, “the situation I am in”)

In the modern era, you’re more likely to see the schema (assuming you even have one) owned by the dev team. Individual developers can make changes without consulting a schema guru of any kind. This is empowering and fast. For the most part, assuming you have good migration practices and a solid quality process you will do just fine.

Unfortunately, when you scale your app you may well start to find that there are performance problems. With all of the responsibility and none of the expertise, you will ask a DBA for help (assuming you still employ them). However, the process that used to be routine is now out of the ordinary and it takes even more time. Maybe you still get these guys to maintain your stuff but they certainly won’t know your schema like you do. Sometimes their advice is pretty unhelpful because it amounts to “I probably wouldn’t have build it this way”.

So, for lack of help from your friendly DBA or because you laid him off years ago, you try to solve it yourself and hit StackOverflow. Maybe you manage it, maybe not. You will almost certainly not solve the problem in such a way that it never recurs or you’re completely happy with it.

The Devops solution

This brings me to my main gripe with some applications of “Devops”. A widely held view seems to be that Devops advocates developers training themselves up to be not-quite-DBAs, just like the end game of the full stack solution above.

I don’t think that is right. I don’t think Devops means everyone can do all of the jobs. There’s no way that I’m ever going to be as good as a DBA, so we should employ one. However, the reasons for developers owning more of the schema are still good reasons, so our DBA will probably get bored. If we put him on a project he will spend most of his time twiddling his thumbs. A manager somewhere just had the idea (again) to put the DBA on six projects at the same time. However, anyone who has seen this done will know that this ends up being a nightmare of flood and drought for the DBA. Database problems are like buses…

So what should he do? Well, I think what the Phoenix Project version of Devops tells us to do is to create a team of DBAs who don’t do mundane stuff. Most of the time they built software (and maybe write and deliver training materials) to help developers see for themselves where the bottlenecks and issues are. Sometimes they help diagnose problems.

I like this approach. It sounds like an interesting job for the DBA and a wonderful service for the app developer. Most importantly, I think it respects the expertise of people who spent their career becoming expert in databases, over someone like me plus Stackoverflow. Obviously, while I’ve mentioned DBAs in this case, I think this all applies equally to frontend JavaScript devs, network experts, sysadmins and so on…

To be clear, I’m not suggesting we don’t take an interest in some of these specific activities. I’m curious by nature and not at all bad with databases. But there is still room for experts here.

Back to my problem. We don’t employ any DBAs onsite, and we certainly don’t have a rich set of tools and training for diagnosing things on our own, so I’ll be requesting some consultation from a remote DBA and surfing StackOverflow.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s