Agile methodology and the word “agile” is thrown around boardrooms and scope documents like confetti at a child’s birthday party. Perhaps there is no adjective (or is it a proper noun?) thrown around more actively among data engineers and their managers than the term agile.
Agile Methodology should not be confused with “scrum,” “sprint,” “iterative delivery,” and a whole slew of other such phrases, “agile” as applied to the endeavor of building a Data Vault is a fundamental, important concept. Agile methodology is a topic we obsess over.
Give credit to those that invented the agile methodology idea in 2001 in the context of software development.
After watching failed project after failed project on their watch, the likes of Kent Beck (yea that Kent Beck that wrote about Extreme Programming), and 16 like-minded companions put forth the Agile Manifesto. It was, and very much still is, profound in its seminal influence, yet grounded in what we would today call plain-and-simple common sense.
The Agile Manifesto
The manifesto, for those that haven’t committed it to memory, is comprised of 12 statements, or principles, that can be paraphrased as:
- Put customers first.
- Adapt to requirements changes.
- Deliver value continuously.
- Collaborate actively with business people.
- Encourage and motivate people.
- Measure continuous improvements.
- Repeat successful practices.
- Maintain sustainability.
- Apply good design.
- Minimize the amount of work not done.
- Create productive teams.
- Apply sprint retrospectives.
Did I mention agile methodology is grounded in common sense? Who can argue with what the manifesto calls out on this list?
“No, I would rather sit back and wait for the requirements of the project to coalesce, than hang out in my cave and take my sweet time to deliver precisely what the requirements say. If you changed your mind, too bad for you. I’m delivering what the requirements say and that’s that. I’ll let you know when I’m done.”
Absurd, in any context of software development. But when applied to data projects, you had better be screaming “agile’’ with all your voice and all your muscle. Any other approach to managing a data platform is riddled with risk and likely doomed. Apply an agile methodology to your data projects today.
Agile as a Data Vault 2.0 Standard - Agile Methodology
As we know, data projects are inherently filled with ambiguity and the need to embrace change. There is no getting around it. We don’t appreciate the data until we discover it. And we don’t know how to get the best business value until we give it a go, iterate, and iterate some more. Our job is never, ever done, as the needs of the business change without compromise.
When I invented the Data Vault, the original area of focus was on modeling. And for good reason. Defining a system wherein Hubs, Satellites, and Links form the backbone of a data platform inherently resilient to change was, at the time, groundbreaking and seminal in its own right. Sources of data change, and the data itself changes. Data vault is set up, by its very nature, to intrinsically accommodate.
But soon thereafter I realized that defining a development methodology, also up to the challenge of dealing with change, was going to be important. I wanted to evolve the Data Vault standard to explicitly include such a methodology. So that gets us to Data Vault 2.0.
In the Data Vault 2.0 standard, making the development and delivery process (1) explicit, (2) prescriptive led me to Scott Ambler, his colleague Mark Lines, and their approach to delivery called, Disciplined Agile, or DAD for Disciplined Agile Delivery.
Now part of the Project Management Institute, DAD is a way-of-working (WoW) that provides for Data Vault implementations based on standard process. And with this approach, we treat each datum as an entity that goes through a life cycle, from inception to retirement.
Essentially, we have applied DAD’s “mindset,” to a rigorous, repeatable, and scalable standard in the form of Data Vault 2.0. It is pragmatic. And empirical evidence tells us that it works. There are debates about what type of character to type when spelling out agile methodology.
Agile vs. agile
There is, amusingly, some debate that we witness between those that take the “capital A” side of the agile argument, and those that prefer the more informal “small a” side. Best I can tell, the debate is a waste of time.
The informal folks say that the manifesto is too hard-wired, and besides, invented for pure software development applications. But look at the 12 principles. In my view, they form a spirit rather than a rule book. So call it what you want, Agile or agile. I’m indifferent. The point is, regardless of capitalization, embrace it. Commit to it.
Which gets me to the last subject I want to address in this article. How do we define “delivery?” By definition, in an agile approach it is iterative and (largely) continuous.
But who decides what delivery is, and what actually gets delivered? In the Data Vault 2.0 Standard, the answer is crystal clear – it is the business leaders who decide. Collaboration is not a nice-to-have, but the backbone of a successful Data Vault project.
Further, consider that business people by and large don’t much care about the back end; the Raw Vault and Business Vault are mostly off the grid from their daily interaction with the data.
It is, of course, the Information Mart layer wherein the business facing data resides in an easy-to-consume format. Yes, the back end tables must evolve for a successful iteration of a Data Vault.
But the point is this: be very careful getting consumed with back end work, banging away at the keyboard, only to find business people who are potentially frustrated with lack of delivery. It’s an argument you won’t win, and really, shouldn’t win.
Delivery, as defined by the business, is about what they experience with the data at the precise level of their interaction.
The Bottom Line of Agile Methodology
We, as Data Vault practitioners, understand how all the pieces of a Data Vault fit together. Raw Vault and Business Vault are essential, integral parts of any successful implementation.
But when it comes to agile delivery, be sure and put the measure of success in the hands of those that rely on you for the benefits of the Data Vault.
Put the customer first. Accept that requirements are going to change. Deliver something valuable continuously. Collaborate. Talk. Keep it real. Reflect, make adjustments, and carry on.
Agile methodology. It’s not just a manifesto. And it’s certainly not something to be lost in a debate about the characters a vs. A. It’s good sense. And the Data Vault 2.0 Standard embraces agile methodology enthusiastically.
Let us know if this blog has helped you understand what agile methodology is and where it came from.
