Navigating Your AI and Data Science Initiatives to Success with Data Vault
Artificial Intelligence can be complex and difficult for businesses to adopt and implement successfully. Business leaders can be nervous about going into the breach with AI. Business executives may even float plans to implement AI while privately leaving it to the next CTO that comes along. Once we get past the marketing spin and buzzwords, AI can benefit businesses through innovation in various ways, from automation to increased customer engagement. What does Data Vault give to organizations embarking on an AI journey?
Data Vault solves many of the real problems in AI implementation and adoption for businesses. Artificial intelligence needs to be set up for success, and there are many challenges along the journey to AI. Data Vault helps the business along the journey from Business Intelligence through to Artificial Intelligence.
Data Vault is a scalable enterprise Business Intelligence methodology that offers a range of benefits to businesses that are implementing enterprise Business Intelligence solutions. Data Vault offers a number of cornerstone benefits that will support the business in implementing Artificial Intelligence successfully. The Data Vault methodology focuses on the business objectives and the goals. With AI, it is very easy to deviate into a route of discovery that is not meaningful. Data scientists can get very hung up on a detail rather than the bigger picture. Therefore, it is crucial to keep the business goals front-and-center of the AI project not to get blown off course.
Data Vault can help by de-risking certain areas of data projects for the business, as well as, by adding in additional benefits that support the business on this journey. However, it is difficult to trust AI since we are accustomed to having control over decisions and decision-making. Artificial intelligence is easy to misunderstand, and it is key to earn trust in the results through diligent testing. The Data Vault methodology assists in the process by offering repeatability as a matter of course. The idea here is that the business gets repeatable, consistent outputs in response to inputs going through Artificial Intelligence algorithms.
Artificial Intelligence is hungry for data
Artificial Intelligence is hungry for data. Although there is ongoing academic research that aims to reduce the amount of data that AI requires to perform predictive analytics well, it is a long way out. Thus, it is crucial that AI is supported by a framework that offers scalability along with the ability to respond flexibly to requests for the addition of new data. Fortunately, this is a key benefit that Data Vault offers to AI implementations since it can respond to data model changes flexibly while systems are in flight. AI implementation is not always a greenfield site, where we have the luxury of designing from scratch. Often, it has to integrate with other systems and receive new data inputs. AI implementation is not static; it moves with the business, at the speed of the business.
Project management often gives us a sense of navigating control of the project. Businesses like plans that look forward to years to come as well as financial controls. However, a waterfall project approach may serve to increase nervousness about AI implementation because it does not show results quickly or easily.
To show success, visionary business leaders will adopt a ‘gently, gently, incrementally’ approach to Artificial Intelligence. The advantage of the agile approach is that the business can follow success quickly and in a cost-efficient manner while minimizing the impact of any failures. It is a matter of human nature to align to success rather than failure. One of the benefits of Data Vault is that it aligns with agile methodology to support business processes. Fundamentally, the agile approach is a very kind approach to development since it allows for quick recovery from failure as well as sharing team success. Low impact changes are very helpful in building trust in Business Intelligence and Artificial Intelligence.
A rising tide lifts all boats.
Data Vault offers ways to derisk Business Intelligence implementations, which, in turn, supports AI implementations too. Data Vault helps to manage risk through robust data governance and versioning.
Versioning is key for AI implementations, which rely on training and test datasets to produce reliable results. Hackers have realized that one potential route of a cyberattack on AI systems is to destroy the training and test systems. Consequently, it is impossible to work out how the AI systems made their decisions since the training and test datasets have gone. This scenario effectively destroys the AI system since it cannot be trusted, so the business often has no other choice but to start the AI project again from the start.
A key part of understanding data drift is auditing; if this is not in place, the AI engineer will not be able to understand the data and the data journey very well. Fortunately, auditing is a crucial part of the Data Vault methodology, so this challenge is mitigated through its careful implementation. Auditability is a key area of importance here. Data Vault can help follow the breadcrumbs of data throughout the data journey because it helps with the ‘right to an explanation’ of how the AI came to a decision.
Drifting, without aim or purpose, is the first cause of failure. – Napoleon Hill
Artificial Intelligence systems also suffer from data drift, which means that the data changes over time, but the model does not change in response. The best approach to data drift is to retrain the model in response to the changes in the data. One example of data drift includes changes to the descriptive statistics within the dataset, which results in variation that was not there at the point of training. Another example of data drift is the inclusion of new features or characteristics in the data, which account for variance in the result. Versioning can support data drift since it helps the AI developer to understand the differences between the original training data and the more recent datasets. Data Vault can support the AI engineer by assisting in the understanding of data drift, thereby offering a quicker resolution to model training.
Putting the science in data science?
Data literacy assumes that people are mostly data illiterate in some way. It is based on the idea that many people don’t have data skills, and if they learn data skills, they will come critical thinkers who can accept insights based on the data. The route from data illiteracy to data-driven is considered to be a linear, brief journey. Furthermore, it is not possible to assume that people don’t have data skills. The myth of data literacy does not acknowledge that people sometimes simply do not want to use data skills. Intentional or not, the challenge is that people don’t always put science into data science, business intelligence, or AI. Often, they choose the data that fits their particular story at the time, ignoring data that does not fit the story that they want to tell and putting a spotlight on favorable data. Skewing data does not produce fair, unbiased results. However, giving people access to raw data does not always build confidence and trust in the data; it may only suit some people if they choose to have confidence in the data and it suits their ends. This situation is complex enough with Business Intelligence, and Artificial Intelligence exacerbates it since people may not want to delegate access or control to these systems.
“Sail far. Sail fast.” ― George R.R. Martin, Fire & Blood
How can we clear the way to better data fluency and transparency? Data Vault introduces the concept of hard rules and soft rules. Hard business rules are the technical rules that align the data domains, and they are contrasted with soft rules where the business changes the data or changes the meaning of the data. With data, it is important to understand these concepts before the business proceeds to do any Artificial Intelligence processing on the data. Bias in AI needs to be minimized, and the use of hard and soft rules can help eliminate bias in AI since they help clarify any bias or changes introduced through the business. If these rules are not well understood, then this could subtly influence the results of AI. Data Vault draws this key distinction, and it impacts both Business Intelligence and Artificial Intelligence solutions.
Artificial Intelligence usually needs a lot of data in order to increase accuracy. Data Vault can set businesses up for success with this requirement by supporting a scalable architecture that allows for careful modeling of the data that is easier for business users to understand intuitively. After all, there is no guarantee that the business teams will create accurate business intelligence calculations by themselves, and AI algorithms are even further away from most people’s understanding.
Developing AI in organizations will require trustworthy decision-making, and Business Intelligence can help to provide better decisions. Before businesses start on their AI journey, they will need to have confidence in the data. If not, the business will not trust the AI results as well as the Business Intelligence results. Data Vault 2.0 architecture defines the separation of data from information, simplifying the process of understanding and interpreting AI results.
“Automation applied to an inefficient operation will magnify the inefficiency.” – Bill Gates
One of the Key benefits of AI is automation
One of the key benefits of AI is automation. AI can help to smooth out business processes, increasing productivity and reducing costs. In this area, the Data Vault methodology is the perfect solution for an organization that is considering implementing AI to reap the benefits of automation. Data Vault can help the organization ensure that the business teams are automating the right things using good data; while automation is a good thing, remember that it started with Frankenstein, and it’s important to ensure that the right processes are being automated. Data Vault can help ensure that processes are repeatable, consistent, and standardized, which provides a good foundation for successful AI implementation.
AI and automation are only beneficial if the business processes are correct in the first place. If the business processes are not correct, then AI will not deliver as expected. The failure of AI to deliver not due to a technology issue. If the processes are not correct, Artificial Intelligence – or any other technology – is not going to magically solve the problems. Automation does not learn from its own mistakes, so it is not possible to rely automatically on technology or buzzwords to produce results for the business. Automation will only do what you tell it to do, and there is no magic wand to fix broken processes. If the AI technology is implemented to support a broken process, it will only produce broken results.
All aboard!
Data Vault is a key part of the AI journey because it can help develop AI sensibly while producing sustainable results. Data Vault prioritizes the importance of business processes to deliver measurable results for the business. If your business executives are serious about including AI as part of their technological estate, then the business needs to get aboard with Data Vault as a way of minimizing the challenges with AI implementation to navigate towards success.
Authored by:
Other articles by Jen on DataVaultAlliance:
Jennifer Stirrup is the Founder and CEO of Data Relish, a UK-based AI and Business Intelligence leadership boutique consultancy delivering data strategy and business-focused solutions. Jen is a recognized leading authority in AI and Business Intelligence Leadership, a Fortune 100 global speaker, and has been named as one of the Top 50 Global Data Visionaries, one of the Top Data Scientists to follow on Twitter and one of the most influential Top 50 Women in Technology worldwide.
Jen has clients in 24 countries in 5 continents, and she holds postgraduate degrees in AI and Cognitive Science. Jen has authored books in data and artificial intelligence has been featured on CBS Interactive and the BBC as well as other well-known podcasts, such as Digital Disrupted, Run As Radio and her own Make Your Data Work webinar series.
Jen has also given keynotes for colleges, universities, as well as donating her expertise to charities and non-profits as a Non-Executive Director. Jen’s keynotes are about AI Leadership, Diversity and Inclusion in Technology 3. Digital Transformation 4. Business Intelligence. All of Jen’s keynotes are based on her two decades plus years of global experience, dedication and hard work.
