Why Most Data Projects Fail and How to Avoid It

GOTO Amsterdam 2023

Unfortunately, the majority of data projects fail. Yet, they fail for the same reasons. Most management and data teams don’t know the reasons a project succeeds or fails. It just appears to be random, hard work, or luck. To help understand the reasons teams succeed, we will introduce the who, what, when, where, and how of data projects. By answering these questions, teams will understand what they’re trying to accomplish far better.

Who: Data teams all start with people. This needs to be the right people, with the right skills, and at the right ratios. You will need data scientists, data engineers, and operations all working together.

What: Just saying you want AI isn’t enough. You need to know what business value will be generated. There should be a clear and attainable path to value creation. You have to clearly state what you are going to do to create value.

When: Unattainable timelines aren’t feasible and neither are “when it’s ready” timeframes. Data projects need to deliver value on a sane timeline. This will include delivering in tranches so the team can gain velocity.

Where: Clusters need to be spun up somewhere. Data needs to be stored somewhere. The data needs to come from somewhere. Data teams need to have a clear plan and architecture of where each piece will be done.

How: Data teams need a clear plan that they are executing. This plan needs a singular focus or the work will go in different directions. There need to be clear technical choices and specific technologies chosen.

Main Takeaways: We need all three teams to be successful with data projects. We need to answer who, what, when, where, and how for a successful data project. Management is at the center of planning for data projects.