WWDVC Presentation: Using Snowflake, dbt & dbtvault to Implement Data Vault 2.0



July 15, 2021
 / 
Xponent21
 / 


PRESENTATION OVERVIEW:


Our client is in a highly competitive and fast-paced industry, where acquisitions happen frequently, and significant business changes are routine. The client’s data team was using Snowflake with snapshots and time travel; even so, they had lost trust with the business because they could not provide stable numbers from one month to the next. In addition, they could not provide answers as to why the reported metrics changed from one time period to the next. Therefore, the data team was under pressure to “correct” the data or adjust business rules on the fly, resulting in the accumulation of technical debt.

Impact Makers proposed data vault architecture coupled with investments in data governance maturity to solve the problem. The client data team was initially resisted data vault, believing it was too complex to implement and wouldn’t solve their problem. We agreed to perform a proof of concept of the value of data vault with the most challenging entity in the organization. We modeled, implemented, and demonstrated the solution for the demanding business entity in two weeks via the data vault methodology. As a result, the scope of our engagement expanded to deliver a critical and previously problematic business metric used in board-level reports.

We have delivered a fully functioning raw data vault, business vault, and information mart that is accurate, auditable, and consistent. During the development process, we upskilled the data team’s capabilities to deliver components of the data vault. The biggest detractor on the team has become the most prominent advocate of the Data Vault and a client that is now actively building out the rest of their data vault. Using the data vault methodology combined with the enhanced data governance maturity, they have reduced and are eliminating the technical debt. Upon completing Data Vault certification, they will be ready to expand the use of data vault across all of their critical metrics.

AGENDA


  • Problems with delivering consistent analytics
  • Proposed architectural solution
  • Overcoming resistance
  • Using dbt and dbtvault to deliver the raw data vault solution
  • Implementing business rules and the business vault
  • Taking advantage of Snowflake and ANSI SQL features
  • Upskilling client resources
  • Integrating data governance, data vault and the SDLC

LEARNING OBJECTIVES


  • Overcoming resistance to data vault
  • Using dbt and dbtvault
  • Using Snowflake and ANSI SQL to save time
  • Upskilling of client resources
  • Integrating Data Vault and Data Governance
  • Reference Data Management with Data Vault