Notifications
Clear all

Building an Agile Data Lake with Talend/Snowflake & Data Vault 2.0  


Posts: 5
Topic starter
(@daleanderson)
Active Member
Joined: 2 years ago

Hello from Sunny California!  - at least the smoke is gone, for now...

Many of you know me, some don't.  So, quickly: I found Data Vault in 2009 and knew right away it's significance.  An early adopted (to some extent).  Interestingly, I found Talend at the same time.  I love both.  Now we have Snowflake - Wow what a TRIO!

I've been building data warehouses since 2001, previously I built enterprise database applications (starting with COBOL, then VB, then C# - I worked on MS SQL Server 1.0 'pre-release' while at Ashton Tate - anyone remember dBASE IV?); the transition from OLTP to OLAP was not so bad back then but we lacked good tools.  More importantly we lacked reliable solutions. And honestly, I never became a STAR Schema (or Informatica) fan. 

My time at Talend taught me many things (it was great), and yet I really felt there was something more, so when I left Talend mid-summer 2019, I was sure that 'more' would reveal itself.

2020 has been, can I say, different?  COVID-19 has put us all on our knees and most companies finally realize WHY data is so important!  They get it, now.  Trouble is, they struggle on WHAT to do about it, and most simply don't know HOW!  Mostly I am seeing failed attempts, wasted budgets, and limited outcomes (ok, I know there are some successes out there too, mostly thanks to you all).  They are not doing it RIGHT.  They are 'stuck' in legacy mentality!

BUT --- 'Sheltering in Place' has provided me a unique opportunity to DO something about the dichotomy of Why-What-How:

COMMING SOON - iGNOSIS!  a Solution Accelerator for building modern data warehouses and/or agile data lakes with automated, configurable, extensible, Metadata Driven Frameworks and integrated Business Process Rules (a managed catalog of hard & soft, plug-n-play code modules). 

So, I ask you: Would you be interested in standing up a data ingestion process in 3 months instead of 12?  Of course you would.

iGNOSIS is not a tool accelerator offering tips & tricks on how to use a particular technology, but a real, comprehensive 'blue-print' on building data processing workflows.  A Reference & Solution Architecture plus many reusable code modules are included, job design patterns, and best practices, 'out-of-the-box'.  It's an accelerator because it's pre-built, generic Processor Engines already do the heavy lifting.  They take on a persona at execution time based upon metadata.  This way, customers only need to focus on managing metadata and writing the business rules to plug in.  Still not exactly easy, but a LOT easier!  Streamlined & Simplified!

Of course I based it on DV 2.0, who wouldn't (oh unless they didn't know it is the best way to create durable, adaptable data processing solutions).  Of course I built it with Talend, heck that's what I know and Talend is a great tool for building reusable code modules, (which I leverage heavily).  Of course I prefer Snowflake, but frankly, iGNOSIS works with any data store (files, databases, API's, & streams) for either source, temp, or target use.  In fact, other than Talend, iGNOSIS is technology agnostic (and for money -lol-, I'd be happy to consider porting to anther ETL tool).  I will say I am a fan of VaultSpeed and plan to use it whenever possible.  I do like all the other tool accelerators for DV and I acknowledge & embrace these vendors in the fabric of our community.  I do have an open mind!

What I seek from you all is, feedback.  Please let me know if you'd like to see a preliminary iGNOSIS presentation & demo (about 1 hour, allowing for Q&A).  Id like to hear what you think.

Simply drop me a line: daletanderson@gmail.com or call me (408) 375-8371

Cheers,

Dale

stay healthy - stay safe!

Reply