Real success of a data warehouse is defined by number of users who use it on a daily basis, so do not forget to always put their needs in the first place.
It is easier to build successful data warehouse in Agile with cross-functional team players. Make sure each of your team members can play someone's else role
Straightforward Stage layer, normalized Integrated layer, and business oriented, dimensional Access layer - these are 3 gold pillars of successful data warehouse.
15 years of experience in building enterprise data warehouses in Logistics, Telco, Banking, and Pharma. Lubo has a strong passion to observe, connect and make things work. Using the power of abstraction to utilize unexpected patterns.
8 years of experience in Business Intelligence. His pragmatic approach leads to functional and robust solutions that work. He played multiple roles in DWH projects, so he gets different perspectives and angles in order to clearly define problems and solve them.
Version control system is an essential tool for DWH projects, tracking changes in the code during development. GIT is a great choice due to the best fit into agile integration.
This principle is applicable recursively starting with DWH layers, going down to data models, entities, attributes. Experiences show that Separation of concerns makes DWH solution maintainable in the long term.
Clear and well-commented code and configuration files, descriptive documentation of the whole data flow. No black boxes and magician features. The best code is not the shortest but the most self-descriptive one.
Anything what can be automated, has to be automated. Start with packaging, deployment and testing. Analyse all repeating activities and assess which really need to be manual.
There are several traditional approaches helping to keep things simple like Refactoring, YAGNI, Single responsibility, DRY. Perfection is achieved not when there is nothing more to add, but when there is nothing left to take away (Antoine de Saint-Exupery).
Entropy of the DWH systems
“When DWH accumulates unorganized data it tends to experience some kind of Turbo Effect in the entropy increase. The main motivation to start the project DWH Academy was to share the patterns and tools which work to build a successful Data warehouse (…) ”
Proven DWH Design
“There is an unlimited number of ways how to build a data warehouse. Few of them work and some of the few can even survive in the long term. In this post, we are going to describe the high-level design of the typical successful DWH in the long term (…) ”
Agile in Data warehousing
“Working in Agile became a hot topic. Big consulting companies (including big 4) realized that implementing agile methodology is a very lucrative business. But have you ever seen a good implementation of Agile methodology in Data Warehousing? Probably not (…) ”