I have met many organisations that structure their Data Platform teams vertically, that is, they have a team for data engineering, a team for data modelling, a team for analytics, and so on. This is fine for waterfall delivery but does not support agile delivery, even if each team works agile.
The primary reason that it doesn’t work is that one team wants to complete their work before handing over to the next team and so, even if each team works agile, the result is waterfall, or more specifically Phase-Gate method, with each team acting as the phase-gate: modelling phase, engineering phase, reports & analytics phase and then go live.
To run a data platform, delivering projects using agile you need to run a multidsciplinary agile team where the team is comprised of engineers, modellers, report authors and key business users all working together to produce a final product.
The Multidisciplinary Agile Team
A mutlidisciplinary agile team would look something like the following. Note, the team would also have scrum master – not shown. Also, not every team would need every role. Each role may be filled by more than one person or one person may take on more than one role.
Data Engineer. Develops and maintains data pipelines that ingest, clean, transform and load source data
Operations. If using a DevOps model, responsible for platform operation, toolchains
Data Modeller. Creates and maintains target data models in base and analytical layer.
Data Owner. Responsible for data governance, both data quality and data access.
Data Analyst. Creates reports and dashboards. Performs ad hoc data requests.
Data Science. Performs advanced analytics using statistical modelling tools or machine learning solutions
Business Analyst. Requirements analysis, performs playbacks to user community. Gathers business definitions of data maintained in Data Catalog
Product Owner. Responsible for overall solution, represents key stakeholders and user community
Larger organisations that need to run several agile teams in parallel can adopt a matrix organisation where the agile team members are pooled from discipline based teams, for example, the engineering team provides engineering resources to the agile teams. Consistency is acheived through an overaarching architecture & design function supported by standards, design patterns, methods centrally maintained and universally applied.
Consumer Based Solution Design
Another critical change is to move to a data consumer based design, that is, start with understanding what reports, analytics, data science models are required and then work backwards to identify the data required. This customer centric approach is a key tenent of the agile method.
The consequence of this is that the overall solution design must also be consumer based, that is, you start with asking what does the analytical solution look like, then what data structures in the data platform support the analytics and then finally what source data is needed and the data engineering to process the data.
To run a truly agile data platform adopt a multidisciplinary agile team and move to a customer centric design approach.
Al Gulland 