Data product
The job: the consumer side of one table, planned first. It changes no table and no SQL.
Input: the table to serve.
Steps
- Plan first: read the table, its code and the queries of its consumers, and show the plan before writing anything.
- Overview: what the table is for, its grain, who reads it, and the service levels.
- Guide: what to trust, the known issues, and where to start.
- Data model: the tables behind it and how they join.
- Metrics: every measure with its rule, and what is safe to sum.
- Queries: the starter queries, taken from the query logs.
- Quality: the quality rules and their results.
- Releases: the release notes, the number diff between the versions, and the corrections the consumers reported.
- Contract: the data contract with the consumers, in the Open Data Contract Standard.
- Add the product to the data products page, where the consumers browse.
Output
One HTML file, like references/data-products.html: the data products page with every product, and each product with its tabs, from the overview to the contract. Read it for the layout before you write the page.
Reads
- the table, its SQL, the query logs, the corrections the consumers reported, the knowledge
Writes
- the documentation of the product, and the data products page
Never
- Never changes the table or any SQL: a finding is listed as an open point for the data engineer.
- Never sends a release note: it drafts.
