Data should be efficient, changeable, interpretable, and verifiable DRAFT

Data1[1] ‘data’ means the data itself rather than code that produces it. For that, standard code quality precepts will do. should be efficient, changeable, interpretable, and verifiable.

Efficient

  1. Is the data stored correctly?
  • database type
  • data types
  • proximity 2[2] over a network, on a local disk, in memory
  1. Are the data transformations structured well?
  • order matters
    • filter early on both fields as well as records
  • do common things upstream, do idiosyncratic things downstream
  1. Is data retrieval optimized?

Retrieval is optimal when you’re fetching

  • no more and no less than what you need,
  • as soon as it is possible to fetch it, and
  • structured and formatted in a way that you need to do as little work as possible to make it useable.

Changeable

Changeablilty presents a tradeoff between the costs of storage and retrieval. The goal is to adhere as much to the Liskov substitution principle3[3] open to extension, closed to modification as one can with as much precomputation as we need. Precomputation eases retrieval but at the cost of malleability.

Interpretable

how do we make the data easier to understand? data structures should mirror mental models. if they don’t, you’re prone to error.

  1. structure

    • schema design
    • relationships and constraints
    • consistent and intuitive naming conventions
    • data types

  2. documentation

    • formula
    • upstream inputs
    • downstream consumers
    • versioning

  3. quality

    • measure:
    • completeness
    • accuracy
    • freshness
    • communicate known issues or gaps using names4[4] tables, columns or categories for categorical data
    • a stray comment somewhere won’t help if it’s not being read

Verifiable

ensure that data produced is what people expect it to be.