- Python 50.4%
- JavaScript 38.1%
- CSS 9.3%
- HTML 1.3%
- Dockerfile 0.7%
- Other 0.2%
| exemplars | ||
| schema-extract | ||
| site | ||
| src | ||
| topics | ||
| .gitignore | ||
| build.sh | ||
| CONTRIBUTING.md | ||
| datanommer-dictionary.json | ||
| mise.toml | ||
| ORIENTATION.md | ||
| README.md | ||
| site-dev.sh | ||
Datanommer Data Dictionary
Knowledge capture for Datanommer message schemas.
https://datanommer-dictionary.hatlas.mwinters.net/
The Gist
- Datanommer data never changes, so knowledge that we can accumulate about it does not expire.
- However, specific implementations are transient.
Therefore:
- Write down everything we know about this data and how to work with it, but do that in JSON.
- Write transformers that can parse this knowledge and turn it into new tools -- materialized views, dbt flows, BI graphs, LLM instructions for world dominance, etc.
Goals
At a high level, this document performs two functions:
- Document what's in the data.
- For analysis purposes:
- Help the humans find interesting data. ("data cataloging" / "canonical fields")
- Label user identifiers (usernames, etc) so that we can correlate activity across systems to avoid generating misleading community statistics. ("data tagging")
- Categorize user activity data according to our taxonomy to enable bucketing it.
- For privacy purposes:
- Label sensitive data (aka "personal data") including the above-mentioned identifiers, so that we can write programs to pseudonymize it in our Privacy-By-Design operation. ("data classification")
- Declare "minimized" versions of our datasets which our ETL pipelines can build for us. ("data minimization")
- For analysis purposes:
- Document how the data got there (which apps & libraries produced it) and why it exists.
- Mostly for human reference as we try to understand the data.
Non-Goal: Schema Stability
The dictionary schema itself is meant to reflect the sort of questions we want to ask of our data. As such, it should evolve as we ask new questions. It is, at its core, a living document for humans, to capture and share human understanding and make it actionable. And we are still discovering what questions we want to ask.
We of course want to build tools to ingest this data, but we are not prioritizing stability at this time. That said, we do have a version marker, and we can start incrementing that for breaking changes if needed.
Interim Solution
There are some purpose-built solutions for this type of work (Confluence Schema Registry, DataHub, etc), and I expect that we will eventually use those. However, we need a relatively lightweight solution today, and it should be possible to ingest this work into any destination solution.
Domain Language
The word "schema" is heavily overloaded around Fedora, so here is the suggested terminology:
- A "schema" is an abstract term to discuss a particular data structure.
- Example usage: "JSONSchema is a tool for managing schemas in JSON".
- A "schema package" is code that provides an API for working with a particular schema, and
around Fedora we have many "python schema packages".
- Example usage: "The schema package for Noggin is in https://github.com/fedora-infra/noggin-messages ."
- Within a Datanommer message, there is a field in the
headerscolumn calledfedora_schema_name. This is a text string used by schema packages to declare which json schema they are using to encode the message. These are called "schema names".- Example usage: "The
noggin-messagesschema package declares schema namenoggin.group.member.sponsor.v1for messages whose schema represents a user joining a FAS group."
- Example usage: "The
More
See next: