Knowledge capture for Datanommer message schemas
  • Python 50.4%
  • JavaScript 38.1%
  • CSS 9.3%
  • HTML 1.3%
  • Dockerfile 0.7%
  • Other 0.2%
Find a file
2026-08-13 15:00:05 -05:00
exemplars rm exemplars 2026-08-13 15:00:05 -05:00
schema-extract Schema extract 2026-08-06 18:11:49 -05:00
site Topic review status in topic list 2026-07-27 14:08:42 -05:00
src Add missing schema packages 2026-08-06 19:33:26 -05:00
topics rm exemplars 2026-08-13 15:00:05 -05:00
.gitignore Schema extract 2026-08-06 18:11:49 -05:00
build.sh Major refactor 2026-06-25 09:44:18 -05:00
CONTRIBUTING.md Schema extract 2026-08-06 18:11:49 -05:00
datanommer-dictionary.json Add missing schema packages 2026-08-06 19:33:26 -05:00
mise.toml Schema extract 2026-08-06 18:11:49 -05:00
ORIENTATION.md Major refactor 2026-06-25 09:44:18 -05:00
README.md docs updates 2026-06-23 17:29:09 -05:00
site-dev.sh Timestamp WKF, partial Discourse topics 2026-07-17 13:52:35 -05:00

Datanommer Data Dictionary

Knowledge capture for Datanommer message schemas.

https://datanommer-dictionary.hatlas.mwinters.net/

The Gist

  • Datanommer data never changes, so knowledge that we can accumulate about it does not expire.
  • However, specific implementations are transient.

Therefore:

  1. Write down everything we know about this data and how to work with it, but do that in JSON.
  2. Write transformers that can parse this knowledge and turn it into new tools -- materialized views, dbt flows, BI graphs, LLM instructions for world dominance, etc.

Goals

At a high level, this document performs two functions:

  1. Document what's in the data.
    • For analysis purposes:
      • Help the humans find interesting data. ("data cataloging" / "canonical fields")
      • Label user identifiers (usernames, etc) so that we can correlate activity across systems to avoid generating misleading community statistics. ("data tagging")
      • Categorize user activity data according to our taxonomy to enable bucketing it.
    • For privacy purposes:
      • Label sensitive data (aka "personal data") including the above-mentioned identifiers, so that we can write programs to pseudonymize it in our Privacy-By-Design operation. ("data classification")
      • Declare "minimized" versions of our datasets which our ETL pipelines can build for us. ("data minimization")
  2. Document how the data got there (which apps & libraries produced it) and why it exists.
    • Mostly for human reference as we try to understand the data.

Non-Goal: Schema Stability

The dictionary schema itself is meant to reflect the sort of questions we want to ask of our data. As such, it should evolve as we ask new questions. It is, at its core, a living document for humans, to capture and share human understanding and make it actionable. And we are still discovering what questions we want to ask.

We of course want to build tools to ingest this data, but we are not prioritizing stability at this time. That said, we do have a version marker, and we can start incrementing that for breaking changes if needed.

Interim Solution

There are some purpose-built solutions for this type of work (Confluence Schema Registry, DataHub, etc), and I expect that we will eventually use those. However, we need a relatively lightweight solution today, and it should be possible to ingest this work into any destination solution.

Domain Language

The word "schema" is heavily overloaded around Fedora, so here is the suggested terminology:

  • A "schema" is an abstract term to discuss a particular data structure.
    • Example usage: "JSONSchema is a tool for managing schemas in JSON".
  • A "schema package" is code that provides an API for working with a particular schema, and around Fedora we have many "python schema packages".
  • Within a Datanommer message, there is a field in the headers column called fedora_schema_name. This is a text string used by schema packages to declare which json schema they are using to encode the message. These are called "schema names".
    • Example usage: "The noggin-messages schema package declares schema name noggin.group.member.sponsor.v1 for messages whose schema represents a user joining a FAS group."

More

See next: