How to have an analytics datanommer

What's a Datanommer?

Process

1. Have a database:
   • Restore a full upstream backup (~17 hours)
   • Congrats! You can't do very much because the messages are json in a text field 🙃. (This is because some of the incoming json is >256M or has \u0000 JSON nulls in it.)
2. Have a database you can actually query
   • For an overview of all changes made, see: All changes to the public schema.
   • To implement this on your own replica, see:
     • Tuning (this can dramatically improve the performance of the rest of these steps!)
     • Upgrade a PG13 datanommer to PG18 (~12 hours)
     • Convert the json text to jsonb (~200 hours, 36x faster queries!)
     • Convert the topic column to ltree representation (~20 hours, 2x faster queries)
3. Have fun!
   • Keep your replica updated
   • Export some parquet
   • Make some room for your friends

General Info

• Facts (schema, etc)
• Things you can't do with datanommer