No description
  • Shell 59.4%
  • PLpgSQL 27.2%
  • Python 8.8%
  • Dockerfile 4.6%
Find a file
barnabe d991428df0 Fix: Ingest bug when month < 10
Path for parquet is using single digit month when `fill_tables_parquet.sh` takes 2 digit month as input.
2026-06-05 22:47:53 +00:00
container-utils Initial commit 2026-05-20 21:31:46 +02:00
src Fix: Ingest bug when month < 10 2026-06-05 22:47:53 +00:00
.env.example Initial commit 2026-05-20 21:31:46 +02:00
.gitignore Initial commit 2026-05-20 21:31:46 +02:00
add_all_days.sh Initial commit 2026-05-20 21:31:46 +02:00
add_day_data.sh Add parquet import support via --parquet flag 2026-05-31 18:39:37 +02:00
demo.py Initial commit 2026-05-20 21:31:46 +02:00
Dockerfile Add parquet import support via --parquet flag 2026-05-31 18:39:37 +02:00
query_db.sh Initial commit 2026-05-20 21:31:46 +02:00
README.md add section about --parquet import 2026-06-01 15:45:48 +00:00

datanommer-toolbox

This repo contains utilities that aims to facilitate the use of datanomer2 data in order to create analytics.

Getting started

After cloning the repo, you can start the PostgreSQL/TimescaleDB server by building and starting the Docker image.

First copy .env.example to .env.

cp .env.example .env

Then you can start building the image.

sudo docker build -t datanommer2 .

Start the container with the container-utils/start_db.sh script. It can take up to 5-10 minutes to complete.

sudo ./container-utils/start_db.sh

When started, it will create a pg_data directory. You can change this path by modifying the HOST_BASE variable in the .env file.

It will automatically download data from 2 days ago and populate the postgre database with it.

You can add more data using the following command. The script take a date at YYYY-mm-dd format as an argument.

sudo ./add_day_data.sh YYYY-mm-dd

You can also import data from Hatlas parquet files instead of daily dumps. This requires datanommer-parquet-download and a Hatlas account (see its README for setup instructions).

Once you have oidc-agent configured, you can import data for a specific month:

./add_day_data.sh --parquet YYYY-MM /path/to/datanommer-parquet-download

For example:

./add_day_data.sh --parquet 2024-03 ~/datanommer-parquet-download

To test it:

sudo ./query_db.sh "select distinct date(timestamp) from messages;"

Use with Python

You can connect to database in Python with psycopg or DuckDB. demo.py demonstrates how to do so.

Clean database

If you neeed to reset the data for some reasons (e.g. bad db initialization) you can restore a clean state with:

sudo ./container-utils/clean.sh

Purpose

datanommer-toolbox is meant to provide easy-access tools to:

  • host a small replica of the datanommer2 database
  • download only the data you need
  • make it easy to create your first charts
  • (mirror ingestion processes for development and testing?)