- Shell 59.4%
- PLpgSQL 27.2%
- Python 8.8%
- Dockerfile 4.6%
Path for parquet is using single digit month when `fill_tables_parquet.sh` takes 2 digit month as input. |
||
|---|---|---|
| container-utils | ||
| src | ||
| .env.example | ||
| .gitignore | ||
| add_all_days.sh | ||
| add_day_data.sh | ||
| demo.py | ||
| Dockerfile | ||
| query_db.sh | ||
| README.md | ||
datanommer-toolbox
This repo contains utilities that aims to facilitate the use of datanomer2 data in order to create analytics.
Getting started
After cloning the repo, you can start the PostgreSQL/TimescaleDB server by building and starting the Docker image.
First copy .env.example to .env.
cp .env.example .env
Then you can start building the image.
sudo docker build -t datanommer2 .
Start the container with the container-utils/start_db.sh script. It can take up to 5-10 minutes to complete.
sudo ./container-utils/start_db.sh
When started, it will create a pg_data directory. You can change this path by modifying the HOST_BASE variable in the .env file.
It will automatically download data from 2 days ago and populate the postgre database with it.
You can add more data using the following command. The script take a date at YYYY-mm-dd format as an argument.
sudo ./add_day_data.sh YYYY-mm-dd
You can also import data from Hatlas parquet files instead of daily dumps. This requires datanommer-parquet-download and a Hatlas account (see its README for setup instructions).
Once you have oidc-agent configured, you can import data for a specific month:
./add_day_data.sh --parquet YYYY-MM /path/to/datanommer-parquet-download
For example:
./add_day_data.sh --parquet 2024-03 ~/datanommer-parquet-download
To test it:
sudo ./query_db.sh "select distinct date(timestamp) from messages;"
Use with Python
You can connect to database in Python with psycopg or DuckDB. demo.py demonstrates how to do so.
Clean database
If you neeed to reset the data for some reasons (e.g. bad db initialization) you can restore a clean state with:
sudo ./container-utils/clean.sh
Purpose
datanommer-toolbox is meant to provide easy-access tools to:
- host a small replica of the datanommer2 database
- download only the data you need
- make it easy to create your first charts
- (mirror ingestion processes for development and testing?)