Hatlas ArgoCD

This is the Kubernetes "ops" repo for Hatlas, leveraging ArgoCD. Join us in #data!

Overview / Setup

Mise

@mwinters heavily uses mise (basically direnv + homebrew), and there is a mise.toml file in the root directory. This is the easiest way to ensure that you're using the correct versions of the CLI tools.

Secrets Handling

1. We use fnox + age to share encrypted secrets between operators. This uses your SSH key(s) as encryption keys. See the quickstart for setup, though you'll need to have an existing admin add your key first.
   • Note: you must use an SSH key which is not password protected. Yes, this sucks. Don't use your golden keys.
2. We use the Sealed Secrets operator to deliver secrets to the cluster. Anywhere there is a secret, there is also a generate_secrets.sh or similar which reads the raw secret from fnox (or a manual override) and creates the sealed secret which is safe to commit to git.

For onboarding, someone with existing access will have to add your public key to fnox.toml and run fnox reencrypt. Then to decrypt, you can just put this in mise.local.toml:

[env]
FNOX_AGE_KEY_FILE="$HOME/.ssh/your-passwordless-private-key"

To add a new secret:

# Otherwise it will make a new one
cd wherever-fnox.toml-is

fnox set FOO
# It will prompt you for the value

Argo App-of-Apps dependencies

1. kubectl apply -k
2. core 1. admin 2. shared-services 3. fdwg (in default project; declares the fdwg and fdwg-bridge AppProjects)
   1. fdwg-apps (in fdwg-bridge project; constrained to produce only Application objects in the fdwg-argocd namespace)
   2. ... (child Applications in fdwg project, sourced from fdwg repo

Repo Layout

• /bootstrap (kubectl apply -k):
  • argocd
• /core:
  • argocd (Argo App)
  • openEBS
  • Traefik
  • Sealed Secrets operator
  • Cilium
• /admin:
  • headlamp
  • metrics-server
  • Prometheus
    • TODO: Use sealed secret operator to set initial admin password
    • TODO: Move this to shared-services
  • X Trivy Operator
  • X Kubescape
  • TODO: Renovate
• /shared-services:
  • CloudNativePG
  • Keycloak
    • TODO: Move this to core
  • TODO: Prometheus
• /fdwg:
  • Namespaces and PVs for FDWG apps
  • The fdwg-apps Application, which pulls in https://codeberg.org/fedora-mwinters/hatlas-fdwg-apps.git
    • www-hatlas
    • datanommer
    • etc

DNS

Only @mwinters has control of the domain at the moment, so any new subdomains will need to be created by him.

These subdomains have been created in advance and are ready to have something deployed:

• alertmanager
• datagreppar
• datagrepper
• datanommer
• duckdb
• duckdb-ui
• ducklake
• grafana
• iceberg
• jupyterhub
• logs
• mail
• metabase (neat but no OIDC)
• ntfy
• ozone
• pgadmin
• s3
• status
• superset
• todos
• trino
• trino-ui
• workflows

Onboarding new analysts (new FDWG members)

Current process as of 2026-05-08:

1. Pre-reqs: Validate user

• They must have a FAS account with an SSH key and a GPG key set. You can verify this by logging into FAS and then editing the username in the URL.
• They must sign the latest version of the Analytics Volunteer Agreement.
• TODO: The FDWG maintainers must not object to adding this person.
  • Right now, trust is a bit of a judgement call. We don't want to grant access to anyone who walks through the door and demands some PII.
  • We currently lean heavily on transitive trust, i.e. are they already contributing elsewhere around the project? Are they an Outreachy intern? etc.

2. Add them to FAS

• New users get added to fdwg. This is intended to grant "analyst" level access.

3. Add them to Keycloak

• Username is their FAS username prefixed with h-, e.g. h-mwinters.
• Copy their full name and email from the Volunteer Agreement signatories file.
• Ensure that you've set "Email verified" to true.
• Add them to the fdwg-analysts group.
• Set a random temporary password (not "abc123" or similar).
• Set their Required User Actions to "Configure OTP" and "Update Password".

4. Send them their temporary password

I prefer to do this using their GPG key from FAS because that is another validation that "users accessing Hatlas" are the same as "users accessing FAS". This matters for legal purposes. (And we really should get OIDC federation set up!)

# Find and import their key
gpg --keyserver hkps://keys.openpgp.org --search-keys F72C1429A9440862
gpg --keyserver hkps://keys.openpgp.org --search-keys fedora@mwinters.net
gpg --keyserver hkps://keyserver.ubuntu.com --search-keys F72C1429A9440862
gpg --keyserver hkps://keyserver.ubuntu.com --search-keys fedora@mwinters.net

# Write a plaintext file
echo 'asecret' > passwd.txt
# Encrypt with their key
gpg --encrypt --armor -r <recipient@email.com> passwd.txt

# Now send them passwd.txt.asc, ideally directly in the DataWG Matrix channel.

# ----------------------------------------------------------------------------

# Instructions to decrypt:
gpg --decrypt passwd.txt

5. Add them as a system user

TODO: You can't do this yet because the Ansible is not public. @mwinters is working on it.

6. Add them to FDWG fnox

If they will need the ability to manage secrets in the dev repo then add their ssh key to fnox.toml there and run fnox reencrypt.

Onboarding new Ops

TODO