Hatlas ArgoCD

This is the Kubernetes "ops" repo for Hatlas, leveraging ArgoCD. Join us in #data!

Overview / Setup

Mise

@mwinters heavily uses mise (basically direnv + homebrew), and there is a mise.toml file in the root directory. This is the easiest way to ensure that you're using the correct versions of the CLI tools.

Secrets Handling

1. We use fnox + age to share encrypted secrets between operators. This uses your SSH key(s) as encryption keys. See the quickstart for setup, though you'll need to have an existing admin add your key first.
   • Note: you must use an SSH key which is not password protected. Yes, this sucks. Don't use your golden keys.
2. We use the Sealed Secrets operator to deliver secrets to the cluster. Anywhere there is a secret, there is also a generate_secrets.sh or similar which reads the raw secret from fnox (or a manual override) and creates the sealed secret which is safe to commit to git.

For onboarding, someone with existing access will have to add your public key to fnox.toml and run fnox reencrypt. Then to decrypt, you can just put this in mise.local.toml:

[env]
FNOX_AGE_KEY_FILE="$HOME/.ssh/your-passwordless-private-key"

To add a new secret:

# Otherwise it will make a new one
cd wherever-fnox.toml-is

fnox set FOO
# It will prompt you for the value

Repo Layout

• /bootstrap (kubectl apply -k):
  • argocd
• /core: Foundational Services
• /admin: Tools for Admins
• /shared-services:
• /fdwg:
  • Creates privileged (cluster-scoped) resources for FDWG apps: namespaces, PVs, etc
  • Sets the allowlist for fdwg namespaces in fdwg/resources/proj-fdwg.yaml
  • The fdwg-apps Application
    • Pulls in https://forge.fedoraproject.org/fdwg/hatlas-fdwg-argocd

Argo App-of-Apps dependencies

1. kubectl apply -k
2. core 1. admin 2. shared-services 3. fdwg (in default project; declares the fdwg and fdwg-bridge AppProjects)
   1. fdwg-apps (in fdwg-bridge project; constrained to produce only Application objects in the fdwg-argocd namespace)
   2. ... (child Applications in fdwg project, sourced from fdwg repo

DNS

Only @mwinters has admin on the Cloudflare account, but we are running external-dns with a Kyverno policy that allows any subdomain of hatlas.mwinters.net to be created by DNSEndpoint objects within the cluster.

Our pattern is to generally create all Ingress / IngressRoute objects at ArgoCD sync wave 10, and ensure that the DNSEndpoint objects are created in a previous wave. The sync wave should give us a 50% chance of the subdomain being created before Traefik tries to provision a cert for it, and retries will handle the other 50%.

Config you can't see here

Keycloak

• Realm is name "hatlas"
• Realm setting for access token timeout is 12 hours
• We have configured Fedora Account System (FAS) as an OIDC identity provider with group mapping:
  • FAS fdwg -> fdwg-analyst
  • FAS fdwg-maintainer -> fdwg-maintainer

Onboarding new Ops

See: https://forge.fedoraproject.org/fdwg/fdwg/src/branch/main/work-areas/ops