Allow SSH repo access

ryanlerch commented

2025-03-19 06:02:42 +00:00

(Migrated from codeberg.org)

Currently, on the staging instance, accessing a repo via SSH is not working, for example:

git clone git@forgejo.apps.ocp.stg.fedoraproject.org:Infrastructure/ansible.git

times out

Currently, on the staging instance, accessing a repo via SSH is not working, for example: ``` git clone git@forgejo.apps.ocp.stg.fedoraproject.org:Infrastructure/ansible.git ``` times out

👍 2

nirik commented

2025-03-19 19:01:20 +00:00

(Migrated from codeberg.org)

Yes, I noted that this will not work at all in the matrix channel a bit back:

"The instance should serve git repositories over ssh port 22" ... this might be tricky. Our openshift compute nodes are internal and behind proxies. NATting in port 22 to one might work, but which worker has the application will change everytime it's rolled out. I'm not sure how to make that work... perhaps we could change to https serving there?

Yes, I noted that this will not work at all in the matrix channel a bit back: "The instance should serve git repositories over ssh port 22" ... this might be tricky. Our openshift compute nodes are internal and behind proxies. NATting in port 22 to one might work, but which worker has the application will change everytime it's rolled out. I'm not sure how to make that work... perhaps we could change to https serving there?

ryanlerch added this to the Sprint 2 project

2025-11-03 04:33:24 +00:00

mwinters commented

2025-11-15 21:58:15 +00:00

I don't know OpenShift, but from a pure Kubernetes angle this is pretty simple. It's no different to route an https request to the correct pod (regardless of what node it's running on) than it is to route an ssh session.

Here's an example: https://octopus.com/blog/ssh-into-kubernetes-cluster

I don't know OpenShift, but from a pure Kubernetes angle this is pretty simple. It's no different to route an https request to the correct pod (regardless of what node it's running on) than it is to route an ssh session. Here's an example: https://octopus.com/blog/ssh-into-kubernetes-cluster

ryanlerch commented

2025-11-16 23:37:48 +00:00

Owner

Going to re-open this one as a spike.

Things may have changed since this was first looked at.

Going to re-open this one as a spike. Things may have changed since this was first looked at.

ryanlerch modified the project from Sprint 2 to (deleted)

2025-12-02 11:21:01 +00:00

kevin referenced this issue

2025-12-10 01:09:52 +00:00

Enable SSH authentication #325

kevin commented

2025-12-10 01:18:17 +00:00

The kubernetes part of it is simple. However, our clusters are behind our proxies, they are not directly on the internet.

https does:

user -> RHIT firewall external ip -> nat -> proxy (httpd) with all the workers load balanced -> openshift worker

so for ssh we would need something like more external ip address(es) with port 22 natted into some port on proxies where haproxy listens and load balances some port on the workers.

I'm sure it could be made to work... just seems a lot of trouble.

The kubernetes part of it is simple. However, our clusters are behind our proxies, they are not directly on the internet. https does: user -> RHIT firewall external ip -> nat -> proxy (httpd) with all the workers load balanced -> openshift worker so for ssh we would need something like more external ip address(es) with port 22 natted into some port on proxies where haproxy listens and load balances some port on the workers. I'm sure it could be made to work... just seems a lot of trouble.

simo commented

2025-12-10 20:01:49 +00:00

ssh is more secure and easier authentication-wise. I've personally used a yubikey with keys on HW in forver, would rather not downgrade to passwords stored in the clear on the disk.

mwinters commented

2025-12-23 18:26:21 +00:00

@kevin I'm not sure from your flow description where haproxy lives. Is that the first "nat" instance?

If so, there is a trick we might use to perform essentially the same sort of "virtual host" routing that we do with https. You basically wrap the SSH connection in TLS with the encryption turned off. Sounds dumb, but that allows you to specify the "target" host in the TLS SNI and route it in haproxy accordingly.

https://www.haproxy.com/blog/route-ssh-connections-with-haproxy

Alternatively, I'm guessing the community wouldn't find it burdensome to simply say, "Forge SSH happens on port 2222" or whatever. No dynamic routing needed.

@kevin I'm not sure from your flow description where haproxy lives. Is that the first "nat" instance? If so, there is a trick we might use to perform essentially the same sort of "virtual host" routing that we do with https. You basically wrap the SSH connection in TLS with the encryption turned off. Sounds dumb, but that allows you to specify the "target" host in the TLS SNI and route it in haproxy accordingly. https://www.haproxy.com/blog/route-ssh-connections-with-haproxy Alternatively, I'm guessing the community wouldn't find it burdensome to simply say, "Forge SSH happens on port 2222" or whatever. No dynamic routing needed.

kevin commented

2026-01-22 23:44:47 +00:00

My email notifications were not working as expected so I didn't see this comment when you made it. Sorry. ;(

no, haproxy is being the first nat.

So it would need to be:

External proxy ips, ie, 38.145.32.20 / 38.145.32.21 / 2620:52:6:1121:bead:cafe:feed:fed5 / 2620:52:6:1121:bead:cafe:feed:fed6
We would need to tell RHIT what exact port on those to nat in to what port on the internal servers.
(which could be 22)

That nats into say 2222 on the actual proxy machine where haproxy listens and forwards it to one of the pool of openshift workers.

So, it could be you push something and it goes 38.145.32.20:22 (proxy01.fedoraproject.org ) -> 10.16.163.74:2222 (proxy01.rdu3.fedoraproject.org) -> (worker01.ocp) 10.16.163.123:2222

But the next push could go proxy10 -> proxy10.rdu3 -> worker04.ocp

So how would ssh host keys work there?

My email notifications were not working as expected so I didn't see this comment when you made it. Sorry. ;( no, haproxy is being the first nat. So it would need to be: External proxy ips, ie, 38.145.32.20 / 38.145.32.21 / 2620:52:6:1121:bead:cafe:feed:fed5 / 2620:52:6:1121:bead:cafe:feed:fed6 We would need to tell RHIT what exact port on those to nat in to what port on the internal servers. (which could be 22) That nats into say 2222 on the actual proxy machine where haproxy listens and forwards it to one of the pool of openshift workers. So, it could be you push something and it goes 38.145.32.20:22 (proxy01.fedoraproject.org ) -> 10.16.163.74:2222 (proxy01.rdu3.fedoraproject.org) -> (worker01.ocp) 10.16.163.123:2222 But the next push could go proxy10 -> proxy10.rdu3 -> worker04.ocp So how would ssh host keys work there?

gotmax23 commented

2026-01-23 01:06:44 +00:00

Is it possible create ssh host keys and copy the same ones to each worker?

simo commented

2026-01-23 18:38:12 +00:00

Given this is a single service from the pov of users I would expect to see a single key indeed.

I am not sure how this service is deployed but if it uses containers and is orchestrated I would expect the keys to be "container secrets" stored in some sort of vault and deployed when the container is instantiated in ephemeral storage that goes away when the container is stopped.

If this is not done via container orchestration but managed hosts I would still expect keys to be managed by deploying them from a secure location to each host that needs them and removed when the host is retired.

Given this is a single service from the pov of users I would expect to see a single key indeed. I am not sure how this service is deployed but if it uses containers and is orchestrated I would expect the keys to be "container secrets" stored in some sort of vault and deployed when the container is instantiated in ephemeral storage that goes away when the container is stopped. If this is not done via container orchestration but managed hosts I would still expect keys to be managed by deploying them from a secure location to each host that needs them and removed when the host is retired.

mwinters commented

2026-02-02 09:22:33 +00:00

how would ssh host keys work there?

Kubernetes can natively mount NFS volumes into pods. I don't know the ForgeJo side of key management, but the k8s side of having multiple pods share network storage is generally a good idea only if they're mounting it read-only. If it's possible to pull SSH keys from FAS and put them where ForgeJo can find them, that would be pretty slick actually.

> how would ssh host keys work there? Kubernetes can natively mount NFS volumes into pods. I don't know the ForgeJo side of key management, but the k8s side of having multiple pods share network storage is generally a good idea only if they're mounting it read-only. If it's possible to pull SSH keys from FAS and put them where ForgeJo can find them, that would be pretty slick actually.

simo commented

2026-02-02 13:46:00 +00:00

Sharing private ssh keys over NFS is a pretty bad idea. If the environment is kurbernetes based they should be deployed as "secrets" at container instantiation.

mwinters commented

2026-02-09 15:31:43 +00:00

These are public keys, not private.

simo commented

2026-02-09 17:09:10 +00:00

@mwinters WAT?
Each SSH server needs the hosts private keys of course, what would be the utility of sharing just public keys?

@mwinters WAT? Each SSH server needs the hosts private keys of course, what would be the utility of sharing just public keys?

ferdnyc commented

2026-02-23 16:07:35 +00:00

@mwinters wrote in #27 (comment):

If so, there is a trick we might use to perform essentially the same sort of "virtual host" routing that we do with https. You basically wrap the SSH connection in TLS with the encryption turned off. Sounds dumb, but that allows you to specify the "target" host in the TLS SNI and route it in haproxy accordingly.

https://www.haproxy.com/blog/route-ssh-connections-with-haproxy

It's so strange to me, that that blog post never even touches on the question of SSH host keys. I've posted a comment on the article, asking:

How are SSH host keys usually handled in such a setup? Given that all of the connections use a single SSH hostname, the client would expect to receive the same host keys no matter which backend server the request is routed to. Is there a good, secure way to share SSH host keys among multiple backend servers?

It's currently pending approval.

@mwinters wrote in https://forge.fedoraproject.org/forge/forge/issues/27#issuecomment-259202: > If so, there is a trick we might use to perform essentially the same sort of "virtual host" routing that we do with https. You basically wrap the SSH connection in TLS with the encryption turned off. Sounds dumb, but that allows you to specify the "target" host in the TLS SNI and route it in haproxy accordingly. > > https://www.haproxy.com/blog/route-ssh-connections-with-haproxy It's so strange to me, that that blog post never even touches on the question of SSH host keys. I've posted a comment on the article, asking: > How are SSH host keys usually handled in such a setup? Given that all of the connections use a single SSH hostname, the client would expect to receive the same host keys no matter which backend server the request is routed to. Is there a good, secure way to share SSH host keys among multiple backend servers? It's currently pending approval.

mwinters commented

2026-02-23 16:47:20 +00:00

@simo Maybe we're talking about different things, I'm not sure. When you SSH in (whether for TTY or git), you have the private key on your client machine (your laptop) and the public key on the destination. I thought we were talking about distributing public keys to the ForgeJo hosts (and/or SSH proxies), so that they can authenticate incoming connections.

ferdnyc commented

2026-02-23 17:34:06 +00:00

@mwinters wrote in #27 (comment):

@simo Maybe we're talking about different things, I'm not sure. When you SSH in (whether for TTY or git), you have the private key on your client machine (your laptop) and the public key on the destination. I thought we were talking about distributing public keys to the ForgeJo hosts (and/or SSH proxies), so that they can authenticate incoming connections.

Forgejo already handles that — there's an SSH key preferences page where user authentication keys can be registered, and those keys will be used for any forgejo SSH connection. (I've already registered mine — actually, a new one, since my existing fedoraproject SSH key didn't meet the minimum 3072-bit complexity requirements for Forgejo.) But that page currently notes that "SSH is currently disabled so these keys are only used for commit signature verification."

Thing is, none of that addresses the problem of server host keys when multiplexing backend SSH servers. Hence my question to the authors of that blog post you referenced.

@mwinters wrote in https://forge.fedoraproject.org/forge/forge/issues/27#issuecomment-544159: > @simo Maybe we're talking about different things, I'm not sure. When you SSH in (whether for TTY or git), you have the private key on your client machine (your laptop) and the public key on the destination. I thought we were talking about distributing public keys to the ForgeJo hosts (and/or SSH proxies), so that they can authenticate incoming connections. Forgejo already handles that — there's an [SSH key preferences page](https://forge.fedoraproject.org/user/settings/keys) where user authentication keys can be registered, and those keys will be used for any forgejo SSH connection. (I've already registered mine — actually, a new one, since my existing fedoraproject SSH key didn't meet the minimum 3072-bit complexity requirements for Forgejo.) But that page currently notes that "SSH is currently disabled so these keys are only used for commit signature verification." Thing is, none of that addresses the problem of server host keys when multiplexing backend SSH servers. Hence my question to the authors of that blog post you referenced.

hricky referenced this issue from atomic-desktops/tracker

2026-03-21 18:48:00 +00:00

Docs: Unify the Fedora Atomic Desktops docs in a single place #10

adamwill commented

2026-04-05 15:31:39 +00:00

Has anyone had time to look into this further?

With the current concern around supply chain attacks, it's a bit of a problem. I would very much like to use an on-token ssh key for Forgejo, same as I now do for GitHub and GitLab, so that even if my machine is compromised, it's hard for an attacker to do pushes to Forgejo. I can't find a decent way to do this with HTTPS authentication, though.

Has anyone had time to look into this further? With the current concern around supply chain attacks, it's a bit of a problem. I would very much like to use an on-token ssh key for Forgejo, same as I now do for GitHub and GitLab, so that even if my machine is compromised, it's hard for an attacker to do pushes to Forgejo. I can't find a decent way to do this with HTTPS authentication, though.

jibecfed referenced this issue from localization/tickets

2026-04-18 08:47:58 +00:00

Docs localization migration from pagure.io to forge.fedoraproject.org #52

darknao commented

2026-04-18 09:54:20 +00:00

Just to clarify, I think you missed one last step here:

But the next push could go proxy10 -> proxy10.rdu3 -> worker04.ocp

It's actually proxy10 -> proxy10.rdu3 -> worker04.ocp -> forjego-ssh service (port 22) -> forjego pod:22
There is only one SSH host key, and it's served by the only one forjego pod in the cluster where the ssh server is running. The OCP host doesn't matter; they are only forwarding the request.

Just to clarify, I think you missed one last step here: > But the next push could go proxy10 -> proxy10.rdu3 -> worker04.ocp It's actually proxy10 -> proxy10.rdu3 -> worker04.ocp -> forjego-ssh service (port 22) -> forjego pod:22 There is only one SSH host key, and it's served by the only one forjego pod in the cluster where the ssh server is running. The OCP host doesn't matter; they are only forwarding the request.

ferdnyc commented

2026-04-18 19:45:09 +00:00

My question-comment on that blog post @mwinters linked to was never approved by the moderators, so it never posted (visibly). The method presented in that post was complex, anyway, and required client-side configuration of the ingress tunnels so you were actually doing SSH-over-TLS.

It feels like it should be possible to do something as seamless as what GitHub or GitLab do, where you just make an SSH connection to port 22 of github.com / gitlab.com, like any other SSH server. (For no other reason than, well, THEY do it!)

But the data exchanged when making a connection to either of their SSH services indicates they're both running custom SSH servers.

An SSH connection to github.com port 22 receives a server identification that claims it speaks the SSH 2.0 protocol, and is "remote software version 3992d52", which Google has no meaningful matches for, so I feel safe in assuming it's a custom daemon.

gitlab.com, OTOH, has both IPv4 and IPv6 address records, and an SSH connection to port 22 receives an SSH 2.0 protocol server identification as simply "remote software version GitLab-SSHD". So they don't even expose any versioning for their clearly-custom sshd.

But for either service, there must be proxied servers hiding behind the same host identification, not to mention just a single public IP. I assume host keys would be the least of the complexity involved in making that work.

My question-comment on that blog post @mwinters linked to was never approved by the moderators, so it never posted (visibly). The method presented in that post was complex, anyway, and required client-side configuration of the ingress tunnels so you were actually doing SSH-over-TLS. It feels like it _should_ be possible to do something as seamless as what GitHub or GitLab do, where you just make an SSH connection to port 22 of `github.com` / `gitlab.com`, like any other SSH server. (For no other reason than, well, **THEY** do it!) But the data exchanged when making a connection to either of their SSH services indicates they're both running custom SSH servers. An SSH connection to `github.com` port 22 receives a server identification that claims it speaks the SSH 2.0 protocol, and is "remote software version 3992d52", which Google has no meaningful matches for, so I feel safe in assuming it's a custom daemon. `gitlab.com`, OTOH, has both IPv4 and IPv6 address records, and an SSH connection to port 22 receives an SSH 2.0 protocol server identification as simply "remote software version GitLab-SSHD". So they don't even expose any versioning for their clearly-custom sshd. But for either service, there _must_ be proxied servers hiding behind the same host identification, not to mention just a single public IP. I assume host keys would be the least of the complexity involved in making that work.

peartown added a new dependency

2026-04-19 18:29:26 +00:00

localization/tickets#52 - Docs localization migration from pagure.io to forge.fedoraproject.org

mwinters commented

2026-04-22 22:47:20 +00:00

I'm re-reading everything here, including Kevin and Darknao's posts, and I think I have a much clearer picture. And we're talking about two separate problems in this thread :)

Problem #1: Networking / routing

@kevin : I totally misunderstood what you were saying last time. If I understand you now, I think your question is:

Given that we have X number of worker nodes (let's say 50 because I don't know) ...
And we have Y number of Forgejo pods which is less than X (let's say 10) ...
How do we route traffic through the proxies to a node where ForgeJo can handle the request?

Do I have that right? If so, the answer will depend on how our OpenShift is deployed. (I have no idea).

What doesn't work: OpenShift Routes

"Routes" are an OpenShift-ism and they're supposedly the thing you should use everywhere. But they only support HTTP / HTTPS traffic, so this is a no-go.

Universal Answer: NodePort Service

This should work on any k8s cluster configuration.

When you define the Service inside of OpenShift to receive the traffic, you set type: NodePort and pick a port -- let's say 2222. That port will get exposed on all worker nodes, whether or not they're running a ForgeJo pod at the moment. So now, just like you're routing all http traffic to all worker nodes, you can route all forge SSH traffic to any/all worker nodes on port 2222. If a pod is not running on the node that receives the traffic, the cluster will just route that traffic internally via the cluster network and haproxy won't need to know.

Traffic Examples

"Lottery winner" traffic: hits a worker that is running ForgeJo pod:
- proxy01.fedoraproject.org:22 [public IP]->
- proxy01.rdu3.fedoraproject.org:2222 [10.16.x.x]->
- worker01.ocp:2222 [10.16.x.x] ->
- The internal ClusterIP address of forgejo-ssh NodePort service [10.244.x.x] ->
- The ClusterIP of forgejo01 pod running on worker01.ocp [10.244.x.x]
"mwinters" traffic (loser): hits a worker that is not running a ForgeJo pod
- proxy01.fedoraproject.org:22 [public IP] ->
- proxy01.rdu3.fedoraproject.org:2222 [10.16.x.x] ->
- worker02.ocp:2222 (dud node) [10.16.x.x] ->
- The internal ClusterIP address of forgejo-ssh NodePort service [10.244.x.x] ->
- worker02.ocp's cluster networking (CNI) [10.16.x.x] ->
- worker01.ocp's cluster networking [10.16.x.x] ->
- The ClusterIP of forgejo01 pod running on worker01.ocp [10.244.x.x]

Note: I actually suggest avoiding 2222 just because this is basically a "forever" port once you allocate it, and other things like to use those common "alternate" ports. So maybe pick something random like 29292 and keep 2222 in your back pocket for the next kludge.

Maybe better answer: LoadBalancer Service

In Kubernetes, a Service of type: LoadBalancer is a generic representation of some sort of dedicated endpoint that can receive public traffic and route it to a worker which can service the request. There are lots of different LoadBalancer implementations to abstract you from the painful details.

For example, if you are running in AWS, then this manifests as an AWS LoadBalancer which routes traffic to EC2 nodes (VMs). As pods shift around inside the cluster, Kubernetes will keep the LoadBalancer informed about which nodes can currently handle the traffic, and the AWS LoadBalancer will route accordingly. If you're on Azure, the same exact config will result in an Azure LoadBalancer. etc etc.

If you're on-prem and have a SDN (I don't know!), this might get handled by provisioning a "static" private IP on the network that the upstream proxies can route to. The same bookeeping about current nodes will happen, just at the SDN level. There are lots of ways to implement a LoadBalancer, including some that get gnarly and do BGP or whatever.

The advantage of using LoadBalancer over NodePort is that traffic only gets routed directly to nodes that can handle the traffic, instead of potentially getting bounced around inside the cluster network and consuming more total bandwidth. So it's worth doing if it's easy / you like your SDN / etc. But if your LoadBalancer options are a pain or unreliable, you can accomplish the same efficient traffic topology via NodePort by using Node labels and Pod affinity.

See: LoadBalancer Service docs

Problem #2: SSH Keys (both client and host)

I did some research here, and ForgeJo actually has two modes of operation (which led to some of the confusion earlier in this thread):

Passthrough to the "system" ssh server
A built-in SSH server, which can be configured with START_SSH_SERVER=true

If you use the built-in SSH server (which we should do), ForgeJo handles:

Centralized storage of client keys, so no need to handle distributing them to all worker nodes.
Configuration of the host key, which can be automatic or manual.

You configure the host key via something like SSH_SERVER_HOST_KEYS = ssh/ssh_host_ed25519_key, ssh/ssh_host_ecdsa_key, ssh/ssh_host_rsa_key. (The paths are relative to APP_DATA_PATH.) Configure this once in the k8s manifest and it should result in all pods presenting the same host key.

Ok, what did I miss? What questions remain?

And: how can I help????

I'm re-reading everything here, including Kevin and Darknao's posts, and I think I have a much clearer picture. And we're talking about two separate problems in this thread :) --- ## Problem #1: Networking / routing @kevin : I totally misunderstood what you were saying last time. If I understand you now, I think your question is: - Given that we have X number of worker nodes (let's say 50 because I don't know) ... - And we have Y number of Forgejo pods which is less than X (let's say 10) ... - How do we route traffic through the proxies to a node where ForgeJo can handle the request? Do I have that right? If so, the answer will depend on how our OpenShift is deployed. (I have no idea). ### What doesn't work: OpenShift Routes "Routes" are an OpenShift-ism and they're supposedly the thing you should use everywhere. But they only support HTTP / HTTPS traffic, so this is a no-go. ### Universal Answer: NodePort Service This should work on any k8s cluster configuration. When you define the Service inside of OpenShift to receive the traffic, you set `type: NodePort` and pick a port -- let's say 2222. That port will get exposed on _all_ worker nodes, whether or not they're running a ForgeJo pod at the moment. So now, just like you're routing all http traffic to all worker nodes, you can route all forge SSH traffic to any/all worker nodes on port 2222. If a pod is not running on the node that receives the traffic, the cluster will just route that traffic internally via the cluster network and haproxy won't need to know. #### Traffic Examples - "Lottery winner" traffic: hits a worker that is running ForgeJo pod: - proxy01.fedoraproject.org:22 \[public IP]-> - proxy01.rdu3.fedoraproject.org:2222 \[10.16.x.x]-> - worker01.ocp:2222 \[10.16.x.x] -> - The internal ClusterIP address of `forgejo-ssh` NodePort service \[10.244.x.x] -> - The ClusterIP of `forgejo01` pod running on worker01.ocp \[10.244.x.x] - "mwinters" traffic (loser): hits a worker that is not running a ForgeJo pod - proxy01.fedoraproject.org:22 \[public IP] -> - proxy01.rdu3.fedoraproject.org:2222 \[10.16.x.x] -> - worker02.ocp:2222 (dud node) \[10.16.x.x] -> - The internal ClusterIP address of `forgejo-ssh` NodePort service \[10.244.x.x] -> - worker02.ocp's cluster networking (CNI) \[10.16.x.x] -> - worker01.ocp's cluster networking \[10.16.x.x] -> - The ClusterIP of `forgejo01` pod running on worker01.ocp \[10.244.x.x] Note: I actually suggest avoiding 2222 just because this is basically a "forever" port once you allocate it, and other things like to use those common "alternate" ports. So maybe pick something random like 29292 and keep 2222 in your back pocket for the next kludge. #### See Also - [NodePort Service docs](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport) ### Maybe better answer: LoadBalancer Service In Kubernetes, a Service of `type: LoadBalancer` is a generic representation of some sort of dedicated endpoint that can receive public traffic and route it to a worker which can service the request. There are lots of different LoadBalancer implementations to abstract you from the painful details. For example, if you are running in AWS, then this manifests as an AWS LoadBalancer which routes traffic to EC2 nodes (VMs). As pods shift around inside the cluster, Kubernetes will keep the LoadBalancer informed about which nodes can currently handle the traffic, and the AWS LoadBalancer will route accordingly. If you're on Azure, the same exact config will result in an Azure LoadBalancer. etc etc. If you're on-prem and have a SDN (I don't know!), this might get handled by provisioning a "static" private IP on the network that the upstream proxies can route to. The same bookeeping about current nodes will happen, just at the SDN level. There are lots of ways to implement a LoadBalancer, including some that get gnarly and do BGP or whatever. The advantage of using LoadBalancer over NodePort is that traffic only gets routed directly to nodes that can handle the traffic, instead of potentially getting bounced around inside the cluster network and consuming more total bandwidth. So it's worth doing if it's easy / you like your SDN / etc. But if your LoadBalancer options are a pain or unreliable, you can accomplish the same efficient traffic topology via NodePort by using Node labels and Pod affinity. - See: [LoadBalancer Service docs](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer) --- ## Problem #2: SSH Keys (both client and host) I did some research here, and ForgeJo actually has two modes of operation (which led to some of the confusion earlier in this thread): 1. Passthrough to the "system" ssh server 2. A built-in SSH server, which can be configured with `START_SSH_SERVER=true` If you use the built-in SSH server (which we should do), ForgeJo handles: - Centralized storage of client keys, so no need to handle distributing them to all worker nodes. - Configuration of the host key, which can be automatic or manual. You configure the host key via something like `SSH_SERVER_HOST_KEYS = ssh/ssh_host_ed25519_key, ssh/ssh_host_ecdsa_key, ssh/ssh_host_rsa_key`. (The paths are relative to `APP_DATA_PATH`.) Configure this once in the k8s manifest and it should result in all pods presenting the same host key. --- Ok, what did I miss? What questions remain? And: _how can I help????_

mwinters commented

2026-04-22 22:51:02 +00:00

I forgot to mention: the ForgeJo SSH server supports PROXY protocol if we want it, so that the ForgeJo logs show the source client IPs in the logs / etc. Kubernetes doesn't care -- it just sees TCP.

darknao commented

2026-04-23 11:22:10 +00:00

Quick addition to mwinters comment:
About problem #1:
MetalLB is the correct answer, but that requires at least one (or more) dedicated IPs for the address pool.
But that's only part of the problem. This only solves how the service is exposed from the OpenShift cluster, but that cluster is not exposed to internet (not directly).
Routing this with HAproxy to the LB VIP should not be a big issue, even if we use NodePort in the end. Note that the worker that receives the request doesn't matter; the request will always go through the cluster network and reach the pods where it actually runs.
Assuming the public IP of all proxies has port 22 available, you get *.fedoraproject.org:22 dedicated to the forge SSH forever.
Best would be to have a dedicated public VIP assigned to something like forge-ssh.fedoraproject.org that load balances traffic to the proxies. But not sure if we can have that (otherwise we would use that instead of the DNS-based load balancing we use for the proxies, right?).

About problem #2:
I think it's a non-issue. Forgejo currently runs on 1 pod, and even if it were running on several, providing the same ssh host key via kubernetes secrets is easy. There is also no "system" ssh server as we run inside a container, and there is no point adding an ssh server to that container if forgejo can handle that with the user ssh keys being stored in the postgresql database.

Quick addition to mwinters comment: About problem #1: MetalLB is the correct answer, but that requires at least one (or more) dedicated IPs for the address pool. But that's only part of the problem. This only solves how the service is exposed from the OpenShift cluster, but that cluster is not exposed to internet (not directly). Routing this with HAproxy to the LB VIP should not be a big issue, even if we use NodePort in the end. Note that the worker that receives the request doesn't matter; the request will always go through the cluster network and reach the pods where it actually runs. Assuming the public IP of all proxies has port 22 available, you get *.fedoraproject.org:22 dedicated to the forge SSH forever. Best would be to have a dedicated public VIP assigned to something like forge-ssh.fedoraproject.org that load balances traffic to the proxies. But not sure if we can have that (otherwise we would use that instead of the DNS-based load balancing we use for the proxies, right?). About problem #2: I think it's a non-issue. Forgejo currently runs on 1 pod, and even if it were running on several, providing the same ssh host key via kubernetes secrets is easy. There is also no "system" ssh server as we run inside a container, and there is no point adding an ssh server to that container if forgejo can handle that with the user ssh keys being stored in the postgresql database.

kevin commented

2026-04-30 17:23:55 +00:00

There's no external load balancer in rdu3. So, the best we could do I think is:

externalip1 nat mapped into proxy01 port something, haproxy sends that to a backend pool of all nodes on some ip/port
externalip2 nat mapped into proxy10 port something, haproxy sends that to a backend pool of all nodes on some ip/port

dns has forge-ssh round robin resolve to externalip1/externalip2

Anyone see problems with that?

There's no external load balancer in rdu3. So, the best we could do I think is: externalip1 nat mapped into proxy01 port something, haproxy sends that to a backend pool of all nodes on some ip/port externalip2 nat mapped into proxy10 port something, haproxy sends that to a backend pool of all nodes on some ip/port dns has forge-ssh round robin resolve to externalip1/externalip2 Anyone see problems with that?

simo commented

2026-04-30 17:59:08 +00:00

not a problem in principle, but didn't you say forgejo runs on a single node?

darknao commented

2026-04-30 19:21:15 +00:00

@kevin wrote in #27 (comment):

Anyone see problems with that?

Seems good to me

@simo wrote in #27 (comment):

not a problem in principle, but didn't you say forgejo runs on a single node?

OpenShift handles that part and redirects traffic to the node where the forgejo pod is running.

@kevin wrote in https://forge.fedoraproject.org/forge/forge/issues/27#issuecomment-677409: > Anyone see problems with that? Seems good to me @simo wrote in https://forge.fedoraproject.org/forge/forge/issues/27#issuecomment-677430: > not a problem in principle, but didn't you say forgejo runs on a single node? OpenShift handles that part and redirects traffic to the node where the forgejo pod is running.

mwinters commented

2026-05-04 22:12:46 +00:00

@kevin wrote in #27 (comment):

Anyone see problems with that?

This is the simplest option (NodePort) and it will work just fine.

There will be some amount of network traffic between OpenShift worker nodes which we could avoid by going the more complex route. But no need to optimize this away unless you're tight on internal bandwidth.

@kevin wrote in https://forge.fedoraproject.org/forge/forge/issues/27#issuecomment-677409: > Anyone see problems with that? This is the simplest option (NodePort) and it will work just fine. There will be some amount of network traffic between OpenShift worker nodes which we could avoid by going the more complex route. But no need to optimize this away unless you're tight on internal bandwidth.

humaton commented

2026-06-11 20:55:53 +00:00

Owner

So it seems that guys here came up with a good solution. Thank you everyone!
Should we create actionable tickets from this Spike and put it in the backlog?
@ryanlerch WDYT?

So it seems that guys here came up with a good solution. Thank you everyone! Should we create actionable tickets from this Spike and put it in the backlog? @ryanlerch WDYT?

kevin commented

2026-06-13 07:44:23 +00:00

I've been working on this slowly in the background.

I got to the point to request nat mappings. However, networking will not do different port mappings, so if we map port 22 on proxy01/10 to an internal ip, it must also be port tcp/22, which means we have to move our sshd to access those to another port, which we can do, it's just more work.

I've been working on this slowly in the background. I got to the point to request nat mappings. However, networking will not do different port mappings, so if we map port 22 on proxy01/10 to an internal ip, it must also be port tcp/22, which means we have to move our sshd to access those to another port, which we can do, it's just more work.

❤️ 2

ryanlerch added the

labels

2026-07-20 00:20:28 +00:00