packages app is down: Error communicating with Solr #13240
Labels
No labels
announcement
anubis
authentication
aws
backlog
blocked
bodhi
ci
cloud
communishift
copr
database
day-to-day
dc-move
deprecated
dev
discourse
dns
downloads
easyfix
epel
firmitas
forgejo_migration
Gain
High
Gain
Low
Gain
Medium
gitlab
greenwave
hardware
help wanted
high-trouble
koji
koschei
lists
low-trouble
medium-trouble
mirrorlists
monitoring
Needs investigation
odcs
OpenShift
ops
outage
packager_workflow_blocker
pagure
permissions
Priority
Needs Review
Priority
Next Meeting
Priority
🔥 URGENT 🔥
Priority
Waiting on Assignee
Priority
Waiting on External
Priority
Waiting on Reporter
rabbitmq
release-monitoring
releng
request-for-resources
s390x
security
SMTP
sprint-0
sprint-1
src.fp.o
staging
unfreeze
waiverdb
websites-general
wiki
Backlog Status
Needs Review
Backlog Status
Ready
chore
documentation
points
01
points
02
points
03
points
05
points
08
points
13
Priority
High
Priority
Low
Priority
Medium
Sprint Status
Blocked
Sprint Status
Done
Sprint Status
In Progress
Sprint Status
Review
Sprint Status
To Do
Technical Debt
Work Item
Bug
Work Item
Epic
Work Item
Spike
Work Item
Task
Work Item
User Story
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
infra/tickets#13240
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description of request
The packages app seems down again. The main page does load, but trying a search gives:
Example query: https://packages.fedoraproject.org/search?query=truststore
No deadline, whenever convenient please.
I tried to restart
fedora-packages-staticproject and it didn't help. So we would probably need maintainer of the app to look at it.@kevin Do you know who we can reach to regarding this project?
Looking at it more I think it's related to the deployment change as this was merged yesterday infra/ansible#3151, something must be incorrectly set in the new deployment config.
@phsmoura Could you look into that?
Huh. I checked it after I deployed @phsmoura 's pr... it was running fine?
It also seems to be running fine for me right now?
That might have been when I was moving it from deploymentconfig to deployment and there was some downtime when it had the wrong volume mounts, which I cleaned up...
The error is still there, so I don't think that helped.
If I refresh the page multiple times, I get the page sometimes. It is possible that some AI scraper has found its way into this app and is overloading Solr's connection limit?
I don't see high load on those pods, so I don't think that's it.
Yeah, I don't understand. It's basically always worked fine when I have checked.
Perhaps we could add a zabbix check, and see if there's a pattern to when the errors appear?
I played with it a little today, as I found out that the service needs some changes after DeploymentConfig->Deployment change. It blocked queue processing in the-new-hotness and I noticed similar problems in packages, so I fixed it there as well. Hopefully it will help with the issue.
I haven't seen this since then, but then I wasn't seeing it before then either. :(
Let's close this as fixed as I didn't saw the problem happening lately. If it will happen again, we can always reopen the ticket.