infra/tickets

Fork 2

Outage: Fedorapeople.org outage 2011-05-04 #2753

New issue

Closed

opened 2011-05-05 04:15:00 +00:00 by kevin · 9 comments

kevin commented

2011-05-05 04:15:00 +00:00

Owner

fedorapeople.org and the xen host it's on are not reachable via the network.

Outage started around 21:50UTC.

A ticket has been filed with bu support to investigate.

fedorapeople.org and the xen host it's on are not reachable via the network. Outage started around 21:50UTC. A ticket has been filed with bu support to investigate.

kevin commented

2011-05-05 04:36:26 +00:00

Author

Owner

Note that fedpkg is affected as it's trying to check jkeating.fedorapeople.org to see if there are new branches setup. ;(

kevin commented

2011-05-05 10:05:45 +00:00

Author

Owner

This issue has been assigned an ID of: #86098 on the enghelp@bu.edu ticketing system. So far there's no ETA or information on cause of the outage. ;(

We do have a new instance that was being prepped to replace this one, and if the outage continues we could restore a backup to the new instance.

This issue has been assigned an ID of: #86098 on the enghelp@bu.edu ticketing system. So far there's no ETA or information on cause of the outage. ;( We do have a new instance that was being prepped to replace this one, and if the outage continues we could restore a backup to the new instance.

kevin commented

2011-05-05 10:23:05 +00:00

Author

Owner

Additional information indicates there is a massive power outage on site. ;(

Hopefully more information on when power and network may be restored will be coming in the morning.

Additional information indicates there is a massive power outage on site. ;( Hopefully more information on when power and network may be restored will be coming in the morning.

nphilipp commented

2011-05-05 16:41:18 +00:00

Member

Replying to [comment:1 kevin]:

Note that fedpkg is affected as it's trying to check jkeating.fedorapeople.org to see if there are new branches setup. ;(

I've let my local webserver serve an empty /reposconverted file and added an entry with its IP to /etc/hosts for jkeating.fedorapeople.org to work around this.

Replying to [comment:1 kevin]: > Note that fedpkg is affected as it's trying to check jkeating.fedorapeople.org to see if there are new branches setup. ;( I've let my local webserver serve an empty /reposconverted file and added an entry with its IP to /etc/hosts for jkeating.fedorapeople.org to work around this.

kevin commented

2011-05-05 20:32:15 +00:00

Author

Owner

The site has restored power now, but the xen host that fedorapeople.org is running on is not coming up correctly yet. On site folks are working on it.

kevin commented

2011-05-06 01:03:22 +00:00

Author

Owner

We have moved jkeating.fedorapeople.org over to allow fedpkg to work. As soon as you have refreshed DNS information this should start working normally now.

We are also syncing over a backup of the data to a new instance. This will allow us to bring up an instance thats available, but a day out of date.

We have moved jkeating.fedorapeople.org over to allow fedpkg to work. As soon as you have refreshed DNS information this should start working normally now. We are also syncing over a backup of the data to a new instance. This will allow us to bring up an instance thats available, but a day out of date.

kevin commented

2011-05-06 09:31:24 +00:00

Author

Owner

We are still syncing backup data to the new instance.

There's no further news on the fate of the old machine at this point, so we are moving forward with the idea that we will bring up the new one as soon as the restore is done, sadly losing a day of changes.

Hopefully we can bring up the new instance tomorrow once data has finished syncing to it.

We are still syncing backup data to the new instance. There's no further news on the fate of the old machine at this point, so we are moving forward with the idea that we will bring up the new one as soon as the restore is done, sadly losing a day of changes. Hopefully we can bring up the new instance tomorrow once data has finished syncing to it.

jnovy commented

2011-05-06 14:10:51 +00:00

This affects TeX Live 2010 and 2011 repositories, making their users quite uncomfortable:

http://www.linux.cz/pipermail/texlive/2011-May/000141.html

This affects TeX Live 2010 and 2011 repositories, making their users quite uncomfortable: http://www.linux.cz/pipermail/texlive/2011-May/000141.html

kevin commented

2011-05-06 18:41:15 +00:00

Author

Owner

fedorapeople is back up and operating now finally as of 09:20UTC or so.

We are going to look at migrating it to the new instance soon, but for now, the old one is back up.

fedorapeople is back up and operating now finally as of 09:20UTC or so. We are going to look at migrating it to the new instance soon, but for now, the old one is back up.