pkgs01 remounts /srv/cache/lookaside read-only, preventing uploads. #2647

New issue

Closed

opened 2011-02-25 22:27:53 +00:00 by eseyman · 8 comments

eseyman commented

2011-02-25 22:27:53 +00:00

= phenomenon =
fedpkg does not allow me to upload the source tarball for perl-Math-Random-ISAAC yet replies that it did. Nils Philippsen confirmed this on #fedora-devel.
= reason =
Probably not a very good one.

= phenomenon = fedpkg does not allow me to upload the source tarball for perl-Math-Random-ISAAC yet replies that it did. Nils Philippsen confirmed this on #fedora-devel. = reason = Probably not a very good one.

notting commented

2011-02-25 22:29:24 +00:00

Disk error; volume is currently marked read-only.

kevin commented

2011-02-26 00:19:58 +00:00

Owner

Things should be back to normal now.

Leaving this open to note root cause and how to prevent it moving forward. :)

Things should be back to normal now. Leaving this open to note root cause and how to prevent it moving forward. :)

kevin commented

2011-03-14 23:08:13 +00:00

Owner

This just happened again.

Errors like:

sd 0:0:0:0: timing out command, waited 180s
sd 0:0:0:0: [sdb] Unhandled error code
sd 0:0:0:0: [sdb] Result: hostbyte=DID_IMM_RETRY driverbyte=DRIVER_OK
sd 0:0:0:0: [sdb] CDB: Read(10): 28 00 2b c1 15 18 00 00 08 00
end_request: I/O error, dev sdb, sector 734074136
...
Aborting journal on device dm-1-8.
connection1:0: detected conn error (1020)
EXT4-fs error (device dm-1): ext4_journal_start_sb: Detected aborted journal
EXT4-fs (dm-1): Remounting filesystem read-only

in dmesg.

Mount is a iscsi from a equilogic san.

This just happened again. Errors like: sd 0:0:0:0: timing out command, waited 180s sd 0:0:0:0: [sdb] Unhandled error code sd 0:0:0:0: [sdb] Result: hostbyte=DID_IMM_RETRY driverbyte=DRIVER_OK sd 0:0:0:0: [sdb] CDB: Read(10): 28 00 2b c1 15 18 00 00 08 00 end_request: I/O error, dev sdb, sector 734074136 ... Aborting journal on device dm-1-8. connection1:0: detected conn error (1020) EXT4-fs error (device dm-1): ext4_journal_start_sb: Detected aborted journal EXT4-fs (dm-1): Remounting filesystem read-only in dmesg. Mount is a iscsi from a equilogic san.

pghmcfc commented

2011-03-15 13:23:14 +00:00

It would be good if the upload CGI script detected this and returned a suitable HTTP error status so that curl/fedpkg would know that the upload hadn't actually succeeded.

kevin commented

2011-03-17 09:24:13 +00:00

Owner

There was another 'event' tonight. This time with some xen traces...

we have rebooted the instance into a newer kernel and had it do a full fsck on the lookaside filesystem.

Cause investigation is still ongoing...

There was another 'event' tonight. This time with some xen traces... we have rebooted the instance into a newer kernel and had it do a full fsck on the lookaside filesystem. Cause investigation is still ongoing...

pghmcfc commented

2011-04-01 18:20:11 +00:00

Looks like it's happened again; my upload of Perl-Critic-1.115.tar.gz is reported to have worked, but it didn't.

kevin commented

2011-04-01 18:48:59 +00:00

Owner

sadly so. Fixed again. Please re-upload.

kevin commented

2011-04-22 00:18:37 +00:00

Owner

ok. We have moved pkgs01 to a new kvm rhel6 host.
Hopefully this will fix this from happening again.

I am going to go ahead and close this ticket now, feel free to re-open if this issue ever re-occurs.

ok. We have moved pkgs01 to a new kvm rhel6 host. Hopefully this will fix this from happening again. I am going to go ahead and close this ticket now, feel free to re-open if this issue ever re-occurs.