infra/tickets

Fork 1

bvmhost-a64-03.rdu3.fedoraproject.org DDR init failure #13227

New issue

Open

opened 2026-03-25 23:52:23 +00:00 by kevin · 5 comments

kevin commented

2026-03-25 23:52:23 +00:00

Owner

Description of request

bvmhost-a64-03 didn't come back up from reboots today. ;(

It seems to have bad memory somewhere. We will need to engage support folks to get it fixed.

CP: 0ff01101
DRAM ECC Initialization: [100] [ ==================== ]
DMC RAS register group:
      link_err_count                (0x016c): 0x00000000
      interrupt_status              (0x0510): 0x00000000
      failed_access_int_info_31_00  (0x0538): 0x00000000
      failed_access_int_info_63_32  (0x053c): 0x00000000
      failed_prog_int_info_31_00    (0x0540): 0x00000000
      failed_prog_int_info_63_32    (0x0544): 0x00000000
      link_err_int_info_31_00       (0x0548): 0x00000000
      link_err_int_info_63_32       (0x054c): 0x00000000
      arch_fsm_int_info_31_00       (0x0550): 0x00000000
      arch_fsm_int_info_63_32       (0x0554): 0x00000000
      err0ctlr0                     (0x0708): 0x0000010f
      err0ctlr1                     (0x070c): 0x000000fe
      err1status                    (0x0750): 0x00000000
      err1addr0                     (0x0758): 0xde0d8003
      err1addr1                     (0x075c): 0x00000000
      err1misc0                     (0x0760): 0x0e0d8003
      err1misc1                     (0x0764): 0x0f5ad6bd
      err1misc2                     (0x0768): 0x00000000
      err1misc3                     (0x076c): 0x00000000
      err1misc4                     (0x0770): 0x00000000
      err1misc5                     (0x0774): 0x00000000
      err2status                    (0x0790): 0x00000000
      err2addr0                     (0x0798): 0x00000000
      err2addr1                     (0x079c): 0x00000000
      err2misc0                     (0x07a0): 0x00000000
      err2misc1                     (0x07a4): 0x00000000
      err2misc2                     (0x07a8): 0x00000000
      err2misc3                     (0x07ac): 0x00000000
      err2misc4                     (0x07b0): 0x00000000
      err2misc5                     (0x07b4): 0x00000000
      err3status                    (0x07d0): 0x00000000
      err3addr0                     (0x07d8): 0x00000000
      err3addr1                     (0x07dc): 0x00000000
      err4status                    (0x0810): 0x00000000
      err4addr0                     (0x0818): 0x00000000
      err4addr1                     (0x081c): 0x00000000
      err4misc0                     (0x0820): 0x00000000
      err4misc1                     (0x0824): 0x00000000
      err4misc2                     (0x0828): 0x00000000
      err5status                    (0x0850): 0x00000000
      err5addr0                     (0x0858): 0x00000000
      err5addr1                     (0x085c): 0x00000000
      err5misc0                     (0x0860): 0x00000000
      err5misc1                     (0x0864): 0x00000000
      err5misc2                     (0x0868): 0x00000000
      err6status                    (0x0890): 0x00000000
      err6addr0                     (0x0898): 0x00000000
      err6addr1                     (0x089c): 0x00000000
      err6misc0                     (0x08a0): 0x00000000
      err6misc1                     (0x08a4): 0x00000000
      errgsr                        (0x0920): 0x00000000
ERR: 81f05310
CP: 3ff00800
ERR: bff0c100
ERROR:   DDR initialization failure. STOP!

### Description of request bvmhost-a64-03 didn't come back up from reboots today. ;( It seems to have bad memory somewhere. We will need to engage support folks to get it fixed. ``` CP: 0ff01101 DRAM ECC Initialization: [100] [ ==================== ] DMC RAS register group: link_err_count (0x016c): 0x00000000 interrupt_status (0x0510): 0x00000000 failed_access_int_info_31_00 (0x0538): 0x00000000 failed_access_int_info_63_32 (0x053c): 0x00000000 failed_prog_int_info_31_00 (0x0540): 0x00000000 failed_prog_int_info_63_32 (0x0544): 0x00000000 link_err_int_info_31_00 (0x0548): 0x00000000 link_err_int_info_63_32 (0x054c): 0x00000000 arch_fsm_int_info_31_00 (0x0550): 0x00000000 arch_fsm_int_info_63_32 (0x0554): 0x00000000 err0ctlr0 (0x0708): 0x0000010f err0ctlr1 (0x070c): 0x000000fe err1status (0x0750): 0x00000000 err1addr0 (0x0758): 0xde0d8003 err1addr1 (0x075c): 0x00000000 err1misc0 (0x0760): 0x0e0d8003 err1misc1 (0x0764): 0x0f5ad6bd err1misc2 (0x0768): 0x00000000 err1misc3 (0x076c): 0x00000000 err1misc4 (0x0770): 0x00000000 err1misc5 (0x0774): 0x00000000 err2status (0x0790): 0x00000000 err2addr0 (0x0798): 0x00000000 err2addr1 (0x079c): 0x00000000 err2misc0 (0x07a0): 0x00000000 err2misc1 (0x07a4): 0x00000000 err2misc2 (0x07a8): 0x00000000 err2misc3 (0x07ac): 0x00000000 err2misc4 (0x07b0): 0x00000000 err2misc5 (0x07b4): 0x00000000 err3status (0x07d0): 0x00000000 err3addr0 (0x07d8): 0x00000000 err3addr1 (0x07dc): 0x00000000 err4status (0x0810): 0x00000000 err4addr0 (0x0818): 0x00000000 err4addr1 (0x081c): 0x00000000 err4misc0 (0x0820): 0x00000000 err4misc1 (0x0824): 0x00000000 err4misc2 (0x0828): 0x00000000 err5status (0x0850): 0x00000000 err5addr0 (0x0858): 0x00000000 err5addr1 (0x085c): 0x00000000 err5misc0 (0x0860): 0x00000000 err5misc1 (0x0864): 0x00000000 err5misc2 (0x0868): 0x00000000 err6status (0x0890): 0x00000000 err6addr0 (0x0898): 0x00000000 err6addr1 (0x089c): 0x00000000 err6misc0 (0x08a0): 0x00000000 err6misc1 (0x08a4): 0x00000000 errgsr (0x0920): 0x00000000 ERR: 81f05310 CP: 3ff00800 ERR: bff0c100 ERROR: DDR initialization failure. STOP! ```

james referenced this issue

2026-03-26 01:32:03 +00:00

Planned Outage - update/reboots - 2026-03-25 22:00 UTC #13207

smoliicek added the

labels

2026-03-26 19:09:49 +00:00

kevin commented

2026-03-30 01:43:52 +00:00

Author

Owner

I submitted a support ticket on this issue.

kevin self-assigned this

2026-03-30 01:43:57 +00:00

kevin added this to the Sprint ( 2026-03-23 to 2026-04-06 ) project

2026-03-30 01:44:02 +00:00

kevin commented

2026-04-06 20:51:19 +00:00

Author

Owner

No answer, so I am trying a tech support email now.

kevin modified the project from Sprint ( 2026-03-23 to 2026-04-06 ) to Sprint ( 2026-04-06 to 2026-04-20 )

2026-04-07 13:53:30 +00:00

kevin commented

2026-04-07 23:07:57 +00:00

Author

Owner

ok, got a hold of them and they suggest to reseat all the memory, if that doesn't work, remove all the memory and readd one dimm at a time (there's 8 of them) until we can isolate which one(s) are bad.

I've filed a internal ticket to get DC folks to do this.

kevin commented

2026-04-16 17:23:10 +00:00

Author

Owner

DC folks pulled and reseated all the memory and it came up fine after that.

I don't know if it was just something that needed reseating or if there's a dimm thats possibly going to fail soon, but we can burn that bridge when we come to it.

For now the machine is up and running fine again. :)

DC folks pulled and reseated all the memory and it came up fine after that. I don't know if it was just something that needed reseating or if there's a dimm thats possibly going to fail soon, but we can burn that bridge when we come to it. For now the machine is up and running fine again. :)

kevin closed this issue

2026-04-16 17:23:10 +00:00

kevin commented

2026-07-10 15:43:30 +00:00

Author

Owner

So, this came back this week. When trying to reinstall the machine with rhel10 it came up with the memory error again.

After a lot of testing and first thinking it was a bad memory stick, the problem seems to be in the end that the E0 slot on the motherboard is bad. When any memory is in that slot, the machine fails to boot with that error. ;(

Warentee for the machine unfortunately requires us to ship it back to them for testing/replacement/fixing. :(

While thats happening, the mass rebuild is next week and I don't want to be down a bunch of arm builders, so I am going to move buildhw-a64-02 from being a secure-boot builder to a bvmhost-a64-03 replacement.

So, this came back this week. When trying to reinstall the machine with rhel10 it came up with the memory error again. After a lot of testing and first thinking it was a bad memory stick, the problem seems to be in the end that the E0 slot on the motherboard is bad. When any memory is in that slot, the machine fails to boot with that error. ;( Warentee for the machine unfortunately requires us to ship it back to them for testing/replacement/fixing. :( While thats happening, the mass rebuild is next week and I don't want to be down a bunch of arm builders, so I am going to move buildhw-a64-02 from being a secure-boot builder to a bvmhost-a64-03 replacement.

kevin reopened this issue

2026-07-10 15:43:30 +00:00

kevin modified the project from Sprint ( 2026-04-06 to 2026-04-20 ) to Sprint ( 2026-06-29 to 2026-07-12)

2026-07-10 15:43:37 +00:00

kevin removed this from the Sprint ( 2026-06-29 to 2026-07-12) project

2026-07-13 14:57:36 +00:00