Forum - RE: Dell R720xd iDRAC BIOS recovery

Win-Raid Forum (Drivers - Firmware - Modding) � BIOS Modding � BIOS Modding Guides and Problems � [Problem] Dell R720xd iDRAC BIOS Recovery

Page 10 of 20 � Page 1 ... 5 6 7 8 9 10 11 12 13 14 ... 20 Page �

Topic settings

Print view

Jump to forum

General & Forum related Topics News / Announcements Forum-related Questions & Suggestions Off Topic Important Drivers (AHCI/RAID, NVMe, USB etc.) General: Storage Drivers (AHCI/RAID, NVMe and USB) Specific: Intel AHCI/RAID Drivers Specific: NVIDIA nForce Chipset Drivers (incl. AHCI/RAID) Specific: NVMe Drivers Other Drivers System Performance RAID Performance AHCI/NVMe Performance Solid State Drives BIOS Modding BIOS Modding Guides and Problems BIOS modules (PCI ROM, EFI and others) BIOS Modding Requests Reports: BIOS Modding Results Offers: Already modded special BIOSes Special Topics Intel Management Engine CPU Microcodes NVMe Support for old Systems Other Tools (not BIOS modding related) Interesting PC Hardware & Software OS related Topics Windows 11 Windows 8-10/Server 2012-2019 Windows 7/Vista/Server 2008 Windows XP/2000/Server 2003 DOS/Windows 3.X/9X/ME - Retro Gaming Other Operating Systems

#136 | RE: Dell R720xd iDRAC BIOS recovery datum

Fri Dec 11, 2020 8:41 am (Last edited: Fri Dec 11, 2020 8:47 am)

So I'm another user that got hit by this. Did the same thing as OP, upgraded from too old of version to 2.30.x and it bricked my iDRAC.

Running the commands here in iDRAC7=> prompts give the same results as others have seen `Failure writing blocks [1:196609] RC=0xffffffac`
gist DOT github DOT com SLASH vaskokj SLASH de70dd5241df64dceb4710fa9cf0b2a9 - I can't post links as I'm a new member.

It seems that the only person that has had success fixing this by replacing the chip is @thewooflife?

I assume that once we see the `Failure writing blocks [1:196609] RC=0xffffffac` when trying to run `util recover -emmc -from sd -f firmimg.d7` the eMMC is bad and just had to be replaced and no other solution exists to "revive" it that I might have missed? If so I'm ordering a hot air rework station and the chip that @thewooflife ordered and going to see if I can replace it.

@willard @ldv @XOIIO did any of yall replace your eMMC chip and have any luck?

The only other solutions seem to be "temporary" fixes.

1) tftp solution mentioned here by @ribs
2) booting off sd card? I couldn't find proper solution for this, just people speculating it might work? Unless I missed it?

reply

#137 | RE: Dell R720xd iDRAC BIOS recovery datum

Fri Dec 11, 2020 4:18 pm

I think this thread has people discussing two different iDRAC issues:

1) iDRAC failure during flash/upgrade
2) iDRAC failure due to worn-out eMMC

I believe #1 can be fixed via serial commands, but #2 requires eMMC chip replacement. I don't think anybody has done #2 successfully yet

reply

#138 | RE: Dell R720xd iDRAC BIOS recovery datum

Fri Dec 11, 2020 6:48 pm

Zitat von alexthefifth im Beitrag #137
I think this thread has people discussing two different iDRAC issues:

1) iDRAC failure during flash/upgrade
2) iDRAC failure due to worn-out eMMC

I believe #1 can be fixed via serial commands, but #2 requires eMMC chip replacement. I don't think anybody has done #2 successfully yet

Well I was doing an upgrade to the iDRAC when it "failed". My errors seem to be consistent with the failed eMMC (but was asking my question to validate). There was speculation that doing too large of jump from 1.x to 2.x would cause it to brick, which is what I was doing. I just have the errors suggesting the memory is defective. That is one hell of a coincidence if that is the case though. I can't seem to get the eMMC to take the firmimg.d7 file no matter what I do. I do believe @thewooflife said they replaced their eMMC and it worked (post #111 here, sorry can't create links yet, new to the boards).

reply

#139 | RE: Dell R720xd iDRAC BIOS recovery datum

Sun Dec 27, 2020 2:12 am

rgrimes

Show info

Posts:	45
Registered since:	12.27.2020

Zitat von alexthefifth im Beitrag #137
I think this thread has people discussing two different iDRAC issues:

1) iDRAC failure during flash/upgrade
2) iDRAC failure due to worn-out eMMC

I believe #1 can be fixed via serial commands, but #2 requires eMMC chip replacement. I don't think anybody has done #2 successfully yet

First off, I am new here. I have read the whole thread and am fairly well versed in hardware design. You all have pulled some good info together.
I agree that at least 2 problems are being discussed here, and the 1) above may even have some different failure modes depending on what was being done with things bricked.

At present I a R620 with a non responsive idrac7. I have added the serial header and done tbe A2/B2 jumper. This system gets past Uboot and fails while doing the linux system init with what looks like corrupted file systems. I have been doing some very careful poking around seeing what I can learn about its issues and about the advice being given here.

I also have an R720 with a bricked idrac7 but have not done the serial header or jumper. that shall wait tell I get further with the R620.

I have lots of questions, and hopefully can update some of what is posted here. I believe that many things are missing very important details and it is the reason that what seems to work for one person is not working for another, especially around the booting of an idrac7 via tftp. I have tried to do what is described as:

1
2
 
tftpboot firmimg.d7
go addr 0x81000000
 

that fails for me as it has for others with you just ending up back in uboot. And for whoever has made this work I would like to get some additional details from you. In an effort to sort out what is going on here I have started in on analysis of the 1.57.57 firmimg.d7 file. It is a uImage with many things inside and it is very unlikely that the entry address would be the load address as everyone is stating in this thread, but more likely at some later offset. Using binwalk on the image we get this info:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 
binwalk -v firmimg.d7 
Scan Time:     2020-12-26 23:44:16
Target File:   payload/firmimg.d7
MD5 Checksum:  7b30eadf0c00acb45b657dada2d102a8
Signatures:    344
 
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
512           0x200           uImage header, header size: 64 bytes, header CRC: 0x59CF8328, created: 2014-04-30 18:46:08, image size: 2540437 bytes, Data Address: 0x80001000, Entry Point: 0x80002000, data CRC: 0x4C797D36, OS: Linux, CPU: SuperH, image type: OS Kernel Image, compression type: gzip, image name: "sh-linux"
576           0x240           gzip compressed data, maximum compression, has original file name: "vmlinux.bin", from Unix, last modified: 2014-04-30 18:46:07
2541056       0x26C600        Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 59577844 bytes, 4811 inodes, blocksize: 1048576 bytes, created: 2014-04-30 18:47:06
62305916      0x3B6B67C       CRC32 polynomial table, little endian
62336632      0x3B72E78       PGP armored data,
62336676      0x3B72EA4       PGP armored data,
62481920      0x3B96600       Squashfs filesystem, little endian, version 4.0, compression:gzip, size: 467244 bytes, 613 inodes, blocksize: 1048576 bytes, created: 2014-04-30 18:47:06
62952960      0x3C09600       PGP armored data,
 

my conclusion is that the uboot "go" command is simply the wrong way to start this loaded image file, but "bootm" probably is the right way. From some other reading I found this more complete bootm command:

1
 
=> bootm ${kernel_addr} ${ramdisk_addr} ${fdt_addr}

and I suspect once I get those 3 values right I might get this code to run.

I have also figured out the process for getting the firmimg.d7 extracted and written to the SD card using the fwu utility, but I have not been able to boot from it yet, though I suspect it is a matter of setting the correct uboot env bootcmd string and as another pointed out modifying a few mmcblk0 values in files inside the squashfs. This could be a good long term solution to a failed emmc chip.

For now that is all, if anyone who has this working wants to collaborate on getting the tftpbooting of an idrac7 documented so more might get past this hurtle I am here to help too.

reply

#140 | RE: Dell R720xd iDRAC BIOS recovery datum

Tue Jan 05, 2021 5:39 am

Doesn't the tftpboot require network access? How are you getting a valid IP address that will actually respond on the network at all? I can set an IP address but it won't respond to anything on the network.

Have you tried the boot from SD card method?

reply

#141 | RE: Dell R720xd iDRAC BIOS recovery datum

Tue Jan 05, 2021 6:39 pm

rgrimes

Show info

Posts:	45
Registered since:	12.27.2020

Zitat von kur1j im Beitrag #140
Doesn't the tftpboot require network access? How are you getting a valid IP address that will actually respond on the network at all? I can set an IP address but it won't respond to anything on the network.

Have you tried the boot from SD card method?

Yes, tftpboot requires network access and I left out the details of setting that up, you need to set and save a few environment variables to get the network going. Note the idrac can use either the idrac dedicated ethernet port, or it can be set to use port 1 of the host ethernet and having that setting wrong is sometimes why people can not access there idrac.

To set the IP stuff you need to modify these uboot env variables (I show the defaults values here):

1
2
3
4
5
6
 
ipaddr=192.168.0.120
serverip=192.168.0.100
gatewayip=192.168.0.1
netmask=255.255.255.0
netdev=eth0
 
 

I am not sure exactly what you mean by the "SD boot card" method. If you are referring to where I discuss extracting firmimg.d7 to the SD card I have done nothing more with that at this time, focusing my efforts on getting tftpboot to work.

reply

#142 | RE: Dell R720xd iDRAC BIOS recovery datum

Wed Jan 06, 2021 12:09 am

oh2ftu

Show info

Posts:	17
Registered since:	01.05.2021
Location:	Finland

Hello,
New guy here.
I just received my R720 today. Upgraded iDRAC and LCC to 2.21.21.
An upgrade (from within windows 10) to 2.40.40 failed and it suggested to reboot and retry. Well guess what? didn't work and now it's bricked.
And guess what? it's bricked _properly_. No amber light, no nic activity (watching with wireshark). No nothing. Just a LOT of noise.
I tried the SD card loading, but to no avail.
Took a look at the UART and it goes directly to "Welcome to developer console!".
I can get an interface up and ping, but that's it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
 
U-Boot 2009.08 (Nov 04 2015 - 11:28:55) Avocent (0.0.3) EVB
 
CPU: SH-4A
BOARD: R0P7757LC00xxRL (Cn step) board
BOOT: Secure, HRK generated
DRAM:  240MB
  (240MB of 256MB total DRAM is available on U-Boot)
ENV:   Using primary env area.
In:    serial
Out:   serial
Err:   serial
PCIe:  Bridge loaded with 0x18000 bytes
WDT2:  Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net:   sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:004 U-Boot 2009.08 (Nov 04 2015 - 11:28:55) Avocent (0.0.3) EVB
INFO: 00:007 U-Boot checkin date(08-04-2015) Version(1.1.31)
INFO: 00:006 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31  Minor Ver=0x14 C4 Little endian
             Family=0x10    Major Ver=0x30  Minor Ver=0x0b
INFO: 00:139 BCM54610 OUI=0x00d897  Model=0x26  Revision=0x0a PhyAddr=1
INFO: 00:007 Dedicated monolithic mgmt NIC disabled
INFO: 00:005 SD CARD: is not found
INFO: 00:042 EMMC:   Device: sh_mmcif   Manufacturer ID: 90   OEM: 14a
             Name: HYNIX   Tran Speed: 25000000   Rd Blk Len: 512
             MMC ver 4.0   HC: Yes   cap(<2gb): 0  cap(>2gb): 7569408
             ext_rev: 5    rst_n: 0  ext[96]: 0
INFO: 00:023 CPLD: Major Ver=0x1  Minor Ver=0x0  Maint Ver=0x3
             Planar: Type=0x02  Rev=0x8  Rework=0x0  Scratch/PathRetry=0x00
PASS: 00:014 Coin cell detected good,  AD=0x38f low water=0x2c1
PASS: 00:008 PCIe SH7757_C0 Ver=0.15 MCTP en, CRC=0x8e9b6875 @0x8efbd704 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=normal
INFO: 00:002 reset_cause=BIOS
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=1
INFO: 00:005 wdt0cnt=0  cpld_recover_cnt=0  ddr_recover_cnt=0
PASS: 00:006 SMR0 no sermux env, default 0xd4
INFO: 00:005 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
             MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x6f80feff
             BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=F0:1F:AF:EB:92:3E
             getherc0=F0:1F:AF:EB:92:3F
INFO: 00:007 Fan logic is not modified on Non-AC power up
INFO: 00:053 Env and backup CRC'ed ok
*** no text signature found ***
INFO: 00:649 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:074 Current u-boot1 1.1.31 verified with 'ubootN'
             Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
PASS: 05:847 Verify OS Images N: CRC32 OK: Kernel=0xc4135ec5 Rootfs=0x47eca7d
PASS: 00:007 Boot device=emmc  Boot partition1/N
             Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 06:971
 

WDT2: Disabled
Initialization complete, proceed with boot
OS_WDT0_EN:180 seconds,  Re-load ILRAM:
SPI_WP: Not attempted.
EMMC_WP: Not attempted.
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image ... OK
## Booting kernel from Legacy Image at 88000000 ...
   Image Name:   sh-linux
   Image Type:   SuperH Linux Kernel Image (gzip compressed)
   Data Size:    2625419 Bytes =  2.5 MB
   Load Address: 80001000
   Entry Point:  80002000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
WDT2: Disable in do_bootm_linux()
 
Monolithic/DRB
systemd[1]: could not find module by name='autofs4'
systemd[1]: Failed to insert 'autofs4'
 
Welcome to developer console!
Type "journalctl -b" to view system logs. Type "systemctl reboot" to reboot.
/home/root #
 

So it seems the eMMC might still work. But dmesg gives me this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
 
/home/root # dmesg
 1000/Full
<5>SCSI subsystem initialized
<6>Initializing USB Mass Storage driver...
<6>usbcore: registered new interface driver usb-storage
<6>USB Mass Storage support registered.
<4>mknod /dev/sh_pbi c 228 0
<4>sh_pbi: 4kb Shared memory @ ffcaa000, PBI registers @ ffca0000, IRQ 320
<6>usbmux_init():: Init ..
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 1991385, nr 40, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 1991385
<3>Buffer I/O error on device mmcblk0p12, logical block 98394
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 1991393
<3>Buffer I/O error on device mmcblk0p12, logical block 98395
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 1991401
<3>Buffer I/O error on device mmcblk0p12, logical block 98396
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 1991409
<3>Buffer I/O error on device mmcblk0p12, logical block 98397
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 1991417
<3>Buffer I/O error on device mmcblk0p12, logical block 98398
<4>lost page write due to I/O error on mmcblk0p12
<4>JBD: recovery failed
<3>EXT3-fs (mmcblk0p12): error loading journal
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 2260985, nr 32, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 2260985
<3>Buffer I/O error on device mmcblk0p12, logical block 1056752
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260986
<3>Buffer I/O error on device mmcblk0p12, logical block 1056753
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260987
<3>Buffer I/O error on device mmcblk0p12, logical block 1056754
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260988
<3>Buffer I/O error on device mmcblk0p12, logical block 1056755
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260989
<3>Buffer I/O error on device mmcblk0p12, logical block 1056756
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260990
<3>Buffer I/O error on device mmcblk0p12, logical block 1056757
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260991
<3>Buffer I/O error on device mmcblk0p12, logical block 1056758
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260992
<3>Buffer I/O error on device mmcblk0p12, logical block 1056759
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260993
<3>Buffer I/O error on device mmcblk0p12, logical block 1056760
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260994
<3>Buffer I/O error on device mmcblk0p12, logical block 1056761
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2260995
<3>end_request: I/O error, dev mmcblk0, sector 2260996
<3>end_request: I/O error, dev mmcblk0, sector 2260997
<3>end_request: I/O error, dev mmcblk0, sector 2260998
<3>end_request: I/O error, dev mmcblk0, sector 2260999
<3>end_request: I/O error, dev mmcblk0, sector 2261000
<3>end_request: I/O error, dev mmcblk0, sector 2261001
<3>end_request: I/O error, dev mmcblk0, sector 2261002
<3>end_request: I/O error, dev mmcblk0, sector 2261003
<3>end_request: I/O error, dev mmcblk0, sector 2261004
<3>end_request: I/O error, dev mmcblk0, sector 2261005
<3>end_request: I/O error, dev mmcblk0, sector 2261006
<3>end_request: I/O error, dev mmcblk0, sector 2261007
<3>end_request: I/O error, dev mmcblk0, sector 2261008
<3>end_request: I/O error, dev mmcblk0, sector 2261009
<3>end_request: I/O error, dev mmcblk0, sector 2261010
<3>end_request: I/O error, dev mmcblk0, sector 2261011
<3>end_request: I/O error, dev mmcblk0, sector 2261012
<3>end_request: I/O error, dev mmcblk0, sector 2261013
<3>end_request: I/O error, dev mmcblk0, sector 2261014
<3>end_request: I/O error, dev mmcblk0, sector 2261015
<3>end_request: I/O error, dev mmcblk0, sector 2261016
<7>eth0: no IPv6 routers present
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 2269177, nr 32, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 2269177
<3>end_request: I/O error, dev mmcblk0, sector 2269178
<3>end_request: I/O error, dev mmcblk0, sector 2269179
<3>end_request: I/O error, dev mmcblk0, sector 2269180
<3>end_request: I/O error, dev mmcblk0, sector 2269181
<3>end_request: I/O error, dev mmcblk0, sector 2269182
<3>end_request: I/O error, dev mmcblk0, sector 2269183
<3>end_request: I/O error, dev mmcblk0, sector 2269184
<3>end_request: I/O error, dev mmcblk0, sector 2269185
<3>end_request: I/O error, dev mmcblk0, sector 2269186
<3>end_request: I/O error, dev mmcblk0, sector 2269187
<3>end_request: I/O error, dev mmcblk0, sector 2269188
<3>end_request: I/O error, dev mmcblk0, sector 2269189
<3>end_request: I/O error, dev mmcblk0, sector 2269190
<3>end_request: I/O error, dev mmcblk0, sector 2269191
<3>end_request: I/O error, dev mmcblk0, sector 2269192
<3>end_request: I/O error, dev mmcblk0, sector 2269193
<3>end_request: I/O error, dev mmcblk0, sector 2269194
<3>end_request: I/O error, dev mmcblk0, sector 2269195
<3>end_request: I/O error, dev mmcblk0, sector 2269196
<3>end_request: I/O error, dev mmcblk0, sector 2269197
<3>end_request: I/O error, dev mmcblk0, sector 2269198
<3>end_request: I/O error, dev mmcblk0, sector 2269199
<3>end_request: I/O error, dev mmcblk0, sector 2269200
<3>end_request: I/O error, dev mmcblk0, sector 2269201
<3>end_request: I/O error, dev mmcblk0, sector 2269202
<3>end_request: I/O error, dev mmcblk0, sector 2269203
<3>end_request: I/O error, dev mmcblk0, sector 2269204
<3>end_request: I/O error, dev mmcblk0, sector 2269205
<3>end_request: I/O error, dev mmcblk0, sector 2269206
<3>end_request: I/O error, dev mmcblk0, sector 2269207
<3>end_request: I/O error, dev mmcblk0, sector 2269208
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 2277369, nr 32, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 2277369
<3>end_request: I/O error, dev mmcblk0, sector 2277370
<3>end_request: I/O error, dev mmcblk0, sector 2277371
<3>end_request: I/O error, dev mmcblk0, sector 2277372
<3>end_request: I/O error, dev mmcblk0, sector 2277373
<3>end_request: I/O error, dev mmcblk0, sector 2277374
<3>end_request: I/O error, dev mmcblk0, sector 2277375
<3>end_request: I/O error, dev mmcblk0, sector 2277376
<3>end_request: I/O error, dev mmcblk0, sector 2277377
<3>end_request: I/O error, dev mmcblk0, sector 2277378
<3>end_request: I/O error, dev mmcblk0, sector 2277379
<3>end_request: I/O error, dev mmcblk0, sector 2277380
<3>end_request: I/O error, dev mmcblk0, sector 2277381
<3>end_request: I/O error, dev mmcblk0, sector 2277382
<3>end_request: I/O error, dev mmcblk0, sector 2277383
<3>end_request: I/O error, dev mmcblk0, sector 2277384
<3>end_request: I/O error, dev mmcblk0, sector 2277385
<3>end_request: I/O error, dev mmcblk0, sector 2277386
<3>end_request: I/O error, dev mmcblk0, sector 2277387
<3>end_request: I/O error, dev mmcblk0, sector 2277388
<3>end_request: I/O error, dev mmcblk0, sector 2277389
<3>end_request: I/O error, dev mmcblk0, sector 2277390
<3>end_request: I/O error, dev mmcblk0, sector 2277391
<3>end_request: I/O error, dev mmcblk0, sector 2277392
<3>end_request: I/O error, dev mmcblk0, sector 2277393
<3>end_request: I/O error, dev mmcblk0, sector 2277394
<3>end_request: I/O error, dev mmcblk0, sector 2277395
<3>end_request: I/O error, dev mmcblk0, sector 2277396
<3>end_request: I/O error, dev mmcblk0, sector 2277397
<3>end_request: I/O error, dev mmcblk0, sector 2277398
<3>end_request: I/O error, dev mmcblk0, sector 2277399
<3>end_request: I/O error, dev mmcblk0, sector 2277400
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 2293753, nr 32, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 2293753
<3>end_request: I/O error, dev mmcblk0, sector 2293754
<3>end_request: I/O error, dev mmcblk0, sector 2293755
<3>end_request: I/O error, dev mmcblk0, sector 2293756
<3>end_request: I/O error, dev mmcblk0, sector 2293757
<3>end_request: I/O error, dev mmcblk0, sector 2293758
<3>end_request: I/O error, dev mmcblk0, sector 2293759
<3>end_request: I/O error, dev mmcblk0, sector 2293760
<3>end_request: I/O error, dev mmcblk0, sector 2293761
<3>end_request: I/O error, dev mmcblk0, sector 2293762
<3>end_request: I/O error, dev mmcblk0, sector 2293763
<3>end_request: I/O error, dev mmcblk0, sector 2293764
<3>end_request: I/O error, dev mmcblk0, sector 2293765
<3>end_request: I/O error, dev mmcblk0, sector 2293766
<3>end_request: I/O error, dev mmcblk0, sector 2293767
<3>end_request: I/O error, dev mmcblk0, sector 2293768
<3>end_request: I/O error, dev mmcblk0, sector 2293769
<3>end_request: I/O error, dev mmcblk0, sector 2293770
<3>end_request: I/O error, dev mmcblk0, sector 2293771
<3>end_request: I/O error, dev mmcblk0, sector 2293772
<3>end_request: I/O error, dev mmcblk0, sector 2293773
<3>end_request: I/O error, dev mmcblk0, sector 2293774
<3>end_request: I/O error, dev mmcblk0, sector 2293775
<3>end_request: I/O error, dev mmcblk0, sector 2293776
<3>end_request: I/O error, dev mmcblk0, sector 2293777
<3>end_request: I/O error, dev mmcblk0, sector 2293778
<3>end_request: I/O error, dev mmcblk0, sector 2293779
<3>end_request: I/O error, dev mmcblk0, sector 2293780
<3>end_request: I/O error, dev mmcblk0, sector 2293781
<3>end_request: I/O error, dev mmcblk0, sector 2293782
<3>end_request: I/O error, dev mmcblk0, sector 2293783
<3>end_request: I/O error, dev mmcblk0, sector 2293784
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 2285561, nr 32, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 2285561
<3>end_request: I/O error, dev mmcblk0, sector 2285562
<3>end_request: I/O error, dev mmcblk0, sector 2285563
<3>end_request: I/O error, dev mmcblk0, sector 2285564
<3>end_request: I/O error, dev mmcblk0, sector 2285565
<3>end_request: I/O error, dev mmcblk0, sector 2285566
<3>end_request: I/O error, dev mmcblk0, sector 2285567
<3>end_request: I/O error, dev mmcblk0, sector 2285568
<3>end_request: I/O error, dev mmcblk0, sector 2285569
<3>end_request: I/O error, dev mmcblk0, sector 2285570
<3>end_request: I/O error, dev mmcblk0, sector 2285571
<3>end_request: I/O error, dev mmcblk0, sector 2285572
<3>end_request: I/O error, dev mmcblk0, sector 2285573
<3>end_request: I/O error, dev mmcblk0, sector 2285574
<3>end_request: I/O error, dev mmcblk0, sector 2285575
<3>end_request: I/O error, dev mmcblk0, sector 2285576
<3>end_request: I/O error, dev mmcblk0, sector 2285577
<3>end_request: I/O error, dev mmcblk0, sector 2285578
<3>end_request: I/O error, dev mmcblk0, sector 2285579
<3>end_request: I/O error, dev mmcblk0, sector 2285580
<3>end_request: I/O error, dev mmcblk0, sector 2285581
<3>end_request: I/O error, dev mmcblk0, sector 2285582
<3>end_request: I/O error, dev mmcblk0, sector 2285583
<3>end_request: I/O error, dev mmcblk0, sector 2285584
<3>end_request: I/O error, dev mmcblk0, sector 2285585
<3>end_request: I/O error, dev mmcblk0, sector 2285586
<3>end_request: I/O error, dev mmcblk0, sector 2285587
<3>end_request: I/O error, dev mmcblk0, sector 2285588
<3>end_request: I/O error, dev mmcblk0, sector 2285589
<3>end_request: I/O error, dev mmcblk0, sector 2285590
<3>end_request: I/O error, dev mmcblk0, sector 2285591
<3>end_request: I/O error, dev mmcblk0, sector 2285592
<7>eth1: no IPv6 routers present
<3>sh_mmcif sh_mmcif.0: Error IRQ while waiting for DMA completion!
<3>mmcblk0: error -5 transferring data, sector 1196854, nr 34, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 1196854
<3>end_request: I/O error, dev mmcblk0, sector 1196856
<3>end_request: I/O error, dev mmcblk0, sector 1196858
<3>end_request: I/O error, dev mmcblk0, sector 1196860
<3>end_request: I/O error, dev mmcblk0, sector 1196862
<3>end_request: I/O error, dev mmcblk0, sector 1196864
<3>end_request: I/O error, dev mmcblk0, sector 1196866
<3>end_request: I/O error, dev mmcblk0, sector 1196868
<3>end_request: I/O error, dev mmcblk0, sector 1196870
<3>end_request: I/O error, dev mmcblk0, sector 1196872
<3>end_request: I/O error, dev mmcblk0, sector 1196874
<3>end_request: I/O error, dev mmcblk0, sector 1196876
<3>end_request: I/O error, dev mmcblk0, sector 1196878
<3>end_request: I/O error, dev mmcblk0, sector 1196880
<3>end_request: I/O error, dev mmcblk0, sector 1196882
<3>end_request: I/O error, dev mmcblk0, sector 1196884
<3>end_request: I/O error, dev mmcblk0, sector 1196886
<3>Aborting journal on device mmcblk0p11.
<2>EXT3-fs (mmcblk0p11): error in ext3_new_inode: Journal has aborted
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -110 transferring data, sector 2291281, nr 8, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 2291281
<4>quiet_error: 150 callbacks suppressed
<3>Buffer I/O error on device mmcblk0p12, logical block 1087048
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291282
<3>Buffer I/O error on device mmcblk0p12, logical block 1087049
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291283
<3>Buffer I/O error on device mmcblk0p12, logical block 1087050
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291284
<3>Buffer I/O error on device mmcblk0p12, logical block 1087051
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291285
<3>Buffer I/O error on device mmcblk0p12, logical block 1087052
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291286
<3>Buffer I/O error on device mmcblk0p12, logical block 1087053
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291287
<3>Buffer I/O error on device mmcblk0p12, logical block 1087054
<4>lost page write due to I/O error on mmcblk0p12
<3>end_request: I/O error, dev mmcblk0, sector 2291288
<3>Buffer I/O error on device mmcblk0p12, logical block 1087055
<4>lost page write due to I/O error on mmcblk0p12
<3>mmcblk0: error -110 transferring data, sector 1196620, nr 2, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 1196620
<3>Buffer I/O error on device mmcblk0p11, logical block 290
<4>lost page write due to I/O error on mmcblk0p11
<3>JBD: I/O error detected when updating journal superblock for mmcblk0p11.
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -110 transferring data, sector 1196042, nr 2, cmd response 0x900, card status 0x0
<3>end_request: I/O error, dev mmcblk0, sector 1196042
<3>Buffer I/O error on device mmcblk0p11, logical block 1
<4>lost page write due to I/O error on mmcblk0p11
<3>EXT3-fs (mmcblk0p11): I/O error while writing superblock
<2>EXT3-fs (mmcblk0p11): error in ext3_create: Journal has aborted
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
<3>mmcblk0: error -5 sending stop command, original cmd response 0x900, card status 0xd00
/home/root #
 

Is there ANYTHING to be done? I've read all of this thread, most on reddit..

reply

#143 | RE: Dell R720xd iDRAC BIOS recovery datum

Wed Jan 06, 2021 2:42 am

@oh2ftu Your in a similiar state to me, your actually ARE booting the Linux kernel, but it is not booting all the way. What does a "df" command show? I wonder how many of the file systems got mounted.

reply

#144 | RE: Dell R720xd iDRAC BIOS recovery datum

Wed Jan 06, 2021 8:40 am (Last edited: Wed Jan 06, 2021 8:52 am)

oh2ftu

Show info

Posts:	17
Registered since:	01.05.2021
Location:	Finland

@rgrimes I will have to reboot the uBoot for that. Now it just shows hundreds of lines of the same mount-points. Same for 'mount' and 'df'. I'm able to mount the sd-card and look at it's contents (including firming.d7). Any idea how to boot it?
Mount:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
 
/home/root # mount
rootfs on / type rootfs (rw)
/dev/root on / type squashfs (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=116100k,nr_inodes=29025,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=//lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
tmpfs on /etc/machine-id type tmpfs (ro,relatime,mode=755)
mqueue on /dev/mqueue type mqueue (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
/dev/mmcblk0p9 on /flash/pd9 type squashfs (ro,noatime)
/dev/mmcblk0p14 on /flash/data2 type ext2 (rw,noatime,errors=continue)
tmpfs on /var/volatile type tmpfs (rw,relatime)
/dev/mmcblk0p15 on /mmc2 type ext3 (rw,noatime,errors=continue,barrier=1,data=ordered)
/dev/mmcblk0p13 on /mnt/cores type ext3 (rw,noatime,errors=continue,user_xattr,barrier=1,data=writeback)
/dev/mmcblk0p11 on /flash/data0 type ext3 (rw,noatime,errors=continue,barrier=1,data=ordered)
 

df:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
/home/root # df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                   91904     91904         0 100% /
/dev/root                91904     91904         0 100% /
devtmpfs                116100         0    116100   0% /dev
tmpfs                   116164         0    116164   0% /dev/shm
tmpfs                   116164       136    116028   0% /run
tmpfs                   116164         0    116164   0% /sys/fs/cgroup
tmpfs                   116164       136    116028   0% /etc/machine-id
tmpfs                   116164         8    116156   0% /tmp
/dev/mmcblk0p9            1024      1024         0 100% /flash/pd9
/dev/mmcblk0p14           2011       136      1855   7% /flash/data2
tmpfs                   116164         0    116164   0% /var/volatile
/dev/mmcblk0p15         604736     16880    587856   3% /mmc2
/dev/mmcblk0p13          48541      6036     42505  12% /mnt/cores
/dev/mmcblk0p11           3835      2899       896  76% /flash/data0
/home/root #
 

reply

#145 | RE: Dell R720xd iDRAC BIOS recovery datum

Wed Jan 06, 2021 10:49 am

Alright. So I'm taking a few leaps here and there. I'm able to mount the SD-card. I can use dd, and dump something to the SD-card.
How should I go forth in dumping the SPI flash? which device is it? after that, I could try reflashing it (via the same console?) with a working dump?

reply

#146 | RE: Dell R720xd iDRAC BIOS recovery datum

Wed Jan 06, 2021 8:54 pm

rgrimes

Show info

Posts:	45
Registered since:	12.27.2020

Zitat von oh2ftu im Beitrag #144

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
 
/home/root # df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                   91904     91904         0 100% /
/dev/root                91904     91904         0 100% /
devtmpfs                116100         0    116100   0% /dev
tmpfs                   116164         0    116164   0% /dev/shm
tmpfs                   116164       136    116028   0% /run
tmpfs                   116164         0    116164   0% /sys/fs/cgroup
tmpfs                   116164       136    116028   0% /etc/machine-id
tmpfs                   116164         8    116156   0% /tmp
/dev/mmcblk0p9            1024      1024         0 100% /flash/pd9
/dev/mmcblk0p14           2011       136      1855   7% /flash/data2
tmpfs                   116164         0    116164   0% /var/volatile
/dev/mmcblk0p15         604736     16880    587856   3% /mmc2
/dev/mmcblk0p13          48541      6036     42505  12% /mnt/cores
/dev/mmcblk0p11           3835      2899       896  76% /flash/data0
/home/root #
 

I am not sure why you say you need to reboot the uBoot to do the df, the df is a linux command. Anyway I see you got the output and your in much better shape than I am. My system does not mount ANY of the /dev/mmcblk0p* entries.

I have not completed the boot from sd-card, but I can say some of the instructions in other parts of this thread CAN destroy your SPI flash, so be warry of what you follow. I have more information, but I am intentionally NOT posting it until I have tested and verified that it works and does no damange.

BE VERY VERY careful with the uboot fwu utility if you try to use it!!!

Currently I am spending most of my time on this problem READING the dell open source release of the 1.57.57 idrac code so that I better understand what these commands and messages are really doing and what they are telling me.

One tidbit in reading that code is I have confirmed that Dell puts the emmc chip on the board BLANK, and that the uboot code, if working correctly, partitions the emmc and then goes into a poll mode looking for the firmimg.d7 file. This means that an actual repair of a failed emmc chip should be a mater of an expert with hot air rework replace the chip and someone doing the idrac recovery. One suspect as to a failure mechanism I have is that if this code decides the emmc is blank and does this partitioning at every boot cycle one could rack up a lot of write cycles on the same blocks pretty quickly, this may be a mechanism of wear out, though it should only start down this road if the emmc is already corrupted.

reply

#147 | RE: Dell R720xd iDRAC BIOS recovery datum

Thu Jan 07, 2021 7:43 am

oh2ftu

Show info

Posts:	17
Registered since:	01.05.2021
Location:	Finland

Leaving the system on will leave df and mount with hundreds of duplicate mount points.
I did manage to mount a SD-card and dump both SPI and emmc (possibly?):

1
2
3
4
5
6
7
8
 

mount /dev/mmcblk1p1 /media/card
cd /dev/
for i in mmcblk0boot*;do dd if=/dev/$i of=/media/card/$i.bin;done
for i in mmcblk0p*;do dd if=/dev/$i of=/media/card/$i.bin;done
for i in mtd*;do dd if=/dev/$i of=/media/card/$i.bin;done
umount /media/card
 
 

The last will dump the devices mtd_, mtd_ro and mtdblock_ - all seem identical.

I will have to look at the thread if someone has a pre 2.x version of the SPI-dump. What will need to be modified is at least:
- Service tag
- ethernet mac
- Licenses?

These might be located in multiple places.

I'm not sure if I've got fwu-utility, I only get the root shell of uboot.
Might just try to erase the emmc and see what that does.
The emmc seems to be 0,5 pitch, I've done TFBGA88 before without any issues. Reballing is a PITA, though a new chip would have the balls in place making it much easier.

reply

#148 | RE: Dell R720xd iDRAC BIOS recovery datum

Thu Jan 07, 2021 9:43 pm

oh2ftu

Show info

Posts:	17
Registered since:	01.05.2021
Location:	Finland

Status report:
- Flashed v1.57 with SPI, no luck
- Flashed original (somewhere between 2.21 and 2.40) still the same.
Using the interrupt, I can now get to iDRAC8=>
Still:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 

iDRAC8=> fwu check 0x81000000
Checking image header CRC ... OK
Checking platform env ID..... OK
Checking kernel image CRC ... OK
Checking rootfs image CRC ... OK
Checking u-boot image CRC ... OK
Skipping u-boot update ...... NO
Checking Platform image CRC . OK
Checking processor support .. OK
Done!
iDRAC8=> fwu mmc emmc
Erasing SPI flash at 0x100000...Writing to SPI flash...done
Erasing SPI flash at 0x110000...Writing to SPI flash...done
iDRAC8=> fwu update 0x81000000
 
*** Updating Partition 1 ***
Checking image header CRC ... OK
Checking platform env ID..... OK
Checking kernel image CRC ... OK
Checking rootfs image CRC ... OK
Checking u-boot image CRC ... OK
Skipping u-boot update ...... NO
Checking Platform image CRC . OK
Checking processor support .. OK
Copying kernel image ........ mmc write failed, err=-84
*** fail to write block to device ***
Fail
*** fail to update kernel image ***
*** fail to update image ***
iDRAC8=> mmcinfo
Device: sh_mmcif
Manufacturer ID: 90
OEM: 14a
Name: HYNIX
Tran Speed: 25000000
Rd Block Len: 512
MMC version 4.0
High Capacity: Yes
Capacity: 0
Bus Width: 8-bit
 

I guess it loads something from the emmc to SPI during boot, which is not ideal.

Here's what bothers me:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
 

INFO: 00:077 Env and backup CRC'ed ok
Erasing SPI flash at 0x100000...Writing to SPI flash...done
Erasing SPI flash at 0x110000...Writing to SPI flash...done
[b]*** no text signature found ***
[/b]INFO: 02:563 Sync eMMC/SPI NOR/Alternate u-boot images
[b]PASS: 00:262 Current u-boot1 1.1.31 verified with 'ubootN'
             Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
             U-boot2 in sync with u-boot1 1.1.31[/b]
PASS: 05:803 Verify OS Images N: CRC32 OK: Kernel=0xc4135ec5 Rootfs=0x47eca7d
[b]PASS: 00:008 Boot device=emmc  Boot partition1/N
             Boot Path Retry:P1/N=0 P5/N-1=0[/b]
INFO: 00:071 No 'userexe' defined
 
 

I've ordered two blank emmc's and a few spi:s. But the SPI flashes are redundant now that I've borrowed a programmer.

reply

#149 | RE: Dell R720xd iDRAC BIOS recovery datum

Fri Jan 08, 2021 5:45 am

Yeah @ldv mentioned it here Dell R720xd iDRAC BIOS recovery (6) where you could boot directly from the SD card. Basically an alternative to the tftp option.

I want to see if I can get mine to do the above SD card boot like @ldv mentioned. If not I'll fall back to trying the tftp.

I have a hot air reworkstation and ordered the eMMC, so once it arrives I'm going to try to attempt to replace it.

To me it really doesn't make sense that the eMMC died though.

reply

#150 | RE: Dell R720xd iDRAC BIOS recovery datum

Fri Jan 08, 2021 5:58 am

When you boot from SD card does the system still have the issue of loud fans (for a long time) upon every restart? Or only if iDRAC loses power i.e. system gets unplugged

reply

Turing GPU using Certificate 2.0 Question � � Alienware 15 r3 dead after bios update

Page 10 of 20 � Page 1 ... 5 6 7 8 9 10 11 12 13 14 ... 20 Page �

Please enter a reason for warning.