Zitat von XOIIO im Beitrag #120

---

Hi all, so the video is done and available here if you want to watch for the heck of it, I ended up removing the EMMC chip after, I did damage one pad but it appears to be an unused one, and there are a surprising number of unused pads on the 153 ball chips.

https://www.youtube.com/watch?v=kvSDNAi39YY

I'm going to order a parallel style memory chip for the heck of it as it's only $9, and see if that works and update you all, I'm waiting on digikey to get back to me as I can't add those newer chips to my cart for some reason, but if the TSOP package memory works that's a really good alternative as the BGA is definitely going to be tricky to line up thanks to the solder mask being totally white around it, and it will be easier for people to solder.

Reflowing unfortunately was of no help, and wow this board really sucks up the heat, I had to set my station to 400, I'm going to remove the motherboard to see if there's anything sinking heat away to the chassis underneath but I think it's just a really thick board with multiple ground planes inside it which really pulls heat away.

I'll make videos covering further steps on the process to when I get the parts and tinker some more. I also have an idea for using an arduino to just do the steps of interrupting u-boot and then sending the command to set the fan speeds every time the server powers up, which is less than ideal, but is an option.

---

if we leave the arduino in place, could we make the arduino load the idrac firmware into memory from TFTP? that way at least we have a temporary idrac running.
the reason for this is that then reboots would not take 20 minutes
the only time it would take 20 minutes again is if the arduino has the load the idrac firmware again, which I think would only happen on a *hard* power cut? regular reboots/shutdowns should be fine (idrac would continue to run even with the server shut down)

^^this is if i understand idrac correctly

Yeah, the 20 minute reboot is why I really would want to fix IDRAC properly, with that long of a reboot using this r420 I have as a router is really not feasible.

I have not yet used any ethernet add on cards with an arduino but I did find an FTP server project with a quick google, so it should be doable, you would simply loose port 1 of your motherboard since it would be dedicated to the arduino's ethernet connection. Assuming it hosts the file fine and u-boot has no problem connecting then the arduino could interrupt the boot process and send all the commands to load from ftp when everything powers up.

It would definitely need testing to see how smooth it all goes, in terms of timing for things like the ftp server to get going and for the file to download to the server but it should totally be doable.


One thing is that you don't need to power on the server at all as u-boot starts as soon as the board gets power so you need to find a spot to power the arduino from that's a standby rail. Arduinos take ~8 seconds to fully boot iirc, so that needs to be factored in in terms of everything getting going but after that it could immediately bull the SW2 contact to ground to trip it and start the process.

Alright, so gave it a shot using the TSOP package, TDLR it didn't work but I got some new output on my serial console. I'm not sure if there's a way to change the target from e-mmc to u-nand for IDRAC but if you can then I think using the tsop instead might be a viable option. Based on how it's in the same spot as the EMMC it seems like the idrac system would be able to access it too.

https://pastebin.com/uLXPwQPP

https://www.youtube.com/watch?v=V0l8mT_ZsRA

edit: I was super freaking tired when I was doing this, and I can't recall if I tried manually typing in the recover command specifying the nand flash after I finished recording actually, so once I get some sleep, or copious amounts of caffeine I'll give that a shot. Maybe even if I couldn't spot a command to check the u_nand chip in idrac, manually specifying it for recovery might work.

Of course thanks to the clunky iron tip I have there might be some poor solder joints or something which is why I was hoping to probe the size of the nand flash somehow, but that might also be something I missed due to being up for, well I don't even know right now, 20 ish hours now with only a few hours of sleep before.

edit 2: shit, yeah flinfo might be what I needed to use, I think when I tried erasing flash with the EMMC chip before that failed. I'm going to wait before trying out more though since I'm liable to mess up something right now.

Hello everyone, I have a similar problem on my dell T620 server, message "Initializing iDRAC", I want to try to start it and update it from an SD memory, but can someone tell me what type of memory it is exactly, I mean (size, model)? with a serious image very helpful
Or if anyone can help me, I am desperate, thank you very much.

hey guys I am also having similar problem with idrac on my r720. I can't seem to be able to stop uboot and the uart goes into a linux mode. I was wondering if there was a way from there to try to flush the emmc.
This is the bulk of my error:
WDT2: Disabled
Initialization complete, proceed with boot
OS_WDT0_EN:180 seconds, Re-load ILRAM:
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image ... OK
## Booting kernel from Legacy Image at 88000000 ...
Image Name: sh-linux
Image Type: SuperH Linux Kernel Image (gzip compressed)
Data Size: 2537538 Bytes = 2.4 MB
Load Address: 80001000
Entry Point: 80002000
Verifying Checksum ... OK
Uncompressing Kernel Image ... OK
WDT2: Disable in do_bootm_linux()
sh_eth_init

Monolithic/DRB
init started: BusyBox v1.18.4 (2012-11-04 19:32:12 CST)
starting pid 407, tty '': '/etc/init.d/rcS'
waitfor: pid 407 ...
setting net.ipv6.conf.default.accept_dad based on kernel bootup option
enable normal accept DAD for ipv6
Sat Jan 1 00:00:01 UTC 2000 To execute command __source /etc/sysapps_script/I_1000_mountfs.sh__
Sat Jan 1 00:00:02 UTC 2000 To execute command __source /etc/sysapps_script/I_1001_trigwdt.sh__
<========= Start H/W watchdog first... =========>
Sat Jan 1 00:00:02 UTC 2000 To execute command __source /etc/sysapps_script/I_1100_nfs.sh__
Sat Jan 1 00:00:02 UTC 2000 To execute command __source /etc/sysapps_script/I_1105_ps.sh__
XXXXXX /dev/mmcblk0p11
/dev/mmcblk0p11: recovering journal
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to re-open /dev/mmcblk0p11
e2fsck: io manager magic bad!
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open /dev/mmcblk0p11
Could this be a zero-length partition?
mount: mounting /dev/mmcblk0p11 on /flash/data0 failed: Invalid argument
mounting private storage fail, formatting partition
mke2fs 1.41.14 (22-Dec-2010)
Warning: could not erase sector 2: Attempt to write block from filesystem resulted in short write
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
1024 inodes, 4096 blocks
204 blocks (4.98%) reserved for the super user
First data block=1
Maximum filesystem blocks=4194304
1 block group
8192 blocks per group, 8192 fragments per group
1024 inodes per group

Hey guys,
I managed to interrupt uboot and trying to boot from tftp on my r720.

I manage to download the firmimg.d7 from tftpboot, but when I try to run:
go addr 0x81000000
Uboot restarts and it loads the old firmware therefore failing.
Any suggestions on how I can successfully boot from memory after I download the firmware via tftp?

Sorry dup entry

I have 2x R630 motherboard with iDRAC error. =(
I can access iDAR UART, but cant interrupt uBOOT process (upgraded to latest iDRAC version 2.70.70).

Any help are welcome.

Motherboard 1 log:

To interrupt boot, you must to shortcut 2 pins on mb. I soldered there 2 wires on made a "jumper". Read this thread, there was instructions.

Zitat von XOIIO im Beitrag #124

---

Alright, so gave it a shot using the TSOP package, TDLR it didn't work but I got some new output on my serial console. I'm not sure if there's a way to change the target from e-mmc to u-nand for IDRAC but if you can then I think using the tsop instead might be a viable option. Based on how it's in the same spot as the EMMC it seems like the idrac system would be able to access it too.

https://pastebin.com/uLXPwQPP

https://www.youtube.com/watch?v=V0l8mT_ZsRA

edit: I was super freaking tired when I was doing this, and I can't recall if I tried manually typing in the recover command specifying the nand flash after I finished recording actually, so once I get some sleep, or copious amounts of caffeine I'll give that a shot. Maybe even if I couldn't spot a command to check the u_nand chip in idrac, manually specifying it for recovery might work.

Of course thanks to the clunky iron tip I have there might be some poor solder joints or something which is why I was hoping to probe the size of the nand flash somehow, but that might also be something I missed due to being up for, well I don't even know right now, 20 ish hours now with only a few hours of sleep before.

edit 2: shit, yeah flinfo might be what I needed to use, I think when I tried erasing flash with the EMMC chip before that failed. I'm going to wait before trying out more though since I'm liable to mess up something right now.

---

Thank-you to everyone in this thread, I have been able to recover the iDRAC on my R420 to 1.66.65 after a bad update to 2.21.21 from 1.57.57. In my case the EMMC was working fine it was just complaining that it couldn't contact the BMC.

I soldered pins onto the BMC_UART and used a TTL to USB adapter to connect at 115200. I was able to reboot the idrac just by tapping the "ON" pin of SW2 pair 1. THen I held a bent paperclip to pair 2 of SW2 and rebooted the idrac and watched for "press any key" to interrupt the boot. This left me at a IDRAC8=> prompt which seemed wrong.

I then used the info in @willard post on page 4 to recover from 1.66.65 firmimg.d7 on the SD card with `util recover -emmc -from sd -f firmimg.d7`

I noticed that the U-Boot header changed from
U-Boot 2009.08 (Nov 04 2015 - 11:28:55) Avocent (0.0.3) EVB
to
U-Boot 2009.08-00088-g121cddc (Nov 17 2014 - 05:50:46) Avocent (0.0.3) EVB, Build: jenkins-idrac-yocto-release-505

Here's a pastebin of my terminal log, starting with a reboot when I missed jumping pair 2 of SW2 (you can see that it booted completely) followed by a second boot where I got the "Hit any key to stop autoboot" and hit a key in time.

Then you'll see a few failed attempts to tftp, eventually just going with the `util recover` command.

pastebin.com HUwLWjbp

I'm not sure why the tftp failed, it succeeded in a previous attempt. On that successful attempt, when I typed the `go 0x81000000` command from @ldv post on page 6, it immediately booted the emmc firmware, not the one downloaded from tftp. I never got to the bootm step.

I'm even more grateful because this is my second motherboard. The first one I ran a firmware_erase_all command of some sort thinking I could erase the mmc and trigger the recovery procedure, but it apparently deleted U-Boot! That motherboard doesn't do anything now. I think I'll have to flash a chip somehow, I don't have those tools or skills though.

I also wanted to mention that the pins I soldered onto the motherboard were pulled from a junk system's i/o header (the pins that connects to cables that run up to the power button, led, etc.) This worked because my ttl-usb had compatible ends on it already and I just pushed them on.

imgur.com /a/bLmA3K4

Note: I was following a post titled "iDRAC7 Upgrade Path" on the Dell Community forums to do the updates.
1. BIOS 2.0.21
2. iDrac 1.57.57
3. BIOS 2.1.3
4. iDrac w/ Lifecycle Controller 2.21.21 (the updates merged after this point)
5. BIOS 2.4.2
6. iDrac w/ LCC 2.40.40
7. BIOS 2.5.1
8. iDrac w. LCC 2.63.60.62. (latest)
9. BIOS 2.6.0 (latest)

I was successful up to step 3, and step 4 is when it failed. Maybe I needed to goto 1.66.65 first, or only go to 2.10.10 before 2.21.21. Now that I look at it, I think the power only lists the "Urgent" updates.

Update: I've had to do this a few more times, I can't seem to get 2.10 installed. Updated BIOS to 2.3.3 and 2.4.2 and iDRAC with LifeCycle Controller still fails.

I've also noticed that the SD card can't be full size as that causes the util recover to fail to find firmimg.d7.

DISKPART> list disk
  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          931 GB  1024 KB        *
  Disk 1    Online           14 GB      0 B
DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> clean
DiskPart succeeded in cleaning the disk.
DISKPART> create partition primary size=1000
DiskPart succeeded in creating the specified partition.
DISKPART> format fs=fat32 quick
  100 percent completed
DiskPart successfully formatted the volume.
DISKPART> assign letter d
DiskPart successfully assigned the drive letter or mount point.
DISKPART> list volume
  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0         Windows RE   NTFS   Partition    499 MB  Healthy
  Volume 1     C   OSDisk       NTFS   Partition    929 GB  Healthy    Boot
  Volume 2                      NTFS   Partition    552 MB  Healthy
  Volume 3                      FAT32  Partition    512 MB  Healthy    System
* Volume 4     D                FAT32  Removable   1000 MB  Healthy
DISKPART> exit
Leaving DiskPart...
 
 

I'm unable to get any v2 release to work on my motherboards. Each one is unable to communicate with the rest of the system with v2. It can't read the service tag, temperatures, fans, or anything else. v1.66.65.65 is the last version that works for me.

I especially dislike v2.63 because it (or a predecessor) removes the interactive shell on the serial console (I believe that was done to address a CVE)

It seems that all my hardware is ok, the BIOS chip and the eMMC are functioning.

If I had to do another R420 recovery:
1. Solder the motherboard
- imgur.com a/QWj7GN8
1a. Pins for the serial console
- I get these pins by removing them from old hardware
1b. Solder a bridge across SW2.2 (Even when the system is working this only adds 3 seconds to idrac boot time). I used a 8 inch length of stripped Cat 5 with the end bent to the width of the SW2.2 solder pads. Tinned and then soldered, then the excess trimmed off.
3. Connect TTL Serial to USB and identify the com port via Device Manager
- (Mine is an old fake PL2303A and Device manager lists it as "PL2303HXA phased out since 2012 please contact your supplier" and I found v3.3.2.105 that works in a file called "IO-Cable_PL-2303_Drivers-Generic_Windows_PL2303_Prolific.zip". I need to "update" to the old driver every boot.)
4. Launch PuTTY and connect to the serial port at 115200.
5. Connect power and monitor serial for data (trial/error to get tx/rx/gnd correct)
5a. If the server auto powers on, hold the power button until it turns off.
6. Review the serial output for signs of hardware failure, I/O errors etc.
7. If the IDRAC o/s boots, see if I get a console, otherwise goto 11 below.
8. Erase the eMMC from the console with `dd if=/dev/zero of=/dev/mmcblk0 bs=1M`
8a. Hopefully no errors, otherwise eMMC has likely failed.
8b. The dd command will eventually trigger a reboot into recovery mode
9. Recovery mode will watch for an SD Card with firmimg.d7 extracted from 1.66.65.65 on it. But the volume on the SD card can't be too big. i.e. It won't read from a 16-GB volume.
9a. Monitor the recovery for write failures to the eMMC, those will indicate it's probably dead, but you can still run iDRAC from a tftp download, but that's something you need to do every power cycle.
10. If the write is successful it's probably be good with 1.66.65.65.
11. If the serial doesn't give a console prompt, then I'll disconnect power and get out the CH341A. Using "CH341A Programmer v1.34", write a 1.66.65.65 U-Boot mega.nz folder/aQc3GIhI#Rgvu-_2ZPpPfUOEIoYanGg
12. Connect power and watch for "press a key to interrupt autoboot". If you miss this I think U-Boot will update itself from the EMMC and you won't be able to interrupt it again and make you go back to step 11.
13. Interrupting U-Boot should leave you at an IDRAC7=> or IDRAC8=? prompt. With the same SD card as step 9 inserted;
13a. iDRAC8=> protect off all
13b. iDRAC8=> util recover -emmc -from sd -f firmimg.d7
14. If the eMMC is functioning, that should get 1.66.65.65 onto the eMMC and you should be good with 1.66.65.65.

Server Information
Power State: OFF
System Model: PowerEdge R420
System Revision: I
System Host Name: HV-C1C6CZ1
Operating System:
Operating System Version:
Service Tag: C1C6CZ1
Express Service Code: 26202306349
BIOS Version: 2.4.2
Firmware Version: 1.66.65 (Build 07)
IP Address(es): 192.168.137.198
iDRAC MAC Address: 90:B1:1C:35:19:86
DNS Domain Name:
Lifecycle Controller Firmware: 2.63.60.62 (!)
IDSDM Firmware Version: N/A

I've been unable to downgrade the Lifecycle controller to a matching version, e.g. 1.4.2

I've done this with two R420 motherboards with functioning eMMC, and the thread has instructions on how to tftp boot the iDRAC if the eMMC has failed.

My guess is that there's some other component on my motherboards that need to be upgraded before iDRAC v2 will work. It's not the BIOS as I'm on a fairly recent BIOS 2.4.2

Just for fun I downloaded the Platform Specific Bootable ISO from here dell.com support/article/en-ca/sln296511/update-poweredge-servers-with-platform-specific-bootable-iso?lang=en for the R420 dell.com support/home/en-ca/drivers/driversdetails?driverId=MRRRF&oscode=naa&productcode=poweredge-R420

It updated my NetXtreme BCM5720 to 21.40.21 from 7.2.0, then the BIOS to 2.9.0 from 2.4.2 and then iDRAC-with-Lifecycle-Controller to 2.65.65.65, and of course, broke idrac.

My first recovery attempt was to login to the web console (which shows null for the platform) but that just crashed/restarted iDRAC (possibly triggered a watchdog? It's very slow) or I didn't wait long enough for the system to stabilize. I've also seen iDRAC reject passwords if I tried to login too quick.

I didn't lose my Enterprise license this time and I was able to ssh into idrac and use the `racadm fwupdate -g -u -a 192.168.137.1` command to downgrade to 1.66.65.65. (I was using tftpd32 as my tftp server, not sure what's up with Chrome and Firefox saying the site has malware, maybe because rogue dhcp servers are a problem?)

ssh log
pastebin.com embed_js/7u11E6dB

and the serial console log
pastebin.com embed_js/akkf06Y9

So if someone else "bricks" their idrac, see if it's still getting an IP address and try ssh & tftp, or http, to do a downgrade.

That was a fairly easy downgrade and it looks like everything is working. My main reason for v2 was HTML virtual console, but I can live with gist.github .com xbb/4fd651c2493ad9284dbcb827dc8886d6

I wonder what component is stopping iDRAC v2 from working?

Edit: The platform specific bootable iso didn't actually update the bios. It added a job, which ran after I completed the downgrade and rebooted.

Some one tried to connect to idrac using USB?
https://www.dell.com/support/manuals/en-...cb93&lang=en-us

https://www.manualslib.com/manual/124429...8.html?page=269

or this way
https://github.com/kamermans/docker-openmanage/issues/4