To use DOS-based flash tools, you must boot in BIOS mode with CSM (and OPROMs too, I think) enabled. If you’re booting without a CSM, use an EFI shell with the EFI flash executables.

Number one cause of random hard crashes/hangs is RAM. Re-seat it, replace it, down-clock it, run a single stick, do everything you can to either rule it out as a problem, or to isolate the problem to a particular module or channel.

Maybe it’s clear this way:

For every 2 lanes you want allocated to the PCIe slot (up to 4), you lose two SATA lanes. Since there are 8 lanes total, but 12 possible lane destinations, they pre-made combinations of destinations that they think would be useful:

• All 8 lanes to SATA, 4 onboard and 4 through MiniSAS
• 2 lanes to PCIe and 6 to SATA, 2 onboard and 4 through MiniSAS
• 4 lanes to PCIe and 4 to SATA, either activating the 4 onboard ports or the MiniSAS (but not both)

Commscope/Ruckus/Brocade ICX7650-48ZP will do it. Be prepared for sticker shock.

Make the 10.0.0._ addresses loopback addresses, and do point-to-point connections from each box to every other box. No idea how to do this in ESXi, but it’s straightforward in *nix/BSD.

CPU1 handles almost everything about being a normal computer: booting, chipset, most of the I/O, etc. CPU2 is along for the ride and handles its own I/O lanes (PCIe) and whatever work the kernel wants to send to it. The load is not symmetrical, so if you have turbo enabled, CPU1 will be consistently boosting more than CPU2 as it is handling all of its tasks —> warmer CPU1. This is why “tandem” dual-CPU setups have CPU1 upstream in airflow from CPU2.

2667v2 and the 2697/2696v2 are really tops for this generation.

You could desolder it and solder a new one on, or possibly even solder one on top of the existing LED. Same as replacing any other on-board component.

In that case, just copy someone else’s homework. Look up what Supermicro is using for wattage in 1U non-GPU servers, and use those numbers.

You’ll definitely need something with fast PCIe lanes for NVMe. Something with either PCIe 4.0 x4 coupled with a very fast SSD, or something with a lot of PCIe 3.0 lanes.

Is your RAM on the QVL? Ryzen’s notorious pickiness about RAM carries over to TR and EPYC, too. One of the first things before POST and BIOS splash display is memory training. If it can’t get past that, something about memory needs adjustment. Have you tried downclocking it?

How close do you want to get? Budgeting about 200W per socket for “normal”-ish CPUs and 400-450W per socket for latest EPYC should get you in the right range.

I’ve had drive failures bring down entire systems. Replace sda and see if the problems continue.

E3-1230v2 is completely different to an E5v2 or an E5v3. The E3’s were all 4-core dies. E5’s were built from two different dies, both of which had well more than 4 cores. The chipset is different between E3 and E5, the memory controller is different, and the PCIe lane count is different. You can’t directly compare an E3 to an E5.

Idle power can be estimated (CPU+RAM+chipset+drives+GPU), but is also majorly affected by:

• Efficiency of the PSU: big OEMs were pretty good about putting efficient PSUs in their workstation products, but below 10% of rated load capacity, efficiency will be crap. Exactly how crap, depends on the specific PSU.
• BIOS settings: there are a dozen different BIOS settings that can dramatically change how the CPU behaves, and the defaults vary for each system. Sometimes the defaults do not allow the processor to throttle all the way back under low load.

Your measured power consumption of 100W at low/no load is about what I’d expect. Can you reach lower? Maybe with the right combination of settings, and switching to slower/lower voltage memory, and making sure that the GPU is also throttling down, you could reach 65-80W idle. But I wouldn’t expect less than that.

Tried a different cable, just in case you’re capping out at 100Mbps?

Which protocol are you using to test file transfer? If it’s anything involving encryption, that CPU will be hurt real bad, with no AES-NE.

Xeon E3’s are unbuffered DIMMs only.

Based on chip family, this is the only one: https://www.gigabyte.com/Enterprise/Server-Motherboard/MX33-BS0-rev-1x

Get an ATX case with lots of drive bays, a SAS HBA (and SAS expander if you want lots of drives), cheap mobo/CPU/RAM and OS of your choice. Spin it up, play with it, settle on an OS you like, and upgrade components as needed.

OPNsense, vyos, pfSense, TNSR. TNSR is extremely fast at routing, with some stringent hardware requirements. vyos is Linux-based and very fast at routing virtualized in KVM. The *senses are FreeBSD-based and have their quirks, but if all of your routing is ~Gbit symmetrical, you should be fine.