Correct me if wrong: your issue is that pxe-boot process is not trying/attempted on port net1 because it's stuck at port net0?
Can you going to flexboot menu of good and bad server after reboot by pressing Ctrl-B and share comparison.
You are correct. I'm attempting to pxe boot off of net1 but not because its stuck at net0. I'm required to use net1. But I cannot get passed net0.
I'm unable to get get into the mellanox configuration screen on the bad server since im going through the HP ILO console. I will update the thread if I can.
Here is a screenshot of the good server: