During a project recently, we discovered a switch that was part of a stack that had failed its way all the way to ROMMON. Upon issuing the ‘boot’ statement on the offending switch we watched it rebooting a number of times and by scouring the console messages we were able to discover what was going on.

Apparently, during PnP discovery with DNA Center a command was issued to this switch stack that it didn’t like, causing this Catalyst 9300 switch to experience a “bulk-sync failure”. Or, in other words, the active switch was trying to give the secondary switch its configuration and it was experiencing an issue, causing the secondary switch to reboot in an attempt to recover. After enough reboot cycles, the switch will decide that there’s a problem it can’t solve by rebooting and simply fail back to ROMMON.

The exact lines were:

The solution is fairly simple but not exactly intuitive. Run the command “show redundancy config-sync failures prc” on the non-offending switch. You should receive the rogue command in the output.

Do a ‘show run’ and search for failed command and remove it. If you’re coming across this exact bug and trying to resolve it, this particular command is located under the “archive” command.

The last step is to simply save the configuration and allow the other switch to fully boot.