On Saturday, October 29th at the PNW Block Party Offseason Competition, FRC 7461 experienced a major failure of the Power Distribution Hub. This was the leading cause of FRC 7461’s sub-optimal performance.
This report explains the events that took place, provides tangible action items to avoid such events in the future and concludes the root cause of the issue.
Analysis
To understand the root cause of PDH failure, it is important to understand the electrical system architecture. Power and Ground wires from the battery route into the PDH. The power wire is spliced by the Circuit Breaker while the ground wire is directly routed into the PDH. During battery replacement, both wires are tensioned. The power wire is mounted on the circuit breaker rendering it immune to said tension. However, the ground wire places direct tension onto the PDH port. Previous generation distribution panels (CTRE PDP) use a nut and washer assembly to mount the ground wire which is far more resilient to external forces. The PDH, in contrast, used WAGO connectors which function based on metallic cage clamps exerting force onto the wires. This system does not have a bolt preventing unsanctioned removal. Over multiple battery replacements, the ground wire can be slipped out of the port.
This was the most likely cause. The lack of thorough electrical inspections did not help in any regard. It is likely the case that the robot entered Block Party with loose wires.
Additionally, after the practice match, or quals 1, the battery replacement may have been aggressive causing the wires to further shift out of the WAGO connectors.
By match 8, the robot was experiencing issues with the breaker tripping. The likely cause was a minor short circuit between the power and ground wires. It was small enough to be undetectable but large enough to cause the breaker to trip.
Breakers are designed so that the threshold for tripping lowers each time it trips. For this reason, breakers tripped in subsequent matches.
The pit crew replaced the breaker before match 19 but did not inspect the PDH directly after. The power wire may have slipped out further in the instance. In match 19, the robot was heavily defended by 4911. During this, 4911 slipped under the robot and caused the ground wire to shift more causing a large short circuit. This melted through other power wires causing a fire.
After this match, Pit Crew replaced the wiring but failed to inspect the PDH due to time constraints and the fact that the PDH was buried underwires. The Pit Crew failed to notice molten residue left inside the port and instead mounted the wires back into the WAGO port. This meant that the ground wire did not have a proper connection to the PDH and was not mounted. over the next two matches, the robot experienced power loss as the wire shifted from contact.
In match 27, the wire likely caused a short circuit that melted through the plastic casing on the WAGO connections.
Oversights
Design
-
While some electrical considerations were taken (placement of electrical components, and batteries), the wiring was not thoroughly planned out
-
Components on the electrical board were not placed until the design had been finalized.
-
The electrical subteam was not kept in the loop during the design process (members were absent).
-
Bellypan and electrical board were made of flex poly making damage to the electrical system during heavy defense easy.
-
Bellypan was smoked poly so it had to be removed for every inspection leading to hesitation/time delay for full systems check.
Mechanical/Pit Crew
-
During battery mounting, stress is placed on the connections to the PDH.
-
Electrical fixes were not necessarily prioritized in pits.
Electrical
-
Wiring was not planned well (in conjunction with design).
-
Not all electrical cables were tensioned.
-
Power and Ground wiring were not expected to slip — no effort was made to tension them.
-
Lack of cohesive systems check.
-
Lack of frequent systems checks.
-
PDH breaker and CAN layout not recorded or updated.
-
Wire- protection was not used as extensively as needed.
-
Hot glue for mounting was not used extensively.
Organization
-
Electrical was given minimal time during robot wiring.
-
The electrical subteam has only 3 official members.
Drive Team/Programming
-
No cohesive drive or testing logs led to a lack of contextual data.
-
Battery mounting
Business
- Electrical was not necessarily prioritized in the budget leading to a lack of spares.
Recommended Optimizations
Design
-
Components need to be placed on the CAD.
-
The wiring needs to be taken into consideration.
-
The electrical board needs to be machined
-
The electrical system needs to be better protected (Rigid Poly or integrated electrical)
-
if a belly pan is used, it needs to be transparent.
-
Prebuilt electrical board and drivetrain
Electrical
-
Plan electrical in advance in conjunction with the design
-
Tension all cables
-
Make a cohesive electrical inspection checklist
-
Inspect electrical frequently and between every match
-
Protect all cables
-
Hot glue all connections
-
Record circuit breaker, ports, and CAN layout
Pit Crew/Drive Team/Programming
-
Better battery mounting procedure
-
Prioritize Electrical in pits
Conclusion
This report concludes that the direct cause of PDH failure was due to a combination of gradual degradation of ground wire connection integrity, along with heavy shock and stress experienced during PNW Block Party 2022 Match 19 8. Further failure was caused by the lack of time, organization, and extensive inspection procedures in the pit leading to greater damage in the PDH.
It also concludes that the indirect cause of PDH failure is poor electrical considerations in design, improper battery mounting procedures, and lack of extensive inspection procedures.
This report details recommended optimizations to multiple processes to prevent both the direct and indirect causes of PDH failure.