A Day of Reckoning
Sunday Morning: I woke up too early for a Sunday (4:00 am Denver time) with two things in my mind to accomplish: 1) improve the barrel-avoidance code to give the car some more clearance so it does not drag the sides, and 2) add general purpose collision avoidance, so that if it somehow loses its navigation position it will “intelligently” steer away from obstacles.
I code up both of these changes. The course opened for testing at 8:00 am, so I put a report screen together on the LIDAR avoidance and wander around the hallway of the hotel carrying the car, looking at how it sees and reports obstacles like room service trays and housekeeping carts…it worked reasonably well! I took it outside and tried it against a wall in the parking lot; it worked perfectly. Even when commanded to steer into the wall at a 30-degree angle it would skim along just avoiding the wall. It was perfect…
Practice Sunday morning: On the first run the new barrel-avoidance S/W
was running perfectly. It didn’t touch a barrel. So, I took a good manual
course survey with the car and set up to fine-tune the route. Unfortunately, it
started to act highly unreliably. It ran sometimes and wouldn’t run others.
Sunday heat one: The vehicle makes a bad “decision” on how to go around the very first barrel and gets wedged between the barrel and haybale. In past times, this would not have been an issue as it would push the barrel aside a little bit and continue. This run there is so much loose hay on the course that the car just spins its slick race wheels on the loose hay. I had not planned for loose terrain.
So while other heats are running on the course I go out in the big empty parking lot and test. Something is very wrong; it’s gotten so unreliable. It’s even impossible to steer in manual mode…what is going on!!!!???? I unplug and re-plug the steering servo and it comes back to life for only 30 seconds. I do this about three times, and it comes back to life each time and then dies. On the last plug-in, I join the connector off by one pin, put 5V on a 3.3V pin and the CPU power supply dies. Is this story a tragedy or a comedy? Maybe both.
I rush back to my worktable, reach into my spares bag and pull out a new NetBurner DEV board. “Oh no!” Its then I realize that the boards on the cars have header pins soldered in for the connector while the spare board had just thru-holes, so it won’t work. I disassemble the CPU dev board from the white car and put it on the black car. I take my last spare CPU board from the useless DEV board spare, plug it in and remount it. I also remove the steering servo from one car to the other. I’ve got both the steering servo and the dev board swapped and I have 15 minutes of testing before the heats start.
Let’s take a break and talk about why I do this. I think it’s important that a company eat its own dog food. I’m doing all this development on a new NetBurner software version (3.0, alpha release) -- it will be our new multiplatform release, capable of supporting a variety of microprocessors - ColdFire, ARM, etc. I’m using this new code to find bugs and issues, and improve the quality of our new release. This now bites me hard…so much for being a hero.
Here’s what happened: in version 3.0 we changed the way that IP configuration and setup works to be all web-based with no special tools – which is awesome. Because of this, 95% of our testing has been on DHCP based LANS. With an unconfigured direct cable connection the system is supposed to use AutoIP to setup the IP configuration. It has not been tested recently, and for me at the time, it no longer worked. For quality control this is great news and this bug has been reported and fixed. But for me, in the “heat” of the moment, I’m cooked. I can’t talk to the board to download paths, or set parameters, because it has no IP address! I start hacking the system code to force it to have a static IP address.
In hindsight I could have used the local IPV6 configured address and solved my problem, but I did not think about that at the time. It takes me about 15 minutes to hack in a solution and get the board to boot with a static IP. Now they are calling my heat and I’m out of time. I program in the path, the config settings and run to the race track. The car has not even run 1 inch since I swapped the steer servo and the brains. I’m hoping the servo center is close enough that the steer PID loop can correct.
Sunday Heat 2: The run is almost perfect. The car goes through the barrels and doesn’t touch a thing. It just misses the hoop by about 3 inches (no points there), but it detects the pedestrian, and does an absolutely perfect stop at the stop sign, waits for the path to be clear and continues…BINGO!!!! It drives all the way around the course and is looking great. It approaches the finish line and… And!… AND!!!????… stops 2” short.
What happened?! The car had been programmed to go about 3 feet past the finish line. Alas, with all the loose hay on the course it got 39” of wheel slip and was 3” short. If it had gone 4” farther it would have been the high-scoring run of the entire event. This was just crushingly demoralizing…an EPIC upset.
Photo credit: myromancewithrunning.com. We hope she's fully recoved. We know how that feels, now... at least from a robotics perspective.
Between heat 2 and heat 3: I extended the stop zone by 8 feet and did a practice run. It’s perfect. I turned the speed up 50% and ran it again, it’s perfect. I parked the car and spend the next hour packing up all my stuff into the SUV and getting ready to go home. I don’t touch the car. Now, with everything packed in the car, the only thing left to do is run the last heat and go home and finally relax. Having not finished the first two heats I’m no longer in the running to place. I’m half-tempted to head out before running the last heat. I stay.
Sunday Heat 3: I press the GO button; the car accelerates hard about 30 feet and then just stops. When I walk up to the car the speed controller is flashing a red fault light, either it got too hot (despite being cool to the touch) or I ran the car battery down too far (more likely). Either way, my AVC 2017 is finally done and despite my numerous failures and SNAFUS I have won 2nd place. So much for regaining the championship – but also inspiration for a rematch.
Final thoughts: In the end it did not go as well as I would have liked. I had not expected to have such an intensely stressful and exciting experience either. On six official attempts, I only completed one lap.
- The car was too wide for how tight the barrels were. The barrels were much tighter this AVC than previous AVC’s. I chose the chassis to be able to go insanely fast therefore it had a wide wheelbase, slick race tires and a low profile. Instead, I needed a narrower, more-rugged, and high-clearance chassis. The slick tires just don’t work well in straw. The car would be superior on smooth dry, solid pavement, but not so much on the AVC course. In future AVCs I’ll most likely pick a monster truck chassis and tires.
- The mechanical servo failure was probably the worst issue as it took a lot of troubleshooting and trial and error to narrow down. When I turned the steer servo gain way up to better handle the barrels section I was driving the steer servo too hard. It was overheating and shutting down. A normal RC driver does not continuously drive the servo stop-to-stop that hard and for that long. When in barrel-avoid mode, I was doing just that, and the servo overheated and died. In the hyper-sensitive obstacle avoidance algorithms, I only made this problem worse since under that regime the car might change its steering direction mind significantly while at 50Hz.
- Here's a big one… I was not 100% ready. Ideally, I would have completed all of the bits of software before left for Denver and tested the barrel mode enough to show the steering servo flaw. I should have had my spares all at hand, had a spare for ALL 3D printed parts, and had the wiring done on Car #2.
- A bunch of things worked really well, the RC system, the gyros in the IMU and the two LIDARS were flawless. And not to brag… the NetBurner NANO 54415 CPU hardware was great. Besides my frying it accidentally, it performed exceptionally well as the brains, providing real-time sensing and communication as well as processing control and navigation software.
- I had a lot going on and in the heat of troubleshooting I caused unnecessary damage to the NANO544115 Core Module (e.g. I fried it accidentally during a misplaced cable-swap). Adding insult to injury, when I replaced the NANO with a spare, I had forgotten it was running the version 3.0 alpha NetBurner software (under testing prior to public release). That backfired, as I discovered a bug that slowed me down at a point when I had no time to lose. Valuable debugging on a company level but not great for me on the racetrack.
- Bring spares of everything! One of my six custom 3D printed parts could have been designed stronger. But even worse, after it snapped from a collision on the track, I didn’t have a spare! This was a considerable set back. And make sure all spare items can hot swap… my NANO CPU spare should have been prepped, preconfigured and tested ahead of time.
- Work out a design to compensate for wheel spinning on loose hay, terrain or even mid-air after a ramp jump. These events threw off my path measurements which tied in odometer readings; which in these cases had erroneous travel measurements.
- Projects with a hard, unmoving deadline are painful and hard to coast through. Allocate extra time – start earlier.
- Prepare for your contingencies and worst-case scenarios.
- Roll with the punches and try to learn from and even enjoy failures.
We hoped you enjoyed this epic story of trials and tribulations at the Sparkfun AVC 2017 race track. Perhaps we’ll see you there next year? Keep tuned for more posts on some of the more technical details and how-to’s from our AVC. Please provide us with your comments and questions below.