Re-envisioning Stereo with Neural Networks
Ghost has pioneered a new implementation of stereo perception in the car, utilizing industry-standard camera sensors paired with our KineticFlow visual neural network to deliver real-time, high-reliability distance and velocity measurements for every pixel in a scene. This serves as the foundation of Ghost's perception layer, which combines stereo vision, mono vision and imaging radar to develop scene understanding.
As detailed in other posts, one of the key benefits of this vision-centric perception system is universal obstacle detection – finding and accurately ranging everything in a scene without requiring image recognition, eliminating long-tail recognition errors and the need for lidar.
Upgrading Side Perception with Stereo
We have expanded our stereo implementation to cover the entire scene in every direction, using four separate camera pairs located at the front, rear and each side.
With 360-degree stereo coverage we can now leverage all the benefits of universal obstacle detection and high-reliability ranging to execute a much broader set of driving maneuvers, including changing lanes, navigating cut-ins, and adjusting our lane placement for large or encroaching objects in adjacent lanes. Side-stereo perception also adds significant safety maneuvers, like detecting and avoiding side-swipes or executing a minimal risk condition that safely moves the vehicle to the shoulder in extreme circumstances. In the future this application will enable a host of off-highway capabilities as well, including always-on safety maneuvers on surface streets and parking assistance.
This is an especially novel solution for side perception, delivering high resolution and high reliability in a scalable form factor, overcoming obvious limitations of other alternatives for side-sensing:
- Mono-camera – Unreliable distance estimates which generally rely upon successful object recognition
- Lidar – Expensive, unproven at scale, and typically only forward-facing in consumer auto deployments
- Radar – Unreliable for low/no-speed targets, and most side obstacles are low speed relative to ego car
- Ultrasonics – Low resolution, especially in more complex scenarios
While our forward stereo vision handles both near and far scenarios, the vast majority of maneuvers on the highway only require side ranging to about 8m, enough distance to cover 2 lanes in either direction. At these short distances, our stereo neural networks are highly accurate, with the benefit of never missing an obstacle due to recognition errors.
Turning Stereo on Its Side
We developed a new camera module for stereo side perception to accommodate a new position on the car, altering both the form factor and orientation from our front camera assembly.
- Form Factor – For our side assembly, the stereo camera pair is now 6cm apart (vs. 17cm in the front), improving short range accuracy in a smaller form factor.
- Orientation – The horizontal stereo camera pair from the front is now on turned on its side, utilizing the same stereo geometry but fitting neatly along the vertical geometry of B-pillar. In Ghost's current test vehicles this side camera sits just behind the B-pillar, but fully-integrated OEM implementations can be flexible in integrating the camera assembly into the B-pillar itself, in the side body panel, or in the side mirrors.
For our initial implementation, we did not need to develop and re-train a new stereo model to account for the new vertical disparity form factor, instead opting to apply the horizontal model on transposed images. Our models proved to be highly generalizable, delivering a high accuracy result with minimal changes. This also suggests that we should be able to leverage any stereo model improvements for use in all four directions.
Coming Soon – 360-Degree Stereo Visualizations
Look out for an upcoming post where we stitch it all together. Add your email below to join our mailing list for regular updates on our perception stack and more.
