Navigating San Francisco: A Deep Dive into NVIDIA’s L2++ Autonomous Driving Stack

🚗 What I demonstrated on the streets of San Francisco

I drove an address-to-address route through dense urban streets, highways, interchanges, and suburban blocks with a consumer-grade autonomous driving system that blends classical robotics software and modern end-to-end deep learning. The goal was not a stunt. It was to show a usable, comfortable, safe driving experience in a city that is notoriously difficult for automation: narrow lanes, double-parked vehicles, unpredictable pedestrians, cyclists, and frequent short-range maneuvers.

The platform I used is built on NVIDIA’s Hyperion architecture and runs on a single Drive AGX RN compute module. The vehicle is instrumented with a balanced sensor suite: 10 cameras, 5 radar sensors, and 12 ultrasonics for close-range maneuvers like parking. For this level of functionality—what we call L2++—I intentionally did not rely on LiDAR or preinstalled high-definition maps. That decision makes the solution more cost-effective and portable to mainstream vehicles.

During the drive I demonstrated typical urban challenges: navigating around double-parked cars by borrowing the oncoming lane when necessary, smoothly yielding when a reversing vehicle needed space, detecting and reacting to open vehicle doors, and negotiating unprotected left turns while accounting for pedestrians and oncoming traffic. Throughout, the system kept behavior human-like and predictable, which is critical for acceptance and safety in mixed-traffic environments.

🧠 Two complementary software philosophies working together

The driving stack combines two diverse but complementary approaches: a classical modular stack and an end-to-end neural stack. I use both because they play to different strengths and together they produce safer, more comfortable behavior than either could alone.

The classical stack follows the traditional robotics pipeline. It ingests sensor data, performs perception to detect and classify both dynamic objects—cars, cyclists, pedestrians—and static elements like curbs, trash cans, or construction cones. It builds an online understanding of lane geometry, traffic lights, signs, and rules of the road. Then it predicts how other agents will behave and plans a safe trajectory given the route and rules.

The end-to-end stack is a single large neural model that takes raw sensors and navigation inputs and directly produces trajectories. This model is trained on massive quantities of human driving data and learns driving patterns, timing, and social conventions that are difficult to hand-code. The result is trajectories that feel natural, smooth, and human-like.

At runtime the two systems propose candidate trajectories and a final module chooses the one that best satisfies safety and comfort constraints. In practice the end-to-end model often produces the most natural trajectory while the classical stack provides explicit, interpretable safety checks and rule-based constraints. The blend gives the best of both worlds: human-like behavior with provable guardrails.

🛣️ Handling messy, real-world scenarios

San Francisco is a laboratory of edge cases: double-parked delivery trucks, drivers opening doors, pedestrians stepping into crosswalks mid-block, and sudden cut-ins by other vehicles. I architected the stack to reason through these situations in a way that aligns with how a competent human driver would.

• Double-parked vehicles: The system detects the intent and context—whether a car is actively double parked, loading/unloading, or preparing to re-enter traffic—and computes a safe maneuver, including a controlled borrowing of the oncoming lane when necessary. Borrowing is done conservatively with real-time monitoring of oncoming traffic and an explicit decision to return to the correct lane as soon as conditions permit.
• Reversing vehicles: When another car begins reversing into our lane, the stack yields and gives that vehicle the space it needs to complete the maneuver. Once the situation clears, the system chooses whether to follow or overtake, again guided by comfort and safety metrics.
• Open doors and vulnerable road users: We detect open doors and pedestrians and adjust speed and lateral position to provide a safe margin. The end-to-end policy, trained on human responses to similar occurrences, often produces the smooth, anticipatory adjustments that riders expect.
• Unprotected left turns: These are among the most challenging maneuvers. The stack waits for safe gaps in oncoming traffic and considers pedestrians crossing the intended path. The classical stack explicitly models vehicle right-of-way and stop line behavior, while the learned policy contributes timing and comfort considerations.

These behaviors are not scripted one-off reactions. They are emergent from perception, prediction, rules, and learned human tendencies—combined in a system that continuously evaluates safety and comfort.

🖥️ Simulation and validation at massive scale

Driving safely at scale requires rigorous validation beyond miles driven on public roads. To achieve that, I rely on three simulation and reconstruction pillars: NVIDIA Omniverse, NVIDIA Cosmos, and Omniverse Neural Reconstruction (Nurek).

Omniverse lets me create detailed digital twins of the real world. Because I can represent cameras, radars, and the full physical environment accurately, I can replay sensor streams in simulation and test software changes thousands or millions of times without placing a vehicle on the street.

Cosmos is used to generate variations on those scenes. Small variations—lighting, weather, actor positions—help make sure the system is robust to expected changes. Large variations—different road geometries, novel actor behaviors, or unusual traffic patterns—let me stress-test the stack against rare conditions that are too infrequent to observe in real-world driving alone.

Omniverse Neural Reconstruction (Nurek) is particularly powerful because it converts camera and sensor data from real-world drives into 3D reconstructions. Instead of merely replaying recorded sensor data from the original ego-trajectory, I can render novel viewpoints and modify actors or their behaviors. That means every mile we drive becomes a source of effectively infinite test cases: I can change the timing of a pedestrian crossing, insert an unexpected parked truck, or render the scene from a slightly different vehicle pose to simulate alternative behaviors our software might choose.

Practically, that allows me to validate new software releases across large numbers of realistic, diverse scenarios. Today I can run over one million scene reconstructions and test replays per day, and I plan to scale that number aggressively. This offline, high-fidelity replay capability closes the loop on validation: what was once a single recorded event becomes a family of tests that exercise perception, prediction, planning, and end-to-end learning under controlled variations.

🗺️ Why I do not rely on LiDAR or pre-baked HD maps for L2++

Historically, many autonomous systems have depended on LiDAR and high-definition maps to localize and plan. For true consumer adoption of L2++ across many vehicle models and geographies, that approach poses cost and scalability challenges. I took a different path.

For L2++ I intentionally used a camera-first approach supplemented by radar and ultrasonics, and I build an online HD map in real time from sensor data combined with navigation maps such as Google Maps. Here is why:

• Cost effectiveness: LiDAR and pre-built HD maps add hardware and operational costs that are difficult to amortize across mainstream vehicles. Removing those dependencies makes the technology accessible for a broader segment of the market.
• Geographic scalability: Pre-built HD maps require continuous maintenance as roads change. An online mapping approach creates a fresh, up-to-date scene representation wherever the vehicle goes, which is especially valuable in regions with rapid infrastructure change or limited HD coverage.
• Robustness to unexpected changes: Construction zones, temporary signage, and ad hoc delivery activity are common in urban environments. Building lane geometry, stop lines, and traffic rules at runtime from sensor data allows the system to adapt to temporary deviations from a static map.
• Complementary to navigation maps: I use global navigation maps for route guidance while relying on local, sensor-driven HD map generation for lane-level decisions and traffic-rule associations. The combination keeps the system both globally aware and locally precise.

In practice this means the software recognizes lane lines, determines their connectivity and direction, identifies turn-only lanes, and interprets stop lines, stop signs, and traffic lights. All associations between signs, lanes, and rules are formed on the fly so the vehicle can behave correctly without a preinstalled HD layer.

🎯 Human-like behavior through end-to-end learning

Comfort and predictability are as important as safety. People are more likely to accept autonomous capabilities when the behavior is legible and comfortable. That is where end-to-end learning shines.

I train the end-to-end model on extremely large amounts of human driving data so it internalizes subtle driving conventions: how much gap to leave at an intersection, when to nudge around a parked truck vs wait, how quickly to accelerate out of a turn, and how much lateral offset to give passing cyclists. Those are the sorts of nuanced decisions that are cumbersome to specify with hand-tuned rules.

The end-to-end model produces candidate trajectories that are often the most natural feeling to human passengers. However, because learned models can sometimes generalize unpredictably, I merge their output with the classical stack. The classical stack provides explicable safety checks—explicit collision constraints, adherence to traffic rules when necessary, and interpretable signals when something unexpected happens.

The result is a hybrid control strategy that preserves the smoothness of human-like driving while ensuring that safety constraints are enforced. That blend reduces false alarms, yields fewer abrupt corrections, and delivers a comfortable ride.

🔒 Safety, collaboration, and the always-on approach

Safety is nonnegotiable. For consumer deployments I designed a collaborative driving model that keeps the driver engaged without forcing continual manual control. The system remains engaged and available to assist, even while a driver momentarily overrides its current trajectory.

Here is how the collaboration model works in practice:

• Seamless handover: A driver can take the wheel at any time and nudge the vehicle laterally or change speed. The software continues to run in the background and reasserts control smoothly when the driver lets go. That way the driver never needs to perform a formal re-engagement sequence.
• Continuous assistance: Even while a human is briefly controlling the vehicle, the autonomous system continues to monitor the scene and provide assistance—interpreting intent, warning about hazards, and suggesting maneuvers.
• Conservative borrowing of oncoming lane: When the stack decides to borrow the oncoming lane to pass a double-parked car, it does so with conservative margins and constant re-evaluation. If the driver nudges into that lane, the software will still be ready to assume control and correct if needed.

In short, the system is not a binary “on/off” autopilot. It is an ever-present collaborator that supports a human driver, making shared control natural and safe.

📊 Technical highlights and deployment partners

Key technical and commercial milestones I highlighted during the demonstration include:

• Hyperion reference architecture: One Drive AGX RN compute module, 10 cameras, 5 radars, 12 ultrasonics.
• No LiDAR and no pre-baked HD maps for L2++: Camera-first perception combined with radar and ultrasonics and an online HD mapping layer at runtime.
• Dual-stack software: Classical modular pipeline for perception/prediction/planning plus a large end-to-end neural trajectory model.
• Simulation throughput: Over one million reconstructed scenes and replays per day for offline validation, with plans to scale further.
• Commercial availability: The software is shipping in consumer cars today in the Mercedes-Benz CLA and will be available on platforms from Jaguar Land Rover, Lucid, and Stellantis.

These highlights reflect a balance of advanced research methods, practical deployment engineering, and partnerships with automotive OEMs to bring autonomous functionality into production vehicles.

🌆 What this means for drivers, cities, and mobility

The combination of cost-conscious hardware, mapless operation, and rigorous simulation-based validation has several implications for the near-term future of mobility:

• Broader availability: By removing LiDAR and eliminating the need for prebuilt HD maps at the L2++ level, this approach can be retrofitted or integrated into a wider range of vehicle models at production scale.
• Faster geographic rollout: Online mapping and camera-first perception allow deployments to scale across cities and countries without the overhead of creating and maintaining HD map databases for each region.
• Urban resilience: Cities with frequent temporary changes—construction, event setups, ad hoc delivery activity—benefit because the stack perceives and adapts to the current state of the world rather than relying on stale map data.
• Improved rider experience: The end-to-end model’s human-like behavior helps passengers feel comfortable and confident in the system, which is a key factor for adoption.

From a policy standpoint, this approach also aligns with a phased progression toward higher levels of automation. By deploying L2++ systems that are safe, explainable, and widely accessible, we can responsibly increase automated capabilities while collecting the data and operational experience needed for eventual higher levels of autonomy.

📌 Technical deep dives: perception, prediction, and online HD mapping

To make the system work reliably, each subsystem must meet exacting standards. Here are technical details about the three most critical pieces.

Perception

Perception uses camera and radar fusion to detect and classify actors and static scene elements across a wide field of view. The system recognizes dynamic objects—vehicles, cyclists, pedestrians—and static artifacts like curbs, signposts, and obstacles. Lane-level perception recovers lane markings, turn-only lanes, and stop lines. Because we do not rely on an external HD baseline, perception must be robust to partial or worn lane markings and occlusions from parked vehicles.

Prediction

Prediction models estimate future trajectories of nearby actors using their kinematics, contextual cues, and learned priors about typical behaviors. For example, a pedestrian standing near a curb may be predicted to cross, or a vehicle signaling may be predicted to turn. These predictions feed both the classical planner and the safety evaluations that constrain end-to-end outputs.

Online HD mapping

Rather than a static, cloud-stored HD map, the system builds a lane-level interpretation of the scene live. It associates signs, signals, and rules with lanes. It determines which lanes allow turns and which are one-way. That online map is ephemeral and precise for the current road context, and it is continuously updated as the vehicle moves and new evidence arrives.

🧪 The validation loop: from data collection to deployment

Validation is not a one-off test. It is a continuous loop that starts with data collection on public roads and ends with cautious, validated releases to production vehicles.

1. Data collection: Fleet vehicles gather diverse driving scenarios across seasons, weather, and traffic patterns. Every real-world mile is a valuable sample.
2. Neural reconstruction: Nurek converts these recordings into 3D scenes that can be replayed from arbitrary viewpoints and modified to create additional edge cases.
3. Simulation augmentation: Cosmos and Omniverse generate variations, from subtle sensor noise to major scene edits like added actors or changed lighting.
4. Model training and testing: End-to-end networks receive curated datasets, while classical modules get targeted supervised data for perception, mapping, and prediction.
5. Massive replay testing: The reconstructed scenes are replayed at scale—over a million times per day—validating new software versions across realistic variations.
6. Selective on-road rollout: Only after extensive offline validation and closed-course verification is software released to production vehicles, initially under supervised or limited conditions.

This loop reduces risk by ensuring most failure modes are discovered and mitigated offline before a broad on-road rollout.

🔭 Future directions and L4 readiness

Hyperion is designed to scale. While the current demonstration focuses on L2++—an assisted driving capability that requires driver oversight—the architecture and tooling prepare the system for higher autonomy levels over time.

Key future directions include:

• Scaling simulation: Increasing scene reconstruction throughput and introducing even richer behavioral variations will tighten the validation loop for higher-level autonomy.
• Improved multi-sensor fusion: Continued enhancements to camera-radar fusion will raise robustness in poor visibility and complex interactions.
• Expanded partnerships: Working with more OEMs and vehicle platforms to collect diverse data and enable broader, safer deployments.
• Regulatory alignment: Collaborating with regulators to establish acceptance criteria and deployment guidelines for incremental autonomy.

The path to truly driverless level four capabilities will require both technical progress and operational maturity. The hybrid architecture and the massive simulation and reconstruction capability accelerate that trajectory by letting me validate corner cases that matter most.

🚀 Final summary and availability

What I demonstrated is a practical, deployable L2++ system that handles address-to-address driving across varied urban environments without LiDAR or preinstalled HD maps. It blends classical, interpretable software with end-to-end learned behaviors to produce driving that is both safe and human-like. Robust offline validation powered by Omniverse, Cosmos, and Neural Reconstruction gives me the confidence to scale releases rapidly and conservatively.

The technology is shipping in production vehicles today and will appear on additional OEM platforms in the near term. That combination of commercial availability and rigorous validation marks a notable step toward more capable and widely accessible automated driving.

Driving in San Francisco is still a challenge—but with a sensor-efficient hardware stack, hybrid software architecture, and simulation-driven validation, I believe the right mix of safety, comfort, and scalability is within reach for everyday consumer vehicles.