This SAE Recommended Practice provides DA metrics used to quantify the DDT performance of ADS-operated vehicles.3 Here, the primary focus is on the safety-related DDT performance and includes definitions, taxonomy, characteristics, and usage (along with alternatives) for each metric. DDT performance is a subset of overall operational performance of ADS-operated vehicles. Thus, assessments of DDT Fallback [1], cybersecurity, maintenance, interactions with passengers, etc., while important and could have an indirect impact on the DDT, are out of scope for this document. Note that the DA metrics do not specify the actions and/or maneuvers to be executed by the (ADS-operated) subject vehicle (SV).
While this document presents a set of individual DA metrics, it is important to note that it is out of the scope of this document to describe how these metrics should be applied in practice. This is because the overall context of the scenario or deployment must be considered during DA metrics implementation and different DA metrics could be applicable for different ADS applications. Implementation of the DA metrics recommended in this document may inform a safety assessment of an ADS’s performance of the DDT, and this assessment in turn may inform a broader safety assessment of an ADS-operated vehicle. However, these metrics could be supplemented by additional input for both types of assessments.
A literature review of DA metrics that have been proposed and, in some cases, used in previous studies was conducted. From this literature review, a set of DA metrics has been created with example usage. Evidence of prior research indicating that a metric has a meaningful relationship with safety outcome(s) is included where applicable.4 The set of DA metrics is not necessarily comprehensive, and there may be additional metrics that can be used to assess DDT performance, particularly metrics that measure ADS subsystem performance.5
DA metrics might use parameterized variables, assumptions, and thresholds6 in their formulation. Defining or recommending values for these quantities used in the metrics formulation is outside the scope of this document. It is expected that the methodologies for collecting and assigning such values for these metrics will be developed based on implementation of the metrics in this document in field and other learned experiences with production and prototype vehicles and systems. Values could also be established by regulation, industry best practices, improved technology development, and academic research, among others, and may change based on the operating conditions within the operational design domain (ODD). In the interim, it will be the responsibility of the implementer of these metrics to choose values for these quantities.
The DA metrics can be implemented in both testing and commercialization phases and for various purposes, including for ADS-equipped vehicle development, third-party evaluation, and event reconstruction and analysis. For example, the metrics can be used as part of an ADS-equipped vehicle verification and validation (V&V) scenario-based testing process wherein individual scenarios are tested. A third-party evaluator could also aggregate measurements of the metrics measurements over a period of time or number of scenarios (i.e., a period of time could be considered to be a sequence of scenarios) in on-road operation (which could be during testing or commercialization) in order to monitor driving performance of an ADS-operated vehicle.
As development of Automated Driving System (ADS) technologies progresses, the ability to assess the dynamic driving task (DDT)1 performance of an on-road,2 ADS-operated vehicle is critical. Further, this DDT assessment (DA) should be quantified using well-defined DA metrics. Establishing a set of DA metrics can help enable stakeholders, including the general public, to quantitatively understand various aspects of how well (i.e., the degree of competence with which) an ADS is executing the DDT, including its safety performance. Different stakeholders may use the DA metrics for various purposes, such as ADS-equipped vehicle development, third-party evaluation, and event reconstruction and analysis.
To address the need for DA metrics, this document provides a set of metrics, including definitions, taxonomy, and characteristics, to assess ADS DDT performance. The usage of the DA metrics is explicitly described, including examples, with alternative usages described as well.
This document has been developed according to the following guiding principles. Namely, it should:
a
Be descriptive and informative rather than normative.
b
Provide functional definitions.
c
Be consistent with current industry practice.
d
Be consistent with prior art to the extent practicable.
e
Be useful in various areas, including engineering, law, media, and public discourse.
