NIST AI RMF, MeasureMEASURE 2.5

Make prominent written disclosure of your use of artificial intelligence in utilization review in your policies and procedures, review its outcomes periodically for accuracy, and keep patient data used in that review within its stated purpose under HIPAA.

Before deploying or substantially updating an algorithmic trading system, trading algorithm or algorithmic trading strategy, establish clearly delineated methodologies to develop and test it, have a person designated by senior management authorize the deployment or update, and design the methodologies to ensure that the system, algorithm or strategy does not behave in an unintended manner, complies with your obligations and the rules and systems of the trading venues you access, does not contribute to disorderly trading conditions and continues to work effectively in stressed market conditions; the authorization and the testing methodologies of Article 5(2) to (5) apply only to trading algorithms leading to order execution (Article 5(1), (2), (4) and (6)).

Perform a self-assessment and validation process annually and on that basis issue a validation report that reviews, evaluates and validates your algorithmic trading systems, trading algorithms and strategies, your governance, accountability and approval framework, your business continuity arrangement and your overall compliance with Article 17 of Directive 2014/65/EU; have the risk management function draw up the report, have the internal audit function audit it where one exists, obtain the approval of senior management, and remedy any deficiencies it identifies (Article 9). As part of that self-assessment, test that your algorithmic trading systems and the controls in Articles 12 to 18 can withstand increased order flows or market stresses, with high messaging volume tests and high trade volume tests using the highest volumes of the previous six months multiplied by two, carried out in a way that does not affect the production environment (Article 10).

Take reasonable steps to ensure that the information collected about your clients is reliable, including by ensuring that all tools, such as risk assessment profiling tools or tools to assess a client's knowledge and experience, are fit-for-purpose and appropriately designed for use with your clients, with any limitations identified and actively mitigated through the suitability assessment process (Article 54(7), point (b)).

Document, record, track and report validation findings across all models to Senior Management and the Board at least once a year (Section 3.7.9).

Take reasonable steps to ensure that the information collected about your clients is reliable, including by ensuring that all tools, such as risk assessment profiling tools or tools to assess a client's knowledge and experience, used in the suitability assessment process are fit-for-purpose and appropriately designed for use with your clients, with any limitations identified and actively mitigated, and that the questions used are likely to be understood by clients and capture an accurate reflection of the client's objectives and needs (COBS 9A.2.9R).

Validate a model before its first use. Where an urgent business need requires use before validation is complete, give greater attention to the model's limitations, inform relevant stakeholders of them, and set controls such as limits on use or closer monitoring of performance.

Validate conceptual soundness, compare model outputs with real-world outcomes, and monitor model performance on an ongoing basis, considering adjustment, recalibration or redevelopment when performance deviates meaningfully from expectations.

If you are the provider of a high-risk AI system, design and develop it to achieve an appropriate level of accuracy, robustness and cybersecurity, held consistently throughout its lifecycle.