NIST AI RMF, MeasureMEASURE 2.4

You should have defined standards for model monitoring that cover frequency, scope and evaluation criteria by risk rating and model type, thresholds for breaches, contingency plans for model unavailability, deterioration or failure, and processes for AI/ML challenges such as autonomous decision making, autonomous re-parametrization and model drift.

Keep records of your algorithmic trading and, if you use high-frequency algorithmic trading technique, record precise, chronological data on every order submitted, cancelled and executed and every quotation made on a trading venue (section 82^15(11)).

Institutions should ensure that the performance of the model, including the validity and quality of its outputs, is continuously monitored and that appropriate remediation measures are taken in a timely manner in the case of detected issues (paragraph 53(f)).

Monitor all trading activity that takes place through your trading systems, including your clients' trading, for signs of potential market manipulation with an automated surveillance system that generates alerts and reports, covers the full range of your trading activities and all orders you submit, is reviewed at least once a year, and can generate operable alerts at the beginning of the following trading day or, where manual processes are involved, at the end of the following trading day (Article 13(1) to (3), (6) and (7)).

Monitor in real time, during the hours you are sending orders to trading venues, all algorithmic trading activity that takes place under your trading code, including that of your clients, for signs of disorderly trading, have the trader in charge of the algorithm and the risk management function or an independent risk control function carry out that monitoring, generate real-time alerts within five seconds after the relevant event, and take remedial action as soon as possible after an alert (Article 16(1), (2) and (5)). Continuously operate your post-trade controls, take appropriate action when one is triggered, keep complete, accurate and consistent records of trade and account information, and reconcile your own electronic trading logs with the information provided by trading venues, brokers, DEA providers, clearing members, central counterparties and data providers (Article 17(1) and (3)).

The CBJ expects you to review and validate AI data, models and decisions to detect and mitigate bias, to conduct periodic AI Impact Assessments proportional to risk, to validate and robustness-test models, and to monitor AI performance continuously at a level matched to the use case and risk (sections 2.5, 2.6, 5.2 and 5.3).

The CBJ expects you to log and keep auditable all data, models, decisions, user prompts and authorized overrides, to store logs in a secure, tamper-evident manner, and to provide regulatory authorities with comprehensive audit reports on AI decision-making processes (sections 2.2 and 4.2).

Through your board, keep adequate policies, processes and controls so that your algorithms continue to perform as intended, a robust framework for designing, monitoring and testing them through periodic and random reviews, and competent officers to develop and review their methodologies even if that work is outsourced; do not outsource the key processes and management of your client-facing tools (clause 10).

Keep, in addition to the records required by section 29 of the Financial Services Act, a copy of the advice given and the output of the investment, the client information relied on to generate the advice, and details of all algorithms and software you use (clause 12).

The Commission advises you to establish procedures for monitoring AI systems and clear accountability for AI-related errors or malfunctions, and to run ongoing monitoring and evaluation of the ethical and societal impact of your AI systems.

Give the fraud detection systems the capability to monitor and report metrics and Management Information on data integrity, rule and scenario effectiveness such as the false positive rate, and operational performance (Section 5.2(j)).

Monitor AI continuously in accordance with the Model Management Standards, test automatic updates to AI tools before implementation, have mechanisms to detect, report and remediate performance issues, biases or unintended consequences before implementation and over time, and retain the clear and immediate ability, with human intervention, to cease use of an AI model, system, technology or application (Sections 6(a), 6(c), 6(d) and 6(f)).

Monitor the performance of your models on a regular basis, present monitoring reports to the Model Oversight Committee at least every quarter, and follow a clear process in which that Committee decides whether to continue using a model with further monitoring or to suspend it and work on remediation (Sections 3.6.1, 3.6.4 and 3.6.7).

As a deployer, regularly monitor each such system and take reasonable and proportionate steps to mitigate the known or reasonably foreseeable risk of biased impacts

Periodically review the AI system to ensure its outcomes are accurate and reliable and that an individual's health data is not used beyond its intended or stated purpose

If you are the deployer of a high-risk AI system, monitor its operation on the basis of the provider's instructions for use, and inform the provider where relevant.

Document every use in the relevant police file, and make it available to the market surveillance authority and the national data protection authority on request, excluding sensitive operational data.

Keep the logs the system automatically generates, to the extent they are under your control, as Article 19 requires.

Have the system actively and systematically collect, document and analyze relevant data on the system's performance throughout its lifetime, whether provided by deployers or collected through other sources, so that you can evaluate its continuous compliance with the Chapter III, Section 2 requirements; where relevant, include an analysis of its interaction with other AI systems. This does not cover sensitive operational data of deployers that are law-enforcement authorities.

Allow access only to authorised persons, and record accesses and processing operations automatically in non-modifiable log files kept for five years (art. 10, comma 8).

Keep records of data inputs, training datasets, outputs and performance metrics for at least five years.

An owner of such a heightened-risk system must meet the Cabinet of Ministers' risk-management, transparency, accuracy, reliability, data-quality and technical-documentation requirements, keep operating logs, and declare conformity in a signed digital document published on the owner's site before the system is deployed.

Make the tool available for audit or compliance review by the Insurance Commissioner, and review its performance, use, and outcomes at least quarterly.

Strengthen your management of deep synthesis content: review your users' input data and synthesis output by technical or manual means, and record and retain the related network logs, if you provide a deep synthesis service.

Build and maintain a feature database for identifying illegal and undesirable information, with entry standards, rules, and procedures.

Build automatic event logging into a high-risk AI system so it can record events over the system's lifetime, if you are its provider.

Design the logging capability to support identifying an emerging risk or a substantial modification, post-market monitoring, and a deployer's monitoring of the system's operation.

Keep the logs your high-risk AI system automatically generates under Article 12, to the extent they are under your control, if you are its provider.

Keep the logs your high-risk AI system automatically generates, to the extent they are under your control, if you are its deployer.

Record each use automatically in non-modifiable log files and keep them for five years (art. 9, comma 2).

Maintain a record of processing activities and audit the AI system's operation in proportion to its level of risk.