Teresa Scassa - Blog

A recent decision of the Federal Court of Canada (Ali v. Minister of Public Safety and Emergency Preparedness) highlights the role of judicial review in addressing automated decision-making. It also prompts reflection on the limits of emerging codified rights to an explanation.

In July 2024, Justice Battista overturned a decision of the Refugee Protection Division (RPD) which had vacated the refugee status of the applicant, Mr. Ali. The decision of the RPD was based largely on a photo comparison that the RPD to conclude that Mr. Ali was not a Somali refugee as he had claimed. Rather, they concluded that he was a Kenyan student who had entered Canada on a student visa in 2016, a few months prior to Mr. Ali’s refugee protection claim.

Throughout the proceedings the applicant had sought information about how photos of the Kenyan student had been found and matched with his own. He was concerned that facial recognition technology (FRT) – which has had notorious deficiencies when used to identify persons of colour – had been used. In response, the Minister denied the use of FRT, maintaining instead that the photographs had been found and analyzed through a ‘manual process’. A Canadian Border Services agent subsequently provided an affidavit to the effect that “a confidential manual investigative technique was used” (at para 15). The RPD was satisfied with this assurance. It considered that how the photographs had been gathered was irrelevant to their own capacity as a tribunal to decide based on the photographs before them. They concluded that Mr. Ali had misrepresented his identity.

On judicial review, Justice Battista found that the importance of the decision to Mr. Ali and the quasi-judicial nature of the proceedings meant that he was owed a high level of procedural fairness. Because a decision of the RPD cannot be appealed, and because the consequences of revocation of refugee status are very serious (including loss of permanent resident status and possible removal from the country), Justice Battista found that “it is difficult to find a process under [the Immigration and Refugee Protection Act] with a greater imbalance between severe consequences and limited recourse” (at para 23). He found that the RPD had breached Mr. Ali’s right to procedural fairness “when it denied his request for further information about the source and methodology used by the Minister in obtaining and comparing the photographs” (at para 28).

Justice Battista ruled that, given the potential consequences for the applicant, disclosure of the methods used to gather the evidence against him “had to be meaningful” (at para 33). He concluded that it was unfair for the RPD “to consider the photographic evidence probative enough for revoking the Applicant’s statuses and at the same time allow that evidence to be shielded from examination for reliability” (at para 37).

In addition to finding a breach of procedural fairness, Justice Battista also found that the RPD’s decision was unreasonable. He noted that there had been sufficiently credible evidence before the original RPD refugee determination panel to find that Mr. Ali was a Somali national entitled to refugee protection. None of this evidence had been assessed in the decision of the panel that vacated Mr. Ali’s refugee status. Justice Battista noted that “[t]he credibility of this evidence cannot co-exist with the validity of the RPD vacation panel’s decision” (at para 40). He also noted that the applicant had provided an affidavit describing differences between his photo and that of the Kenyan student; this evidence had not been considered in the RPD’s decision, contributing to its unreasonableness. The RPD also dismissed evidence from a Kenyan official that, based on biometric records analysis, there was no evidence that Mr. Ali was Kenyan. Justice Battista noted that this dismissal of the applicant’s evidence was in “stark contrast to its treatment of the Minister’s photographic evidence” (at para 44).

The Ali decision and the right to an explanation

Ali is interesting to consider in the context of the emerging right to an explanation of automated decision-making. Such a right is codified for the private sector context in the moribund Bill C-27, and Quebec has enacted a right to an explanation for both public and private sector contexts. Such rights would apply in cases where an automated decision system (ADS) has been used (and in the case of Quebec, the decision must be based “exclusively on an automated processing” of personal information. Yet in Ali there is no proof that the decision was made or assisted by an AI technology – in part because the Minister refused to explain their ‘confidential’ process. Further, the ultimate decision was made by humans. It is unclear how a codified right to an explanation would apply if the threshold for the exercise of the right is based on the obvious and/or exclusive use of an ADS.

It is also interesting to consider the outcome here in light of the federal Directive on Automated Decision Making (DADM). The DADM, which largely addresses the requirements for design and development of ADS in the federal public sector, incorporates principles of fairness. It applies to “any system, tool, or statistical model used to make an administrative decision or a related assessment about a client”. It defines an “automated decision system” as “[a]ny technology that either assists or replaces the judgment of human decision-makers […].” In theory, this would include the use of automated systems such as FRT that assist in human decision-making. Where and ADS is developed and used, the DADM imposes transparency obligations, which include an explanation in plain language of:

  • the role of the system in the decision-making process;
  • the training and client data, their source, and method of collection, as applicable;
  • the criteria used to evaluate client data and the operations applied to process it;
  • the output produced by the system and any relevant information needed to interpret it in the context of the administrative decision; and
  • a justification of the administrative decision, including the principal factors that led to it. (Appendix C)

The catch, of course, is that it might be impossible for an affected person to know whether a decision has been made with the assistance of an AI technology, as was the case here. Further, the DADM is not effective at capturing informal or ‘off-the-books’ uses of AI tools. The decision in Ali therefore does two important things in the administrative law context. First, it confirms that – in the case of a high impact decision – the right of the individual to an explanation of how the decision was reached as a matter of procedural fairness. Judicial review thus provides recourse for affected individuals – something that the more prophylactic DADM does not. Second, this right includes an obligation to provide details that could either explain or rule out the use of an automated system in the decisional process. In other words, procedural fairness includes a right to know whether and how AI technologies were used in reaching the contested decision. Mere assertions that no algorithms were used in gathering evidence or in making the decision are insufficient – if an automated system might have played a role, the affected individual is entitled to know the details of the process by which the evidence was gathered and the decision reached. Ultimately, what Justice Battista crafts in Ali is not simply a right to an explanation of automated decision-making; rather, it is a right to the explanation of administrative decision-making processes that account for an AI era. In a context in which powerful computing tools are available for both general and personal use, and are not limited to purpose-specific, carefully governed and auditable in-house systems, the ability to demand an explanation of the decisional process in order to rule out the non-transparent use of AI systems seems increasingly important.

Note: The Directive on Automated Decision-Making is currently undergoing its fourth review. You may participate in consultations here.

Published in Privacy

Ontario is currently holding public hearings on a new bill which, among other things, introduces a provision regarding the use of AI in hiring in Ontario. Submissions can be made until February 13, 2024. Below is a copy of my submission addressing this provision.

 

The following is my written submission on section 8.4 of Bill 149, titled the Working for Workers Four Act, introduced in the last quarter of 2023. I am a law professor at the University of Ottawa. I am making this submission in my individual capacity.

Artificial intelligence (AI) tools are increasingly common in the employment context. Such tools are used in recruitment and hiring, as well as in performance monitoring and assessment. Section 8.4 would amend the Employment Standards Act to include a requirement for employers to provide notice of the use of artificial intelligence in the screening, assessment, or selection of applicants for a publicly advertised job position. It does not address the use of AI in other employment contexts. This brief identifies several weaknesses in the proposal and makes recommendations to strengthen it. In essence, notice of the use of AI in the hiring process will not offer much to job applicants without a right to an explanation and ideally a right to bring any concerns to the attention of a designated person. Employees should also have similar rights when AI is used in performance assessment and evaluation.

1. Definitions and exclusions

If passed, Bill 149 would (among other things) enact the first provision in Ontario to directly address AI. The proposed section 8.4 states:

8.4 (1) Every employer who advertises a publicly advertised job posting and who uses artificial intelligence to screen, assess or select applicants for the position shall include in the posting a statement disclosing the use of the artificial intelligence.

(2) Subsection (1) does not apply to a publicly advertised job posting that meets such criteria as may be prescribed.

The term “artificial intelligence” is not defined in the bill. Rather, s. 8.1 of Bill 149 leaves the definition to be articulated in regulations. This likely reflects concerns that the definition of AI will continue to evolve along with the rapidly changing technology and that it is best to leave its definition to more adaptable regulations. The definition is not the only thing left to regulations. Section 8.4(2) requires regulations to specify the criteria that will allow publicly advertised job postings to be exempted from the disclosure requirement in s. 8.4(1). The true scope and impact of s. 8.4(1) will therefore not be clear until these criteria are prescribed in regulations. Further, s. 8.4 will not take effect until the regulations are in place.

2. The Notice Requirement

The details of the nature and content of the notice that an employer must provide are not set out in s. 8.4, nor are they left to regulations. Since there are no statutory or regulatory requirements, presumably notice can be as simple as “we use artificial intelligence in our screening and selection process”. It would be preferable if notice had to at least specify the stage of the process and the nature of the technique used.

Section 8.4 is reminiscent of the 2022 amendments to the Employment Standards Act which required employers with more than 25 employees to provide their employees with notification of any electronic monitoring taking place in the workplace. As with s. 8.4(1), above, the main contribution of this provision was (at least in theory) enhanced transparency. However, the law did not provide for any oversight or complaints mechanism. Section 8.4(1) is similarly weak. If an employer fails to provide notice of the use of AI in the hiring process, then either the employer is not using AI in recruitment and hiring, or they are failing to disclose it. Who will know and how? A company that is found non-compliant with the notice requirement, once it is part of the Employment Standards Act, could face a fine under s. 132. However, proceedings by way of an offence are a rather blunt regulatory tool.

3. A Right to an Explanation?

Section 8.4(1) does not provide job applicants with any specific recourse if they apply for a job for which AI is used in the selection process and they have concerns about the fairness or appropriateness of the tool used. One such recourse could be a right to demand an explanation.

The Consumer Privacy Protection Act (CPPA), which is part of the federal government’s Bill C-27, currently before Parliament, provides a right to an explanation to those about whom an automated decision, prediction or recommendation is made. Sections 63(3) and (4) provide:

(3) If the organization has used an automated decision system to make a prediction, recommendation or decision about the individual that could have a significant impact on them, the organization must, on request by the individual, provide them with an explanation of the prediction, recommendation or decision.

(4) The explanation must indicate the type of personal information that was used to make the prediction, recommendation or decision, the source of the information and the reasons or principal factors that led to the prediction, recommendation or decision.

Subsections 63(3) and (4) are fairly basic. For example, they do not include a right of review of the decision by a human. But something like this would still be a starting point for a person seeking information about the process by which their employment application was screened or evaluated. The right to an explanation in the CPPA will extend to decisions, recommendations and predictions made with respect to employees of federal works, undertakings, and businesses. However, it will not apply to the use of AI systems in provincially regulated employment sectors. Without a private sector data protection law of its own – or without a right to an explanation to accompany the proposed s. 8.4 – provincially regulated employees in Ontario will be out of luck.

In contrast, Quebec’s recent amendments to its private sector data protection law provide for a more extensive right to an explanation in the case of automated decision-making – and one that applies to the employment and hiring context. Section 12.1 provides:

12.1. Any person carrying on an enterprise who uses personal information to render a decision based exclusively on an automated processing of such information must inform the person concerned accordingly not later than at the time it informs the person of the decision.

He must also inform the person concerned, at the latter’s request,

(1) of the personal information used to render the decision;

(2) of the reasons and the principal factors and parameters that led to the decision; and

(3) of the right of the person concerned to have the personal information used to render the decision corrected.

The person concerned must be given the opportunity to submit observations to a member of the personnel of the enterprise who is in a position to review the decision.

Section 12.1 thus combines a notice requirement with, at the request of the individual, a right to an explanation. In addition, the affected individual can “submit observations” to an appropriate person within the organization who “is in a position to review the decision”. This right to an explanation is triggered only by decisions that are based exclusively on automated processing of personal information – and the scope of the right to an explanation is relatively narrow. However, it still goes well beyond Ontario’s Bill 149, which creates a transparency requirement with nothing further.

4. Scope

Bill 149 applies to the use of “artificial intelligence to screen, assess or select applicants”. Bill C-27 and Quebec’s law, both referenced above, are focused on “automated decision-making”. Although automated decision-making is generally considered a form of AI (it is defined in C-27 as “any technology that assists or replaces the judgment of human decision-makers through the use of a rules-based system, regression analysis, predictive analytics, machine learning, deep learning, a neural network or other technique”) it is possible that in an era of generative AI technologies, the wording chosen for Bill 149 is more inclusive. In other words, there may be uses of AI that are not decision-making, predicting or recommending, but that can still used in screening, assessing or hiring processes. However, it should be noted that Ontario’s Bill 149 is also less inclusive than Bill C-27 or Quebec’s law because it focuses only on screening, assessment or selecting applicants for a position. It does not apply to the use of AI tools to monitor, evaluate or assess the performance of existing employees or to make decisions regarding promotion, compensation, retention, or other employment issues – something which would be covered by Quebec’s law (and by Bill C-27 for employees in federally regulated employment). Although arguably the requirements regarding electronic workplace monitoring added to the Employment Standards Act in 2022 might provide transparency about the existence of electronic forms of surveillance (which could include those used to feed data to AI systems), these transparency obligations apply only in workplaces with more than 25 employees, and there are no employee rights linked to the use of these data in automated or AI-enabled decision-making systems.

5. Discriminatory Bias

A very significant concern with the use of AI systems for decision-making about humans is the potential for discriminatory bias in the output of these systems. This is largely because systems are trained on existing and historical data. Where such data are affected by past discriminatory practices (for example, a tendency to hire men rather than women, or white, able-bodied, heterosexual people over those from equity-deserving communities) then there is a risk that automated processes will replicate and exacerbate these biases. Transparency about the use of an AI tool alone in such a context is not much help – particularly if there is no accompanying right to an explanation. Of course, human rights legislation applies to the employment context, and it will still be open to an employee who believes they have been discriminated against to bring a complaint to the Ontario Human Rights Commission. However, without a right to an explanation, and in the face of proprietary and closed systems, proving discrimination may be challenging and may require considerable resources and expertise. It may also require changes to human rights legislation to specifically address algorithmic discrimination. Without these changes in place, and without adequate resourcing to support the OHRC’s work to address algorithmic bias, recourse under human rights legislation may be extremely challenging.

 

6. Conclusion and Recommendations

This exploration of Bill 149’s transparency requirements regarding the use of AI in the hiring process in Ontario reveals the limited scope of the proposal. Its need for regulations in order take effect has the potential to considerably delay its implementation. It provides for notice but not for a right to an explanation or for human review of AI decisions. There is also a need to make better use of existing regulators (particularly privacy and human rights commissions). The issue of the use of AI in recruitment (or in the workplace more generally in Ontario) may require more than just tweaks to the Employment Standards Act but may also demand amendments to Ontario’s Human Rights Code and perhaps even specific privacy legislation at the very least aimed at the employment sector in Ontario.

Recommendations:

1. Redraft the provision so that the core obligations take effect without need for regulations or ensure that the necessary regulations to give effect to this provision are put in place promptly.

2. Amend s. 8.4 (1) to either include the elements that are required in any notice of the use of an AI system or provide for the inclusion of such criteria in regulations (so long as doing so does not further delay the coming into effect of the provision).

3. Provide for a right to an explanation to accompany s. 8.4(1). An alternative to this would be a broader right to an explanation in provincial private sector legislation or in privacy legislation for employees in provincially regulated sectors in Ontario, but this would be much slower than the inclusion of a basic right to an explanation in s. 8.4. The right to an explanation could also include a right to submit observations to a person in a position to review any decision or outcome.

4. Extend the notice requirement to other uses of AI to assess, evaluate and monitor the performance of employees in provincially regulated workplaces in Ontario. Ideally, a right to an explanation should also be provided in this context.

5. Ensure that individuals who are concerned that they have been discriminated against by the use of AI systems in recruitment (as well as employees who have similar concerns regarding the use of AI in performance evaluation and assessment) have adequate and appropriate recourse under Ontario’s Human Rights Code, and that the Ontario Human Rights Commission is adequately resourced to address these concerns.

Published in Privacy
Tuesday, 21 March 2023 06:50

Explaining the AI and Data Act

The federal government’s proposed Artificial Intelligence and Data Act (AIDA) is currently before Parliament as part of Bill C-27, a bill that will also reform Canada’s private sector data protection law. The AIDA, which I have discussed in more detail in a series of blog posts (here, here, and here), has been criticized for being a shell of a law with essential components (including the definition of the “high impact AI” to which it will apply) being left to as-yet undrafted regulations. The paucity of detail in the AIDA, combined with the lack of public consultation, has prompted considerable frustration and concern from AI developers and from civil society alike. In response to these concerns, the government published, on March 13, 2023, a companion document that explains the government’s thinking behind the AIDA. The document is a useful read as it makes clear some of the rationales for different choices that have been made in the bill. It also obliquely engages with many of the critiques that have been leveled at the AIDA. Unlike a consultation document, however, where feedback is invited to improve what is being proposed, the companion document is essentially an apology (in the Greek sense of the word) – something that is written in defense or explanation. At this stage, any changes will have to come as amendments to the bill.

Calling this a ‘companion document’ also somewhat tests the notion of “companion”, since it was published nine months after the AIDA was introduced in Parliament in June 2022. The document explains that the government seeks to take “the first step towards a new regulatory system designed to guide AI innovation in a positive direction, and to encourage the responsible adoption of AI technologies by Canadians and Canadian businesses.” The AIDA comes on the heels of the European Union’s draft AI Act – a document that is both more comprehensive and far more widely consulted upon. Pressure on Canada to regulate AI is heightened by the activity in the EU. This is evident in the introduction to the companion document, which speaks of the need to work with international partners to achieve global protection for Canadians and to ensure that “Canadian firms can be recognized internationally as meeting robust standards.”

An important critique of the AIDA has been that it will apply only to “high impact” AI. By contrast, the EU AI Act sets a sliding scale of obligations, with the most stringent obligations applying to high risk applications, and minimal obligations for low risk AI. In the AIDA companion document, there is no explanation of why the AIDA is limited to high impact AI. The government explains that defining the scope of the Act in regulations will allow for greater precision, as well as for updates as technology progresses. The companion document offers some clues about what the government considers relevant to determining whether an AI system is high-impact. Factors include the type of harm, the severity of harm, and the scale of use. Although this may help understand the concept of high impact, it does not explain why governance was only considered for high and not medium or low impact AI. This is something that cannot be fixed by the drafting of regulations. The bill would have to be specifically amended to provide for governance for AI with different levels of impact according to a sliding scale of obligations.

Another important critique of the AIDA has been that it unduly focuses on individual rather than collective or broader harms. As the US’s NIST AI Risk Management Framework aptly notes, AI technologies “pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment and the planet” (at p. 1). The AIDA companion document addresses this critique by noting that the bill is concerned both with individual harms and with systemic bias (defined as discrimination). Yet, while it is crucially important to address the potential for systemic bias in AI, this is not the only collective harm that should be considered. The potential for AI to be used to generate and spread disinformation or misinformation, for example, can create a different kind of collective harm. Flawed AI could potentially also result in environmental damage that is the concern of all. The companion document does little to address a broader notion of harm – but how can it? The AIDA specifically refers to and defines “individual harm”, and also addresses biased output as discriminatory within the meaning of the Canadian Human Rights Act. Only amendments to the bill can broaden its scope to encompass other forms of collective harm. Such amendments are essential.

Another critique of the AIDA is that it relies for its oversight on the same Ministry that is responsible for promoting and supporting AI innovation in Canada. The companion document tackles this concern, citing the uniqueness of the AI context, and stating that “administration and enforcement decisions have important implications for policy”, such that oversight and the encouragement of innovation “would need to be [sic] work in close collaboration in the early years of the framework under the direction of the Minister.” The Minister will be assisted by a Ministry staffer who will be designated the AI and Data Commissioner. The document notes that the focus in the early days of the legislation will be on helping organizations become compliant: “The Government intends to allow ample time for the ecosystem to adjust to the new framework before enforcement actions are undertaken.” The ample time will include the (at least) two years before the necessary regulations are drafted (though note that if some key regulations are not drafted, the law will never take effect), as well as any subsequent ‘adjustment’ time. Beyond this, the document is quite explicit that compliance and enforcement should not get unnecessarily in the way of the industry. The AIDA contains other mechanisms, including requiring companies to hire their own auditors for audits and having an appointed Ministerial advisory committee to reassure those who remain concerned about governance. Yet these measures do nothing to address a core lack of independent oversight. This lack is particularly noteworthy given that the same government has proposed the creation of an ill-advised Personal Information and Data Protection Tribunal (in Part II of Bill C-27) in order to establish another layer between the Privacy Commissioner and the enforcement of Bill C-27’s proposed Consumer Privacy Protection Act. It is difficult to reconcile the almost paranoid approach taken to the Privacy Commissioner’s role with the in-house, “we’re all friends here” approach to AI governance in the AIDA. It is hard to see how this lack of a genuine oversight framework can be fixed without a substantial rewrite of the bill.

And that brings us to the reality that we must confront with this bill: AI technologies are rapidly advancing and are already having significant impacts on our lives. The AIDA is deeply flawed, and the lack of consultation is profoundly disturbing. Yet, given the scarcity of space on Parliament’s agenda and the generally fickle nature of politics, the failure of the AIDA could lead to an abandonment of attempts to regulate in this space – or could very substantially delay them. As debate unfolds over the AIDA, Parliamentarians will have to ask themselves the unfortunate question of whether the AIDA is unsalvageable, or whether it can be sufficiently amended to be better than no law at all.

 

Published in Privacy

Artificial intelligence (AI) is already being used to assist government decision-making, although we have little case law that explores issues of procedural fairness when it comes to automated decision systems. This is why a recent decision of the Federal Court is interesting. In Barre v. Canada (Citizenship and Immigration) two women sought judicial review of a decision of the Refugee Protection Division (RPD) which had stripped them of their refugee status. They raised procedural fairness issues regarding the possible reliance upon an AI tool – in this case facial recognition technology (FRT). The case allows us to consider some procedural fairness guideposts that may be useful where evidence derived from AI-enabled tools is advanced.

The Decision of the Refugee Protection Division

The applicants, Ms Barre and Ms Hosh, had been granted refugee status after advancing claims related to their fear of sectarian and gender-based violence in their native Somalia. The Minister of Public Safety and Emergency Preparedness (the Minister) later applied under s. 109 of the Immigration and Refugee Protection Act to have that decision vacated on the basis that it was “obtained as a result of directly or indirectly misrepresenting or withholding material facts relating to a relevant matter”.

The Minister had provided the RPD with photos that compared Ms Barre and Ms Hosh the applicants) with two Kenyan women who had been admitted to Canada on student visas shortly before Ms Barre and Ms Hosh filed their refugee claims (the claims were accepted in 2017). The applicants argued that the photo comparisons relied upon by the Minister had been made using Clearview AI’s facial recognition service built upon scraped images from social media and other public websites. The Minister objected to arguments and evidence about Clearview AI, maintaining that there was no proof that this service had been used. Clearview AI had ceased providing services in Canada on 6 July 2020, and the RPD accepted the Minister’s argument that it had not been used, finding that “[a]n App that is banned to operate in Canada would certainly not be used by a law enforcement agency such as the CBSA” (at para 7). The Minister had also argued that it did not have to disclose how it arrived at the photo comparisons because of s. 22 of the Privacy Act, and the RPD accepted this assertion.

The photo comparisons were given significant weight in the RPD’s decision to overturn the applicants’ refugee status. The RPD found that there were “great similarities” between the photos of the Kenyan students and the applicants, and concluded that they were the same persons. The RPD also considered notes in the Global Case Management System to the effect that the Kenyan students did not attend classes at the school where they were enrolled. In addition, the CBSA submitted affidavits indicating that there was no evidence that the applicants had entered Canada under their own names. The RPD concluded that the applicants were Kenyan citizens who had misrepresented their identity in the refugee proceedings. It found that these factual misrepresentations called into question the credibility of their allegations of persecution. It also found that, since they were Kenyan, they had not advanced claims against their country of nationality in the refugee proceedings, as required by law. The applicants sought judicial review of the decision to revoke their refugee status, arguing that it was unreasonable and breached their rights to procedural fairness.

Judicial Review

Justice Go of the Federal Court ruled that the decision was unreasonable for a number of reasons. A first error was allowing the introduction of the photo comparisons into evidence “without requiring the Minister to disclose the methodology used in procuring the evidence” (at para 31). The Minister had invoked s. 22 of the Privacy Act, but Justice Go noted that there were many flaws with the Minister’s reliance on s. 22. Section 22 is an exception to an individual’s right of access to their personal information. Justice Go noted that the applicants were not seeking access to their personal information; rather, they were making a procedural fairness argument about the photo comparisons relied upon by the Minister and sought information about how the comparisons had been made. Section 22(2), which was specifically relied upon by the Minister, allows a request for disclosure of personal information to be refused on the basis that it was “obtained or prepared by the Royal Canadian Mounted Police while performing policing services for a province or municipality…”, and this circumstance simply was not relevant.

Section 22(1)(b), which was not specifically argued by the Minister, allows for a refusal to disclose personal information where to do so “could reasonably be expected to be injurious to the enforcement of any law of Canada or a province or the conduct of lawful investigations…” Justice Go noted that case law establishes that a court will not support such a refusal on the basis that because there is an investigation, harm from disclosure can be presumed. Instead, the head of an institution must demonstrate a “nexus between the requested disclosure and a reasonable expectation of probable harm” (at para 35, citing Canadian Association of Elizabeth Fry Societies v. Canada). Exceptions to access rights must be given a narrow interpretation, and the burden of demonstrating that a refusal to disclose is justifiable lies with the head of the government institution. Justice Go also noted that “the Privacy Act does not operate “so as to limit access to information to which an individual might be entitled as a result of other legal rules or principles”” (at para 42) such as, in this case, the principles of procedural fairness.

Justice Go found that the RPD erred by not clarifying what ‘personal information’ the Minister sought to protect; and by not assessing the basis for the Minister’s 22 arguments. She also noted that the RPD had accepted the Minister’s bald assertions that the CBSA did not rely on Clearview AI. Even if the company had ceased offering its services in Canada by July 6, 2020, there was no evidence regarding the date on which the photo comparisons had been made. Justice Go noted that the RPD failed to consider submissions by the applicants regarding findings by the privacy commissioners of Canada, BC, Alberta and Quebec regarding Clearview AI and its activities, as well as on the “danger of relying on facial recognition software” (at para 46).

The Minister argued that even if its s. 22 arguments were misguided, it could still rely upon evidentiary privileges to protect the details of its investigation. Justice Go noted that this was irrelevant in assessing the reasonableness of the RPD’s decision, since such arguments had not been made before or considered by the RPD. She also observed that when parties seek to exempt information from disclosure in a hearing, they are often required at least to provide it to the decision-maker to assess. In this case the RPD did not ask for or assess information on how the investigation had been conducted before deciding that information about it should not be disclosed. She noted that: “The RPD’s swift acceptance of the Minister’s exemption request, in the absence of a cogent explanation for why the information is protected from disclosure, appears to be a departure from its general practice” (at para 55).

Justice Go also observed that information about how the photo comparisons were made could well have been relevant to the issues to be determined by the RPD. If the comparisons were generated through use of FRT – whether it was using Clearview AI or the services of another company – “it may call into question the reliability of the Kenyan students’ photos as representing the Applicants, two women of colour who are more likely to be misidentified by facial recognition software than their white cohorts as noted by the studies submitted by the Applicants” (at para 56). No matter how the comparisons were made – whether by a person or by FRT technology – some evidence should have been provided to explain the technique. Justice Go found it unreasonable for the RPD to conclude that the evidence was reliable simply based upon the Minister’s assertions.

Justice Go also found that the RPD’s conclusion that the applicants were, in fact, the two Kenyan women, was unreasonable. Among other things, she found that the decision “failed to provide adequate reasons for the RPD’s conclusion that the two Applicants and the two Kenyan students were the same persons based on the photo comparisons” (at para 69). She noted that although the RPD referenced ‘great similarities’ between the women in the two sets of photographs, there were also some marked dissimilarities which were not addressed. There simply was no adequate explanation as to how the conclusion was reached that the applicants were the Kenyan students.

The decision of the RPD was quashed and remitted to be reconsidered by a differently constituted panel of the RPD.

Ultimately, Justice Go sends a clear message that the Minister cannot simply advance photo comparison evidence without providing an explanation for how that evidence was derived. At the very least, then, there is an obligation to indicate whether an AI technology was used in the decision-making process. Even if there is some legal basis for shielding the details of the Minister’s methods of investigation, there may still need to be some disclosure to the decision-maker regarding the methods used. Justice Go’s decision is also a rebuke of the RPD which accepted the Minister’s evidence on faith and asked no questions about its methodology or probity. In her decision, Justice Go take serious note of concerns about accuracy and bias in the use of FRT, particularly with racialized individuals, and it is clear that these concerns heighten the need for transparency. The decision is important for setting some basic standards to meet when it comes to reviewing evidence that may have been derived using AI. It is also a sobering reminder that those checks and balances failed at first instance – and in a high stakes context.

Published in Privacy

This post is the fifth in a series on Canada’s proposed Artificial Intelligence and Data Act in Bill C-27. It considers the federal government’s constitutional authority to enact this law, along with other roles it might have played in regulating AI in Canada. Earlier posts include ones on the purpose and application of the AIDA; regulated activities; the narrow scope of the concepts of harm and bias in the AIDA and oversight and protection.

AI is a transformative technology that has the power to do amazing things, but which also has the potential to cause considerable harm. There is a global clamour to regulate AI in order to mitigate potential negative effects. At the same time, AI is seen as a driver of innovation and economies. Canada’s federal government wants to support and nurture Canada’s thriving AI sector while at the same time ensuring that there is public trust in AI. Facing similar issues, the EU introduced a draft AI Act, which is currently undergoing public debate and discussion (and which itself was the product of considerable consultation). The US government has just proposed its Blueprint for an AI Bill of Rights, and has been developing policy frameworks for AI, including the National Institute of Standards and Technology (NIST) Risk Management Framework. The EU and the US approaches are markedly different. Interestingly, in the US (which, like Canada, is a federal state) there has been considerable activity at the state level on AI regulation. Serious questions for Canada include what to do about AI, how best to do it – and who should do it.

In June 2022, the federal government introduced the proposed Artificial Intelligence and Data Act (AIDA) in Bill C-27. The AIDA takes the form of risk regulation; in other words, it is meant to anticipate and mitigate AI harms to the public. This is an ex ante approach; it is intended to address issues before they become problems. The AIDA does not provide personal remedies or recourses if anyone is harmed by AI – this is left for ex post regimes (ones that apply after harm has occurred). These will include existing recourses such as tort law (extracontractual civil liability in Quebec), and complaints to privacy, human rights or competition commissioners.

I have addressed some of the many problems I see with the AIDA in earlier posts. Here, I try to unpack issues around the federal government’s constitutional authority to enact this bill. It is not so much that they lack jurisdiction (although they might); rather, how they understand their jurisdiction can shape the nature and substance of the bill they are proposing. Further, the federal government has acted without any consultation on the AIDA prior to its surprising insertion in Bill C-27. Although it promises consultation on the regulations that will follow, this does not make up for the lack of discussion around how we should identify and address the risks posed by AI. This rushed bill is also shaped by constitutional constraints – it is AI regulation with structural limitations that have not been explored or made explicit.

Canada is a federal state, which means that the powers typically exercised by a nation state are divided between a federal and regional governments. In theory, federalism allows for regional differences to thrive within an overarching framework. However, some digital technology issues (including data protection and AI) fit uneasily within Canada’s constitutional framework. In proposing the Consumer Privacy Protection Act part of Bill C-27, for example, the federal government appears to believe that it does not have the jurisdiction to address data protection as a matter of human rights – this belief has impacted the substance of the bill.

In Canada, the federal government has jurisdiction over criminal law, trade and commerce, banking, navigation and shipping, as well as other areas where it makes more sense to have one set of rules than to have ten. The cross-cutting nature of AI, the international competition to define the rules of the game, and the federal government’s desire to take a consistent national approach to its regulation are all factors that motivated the inclusion of the AIDA in Bill C-27. The Bill’s preamble states that “the design, development and deployment of artificial intelligence systems across provincial and international borders should be consistent with national and international standards to protect individuals from potential harm”. Since we do not yet have national or international standards, the law will also enable the creation (and imposition) of standards through regulation.

The preamble’s reference to the crossing of borders signals both that the federal government is keenly aware of its constitutional limitations in this area and that it intends to base its jurisdiction on the interprovincial and international dimensions of AI. The other elements of Bill C-27 rely on the federal general trade and commerce power – this follows the approach taken in the Personal Information Protection and Electronic Documents Act (PIPEDA), which is reformed by the first two parts of C-27. There are indications that trade and commerce is also relevant to the AIDA. Section 4 of the AIDA refers to the goal of regulating “international and interprovincial trade and commerce in artificial intelligence systems by establishing common requirements applicable across Canada, for the design, development and use of those systems.” Yet the general trade and commerce power is an uneasy fit for the AIDA. The Supreme Court of Canada has laid down rules for the exercise of this power, and one of these is that it should not be used to regulate a single industry; a legislative scheme should regulate trade as a whole.

The Minister of Industry, in discussing Canada’s AI strategy has stated:

Artificial intelligence is a key part of our government’s plan to make our economy stronger than ever. The second phase of the Pan-Canadian Artificial Intelligence Strategy will help harness the full potential of AI to benefit Canadians and accelerate trustworthy technology development, while fostering diversity and cooperation across the AI domain. This collaborative effort will bring together the knowledge and expertise necessary to solidify Canada as a global leader in artificial intelligence and machine learning.

Clearly, the Minister is casting the role of AI as an overall economic transformer rather than a discrete industry. Nevertheless, although it might be argued that AI is a technology that cuts across all sectors of the economy, the AIDA applies predominantly to its design and development stages, which makes it look as if it targets a particular industry. Further, although PIPEDA (and the CPPA in the first Part of Bill C-27), are linked to trade and commerce through the transactional exchange of personal data – typically when it is collected from individuals in the course of commercial activity – the AIDA is different. Its regulatory requirements are meant to apply before any commercial activity takes place –at the design and development stage. This is worth pausing over because design and development stages may be non-commercial (in university-based research, for example) or may be purely intra-provincial. As a result, the need to comply with a law at the design and development stage, when that law is premised on interprovincial or international commercial activity, may only be discovered well after commercialization becomes a reality.

Arguably, AI might also be considered a matter of ‘national concern’ under the federal government’s residual peace, order and good government power. Matters of national concern that would fall under this power would be ones that did not exist at the time of confederation. The problem with addressing AI in this way is that it is simply not obvious that provinces could not enact legislation to govern AI – as many states have begun to do in the US.

Another possible constitutional basis is the federal criminal law power. This is used, for example, in the regulation of certain matters relating to health such as tobacco, food and drugs, medical devices and controlled substances. The Supreme Court of Canada has ruled that this power “is broad, and is circumscribed only by the requirements that the legislation must contain a prohibition accompanied by a penal sanction and must be directed at a legitimate public health evil”. The AIDA contains some prohibitions and provides for both administrative monetary penalties (AMPs). Because the AIDA focuses on “high impact” AI systems, there is an argument that it is meant to target and address those systems that have the potential to cause the most harm to health or safety. (Of course, the bill does not define “high impact” systems, so this is only conjecture.) Yet, although AMPs are available in cases of egregious non-compliance with the AIDA’s requirements, AMPs are not criminal sanctions, they are “a civil (rather than quasi-criminal) mechanism for enforcing compliance with regulatory requirements”, as noted in a report from the Ontario Attorney-General. That leaves a smattering of offences such as obstructing the work of the Minister or of auditors; knowingly designing, developing or using an AI system where the data were obtained as a result of an offence under another Act; being reckless as to whether the use of an AI system made available by the accused is likely to cause harm to an individual, and using AI intentionally to defraud the public and cause substantial economic loss to an individual. Certainly, such offences are criminal in nature and could be supported by the federal criminal law power. Yet they are easily severable from the rest of the statute. For the most part, the AIDA focuses on “establishing common requirements applicable across Canada, for the design, development and use of [AI] systems” (AIDA, s. 4).

The provinces have not been falling over themselves to regulate AI, although neither have they been entirely inactive. Ontario, for example, has been developing a framework for the public sector use of AI, and Quebec has enacted some provisions relating to automated decision-making systems in its new data protection law. Nevertheless, these steps are clearly not enough to satisfy a federal government anxious to show leadership in this area. It is thus unsurprising that Canada’s federal government has introduced legislation to regulate AI. What is surprising is that they have done so without consultation – either regarding the form of the intervention or the substance. We have yet to have an informed national conversation about AI. Further, legislation of this kind was only one option. The government could have consulted and convened experts to develop something along the lines of the US’s NIST Framework that could be adopted as a common standard/approach across jurisdictions in Canada. A Canadian framework could have been supported by the considerable work on standards already ongoing. Such an approach could have involved the creation of an agency under the authority of a properly-empowered Data Commissioner to foster co-operation in the development of national standards. This could have supported the provinces in the harmonized regulation of AI. Instead, the government has chosen to regulate AI itself through a clumsy bill that staggers uneasily between constitutional heads of power, and that leaves its normative core to be crafted in a raft of regulations that may take years to develop. It also leaves it open to the first company to be hit with an AMP to challenge the constitutionality of the framework as a whole.

Published in Privacy

The Artificial Intelligence and Data Act (AIDA) in Bill C-27 will create new obligations for those responsible for AI systems (particularly high impact systems), as well as those who process or make available anonymized data for use in AI systems. In any regulatory scheme that imposes obligations, oversight and enforcement are key issues. A long-standing critique of the Personal Information Protection and Electronic Documents Act (PIPEDA) has been that it is relatively toothless. This is addressed in the first part of Bill C-27, which reforms the data protection law to provide a suite of new enforcement powers that include order-making powers for the Privacy Commissioner and the ability to impose stiff administrative monetary penalties (AMPs). The AIDA comes with ‘teeth’ as well, although these teeth seem set within a rather fragile jaw. I will begin by identifying the oversight and enforcement powers (the teeth) and will then look at the agent of oversight and enforcement (the jaw). The table below sets out the main obligations accompanied by specific compliance measures. There is also the possibility that any breach of these obligations might be treated as either a violation or offence, although the details of these require elaboration in as-yet-to-be-drafted regulations.

 

Obligation

Oversight Power

To keep records regarding the manner in which data is anonymized and the use or management of anonymized data as well as records of assessment of whether an AI system is high risk (s. 10)

Minister may order the record-keeper to provide any of these records (s. 13(1))

 

 

Any record-keeping obligations imposed on any actor in as-yet undrafted regulations

Where there are reasonable grounds to believe that the use of a high impact system could result in harm or biased output, the Minister can order the specified person to provide these records (s. 14)

Obligation to comply with any of the requirements in ss. 6-12, or any order made under s. 13-14

Minister (on reasonable grounds to believe there has a contravention) can require the person to conduct either an internal or an external audit with respect to the possible contravention (s. 15); the audit must be provided to the Minister

 

A person who has been audited may be ordered by the Minister to implement any measure specified in the order, or to address any matter in the audit report (s. 16)

Obligation to cease using or making available a high-impact system that creates a serious risk of imminent harm

Minister may order a person responsible for a high-impact system to cease using it or making it available for use if the Minister has reasonable grounds to believe that its use gives rise to a serious risk of imminent harm (s. 17)

Transparency requirement (any person referred to in sections 6 to 12, 15 and 16)

Minister may order the person to publish on a publicly available website any information related to any of these sections of the AIDA, but there is an exception for confidential business information (s. 18)

 

Compliance with orders made by the Minister is mandatory (s. 19) and there is a procedure for them to become enforceable as orders of the Federal Court.

Although the Minister is subject to confidentiality requirements, they may disclose any information they obtain through the exercise of the above powers to certain entities if they have reasonable grounds to believe that a person carrying out a regulated activity “has contravened, or is likely to contravene, another Act of Parliament or a provincial legislature” (s. 26(1)). Those entities include the Privacy Commissioner, the Canadian Human Rights Commission, the Commissioner of Competition, the Canadian Radio-television and Telecommunications Commission, their provincial analogues, or any other person prescribed by regulation. An organization may therefore be in violation of statutes other than AIDA and may be subject to investigation and penalties under those laws.

The AIDA itself provides no mechanism for individuals to file complaints regarding any harms they may believe they have suffered, nor is there any provision for the investigation of complaints.

The AIDA sets up the Minister as the actor responsible for oversight and enforcement, but the Minister may delegate any or all of their oversight powers to the new Artificial Intelligence and Data Commissioner who is created by s. 33. The Data Commissioner is described in the AIDA as “a senior official of the department over which the Minister presides”. They are not remotely independent. Their role is “to assist the Minister” responsible for the AIDA (most likely the Minister of Industry), and they will also therefore work in the Ministry responsible for supporting the Canadian AI industry. There is essentially no real regulator under the AIDA. Instead, oversight and enforcement are provided by the same group that drafted the law and that will draft the regulations. It is not a great look, and, certainly goes against the advice of the OECD on AI governance, as Mardi Wentzel has pointed out.

The role of Data Commissioner had been first floated in the 2019 Mandate Letter to the Minister of Industry, which provided that the Minister would: “create new regulations for large digital companies to better protect people’s personal data and encourage greater competition in the digital marketplace. A newly created Data Commissioner will oversee those regulations.” The 2021 Federal Budget provided funding for the Data Commissioner, and referred to the role of this Commissioner as to “inform government and business approaches to data-driven issues to help protect people’s personal data and to encourage innovation in the digital marketplace.” In comparison with these somewhat grander ideas, the new AI and Data Commissioner role is – well – smaller than the title. It is a bit like telling your kids you’re getting them a deluxe bouncy castle for their birthday party and then on the big day tossing a couple of couch cushions on the floor instead.

To perhaps add a gloss of some ‘independent’ input into the administration of the statute, the AIDA provides for the creation of an advisory committee (s. 35) that will provide the Minister with “advice on any matters related to this Part”. However, this too is a bit of a throwaway. Neither the AIDA nor any anticipated regulations will provide for any particular composition of the advisory committee, for the appointment of a chair with a fixed term, or for any reports by the committee on its advice or activities. It is the Minister who may choose to publish advice he receives from the committee on a publicly available website (s. 35(2)).

The AIDA also provides for enforcement, which can take one of two routes. Well, one of three routes. One route is to do nothing – after all, the Minister is also responsible for supporting the AI industry in Canada– so this cannot be ruled out. A second option will be to treat a breach of any of the obligations specified in the as-yet undrafted regulations as a “violation” and impose an administrative monetary penalty (AMP). A third option is to treat a breach as an “offence” and proceed by way of prosecution (s. 30). A choice must be made between proceeding via the AMP or the offense route (s. 29(3)). Providing false information and obstruction are distinct offences (s. 30(2)). There are also separate offences in ss. 38 and 39 relating to the use of illegally obtained data and knowingly or recklessly making an AI system available for use that is likely to cause harm.

Administrative monetary penalties under Part 1 of Bill C-27 (relating to data protection) are quite steep. However, the necessary details regarding the AMPs that will be available for breach of the AIDA are to be set out in regulations that have yet to be drafted (s. 29(4)(d)). All that the AIDA really tells us about these AMPs is that their purpose is “to promote compliance with this Part and not to punish” (s. 29(2)). Note that at the bottom of the list of regulation-making powers for AMPs set out in s. 29(4). This provision allows the Minister to make regulations “respecting the persons or classes of persons who may exercise any power, or perform any duty or function, in relation to the scheme.” There is a good chance that the AMPs will (eventually) be administered by the new Personal Information and Data Tribunal, which is created in Part 2 of Bill C-27. This, at least, will provide some separation between the Minister and the imposition of financial penalties. If this is the plan, though, the draft law should say so.

It is clear that not all breaches of the obligations in the AIDA will be ones for which AMPs are available. Regulations will specify the breach of which provisions of the AIDA or its regulations will constitute a violation (s. 29(4)(a)). The regulations will also indicate whether the breach of the particular obligation is classified as minor, serious or very serious (s. 29(4)(b)). The regulations will also set out how any such proceedings will unfold. As-yet undrafted regulations will also specify the amounts or ranges of AMPS, and factors to take into account in imposing them.

This lack of important detail makes it hard not to think of the oversight and enforcement scheme in the AIDA as a rough draft sketched out on a cocktail napkin after an animated after-hours discussion of what enforcement under the AIDA should look like. Clearly, the goal is to be ‘agile’, but ‘agile’ should not be confused with slapdash. Parliament is being asked to enact a law that leaves many essential components undefined. With so much left to regulations, one wonders whether all the missing pieces can (or will) be put in place within this decade. There are instances of other federal laws left incomplete by never-drafted regulations. For example, we are still waiting for the private right of action provided for in Canada’s Anti-Spam Law, which cannot come into effect until the necessary regulations are drafted. A cynic might even say that failing to draft essential regulations is a good way to check the “enact legislation on this issue” box on the to-do list, without actually changing the status quo.

Published in Privacy

This is the third in my series of posts on the Artificial Intelligence and Data Act (AIDA) found in Bill C-27, which is part of a longer series on Bill C-27 generally. Earlier posts on the AIDA have considered its purpose and application, and regulated activities. This post looks at the harms that the AIDA is designed to address.

The proposed Artificial Intelligence and Data Act (AIDA), which is the third part of Bill C-27, sets out to regulate ‘high-impact’ AI systems. The concept of ‘harm’ is clearly important to this framework. Section 4(b) of the AIDA states that a purpose of the legislation is “to prohibit certain conduct in relation to artificial intelligence systems that may result in serious harm to individuals or harm to their interests”.

Under the AIDA, persons responsible for high-impact AI systems have an obligation to identify, assess, and mitigate risks of harm or biased output (s. 8). Those persons must also notify the Minister “as soon as feasible” if a system for which they are responsible “results or is likely to result in material harm”. There are also a number of oversight and enforcement functions that are triggered by harm or a risk of harm. For example, if the Minister has reasonable grounds to believe that a system may result in harm or biased output, he can demand the production of certain records (s. 14). If there is a serious risk of imminent harm, the Minister may order a person responsible to cease using a high impact system (s. 17). The Minister is also empowered to make public certain information about a system where he believes that there is a serious risk of imminent harm and the publication of the information is essential to preventing it (s. 28). Elevated levels of harm are also a trigger for the offence in s. 39, which involves “knowing or being reckless as to whether the use of an artificial intelligence system is likely to cause serious physical or psychological harm to an individual or substantial damage to an individual’s property”.

‘Harm’ is defined in s. 5(1) to mean:

(a) physical or psychological harm to an individual;

(b) damage to an individual’s property; or

(c) economic loss to an individual.

I have emphasized the term “individual” in this definition because it places an important limit on the scope of the AIDA. First, it is unlikely that the term ‘individual’ includes a corporation. Typically, the word ‘person’ is considered to include corporations, and the word ‘person’ is used in this sense in the AIDA. This suggests that “individual” is meant to have a different meaning. The federal Interpretation Act is silent on the issue. It is a fair interpretation of the definition of ‘harm’ that “individual” is not the same as “person”, and means an individual (human) person. The French version uses the term “individu”, and not “personne”. The harms contemplated by this legislation are therefore to individuals and not to corporations.

Defining harm in terms of individuals has other ramifications. The AIDA defines high-risk AI systems in terms of their impacts on individuals. Importantly, this excludes groups and communities. It also very significantly focuses on what are typically considered quantifiable harms, and uses language that suggests quantifiability (economic loss, damage to property, physical or psychological harm). Some important harms may be difficult to establish or to quantify. For example, class action lawsuits relating to significant data breaches have begun to wash up on the beach of lost causes due to the impossibility of proving material loss either because, although thousands may have been impacted, the individual losses are impossible to quantify, or because it is impossible to prove a causal link between very real identity theft and that particular data breach. Consider an AI system that manipulates public opinion through an algorithm that drives content to individuals based on its shock value rather than its truth. Say this happens during a pandemic and it convinces people that they should not get vaccinated or take other recommended public health measures. Say some people die because they were misled in this way. Say other people die because they were exposed to infected people who were misled in this way. How does one prove the causal link between the physical harm of injury or death of an individual and the algorithm? What if there is an algorithm that manipulates voter sentiment in a way that changes the outcome of an election? What is the quantifiable economic loss or psychological harm to any individual? How could causation be demonstrated? The harm, once again, is collective.

The EU AI Act has also been criticized for focusing on individual harm, but the wording of that law is still broader than that in the AIDA. The EU AI Act refers to high-risk systems in terms of “harm to the health and safety or a risk of adverse impact on fundamental rights of persons”. This at least introduces a more collective dimension, and it avoids the emphasis on quantifiability.

The federal government’s own Directive on Automated Decision-Making (DADM) which is meant to guide the development of AI used in public sector automated decision systems (ADS) also takes a broader approach to impact. In assessing the potential impact of an ADS, the DADM takes into account: “the rights of individuals or communities”, “the health or well-being of individuals or communities”, “the economic interests of individuals, entities, or communities”, and “the ongoing sustainability of an ecosystem”.

With its excessive focus on individuals, the AIDA is simply tone deaf to the growing global understanding of collective harm caused by the use of human-derived data in AI systems.

One response of the government might be to point out that the AIDA is also meant to apply to “biased output”. Biased output is defined in the AIDA as:

content that is generated, or a decision, recommendation or prediction that is made, by an artificial intelligence system and that adversely differentiates, directly or indirectly and without justification, in relation to an individual on one or more of the prohibited grounds of discrimination set out in section 3 of the Canadian Human Rights Act, or on a combination of such prohibited grounds. It does not include content, or a decision, recommendation or prediction, the purpose and effect of which are to prevent disadvantages that are likely to be suffered by, or to eliminate or reduce disadvantages that are suffered by, any group of individuals when those disadvantages would be based on or related to the prohibited grounds. (s. 5(1)) [my emphasis]

The argument here will be that the AIDA will also capture discriminatory biases in AI. However, I have underlined the part of this definition that once again returns the focus to individuals, rather than groups. It can be very hard for an individual to demonstrate that a particular decision discriminated against them (especially if the algorithm is obscure). In any event, biased AI will tend to replicate systemic discrimination. Although it will affect individuals, it is the collective impact that is most significant – and this should be recognized in the law. The somewhat obsessive focus on individual harm in the AIDA may unwittingly help perpetuate denials of systemic discrimination.

It is also important to note that the definition of “harm” does not include “biased output”, and while the terms are used in conjunction in some cases (for example, in s. 8’s requirement to “identify, assess and mitigate the risks of harm or biased output”), other obligations relate only to “harm”. Since the two are used conjunctively in some parts of the statute, but not others, a judge interpreting the statute might presume that when only one of the terms is used, then it is only that term that is intended. Section 17 of the AIDA allows the Minister to order a person responsible for a high-impact system to cease using it or making it available if there is a “serious risk of imminent harm”. Section 28 permits the Minister to order the publication of information related to an AI system where there are reasonable grounds to believe that the use of the system gives rise to “a serious risk of imminent harm”. In both cases, the defined term ‘harm’ is used, but not ‘biased output’.

The goals of the AIDA to protect against harmful AI are both necessary and important, but in articulating the harm that it is meant to address, the Bill underperforms.

Published in Privacy

This is the second in a series of posts on Bill C-27’s proposed Artificial Intelligence and Data Act (AIDA). The first post looked at the scope of application of the AIDA. This post considers what activities and what data will be subject to governance.

Bill C-27’s proposed Artificial Intelligence and Data Act (AIDA) governs two categories of “regulated activity” so long as they are carried out “in the course of international or interprovincial trade and commerce”. These are set out in s. 5(1):

(a) processing or making available for use any data relating to human activities for the purpose of designing, developing or using an artificial intelligence system;

(b) designing, developing or making available for use an artificial intelligence system or managing its operations.

These activities are cast in broad terms, capturing activities related both to the general curating of the data that fuel AI, and the design, development, distribution and management of AI systems. The obligations in the statute do not apply universally to all engaged in the AI industry. Instead, different obligations apply to those performing different roles. The chart below identifies the actor in the left-hand column, and the obligation the column on the right.

 

Actor

Obligation

A person who carries out any regulated activity and who processes or makes available for use anonymized data in the course of that activity

(see definition of “regulated activity” in s. 5(1)

s. 6 (data anonymization, use and management)

s. 10 (record keeping regarding measures taken under s. 6)

A person who is responsible for an artificial intelligence system (see definition of ‘person responsible’ in s. 5(2)

s. 7 (assess whether a system is high impact)

s. 10 (record keeping regarding reasons supporting their assessment of whether the system is high-impact under s. 7)

A person who is responsible for a high-impact system (see definition of ‘person responsible’ in s. 5(2; definition of “high-impact” system, s. 5(1))

s. 8 (measures to identify, assess and mitigate risk of harm or biased output)

s. 9 (measures to monitor compliance with the mitigation measures established under s. 8 and the effectiveness of the measures

s. 10 (record keeping regarding measures taken under ss. 8 and 9)

s. 12 (obligation to notify the Minister as soon as feasible if the use of the system results or is likely to result in material harm)

A person who makes available for use a high-impact system

s. 11(1) (publish a plain language description of the system and other required information)

A person who manages the operation of a high-impact system

s. 11(2) (publish a plain language description of how the system is used and other required information)

 

For most of these provisions, the details of what is actually required by the identified actor will depend upon regulations that have yet to be drafted.

A “person responsible” for an AI system is defined in s. 5(2) of the AIDA in these terms:

5(2) For the purposes of this Part, a person is responsible for an artificial intelligence system, including a high-impact system, if, in the course of international or interprovincial trade and commerce, they design, develop or make available for use the artificial intelligence system or manage its operation.

Thus, the obligations in ss. 7, 8, 9, 10 and 11, apply only to those engaged in the activities described in s. 5(1)(b) (designing, developing or making available an AI system or managing its operation). Further, it is important to note that with the exception of sections 6 and 7, the obligations in the AIDA also apply only to ‘high impact’ systems. The definition of a high-impact system has been left to regulations and is as yet unknown.

Section 6 stands out somewhat as a distinct obligation relating to the governance of data used in AI systems. It applies to a person who carries out a regulated activity and who “processes or makes available for use anonymized data in the course of that activity”. Of course, the first part of the definition of a regulated activity includes someone who processes or makes available for use “any data relating to human activities for the purpose of designing, developing or using” an AI system. So, this obligation will apply to anyone “who processes or makes available for use anonymized data” (s. 6) in the course of “processing or making available for use any data relating to human activities for the purpose of designing, developing or using an artificial intelligence system” (s. 5(1)). Basically, then for s. 6 to apply, the anonymized data must be processed for the purposes of development of an AI system. All of this must also be in the course if international or interprovincial trade and commerce.

Note that the first of these two purposes involves data “related to human activities” that are used in AI. This is interesting. The new Consumer Privacy Protection Act (CPPA) that forms the first part of Bill C-27 will regulate the collection, use and disclosure of personal data in the course of commercial activity. However, it provides, in s. 6(5), that: “For greater certainty, this Act does not apply in respect of personal information that has been anonymized.” By using the phrase “data relating to human activities” instead of “personal data”, s. 5(1) of the AIDA clearly addresses human-derived data that fall outside the definition of personal information in the CPPA because of anonymization.

Superficially, at least, s. 6 of the AIDA appears to pick up the governance slack that arises where anonymized data are excluded from the scope of the CPPA. [See my post on this here]. However, for this to happen, the data have to be used in relation to an “AI system”, as defined in the legislation. Not all anonymized data will be used in this way, and much will depend on how the definition of an AI system is interpreted. Beyond that, the AIDA only applies to a ‘regulated activity’ which is one carried out in the course of international and inter-provincial trade and commerce. It does not apply outside the trade and commerce context, nor does it apply to any excluded actors [as discussed in my previous post here]. As a result, there remain clear gaps in the governance of anonymized data. Some of those gaps might (eventually) be filled by provincial governments, and by the federal government with respect to public-sector data usage. Other gaps – e.g., with respect to anonymized data used for purposes other than AI in the private sector context – will remain. Further, governance and oversight under the proposed CPPA will be by the Privacy Commissioner of Canada, an independent agent of Parliament. Governance under the AIDA (as will be discussed in a forthcoming post) is by the Minister of Industry and his staff, who are also responsible for supporting the AI industry in Canada. Basically, the treatment of anonymized data between the CPPA and the AIDA creates a significant governance gap in terms of scope, substance and process.

On the issue of definitions, it is worth making a small side-trip into ‘personal information’. The definition of ‘personal information’ in the AIDA provides that the term “has the meaning assigned by subsections 2(1) and (3) of the Consumer Privacy Protection Act.” Section 2(1) is pretty straightforward – it defines “personal information” as “information about an identifiable individual”. However, s. 2(3) is more complicated. It provides:

2(3) For the purposes of this Act, other than sections 20 and 21, subsections 22(1) and 39(1), sections 55 and 56, subsection 63(1) and sections 71, 72, 74, 75 and 116, personal information that has been de-identified is considered to be personal information.

The default rule for ‘de-identified’ personal information is that it is still personal information. However, the CPPA distinguishes between ‘de-identified’ (pseudonymized) data and anonymized data. Nevertheless, for certain purposes under the CPPA – set out in s. 2(3) – de-identified personal information is not personal information. This excruciatingly-worded limit on the meaning of ‘personal information’ is ported into the AIDA, even though the statutory provisions referenced in s. 2(3) are neither part of AIDA nor particularly relevant to it. Since the legislator is presumed not to be daft, then this must mean that some of these circumstances are relevant to the AIDA. It is just not clear how. The term “personal information” is used most significantly in the AIDA in the s. 38 offense of possessing or making use of illegally obtained personal information. It is hard to see why it would be relevant to add the CPPA s. 2(3) limit on the meaning of ‘personal information’ to this offence. If de-identified (not anonymized) personal data (from which individuals can be re-identified) are illegally obtained and then used in AI, it is hard to see why that should not also be captured by the offence.

 

Published in Privacy

This is the first of a series of posts on the part of Bill C-27 that would enact a new Artificial Intelligence and Data Act (AIDA) in Canada. Previous posts have considered the part of the bill that would reform Canada’s private sector data protection law. This series on the AIDA begins with an overview of its purpose and application.

Bill C-27 contains the text of three proposed laws. The first is a revamped private sector data protection law. The second would establish a new Data Tribunal that is assigned a role under the data protection law. The third is a new Artificial Intelligence and Data Act (AIDA) While the two other components were present in the bill’s failed predecessor Bill C-11, the AIDA is new – and for many came as a bit of a surprise. The common thread, of course, is the government’s Digital Charter, which set out a series of commitments for building trust in the digital and data economy.

The preamble to Bill C-27, as a whole, addresses both AI and data protection concerns. Where it addresses AI regulation directly, it identifies the need to harmonize with national and international standards for the development and deployment of AI, and the importance of ensuring that AI systems uphold Canadian values in line with the principles of international human rights law. The preamble also signals a need for a more agile regulatory framework – something that might go towards justifying why so much of the substance of AI governance in the AIDA has been left to the development of regulations. Finally, the preamble speaks of a need “to foster an environment in which Canadians can seize the benefits of the digital and data-driven economy and to establish a regulatory framework that supports and protects Canadian norms and values, including the right to privacy.” This, then, frames how AI regulation (and data protection) will work in Canada – an attempt to walk a tightrope between enabling fast-paced innovation and protecting norms, values and privacy rights.

Regulating the digital economy has posed some constitutional (division of powers) challenges for the federal government, and these challenges are evident in the AIDA, particularly with respect to the scope of application of the law. Section 4 sets out the dual purposes of the legislation:

(a) to regulate international and interprovincial trade and commerce in artificial intelligence systems by establishing common requirements, applicable across Canada, for the design, development and use of those systems; and

(b) to prohibit certain conduct in relation to artificial intelligence systems that may result in serious harm to individuals or harm to their interests.

By focusing on international and interprovincial trade and commerce, the government asserts its general trade and commerce jurisdiction, without treading on the toes of the provinces, who remain responsible for intra-provincial activities. Yet, this means that there will be important gaps in AI regulation. Until the provinces act, these will be with respect to purely provincial AI solutions, whether in the public or private sectors, and, to a large extent, AI in the not-for-profit sector. However, this could get complicated since the AIDA sets out obligations for a range of actors, some of which could include international or interprovincial providers of AI systems to provincial governments.

The second purpose set out in s. 4 suggests that at least when it comes to AI systems that may result in serious harm, the federal jurisdiction over criminal law may be invoked. The AIDA creates a series of offences that could be supported by this power – yet, ultimately the offences relate to failures to meet the obligations that arise based on being engaged in a ‘regulated activity’, which takes one back to activities carried out in the course of international or interprovincial trade and commerce. The federal trade and commerce power thus remains the backbone of this bill.

Although there would be no constitutional difficulties with the federal government exerting jurisdiction over its own activities, the AIDA specifically excludes its application to federal government institutions, as defined in the Privacy Act. Significantly, it also does not apply to products, services or activities that are under the control of the Minister of National Defence, the Canadian Security Intelligence Service, the Communications Security Establishment or any other person who is responsible for a federal or provincial department or agency that is prescribed by regulation. This means that the AIDA would not apply even to those AI systems developed by the private sector for any of the listed actors. The exclusions are significant, particularly since the AIDA seems to be focussed on the prevention of harm to individuals (more on this in a forthcoming post) and the parties excluded are ones that might well develop or commission the development of AI that could (seriously) adversely impact individuals. It is possible that the government intends to introduce or rely upon other governance mechanisms to ensure that AI and personal data are not abused in these contexts. Or not. In contrast, the EU’s AI Regulation addresses the perceived need for latitude when it comes to national defence via an exception for “AI systems developed or used exclusively for military purposes” [my emphasis]. This exception is nowhere near as broad as that in the AIDA, which excludes all “products, services or activities under the control of the Minister of National defence”. Note that the Department of National Defence (DND) made headlines in 2020 when it contracted for an AI application to assist in hiring; it also made headlines in 2021 over an aborted psyops campaign in Canada. There is no reason why non-military DND uses of AI should not be subject to governance.

The government might justify excluding the federal public sector from governance under the AIDA on the basis that it is already governed by the Directive on Automated Decision-Making. This Directive applies to automated decision-making systems developed and used by the federal government, although there are numerous gaps in its application. For example, it does not apply to systems adopted before it took effect, it applies only to automated decision systems and not to other AI systems, and it currently does not apply to systems used internally (e.g., to govern public sector employees). It also does not have the enforcement measures that the AIDA has, and, since government systems could well be high-impact, this seems like a gap in governance. Consider in this respect the much-criticized ArriveCan App, designed for COVID-19 border screening and now contemplated for much broader use at border entries into Canada. The app has been criticized for its lack of transparency, and for the ‘glitch’ that sent inexplicable quarantine orders to potentially thousands of users. The ArriveCan app went through the DADM process, but clearly this is not enough to address governance issues.

Another important limit on the application of the AIDA is that most of its obligations apply only to “high impact systems”. This term is defined in the legislation as “an artificial intelligence system that meets the criteria for a high-impact system that are established in regulations.” This essentially says that this crucial term in the Bill will mean what cabinet decides it will mean at some future date. It is difficult to fully assess the significance or impact of this statute without any sense of how this term will be defined. The only obligations that appear to apply more generally are the obligation in s. 6 regarding the anonymization of data used or intended for use in AI systems, and the obligation in s. 10 to keep records regarding the anonymization measures taken.

By contrast, the EU’s AI Regulation applies to all AI systems. These fall into one of four categories: unacceptable risk, high-risk, limited risk, and low/minimal risk. Those systems that fall into the first category are banned. Those in the high-risk category are subject to the regulation’s most stringent requirements. Limited-risk AI systems need only meet certain transparency requirements and low-risk AI is essentially unregulated. Note that Canada’s approach to ‘agile’ regulation is to address only one category of AI systems – those that fall into the as-yet undefined category of high ‘impact’. It is unclear whether this is agile or supine. It is also not clear what importance should be given to the choice of the word ‘impact’ rather than ‘risk’. However, it should be noted that risk refers not just to actual but to potential harm, whereas ‘impact’ seems to suggest actual harm. Although one should not necessarily read too much into this choice of language, the fact that this important element is left to regulations means that Parliament will be asked to enact a law without understanding its full scope of application. This seems like a problem.

 

Published in Privacy

As part of my series on Bill C-27, I will be writing about both the proposed amendments to Canada’s private sector data protection law and the part of the Bill that will create a new Artificial Intelligence and Data Act (AIDA). So far, I have been writing about privacy, and my posts on consent, de-identification, data-for-good, and the right of erasure are already available. Posts on AIDA, will follow, although I still have a bit more territory on privacy to cover first. However, in the meantime, as a teaser, perhaps you might be interested in playing a bit of statutory MadLibs…...

Have you ever played MadLibs? It’s a paper-and-pencil game where someone asks the people in the room to supply a verb, noun, adverb, adjective, or body part, and the provided words are used to fill in the blanks in a story. The results are often absurd and sometimes hilarious.

The federal government’s proposal in Bill C-27 for an Artificial Intelligence and Data Act, really lends itself to a game of statutory MadLibs. This is because some of the most important parts of the bill are effectively left blank – either the Minister or the Governor-in-Council is tasked in the Bill with filling out the details in regulations. Do you want to play? Grab a pencil, and here goes:

Company X is developing an AI system that will (insert definition of ‘high impact system). It knows that this system is high impact because (insert how a company should assess impact). Company X has established measures to mitigate potential harms by (insert measures the company took to comply with the regulations) and has also recorded (insert records it kept), and published (insert information to be published).

Company X also had its system audited by an auditor who is (insert qualifications). Company X is being careful, because if it doesn’t comply with (insert a section of the Act for which non-compliance will count as a violation), it could be found to have committed a (insert degree of severity) violation. This could lead to (insert type of proceeding).

Company X, though, will be able to rely on (insert possible defence). However, if (insert possible defence) is unsuccessful, Company X may be liable to pay an Administrative Monetary Penalty if they are a (insert category of ‘person’) and if they have (insert factors to take into account). Ultimately, if they are unhappy with the outcome, they can launch a (insert a type of appeal proceeding).

Because of this regulatory scheme, Canadians can feel (insert emotion) at how their rights and interests are protected.

Published in Privacy
<< Start < Prev 1 2 Next > End >>
Page 1 of 2

Canadian Trademark Law

Published in 2015 by Lexis Nexis

Canadian Trademark Law 2d Edition

Buy on LexisNexis

Electronic Commerce and Internet Law in Canada, 2nd Edition

Published in 2012 by CCH Canadian Ltd.

Electronic Commerce and Internet Law in Canada

Buy on CCH Canadian

Intellectual Property for the 21st Century

Intellectual Property Law for the 21st Century:

Interdisciplinary Approaches

Purchase from Irwin Law