Why Can't We Compare Complaints?

When the Accountability Console was first developed in 2017, there were 956 known complaints across 13 Independent Accountability Mechanisms (IAMs). As of this year, after the Research Team at Accountability Counsel completed a comprehensive update of complaint data across 17 IAMs, that number has grown to over 1,800 complaints. The work of updating, standardizing, and maintaining this dataset is a significant use of our research team's time and resources, and is still constrained by the wide differentiation in reporting standards and transparency practices of different IAMs. 

The lack of a shared "complaint vocabulary" across the various IAMs hampers community comprehension, broader public engagement, and system-wide analysis. As the number of complaints grows, so does the need for a standardized method to assess, categorize, and report complaints. This article delves into the importance of developing a shared perspective and vocabulary across the expanding IAM ecosystem, the hurdles that must be overcome to do this effectively, and some ethical considerations for ensuring that data standardization does not result in the further marginalization of community voices.

Toward a Shared Understanding of Complaints

The Importance of Standardization

Given that there are many International Financial Institutions (IFIs) and, by extension, multiple IAMs, it's vital to have a standardized method to: 

1. Compare data across IAMs: Without standardization, drawing comparisons and deriving insights becomes a convoluted process, often leading to misinterpretations and missed opportunities for improvement.
   
   
2. Provide insights into the efficiency and effectiveness of IAM processes: Aggregate data allows us to analyze the volume, nature, and outcomes of complaints, and for stakeholders to gauge the effectiveness of IAMs' outreach efforts, the impact of additional resourcing on complaint management, and the ways that particular policies may impact complaint trajectories.
   
   For example, if one IAM has a notably higher rate of dispute resolution processes leading to an agreement, it might indicate that certain aspects of their mediation or problem-solving approach could be benefiting other IAMs. Conversely, it could highlight that another IAM is not as accessible or well-known to potential complainants. Similarly, the proportion of complaints deemed "eligible" by a single IAM can range from under 7% to over 70%; a disparity that is hard to decipher because of a combination of different eligibility criteria, different outreach and community engagement practices, and different reporting standards. All three require attention and work but setting common reporting standards can at least help identify what the other differences may be. 
   
   
3. Allow for the identification of patterns, trends, and systemic issues: Adopting a shared perspective and data standard across IAMs allows for the identification of patterns, trends and systematic issues that may not be apparent when considering individual complaints.
   
   For instance, we identified earlier this year that complaints raising concerns of displacement have risen sharply in the past few years, but without a shared understanding of how IAMs define and categorize complaint issues, it's prohibitively difficult for others to identify these types of systemic commonalities without countless hours of data processing. 
   
   
4. Identify cases where a project is receiving multiple complaints across multiple IAMs: Perhaps even more pressing, the lack of shared vocabulary around complaint and project identification can make it near-impossible to identify cases where a project is receiving multiple complaints across multiple IAMs. If each IAM is using different complaint names, and different terminology about the project under consideration (if they provide any information on the project at all), it becomes very difficult to know whether complaints at different IAMs are related to one another. Looking at the totality of complaints a project receives is a critical way to understand the harm that project has caused over its lifecycle, but at the moment this is beyond the capacity of even the Accountability Console. 

Promoting Transparency and Proper Disclosure

Transparency is a cornerstone of accountability. Facilitating proper disclosure of complaints ensures that all relevant complaint information is available for scrutiny and systematic improvement. Without clear standards of disclosure, there's a risk that information may be omitted, misrepresented, or misunderstood. 

Maintaining a consistent format and a clear set of guidelines ensure that every IAM discloses information in a manner that is both comprehensive and easy to understand. In order to know how IAMs are working, stakeholders, including affected communities, governments, and civil society organizations, need to be able to access clear and standardized information. Such data is important not only for those involved in a particular complaint who may need to see information relevant to their case, but also for anyone trying to learn from past complaints how to improve project performance and reduce the risk of harm.

The need for confidentiality is all too often a convenient excuse to circumvent meaningful information disclosure. While safeguarding complainants’ security may sometimes necessitate withholding certain information, the uneven information disclosure across mechanisms implies that other considerations are likely at play. For example, in dispute resolution agreements, bank clients may not wish to disclose the full details of their commitments, but, without at least some public disclosure of the nature of these commitments it becomes impossible to assess whether or not these commitments have been meaningfully implemented. To promote healthy transparency, full disclosure of an agreement's commitments should be the norm, with exceptions only made for extenuating circumstances related to the security of complainants. 

Completeness of Complaint Data

Standardization is not just about consistency; it's also about completeness. Every complaint, whether it meets the criteria for formal consideration or not, provides valuable information for both future complainants and the IAMaccountability mechanism itself. For instance, an unregistered or ineligible complaint might indicate areas where IFI policies are not clearly understood or where a gap in understanding might exist.

Several mechanisms do not include ineligible complaints in their case registries, referencing them only in appendices in annual reports or in hard-to-find PDFs with minimal information. By ensuring that even unregistered or ineligible complaints are properly categorized, detailed, and included in complaint registries, IAMs can derive insights into potential areas of improvement. This can range from enhancing outreach and education efforts to refining the complaint submission process to be more accessible and comprehensible.

How Do IAMs Fare?

We've identified 12 criteria for data openness and transparency and have assessed the 10 largest IAMs on each of them. The criteria are as follows:

Standardization

• Complaint ID: Are complaints given a unique identifier beyond the project name?
• Complaint Status: Does the registry display the current status/stage of the complaint?
• Ineligible Complaints: Are unregistered and/or ineligible complaints available on an interactive, searchable webpage?
• Complaint Summary: Does the IAM summarize the complaint issues and the trajectory of the complaint?
• Project Link: Does the complaint registry provide an easy way to access the associated investment page of the bank?
• Complaint Dates: Does the registry include the start and end dates of each stage of a complaint?
• API Availability: Is there an Application Programming Interface to engage with complaint data programmatically? 

Accessibility

• Date Last Updated: The date of the most recent update we found on a complaint registry.
• Complaint Webpage: Are complaints listed on an interactive, searchable webpage?
• Complaint Guide: Does the IAM provide an easily accessible guide for how to file a complaint?
• Complaint Template: Does the IAM provide an example complaint letter, template or online form?
• Complaint Graphic/Flowchart: Is the lifecycle of a complaint represented graphically?
• Access to documents: Does the registry consistently provide access to all relevant (non-confidential) complaint documents?

Dangers of Standardization

It's important to acknowledge that increasing standardization across mechanisms also introduces some risks that must be addressed and managed along the way. A recent paper by Alex Hanna and Tina Park raises some important reflections in this regard that are relevant to the IAM ecosystem:

• Loss of Nuance: One of the primary risks of standardizing data is the potential loss of nuanced information. The process of making data fit into standardized categories might strip away contextual details that are crucial for understanding the intricacies of a complaint or the specific challenges faced by a community. This loss can lead to oversimplified interpretations and misguided solutions.
  
  
• Marginalization: As highlighted in Hannah and Park's exploration of scale thinking, there's a danger in aiming for a "universal" approach that might neglect or marginalize certain groups. In the context of IAMs, a standardized data approach might overlook complaints from certain communities or sectors, especially if these complaints don't fit neatly into predefined categories.
  
  
• Over-reliance on Quantitative Data: While quantitative data can offer valuable insights, it's just one piece of the puzzle. Overemphasis on standardized, quantifiable metrics might overshadow qualitative information, which often provides rich context and depth to the issues at hand.
  
  
• Potential for Misinterpretation: Standardized data, when removed from its original context, can be misinterpreted. For instance, if a complaint is categorized under a broad umbrella term, stakeholders might miss the specific aspects of the complaint that need addressing.

Conclusion

As IAMs grow in scale and scope, the importance of developing a shared vocabulary and approach cannot be overstated. As one of the only bulwarks against harm from IFI-financed projects, we must ensure that IAM processes and outcomes are transparent, accountable, and effective.

Data standardization across IAMs is not just a technical necessity; it's a fundamental step towards building trust, ensuring accountability, and promoting a culture of continuous improvement in the global financial landscape.