The Global Integrity Report comprises about 15,000 local, peer-reviewed assessments every year, all of which is available to public scrutiny. A few of our readers take the time to dig into this raw data and challenge our results. Here’s what that process looks like from the inside.

The short version is this: we benefit tremendously from outside criticism, and do all we can to encourage it. Global Integrity’s approach is to walk the talk on transparency and accountability, with our credibility based on something firmer than “trust us, it’s right”. This starts with rigorous local peer review prior to publication; after we publish, we look hard at our methodology in light of the feedback we receive, so we can learn and improve our future efforts. The following is an example of the latter.

A few weeks ago, a group of American and Canadian researchers had been looking at our data on judicial accountability in Canada related to a Foreign Policy article that used Global Integrity data. They wrote to us:

Response to Global Integrity Report

The Global Integrity Report marks an important effort to in assessing transparency and accountability in governments around the world. We are pleased that our research on judicial appointments has drawn attention to a lack of transparency in the selection of federally-appointed judges in Canada. On the accountability side, it was also fitting that Canada did not get top marks because the Canadian Judicial Council is composed solely of judges and has limited powers to discipline. However, in a number of ways we believe that the study’s methodology and conclusions overstate the lack of transparency/accountability in Canada. In no particular order, here are some reasons why putting Canada in the same category as Kenya and Mexico exaggerates the problem in Canada:

• The study lumps judges and administrative tribunal members together in many of the same categories. While members of administrative boards do adjudicate a number of important issues and deserve scrutiny, they should be treated differently than judges. Note, for example, while that Canada still has lax requirements for board members, judges must have considerable legal experience in order to be appointed. And, judges are able to perform judicial review (to various degrees) on the decisions of administrative boards.

• Whether there is a “confirmation process” is treated as a binary “yes” or “no”. Although sound arguments can be made for judicial confirmations by the legislative branch, one could also argue that confirmation hearings could undermine liberal democratic government by leading to a more politicized appointment system whereby judges are appointed who support the government in power– in turn, this could weaken the ability of the judiciary to act as a check against government and to protect minorities. A more rigorous system of appointment based on a nominating committee consisting of various stakeholders would help improve the situation in Canada, yet such a system would still be coded as “no” under confirmation process.

• The study codes countries according to structures and rules. These are important elements to consider, but the political and legal cultures of countries are usually as important (if not more so) to the functioning of democracy and legal systems as “rules on paper.” Although not without room for improvement, the political and legal culture in Canada does much to promote the “rule of law”. We are not experts in the political or legal systems of Kenya and Mexico, but we suspect that what happens “on the ground” in Canada would likely be more conducive to the rule of law than in those countries.

• The study only looks at federally-appointed judges. In countries like Canada and the US many important judicial decisions are made by judges appointed by provinces or states. In Canada, over the last couple of decades there have been considerable improvements to provincial judicial appointment systems and many of them are now much better than the federal system. With the problems associated with judicial elections in US states being highlighted more frequently in various studies, we think that including sub-national appointment schemes would raise Canada’s score relative to the US (and possibly other countries).

Sincerely,
Troy Riddell, Associate Professor, University of Guelph
Lori Hausegger, Assistant Professor, Boise State University
Matthew Hennigar, Associate Professor, Brock University

Our Managing Director, Nathaniel Heller replied to these points via email:

Dear Dr. Riddell — Many thanks again for you and your colleagues’ feedback…

To briefly engage on each of your points:

1) I agree that our research team “dug deeper” in the 2007 Canada assessment by assessing both judges and quasi-judicial administrative officials in the context of the judicial accountability indicators. This is a case where our indicators and their scoring criteria could potentially be unpacked further by replicating similar indicators for quasi-judicial administrative officials; we appreciate the insight and will factor it into our ongoing methodological refinement.

2) While we take no normative stance on whether formal parliamentary review of judicial nominees is a good or bad thing, we do believe that, in general, some form of oversight and public review of nominees is helpful. If you review the scoring criteria for indicator 33c (by hovering your mouse over the question mark), I do believe that Canada would/could earn a YES for an alternative, stakeholder review of nominees. The existing criteria for YES — “A YES score is earned if there is a formal process establishing a review of national-level judicial nominees by an agency independent from the body appointing the judges” — would allow for a positive score in the case of an alternative, stakeholder-driven review mechanism (for us, the term “agency” would be interpreted to include such a stakeholder committee). Indeed, the actual indicator question itself allows for such alternative review mechanisms: “In law, there is a confirmation process for national-level judges (i.e. conducted by the legislature or an independent body).”

3) I would agree 100% that implementation is where the rubber meets the road on these issues; that’s why our methodology has always attempted to assess both de jure mechanisms as well as their de facto implementation. In this case, the Implementation Gaps” we published for all three countries support your thesis — overall, Canada has a much smaller implementation gap than either Kenya or Mexico, implying that actual enforcement and implementation is superior in Canada (which makes intuitive sense to us all). In these particular indicators, while the potential for corruption in all three countries may be similar due to similar de jure gaps, I would not doubt that in practice actual judicial corruption is less in Canada, as evinced by the smaller implementation gap.

4) I also agree 100% on the need to look at sub-national mechanisms. In the Global Integrity Report, we strictly define our universe of issues to the national level in order to avoid a slippery slope effect in the data gathering process. You may be interested to learn that we are about to publish some pilot research carried out in Liberia assessing sub-national anti-corruption mechanisms (including sub-national judicial accountability) and are expanding that effort to three Latin American countries this year. Like you, I feel this new sub-national work is at the bleeding edge of the research agenda and we are excited by its potential.

Thanks so much again for the helpful feedback and thoughts.

Finally, we also asked our lead researcher for the Canada report, Duff Conacher (who had generated the results in question to begin with) to look at the final report and evaluate their concerns. Excerpts from his analysis:

re: “The study lumps judges and administrative tribunal members together in many of the same categories.” — I did this on purpose, as administrative tribunal members do adjudicate (as the professors note) and, therefore, I disagree with the professors’ argument that they should be treated differently than judges (in fact, administrative tribunals in Canada are often referred to as “specialized courts”).

– and in fact, many administrative tribunal rulings cannot be judicially reviewed by courts/judges at all, or can only be reviewed on very narrow grounds, because of privative clauses in the statutes that empower the tribunal, which is another reason to treat them the same as judges;

– also in fact, very bad administrative tribunal rulings are often not judicially reviewed simply because the person/organization on the losing end cannot afford the costs of such a judicial review, which is yet another reason to treat such tribunal members the same as judges;

– also in fact, administrative tribunals have been established in some cases in Canada (by the national, provincial and territorial governments) in the past few decades because Cabinet ministers did not want to face the scrutiny of the courts (ie. they were losing in court often) and so the Cabinet wanted to have instead a tribunal that is essentially under the control of Cabinet ministers (politicians) who appoint unqualified people who let the ministers off the hook when they violate the law, thereby greatly damaging the rule of law in Canada;

– Global Integrity could add to the Scorecard a separate category of questions for administrative tribunals, but I am guessing that many governments have not established such tribunals (or, at least, not as many of them as Canada has), and therefore parity of evaluations amongst various governments would be negatively affected by adding such a category (and, again, I see absolutely no reason for Global Integrity to treat such tribunals differently from courts);

– re: “A more rigorous system of appointment based on a nominating committee consisting of various stakeholders would help improve the situation in Canada, yet such a system would still be coded as “no” under confirmation process” question 33c — the professors are incorrect, if there was such a system in law (for example, if the Public Appointments Commission mentioned in my response under 33c is established and the Commission handles judicial appointments), I would put a “Yes” in answer to question 33c; in fact, the nominating committee of various stakeholders that the professors mention already exists for appointments of justices to the Supreme Court of Canada, but because it is not established in law it does not check in any way at all the power of the federal Cabinet ministers (politicians) to appoint whomever they want as a justice (the only legal check is the professional criteria, which I gave credit to as far as credit is due in my response to question 33b).

– re: “Although not without room for improvement, the political and legal culture in Canada does much to promote the “rule of law”. We are not experts in the political or legal systems of Kenya and Mexico, but we suspect that what happens “on the ground” in Canada would likely be more conducive to the rule of law than in those countries.” The reason I like Global Integrity’s methodology is that it provides a
detailed, evidence-based answer to these kind of vague claims that are based on “suspicions” and claims about what “would likely be” happening in any country; I believe that these claims and suspicions are simply biases that many, many people have in favour of their own country and against other countries; if you go to this Democracy Watch webpage, you will see links to news releases from the past 14 years about cases in which more than 30 federal Cabinet ministers (including the past three Prime Ministers) and their staff have violated the most fundamental good government laws and all of them were let off the hook — I “suspect that” these cases are just the tip of the iceberg in terms of how the rule of law is regularly violated in Canada, but I don’t know, which again is why I like Global Integrity’s methodology, because it does not allow my “suspicion” to be included in my responses to the Scorecard questions.

In the end, we decided to stick by our original assessment. However, it’s a useful case study of how complicated these issues can become, and also how deep the data going into these scorecards can be. The exchange above addresses 20 of the roughly 15,000 datapoints making up the Global Integrity Index in 2007.

— Jonathan Werve