Seen the “98% of studies were ignored!” one doing the rounds on social media. The editorial in the BMJ put it in much better terms:
“One emerging criticism of the Cass review is that it set the methodological bar too high for research to be included in its analysis and discarded too many studies on the basis of quality. In fact, the reality is different: studies in gender medicine fall woefully short in terms of methodological rigour; the methodological bar for gender medicine studies was set too low, generating research findings that are therefore hard to interpret.”
They all use the same Newcastle-Ottawa system to score studies based on their likelihood of bias in the exact same way the Cass reviews do. The method you described as a joke.
It’s not an indicator of bias, no causal study has been done to show that there is a relationship between bias and the Newcastle Ottawa scale
Studies that self select their cohort and don’t include adequate controls are more susceptible to bias than those that do otherwise. Evaluating studies based on their susceptibility to bias is a vital part of the systematic review process.
You can read more about it here https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp
But not actually proof of bias.