Some comments on results analysis

Question

Like many others on here, I have studied the result analysis many times before taking the exam (mostly to get my head of the hard stuff). Although I find your analysis interesting and I do not want to be seen as the advocatus diaboli with my second post, I feel I have to make you aware of some things, that might have biased your results. I am a PhD student and am obliged to read, analyze and tear apart the work of others on a daily basis and therefore I find it easy to spot loopholes in other peoples analyses, without claiming that I could do any better.

So here we go:
1) the estimation for the MPS seems solid - so much for that :)
2) my critique mainly concerns the part where passing candidates are compared to non-passing candidates concerning their work and educational background. My suspicion is that there is something more hidden in the data, that you did not ask for. To me, it makes no sense at all that people with finance education only background would perform so badly compared to people who have only work experience. My guess is that the real causal relationship here exists between the country of origin of test takers and fail/pass as far as the education only category is concerned. Why? I think we can all agree that most edu-only test takers live in countries outside the US/EUROPE etc., because it is commonly known that students in say India or Russia try to obtain the CFA designation to compensate for the fact that they did not have to opportunity to study at Harvard or Wharton. Do I intend to say that those people are not as smart as their US counterparts? Not at all! I will explain why this group is a drag to the passing percentage in the following points.
3) LANGUAGE BIAS! It is so obvious, but you did not ask in your analysis whether the exam takers were English native speakers! I do not intend to say that failing candidates might have poorer language skills but we can all agree that the CFA exams are a challenge by themselves; taking the exams in a language that you are not native to, however, adds significantly to the level of difficulty!
4) CULTURAL BIAS! It may sound strange at first but culture does play a role when taking the CFA exam. I am not only talking about different exam taking styles in different cultures (if your from the US you are highly familiar with exams of that type, if you are from say France where you have probably never dealt with MCQ in your live you are in for a treat). Closely related is also the notion of some common sense type knowledge. If you were born and raised in the US, the term T-Bill was probably used a million times in the news you watched, the papers you read or even the discussion you had with family and friends; you already know what it is. For someone from Italy the word T-Bill entered the persons lifeworld maybe only a few years ago and is still shrouded in mystery and ambiguity.

My suggestions for upcoming analyses:
+ ask people for their English language profeciency
+ ask people for their country of origin
+ use a non-parametric regression model to obtain confidence estimates

PS: I opened a thread for this topic so other people could share their thoughts too. I do not intend to criticize the CFA exams in any manner for being US-centric, since everyone knows what they are in for when signing up for these exams.

christine · Answer

That's great feedback @kocmohabt. We've had similar thoughts as well on your point #2, and we're constantly evolving the questions to better analyse the results.

We try and ask the minimum number of questions to keep the survey as short as possible, but we will look into including these questions in the next exam cycle.

Thanks again for the suggestions!

Sophie Macon · Answer

@KOCMOHABT, as a fellow statistic nerd, I fully understand where you're coming from. As per @Christine's comment, we thought of these points when implementing the survey at the beginning, but decided that too many questions may reduce response rate, and we all know we want a sufficiently representative pool of people to study on. We may add the other question in the next cycle, and study if these factors are statistically significant, controlling for other factors.