Surgery does not look difficult to me

I’ve butchered a deer, dissected a cat, and sewn buttons and hems. The first time I observed a cardiac bypass I honestly thought: “That does not look very difficult—I could do that.” (Bear in mind, my degrees are in psychology and philosophy.)

Of course, surgeons spend 5-7 years in residency training after completing medical school. They usually know which incisions provide best access to targeted parts and which incisions heal best; how to avoid infections; where to find spare tissue for grafts; and when to ties things up and call it a day. Some develop the skill to tie knots they cannot see; some have incredible manual dexterity and can close an incision without leaving a scar. I know its best to leave surgery to surgeons.

I’m always astonished at the people who decide they’re just going to do a survey or conduct a few interviews. When they do I’m not surprised that their results are invariably worthless or harmfully bad.

As a general rule, you should not design a survey if you cannot explain:

-       What is a construct? And how do we know if we are measuring what we hope to measure (and not something very different)?

-       What is a representative sample, and when does representativeness matter?

-       How to estimate the power of a study.

-       What kinds of reliability are there? Which matter for your study?

-       How to set up a database, clean data, and analyze data.

No, you should not even write the survey items if you cannot explain:

-       Why the way you phrase an item will determine which statistic must be used (or whether any statistic can be meaningfully used)

-       What is a well-formed item or question? And why the word “and” should never appear in an item.

-       Why true/false items are famously bad.

-       What is content validity and how does it differ from construct validity?

A good methodologist knows the answer to all of these questions and has the skills to design your study and analyze your data.

To be clear, methodologists cannot design a good survey by themselves. Presumably, if you want to design a survey on a given topic (say, public attitudes toward biobanking), then you have some knowledge about the topic. This knowledge will be essential to writing items that have good content validity—and your average methodologist or statistician will lack this knowledge. This is why so few social science articles have just one author: A good survey usually requires the complementary expertise of multiple individuals.

However, collaboration alone is not sufficient to guarantee the quality of a survey. You need to collaborate with competent individuals, and the timing of the collaboration matters. Meet with a methodologist before you finalize your questionnaire items or interview guide. Even though you may think these matters pertain to content, the concrete form that survey items or interview questions take will influence the validity and reliability of your study.

Conclusion: Collaborate with a methodologist, and do it early in your project.