A2: Drafting Survey Instruments

Published

October 25, 2024

Overview

Welcome to second assignment where you will start to design your survey instruments. Your goals for this assignment are:

  1. Clarify your research question
  2. Draft at least five questions in Qualtrics that will let you address this question
  3. Download a pdf of your draft questions and upload that pdf to Canvas.

Before Section:

Before section on Wednesday please do the following:

  1. Create shared Google drive folder where you can store documents for project and collaborate with your peers.
  2. Create a common Google document where you can share draft questions and materials.
  3. Make sure everyone in your group as well as myself have access to the folder and the materials
  4. Preview the draft of the common content for our class survey here. Is there anything you think we should add?
  5. Log on to Qualtrics with your university ID: http://brown.qualtrics.com/ Play around with the questions.

Section.

In section, we will demo some basics of designing surveys in Qualtrics.

Then you will collaborate with your group and work on designing your modules (sections of the survey).

One person should create the survey instrument and invite the other members of the group to collaborate on their version of the instrument (Cory will walk through the specifics.)

By Sunday, you will have a draft of at least five “questions” that speak your group’s research question.

Questions include experimental treatments. So if you want to randomly assign some survey takers to read a news article, that article counts as one question. If your article is really long, or if you’re using a video, that experimental treatment may count toward two of your allotted questions.

You may include additional questions beyond your five, but we may ask you to cut them. You may also make the case that some questions should be included as part of the common content. You may want to check with other groups to get a sense of what they’re doing and if there’s over lap in your research interests.

What to Submit by Sunday March 5th, on Canvas

By Friday October 25th, please:

  • Download a pdf of your draft instrument (I will show you how) and upload that to Canvas and a 1 page memo summarizing your revised research question and any additional questions/requests you have.
  • I’ll provide you comments feedback so that for your next section on, you can finalize and revise your instrument and start thinking about how you will analyze your results.
  • Please invite me to collaborate on your survey instrument as well.

General Adivce:

  • Keep it simple. Less is more. Focus on answering one or two question really well, rather than trying to test a bunch of different hypotheses

  • If doing an experiment, try to focus on a simple treatment and control comparison, or at most a two-by-two factorial in which two things are being randomly assigned (e.g. you vary the gender and partisanship of a candidate). Don’t worry about the specifics of implementing the randomization, we will help you with that.

  • If doing an observational/correlational study, think deeply about what your outcome is, what you think predicts variation in that outcome, and factors that might be related to both your outcome and your key predictor. How will you measure these concepts in your study.

  • Put yourself in the survey takers’ shoes. How would you feel answering this question or reading this treatment?

  • If you can format a question so that they have the same response structure (strongly agreee to strongly disagree), you can get more bang for your buck.

  • When in doubt, randomize. If you’re worried the order or wording of question may influence the results, consider a randomly assigning the order or format of the question (in addition to any other randomizations you may be thinking about).

  • Online surveys typically require less explanation than phone or face-to-face surveys. If you’re copying question formats from the NES or Pew or Gallup, consider simplifying the language if it improves the clarity.

  • Online survey takers hate writing. Analyzing writing is very hard and labor intensive. For better or worse, fixed response options are probably best for this context.

  • Video is appealing but harder to manipulate. You have more freedom and control working with text and images.