Article Text
PDF
Abstract
Many family medicine and community health researchers use surveys as an original research methodology. Our purpose is to illustrate how survey research provides an important form of quantitative research that can be effectively combined with qualitative data to form a mixed methods study. We first provide an overview of the key principles in survey research and in mixed methods research. We review the various ways that survey can be used in mixed methods studies, citing options such as beginning a study with a survey, using a survey as the second form of data collection, or combining a survey and a form of qualitative data in a single data collection procedure. Finally, we illustrate in a specific example six steps in conducting a mixed methods study using survey research. In a mixed methods study using a survey, primary care researchers should consider six steps. Step 1. Articulate the rationale for mixed methods study. Step 2. Detail quantitative and qualitative databases. Step 3. Identify a mixed methods design. Step 4. Analyse and report the results of the quantitative and qualitative databases. Step 5. Present and show integration. Step 6. Explicate the value of using mixed methods. The ability to combine and integrate survey research into a mixed methods study provides a more rigorous approach to research than conducting only a survey or conducting just a qualitative interview. While requiring skills beyond traditional survey approaches, surveys in primary care offers an opportunity for a high level of sophistication in research methodology.
- mixed methods research
- survey research
- qualitative research
- family medicine
- general practice
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Many primary care researchers consider the implementation of a survey as means for original data collection. While there are many resources guiding the conduct of a survey, these resources typically have not been written with primary care researcher in mind. Due to many of the complex questions that arise, primary care researchers are often interested in assessing a phenomenon both quantitatively and qualitatively.
Both survey methods and mixed methods research are distinct methodology approaches in the health and social sciences. However, they can be combined in a single mixed methods study with appropriate planning and thought about their combined use. The purpose of this article is to present an applied discussion about how to develop surveys (or questionnaires) and use them in a mixed methods study. To this end, we will first discuss the basic principles involved in mixed methods research, highlight the essential characteristics of survey research and end with a discussion about the steps for using surveys in a mixed methods study. We will further explain the steps using a published mixed methods study in the health sciences.
Basic principles of mixed methods research
Founding
Mixed methods research as we know it today began between 1985 and 1990. At that time, several individuals founded this new methodology. They came from various fields and countries, such as evaluation, management, sociology, medicine and education.1 They were writing from Great Britain and the USA, and thus mixed methods can be seen as primarily an Anglo-American invention. They were not in touch with each other, and perhaps the most well-known writers were Greene from the USA and Bryman from England. Their impetus for developing mixed methods was a perceived need that both quantitative and qualitative research had value, and there was no need to keep the two forms of social and health research separate. This is despite differences in philosophical stances between the quantitative and qualitative researchers within social science sociology, anthropology, education, and evaluation perspectives and those with epidemiological orientations.
Designs and philosophies
Over time, probably during the mid-1990s, the idea began to form that mixed methods was collecting and analysing both quantitative and qualitative data, and also that additional insight might be gained from combining or integrating the two databases and linking them in a creative way. Thus, qualitative research explores phenomenon while quantitative explains the results of tests of hypotheses or research questions. Combined, mixed methods provides the insight of both exploration and explanation. By 2003, in the Handbook of Mixed Methods in Social and Behavioral Research,2 authors advanced specific designs or procedures for conducting mixed methods research. This was not an unusual development in research methodology in that specific designs were well known in types of quantitative research, such as experimental research,3 and in forms of qualitative research such as grounded theory.4 At the same time, individuals were developing an increased understanding of the philosophical underpinnings of mixed methods research that have unfolded in approaches such as pragmatism, critical realism, dialectic pluralism and yin/yang Eastern philosophy.5 These philosophical assumptions advanced the key idea that researchers bring to their mixed methods study core assumptions or beliefs that shape the types of procedures used in their studies. These twin developments—the designs or procedures and the philosophies—have led us to today a complete methodology called ‘mixed methods research’. Thus, to characterise mixed methods is to consider how it spans the process of research that includes a philosophy, an orientation to research problems and questions, specific approaches to collecting and analysing quantitative and qualitative data, and implications from the research that provides additional insight beyond what might be gained from simply collecting and analysing quantitative data or qualitative data.
Integration and basic characteristics
Further, mixing or ‘integration’ of the two forms of data has become a central feature of mixed methods research. Integration has developed as the buzzword for the innovative feature of mixed methods research, and it provides insight beyond what is learnt from the quantitative and qualitative databases separately. For example, the ability to compare the results of both databases provides a more complete understanding than either database alone. This procedure is called a ‘convergent mixed methods design’. It provides the opportunity to check one database against the other (how do people respond when they rate questions on a questionnaire vs when they are asked in person in an interview?).1 Insight can emerge from collecting survey data initially and then following up with interviews to help explain the survey results in more detail. This type of design is call an ‘explanatory sequential design’. Insight can also emerge from exploring first with interviews to understand the culture and specific perspectives of individuals, and then designing questionnaires or experimental interventions that respond to these cultural features (called an ‘exploratory sequential design’). A new feature to emerge in mixed methods is to think beyond these insights that come through integrating these ‘core designs’ into more complex procedures, such as experimental trials, social network analyses, evaluation procedures or into community-based health practices. These designs are called ‘complex mixed methods designs’.1 In short, what has evolved in recent years is an understanding of the key characteristics of a well-designed mixed methods study as shown in box 1.
Key characteristics of a well-designed mixed methods study
Researcher collects and analyses both quantitative and qualitative data.
Researcher engages in rigorous procedures with both databases, such as systematic sampling, adequate sample size, use of good instruments and protocols, multistep analysis and standards of qualitative such as validity, replicability, generalisability and accuracy of the findings.
Researcher combines (or mixes) the two databases in a systematic procedure called ‘integration’, where the overall intent is to bring the databases together, and to conduct analysis that accomplishes this aim and provides new insights.
Researcher conducts the integration within a type of mixed methods design, such as converging or comparing the responses, explaining the findings of one phase, or exploring first before measuring or assessing perspectives to build in a contextual, cultural feature of the study.
Researcher frames the study within larger philosophical assumptions, beliefs or orientations that the researcher brings to a study and incorporates into the study design a theory or conceptual model that informs what the researcher hopes to learn from the study.
Understanding the etiology of mixed methods research, how integration operates and the key characteristics of a good mixed methods project are essential understandings that lead to a contribution of mixed methods in family medicine. For researchers in the health sciences, the ‘Best Practices in Mixed Methods in the Health Sciences’ from the National Institute of Health6 is a useful guideline for understanding an applying mixed methods research.
Basic principles of survey research
History
Survey research is a quantitative approach to social and health science research. Survey research designs are a set of research procedures in which investigators administer a survey to a sample or to the entire population of people to describe the attitudes, opinions, beliefs, perceptions, behaviours or characteristics of the population. In this procedure, survey researchers collect quantitative, numbered data using questionnaires (eg, mailed questionnaires) or interviews (eg, one-on-one interviews) and statistically analyse the data to describe trends about responses to questions and to test research questions or hypotheses. Researchers also interpret the meaning of the data by relating results of the statistical test back to past research studies. Surveys typically involve quantitative items, but researchers might include qualitative open-ended questions as well.
Survey research evolved as a research methodology during the 20th century. During World War II, for example, US surveys examined issues central to the war effort, such as the morale of soldiers, production capacity for weapons and the effectiveness of strategies. Through these studies, survey researchers refined and developed their techniques of large-scale assessments, enabling the emergence of large social research organisations in American universities after the war. For example, investigators established social research centres at the University of California, Berkeley (Survey Research Center), at the University of Chicago (National Opinion Research Center) and at the University of Michigan (Institute for Social Research). In addition, opinion polling organisations, such as Gallup, Roper and the RAND Corporation, furthered the understanding of large-scale data collection. The founding of polling and survey organisations, combined with the use of computers, the availability of data archives and storage, and funding from the federal government, helped to establish the popularity of surveys in research by the mid-20th century. In recent years, both federal and state governments have funded national and state surveys. Recently, individuals have increasingly used the Internet to collect survey data. Researchers can now generate an online survey to easily administer to anyone with Internet access.7
Types of surveys
Surveys can be conducted over time, longitudinal surveys, or administered at one point in time, cross-sectional surveys. In mixed methods research, cross-sectional surveys are frequently used. The key characteristics of good survey work include several features. The participants who fill out a survey are individuals in a specific population (eg, individuals with chronic heart failure). Survey researchers then select a sample from this population to identify individuals to complete the survey. The most rigorous form of survey sampling is probability sampling, where each individual in the population has an equal chance of being selected. The idea, then, is that the researcher draws conclusions from this sample to make inferences about the entire population. Thus, it is not necessary to sample the entire population. However, the size of the sample receiving the survey is important, and it is useful to select as large a sample as possible. Sample size tables in survey texts help identify the appropriate number.8
Different forms of surveys exist, and they can be broadly grouped into questionnaires and interviews. A questionnaire is a form used in a survey design in which participants in a study complete a mailed instrument and return it to the researcher. The participant chooses answers to questions and supplies basic personal or demographic information. An interview survey, however, is a form on which the researcher records answers supplied by the participant in the study.
A survey researcher does have the choice of different types of questionnaires and interviews: a questionnaire mailed to the participant to be filled out, a questionnaire administered online through the Internet or through email, an interview used in a one-on-one interview or with a small focus group, or a telephone interview.5 Of these types, mailed questionnaires are the most popular form used in mixed methods research. Also, increasingly popular is the use an Internet survey with a software program such as Qualtrics9 or Survey Monkey.10 In the health sciences, focus groups are frequently used, and in these the researcher identifies an instrument to record data, convenes a small group of individuals (ie, typically six) and holds a discussion about questions on the instrument.11
Use of a survey and common problems
In terms of a survey instrument to use, researchers can develop their own survey instrument (requiring skills in scale development and design), use an existing instrument or modify (with the authors’ permission) an existing instrument. If the researcher chooses to design their own instrument, the instrument will involve different types of attitudinal or behavioural questions, the instrument needs to have psychometrically rigorous question construction and the instrument should to be pilot-tested with a few individuals before its general administration. In a pilot test, the researcher administers the instrument to a small group of participants, asks them to comment on any weaknesses in the instrument, and then modifies or changes the instrument. Further, conducting cognitive interviews of the items to ascertain that the survey participant interprets the meaning of questions as intended is important. The types of questions the instrument will include closed-ended questions with items that require a check for the most appropriate answer, such as strongly agree, agree, undecided, disagree and strongly disagree. Also, the instrument may include open-ended questions which will result in collecting qualitative data from questions that permit the participant to give short answers. Common problems in constructing these questions include using vague words, asking multiple questions in a single question, writing wordy or lengthy questions with many parts, using negative or jargon language, having response categories overlap or be unbalanced, and having a mismatch between the question and the responses.
Analysing a survey
It is useful to consider the major components of a good mailed survey instrument and how it is analysed.5 It should include a cover letter to the participants asking for their input, have an introduction to the survey stating the reasons for the survey, include questions of a length that could be reasonably answered by the participants and have closing instructions thanking the participant for their help in providing data. The questions or items on a survey instrument need considerable thought. Basically, four to five items can be grouped into a variable that measure attitudes and behaviours. Then these variables can further be grouped into scales. An instrument thus would contain several scales. Knowing this, the data analysis steps on data collected on an instrument can now be identified as illustrated in table 1.5
Steps during the analysis of survey data
Various ways surveys can be used in mixed methods research
Up to this point, we have discussed the basic ideas involved in both mixed methods research and survey research. Both of these can be viewed as distinct methodological approaches for a study in the health sciences and they need to be rigorously conducted. The options for researchers to bring surveys into mixed methods research can be seen in table 2.
Options for using survey in a mixed methods research investigation
Example of using survey research in a mixed methods study
We will use an explanatory sequential design in mixed methods because this is a design that honours a strong quantitative initial data collection, is easy to administer and is one often used by beginning mixed methods researchers. Our specific example would be the study by Sonnenberg et al 12 on an assessment of resident physicians’ communication and collaboration competencies by an interprofessional (IP) team (clinicians, pharmacists, laboratory technicians, therapist and others).12 This is a study in competency-based medical education where there is increasing emphasis on assessing resident performance in abilities to communicate and collaborative effectively in the workplace. This assessment can take place by non-physician members on IP teams, and such teams who have valuable expertise have not been adequately engaged in resident skill assessment.
Thus, the authors in the Sonnenberg et al 12 study used an explanatory sequential mixed methods design to examine the ability of the IP clinicians to provide feedback to pediatric residents during their rotation.12 Using survey research in the first phase, the researchers compared IP supervisors and physician supervisors in terms of communication and if collaborative training objectives were met. These objectives were based on the widely used CanMEDS Roles Framework developed by the Royal College of Physicians and Surgeons of Canada. In this Framework, seven essential skills for medical practice were identified (eg, communicator, collaborator, leader, health advocate, scholar, professional and expert). Then, in the second phase, the researchers conducted follow-up qualitative focus groups to probe a contextual understanding of the factors that influenced the process of assessment. Thus, the purpose of this mixed methods study was to examine IP clinicians’ perceptions of their ability to provide formative feedback compared with physician supervisors on the rotation and to qualitatively explore potential barriers to the feedback process.
What were the steps they engaged in to conduct this study? These steps could be applied to many mixed methods projects using survey research. See table 3 for information from the Sonnenberg et al example.12
Steps in a mixed methods survey investigation as illustrated by the Sonnenberg study
Step 1. Rationale for mixed methods. Determine if mixed methods research is the best methodology to use to answer your research questions. Provide a reason for the use of mixed methods in the project.
Step 2. Quantitative and qualitative databases. Identify the types of quantitative and qualitative data to be collected and analysed. Consider these two types of data as distinct databases. In a mixed method research study, it is important to collect and analyse both quantitative and qualitative data. Use rigorous survey procedures and rigorous qualitative procedures.
Step 3. Mixed methods design. Determine what mixed methods design you will use. Draw a diagram of the design. Mixed methods researchers like to have a visual picture of their procedures. The diagram needs to be simple and straightforward without complicated components.
Step 4. Analyse and report the results of the quantitative and qualitative databases. Present the results of the study showing the quantitative statistical results and the qualitative thematic results. Keep these discussions of analysis separate under distinct headings. They should also mirror the steps in the design, and show a clear linkage between the quantitative and qualitative components in the study.
Step 5. Present and show integration. Pay attention to the point of integration of the two databases and make it specific in the diagram of the design. Use an arrow in the diagram to show the point of integration of the two databases. Discuss the ways integration appears in the study. There are two major ways of representing integration. One is to simply discuss the quantitative and qualitative results side by side in a discussion. The other way is becoming popular in mixed methods research: a joint display. A joint display is typically a table in which the quantitative and qualitative results appear side by side. In this way, comparisons can be made between the quantitative and qualitative results. In an explanatory sequential design, the quantitative results are first reported in a first column, and the qualitative results that help explain the quantitative results appear in a second column. Further, a third separate column shows the impact of the integration in the study. A template is useful for considering the type of joint display that can emerge using an explanatory sequential design as an example.
Step 6. Explicate the value of using mixed methods. In a discussion section at the end of the study, include comments about the value of using mixed methods as a methodology. In this way, readers of the study will see the utility of mixed methods in research. This means drawing the implications of using both quantitative and qualitative data in a mixed methods study.
Additional resources to consider
In addition to Creswell and Creswell, when conducting mixed methods survey studies, it is important to study survey research and consult books on it by leading writers such as Babbie, Fowler and Dillman.5 8 13 14 In mixed methods, consider introductory mixed methods texts by Plano Clark and Ivankova, Creswell and Creswell, Plano Clark, and Creswell, and Guetterman as important resources.1 15–17 Look for practical guides to both survey and mixed methods research.
Discussion
The illustration of using a survey in mixed methods based on an explanatory sequential design is simply one possibility of using surveys. However, it provides an opportunity to see how a survey fits into mixed methods. In order to conduct mixed methods research, investigators need to know the basics of both mixed methods research and survey research, and recognise the various ways the two methodologies can be combined. Further, this combination is a rigorous design that takes time for both the quantitative and qualitative components. Some researchers may find it more economical and time-saving, to employ only quantitative or qualitative research. Knowing how to effectively combine the two methods requires knowledge across multiple research methods. Still, surveys combined with mixed methods research leverage two useful approaches.
Footnotes
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; internally peer reviewed.