Third Millennium Analytics, Inc
Third Millennium Analytics, Inc
Research Methods

The effectiveness of data-driven decisions is highly dependent upon the validity and reliability of the data from which those decisions are based.  We, at Third Millennium Analytics, Inc. can help you maximize the quality of your data by selecting the most appropriate research design for your project, implementing statistical power analysis, selecting appropriate sampling designs and procedures, designing surveys and constructing questionnaires, selecting or constructing measurements, scales or indexes, conducting reliability and validity assessments, and factor analysis.  Each of these services is described in more detail below.

Research Design.  After carefully considering study objectives, resource constraints, and ethical considerations, we can assist in determining the most appropriate and useful research design for your project.  We are familiar with a wide variety of designs, including cross-sectional, pre-experimental, experimental (e.g., classical, double-blind, Solomon four-group, Latin Square designs, fractional, factorial), and quasi-experimental designs (e.g., time-series, multiple time-series, nonequivalent control group designs), strategies for designing effective longitudinal studies (trend, cohort, and panel studies), and nested designs (e.g., cluster- or group-randomized trials). 

Statistical Power Analysis.  Statistical power pertains to the probability that a significant difference will be detected in a sample if a difference actually exists in the population.  Ideally done at the early stages of a research project, power analysis can be implemented to determine the sample size that is most appropriate for your study, so that the study will be “sufficiently powered” for statistical testing.  Decisions about proper sample size are usually also made while considering the level of precision desired and available resources.  Larger samples usually have more power than smaller ones.  Oversampling would clearly lead to a waste of valuable resources, as would under-sampling, because with insufficient power, a significant association or difference can go undetected.

Sampling Frames and Procedures.  Ideally samples are representative subsets of populations from which they are drawn.  Representative samples give you confidence to generalize the results to the population you're seeking to understand.  While probability sampling methods provide ways of deriving representative samples, there will always be some degree of sampling error (i.e., samples will never provide perfect representations of populations).  Common sampling techniques include simple random sampling, systematic sampling, stratified sampling, multistage cluster sampling.  Though easier and cheaper, non-probability sampling methods (e.g., purposive and quota sampling) are less reliable and thus are rarely recommended.

Survey Research and Questionnaire Design.  Surveys that respondents find interesting and engaging, and are easy to complete (because of a variety of secure survey options), result in high quality data, that are more complete and useful.  We can reach survey respondents through a variety of channels, including online and mobile, in person (e.g., face-to-face interviews, paper-and-pencil, ACASI), mailed, or by phone, as well as in a variety of languages.  Our application of multimodal research can help enhance response rates and data collection efficiency, help maintain consistent sampling across modes, and improve data quality (e.g., by limiting missing data).  Our experience with mailed surveys includes the effective design of cover letters with questionnaire instructions, as well as various techniques for maximizing response rates.

Assistance with survey design and questionnaire construction is available, including differentially structured questions based on factual or subjective experiences, question formatting (rating, matrix questions, semantic differentials, and rankings), properly sequencing questions, as well as advice on how to avoid various pitfalls that may cause questionnaire bias.  We can help with proper question wording, and advise when closed-ended, open-ended or contingency questions are warranted.  We also provide guidance to avoid response set bias, and leading, threatening, or double-barreled questions.  

Measurement/Scale and Index Construction.  When pre-existing scales/indices (i.e., with acceptable and well-established psychometric properties) are not available or amenable for your purposes, we can assist with the conceptualization and construction of new and innovative composite measures.  We can help select the most appropriate scaling procedures for your measures (e.g., Bogardus social distance, Thurstone , Likert, semantic differentials, Guttman scaling), and conduct validity and reliability assessments.  Assistance can be provided for typology construction and analysis, which is needed when the intersection of two or more variables need to be summarized, resulting in the creation of sets of nominal categories or types.

Factor Analysis.  Factor analysis is applied to identify relatively few unobserved variables called “factors” by describing the variability among a set of more numerous, observed, and correlated variables.  The aim is to detect dimensions among a large number of interrelated variables.  Exploratory factor analysis (including the related techniques: principal component analysis, canonical factor analysis, and common/principal factor analysis) is employed when there are no a priori expectations about the number of factors, and any indicator can be associated with (or “loaded onto”) any factor.  When conducting confirmatory factor analysis, however, hypotheses are tested about the number of factors and loadings in order to confirm whether they support theoretical assumptions.