Innis E. Bryant, Sr. School of Business, Northcentral University BUS-7320 v4: Quantitative Research Design and Methodology Lawrence R. Ness, Ph.D. November 13, 2022

Innis E. Bryant, Sr. School of Business, Northcentral University BUS-7320 v4 : Quantitative Research Design and Methodology Lawrence R. Ness, Ph.D. November 13, 2022.

[Audio] This presentation will zero-in on internal and external validity term, their threats, examples of those threats, and provide plausible research strategies that could be used to ameliorate the threats. It is imperative that the research one conducts carries valid and reliable information ( Jimenez-Buedo & Russo, 2021). While, the internal validity poses several probes regarding the research in order to address the 'how' components of the methodology, the external validity foci are on how practical the end state data are in an every day life ( Salkind, 2012b). Combined, both internal and external validity complement each other's aspects of the research in that they focused on inside out and outside in questions of the research ( Jiminez-Buedo & Russo, 2021). Why is this important? This is important in order to connect the tenets of research to every day life and vice versa. A researcher may be able to purport a valid reason why a factor is entirely applicable to someone in every day life; however, when it comes to justifying the validity of that statement, without the use of research based protocol that can support the claim, such statement becomes powerless. The complements of internal and external validities scissor their way through reliability and relevance of the research and outcomes (Salkind, 2012b). Are internal and external validity characteristics intertwined? Both internal and external validity characteristics share common denominators. Considering these common denominators is crucial in the planning stage of the research in order to garner meaningfulness throughout the research (Jimenez-Buedo & Russo, 2021). As such, they are not mutually exclusive. A researcher's task is to determine the impact of intersection internal and external powers have on the intent of the final outcome (Jimenez-Buedo & Russo, 2021). What are their similarities and differences? Both internal and external validity are individual values in a research within a framework. External validity has a task of determining the relatability of the conclusions to the real life 9Salkind, 2012b). Internal validity will demonstrably show variations that stem from the impact of the independent (manipulated) and dependent (measurable) variables. External validity will demonstrably define and generalize the applicability of the research in an everyday life. (Salkind, 2012b)..

[Audio] Definition: Internal validity is a statement of confidence in relation to the relationship between the manipulated variable, which is manipulated, and the independent variable, which is measured. What does internal validity bring to the research table? It brings to the table the manner in which a study manifests a dependable cause and effect relationship between the value of treatment and the value of consequence ( Salkind, 2012b). Internal validity has a task of ruling out secondary and tertiary (alternate) causes for a result. For instance, in a clinical setting a medical researcher studying alcoholism could use a drinking interruption method must be confident that when improvement in behavioral outcomes are manifested they manifested because of the methodology. Does this mean that internal validity is a zero-sum method? Internal validity is not a binary method of yes and no ( Lin et al, 2021). Each researcher must examine their competency range in that there should not be any room for pitfalls that can compromise the integrity of the results. This means that a researcher should scrub any ambiguity off the research in order for the results to show confidence ( Lin et al., 2021). What ambiguity is one to watch for? Ambiguities are impacts from the external variables that have not ben anticipated and discombobulate the results of the research ( Slocum et al., 2022). The simplest way to unveil internal validity is to ask: Do the results represent the population in the research? If not, they are adulterated and invalid..

[Audio] Internal validity faces several challenges that can destabilize it. There are multiple threats to internal validity. 3 Categories: Single Group Threats, Multiple Group Threats, and Social Threats Specifically, there are at least 12 common threats, of which this presentation will only focus on one- third of them. The subsequent slides will provide the specifics of each threat; for now, the presenter will define each of the threats in order to provide some content backdrop for the audience to refer to throughout the presentation. For the purpose of this presentation, following are the threats that will be defined and exemplified. Diffusion: diffusion threat spreads from the treatment group into the control group in which there are no boundaries between the groups, such as interaction and observation. Instrumentation: unintended or intended guiding of the research group that will cause it to behave in ways that otherwise, when left alone, the group would not behave in that manner. Measures are completely different between the pre and post tests. Maturation and/or Death: when the aspect of time impacts the results to the point that they would be questioned, maturation/ death threat will compromise the study. Geriatric medical protocols suffer with this threat as the patience either age or succumb to the side effects of the illness or treatment. This can also be present when wholly new subjects are added to the existing gaps among already experienced subjects, which causes misbalance from the moment the newly added subjects complete the pretest survey. Testing: pre-test values impacts post-test outcomes. Test retest reliability brings flawed results as using the same test in pre-test and post-test events, or recurrently, will enable the subjects to use immediate and distant memory to cheat the test..

[Audio] Diffusion Definition and Example Diffusion threat spreads from the treatment group into the control group in which there are no boundaries between the groups, such as interaction and observation. In diffusion, threat methods of research widens and crosses over to another population (control group). For instance, a research in which one population of body builders received information about a muscle mass increase and begins to see the results. The second population only receives information about muscle mass increase but they notices the differences between the two populations and decide to cross-over without the researcher's permission. Their behavior, caused by study's demoralization interferon now threatens the validity of the study ( Salkind, 2012b). Another example of diffusion takes place in public schools today. Schools oftentimes compound on reading interventions for students who cannot decode. Desperate to get the students to decode (which is a requirement by grade 2), students in middle and high school who are still learning how to read hear and observe other students who learn how to decode quicker then they and step over from their reading program to another program ( Department of Education Common Core Standards, 2006). A notorious Hooked on Phonics reading lab and Orton Gillingham reading lab are two programs that take reading study completely differently. The former is pattern based on consonant-vowel matrix, while the latter is based on sensory motor integration and en/ decoding. Moving between the two reading labs will diffuse the study that would prove which group (treated or informational) gets to read sooner..

[Audio] Instrumentation is an unintended or intended guiding of the research group that will cause it to behave in ways that otherwise, when left alone, the group would not behave in that manner. Measures are completely different between the pre and post tests. Instrumentation threats answer HOW instruments are being used in order to determine when and if the process impacted the study negatively and became a threat to the internal validity of a research. This can also happen when there are modifications in the instruments used, available observers, and or designated scoring personnel that may bring changes in outcomes. To focus on consistency which can add value to the process of research by avoiding modification at each observation point. Instrumentation threat can also take place due to mathematical functioning of the programs one uses. This threat is easily ameliorated, while the human threat in instrumentation is harder to fix. A human is actively participating in the process of study. Mood congruent behaviors are not unlikely. Humans are susceptible to masking their own behaviors out of overcompensation that can lead to tainted results (i.e. boredom, incompetence, unintended errors) ( Slocum et al., 2022). Another example of instrumentation threat is in pre-test window during which student satisfaction was probed within a time limit of 15 minutes, while the post-test allotted 30 minutes for the same topic..

[Audio] Maturation and/or Death threat is an occurrence when the aspect of time impacts the results to the point that they would be questioned, maturation/ death threat will compromise the study. Geriatric medical protocols suffer with this threat as the patience either age or succumb to the side effects of the illness or treatment. This can also be present when wholly new subjects are added to the existing gaps among already experienced subjects, which causes misbalance from the moment the newly added subjects complete the pretest survey. This threat can also become concomitant with Mortality or Death threat. When someone opts not to continue with a study, or someone was taken out of a study, poses a threat. Participants can choose to leave, be selected to leave, or their end-stage of life may cause them to leave ( Salkind, 2012b). This takes place when a very ill subject in a medical protocol passes away; a participant is no longer confident that the study will bring the benefit they desire; a participant is no longer accessible (incarcerated, deported, relocated, forbidden, etc.), and the line. When this happens, one can easily see the confluent impact of attrition threat that doubles up against a given validity..

[Audio] Testing threat is an occurrence when pre-test values impact post-test outcomes. Test retest reliability brings flawed results as using the same test in pre-test and post-test events, or recurrently, will enable the subjects to use immediate and distant memory to cheat the test. If a group of students was probed by way of pre-test and then again probed by way of post test, using the same metric and language economy, testing may be the interferon that can question validity of the study. The reason WHY the outcome was higher during the post test must be determined in order to identify the drastic change in the outcome of it. When repeated opportunities to the same content are awarded, the test-retest reliability becomes moot as the redundancy of exposure will automatically enable the participants to respond to queries that have been previously memorized. This transactional outcome can be sued for research that calls for content regurgitation. For research that requires metagonitive transformation, to think about thinking, the test-retest method would be flawed as this task requires critical thinking..

[Audio] Diffusion Control: the purpose of diffusion control is to focus on methodological characteristics toward the end of upping the chances of ruling out alternative hypotheses ( Slocum et al., 2022). One of the ways to accomplish this is by way of Randomization. In randomization protocol, participants are allocated to treatment and control cohorts randomly, which will help avoiding bias between subgroups ( Salkind, 2012b). Whether complete or unrestricted randomization is used, a researcher will beat the odds by using multivariate sources to gain participants without a very low, if any, chance to show bias toward one or another participant or group. To do this, a researcher might utilize sequence generation, allocation of concealment, and implementation. Mitigation Testing threat would involve creating and implementing specific framework of research in order to avoid negative consequences. For example, if the research topic is the same for both the treatment group and the control group, then approach for both groups would remain the same instead of using one method with a treatment group and another method with the control group (Salkind, 2012b). It is key that a researcher stays true to internal and external validity by doing the following: human surveyors cannot know who the subjects are, what their groups are, what the individual and group demographics are, with exception to maturation/ death threat instances. In this case, not the surveyor, but someone ese overseeing the data collection will review the group infrastructure in order to make sure that a departure of one or several participants will not negatively impact the remainder of the cohort or the outcomes (Salkind, 2012a)..

[Audio] External validity brings to the fore how the study applies to the outside world and in different situations ( Salkind, 2012a). It is important that its general applicability can be housed with as many groups, individuals, semiotic placements, and contexts ( Bracht & Gals, 2011). One applicable example would be in environmental science. External validity in this case is heavily connected to the internal validity as internal validity impacts the external generalization ( Bracht & Glass, 2011). Research in science intends to broaden the scope of the study from a smaller population size to as greater population size as possible ( Bo & Galiani, 2021). In pharmacology, for instance, having treatment for larger population size is far more beneficial than creating an expensive product for a small population. This means that medical research uses the values of expansion from artificial, limited, population terrain, to larger, realistic, population plane. (Bracht & Glass, 2011). A researcher should ask: 1- How is your research applicable to the larger world? 2- How is the research applicable in relation to other experiments? 3- How is it applicable to different settings and different people? 4- What is its semiotic impact now and is it commutable to the future and open for advancement? If the answers of these questions are all in the affirmative, and or yes, then a researcher can determine at this pont that the external validity of this research is high..

[Audio] Sampling Bias Example: a number pf participants with a preset metric for a given plane of study. Or it can be a scenario in which a set number of people are chosen as by doing that the impact of generalizability is lowered. Another example of this threat can take palce during national elections. Sampling bias takes place when participants of a desired population are chosen wrongly. They were chosen wrongly because they have either a higher or lower chance of being selected. Most common sampling biases are: Observer bias who unconsciously project their intentions on the study; Self selection and or voluntary response bias happens when someone selects themselves for the research and with it injects the unfavorable conditions in the overall samples that can impact the validity of the entire process; Survivor bias levies the research with an unintended negative outcomes of past researches that failed and over-focuses on small successes. While this is a positive look on the research, one cannot dismiss the failures that took place in order to hyperbolize the successes; 4. Advertising Bias or otherwise knowns as prescreening bias that focuses on the emphasis of one population type over another and in that way recruit participants who share the same, as opposed to random, characteristics. 5. Healthy User bias finds that medical research participants barely meet the criteria for the study. This is so because they chose themselves ( self selection) for the study; 6 . Non response bias surrounds itself about poor construction and design of the survey or the complexity of the survey was compounded to the extent that it demotivated the participants from completing it ( Chen et al., 2022). Reactive Arrangement example is an occurrence during which individuals alter their behavioral outcome in response to the fact that they know they will be observed. As such, the outcome of their behavior is artificial and temporary due to duress. In business models, this threat is seen when employees are asked to be 'accountable' for their jobs and roles. This is applicable for as long as there is someone or something that makes them be accountable. When such accountability force is removed, accountability regresses to its former state. To mitigate this problem, employees should be taught Kolbe's individual growth matrix of conation that stems from one's inner desire to perform whether or not a compulsory mechanism is influencing them (Kolbe, 2004)..

[Audio] There are four things that a researcher should keep an eye on: Generalize the samples; Watch for selection biases; Determine causative correlations within the variations; and Be aware that sample features generating confounders limit generalizations. It is important to recreate all possible threats so that a researcher can learn form it. This can be done by way of widening the scope of generalization in populations, enhancement of conditions, and broadening of the settings. Introducing these modifications in natural settings is important in that it would allow field experimentation to counter testing ( Dehejia et al., 2021). To keep biases off limits, a researcher must have a protocol that would allow them to stay focused on the study instead of showing affinity for a desired outcome or behavior in the participant population. This behavior is at times very difficult to control as humans unconsciously show affinity to a desirable outcome. Partnering up the observational forces, coupled with safety procedures of an assistive computer software program that would be sensitive to data propensity values would be of great importance ( Dehejia et al., 2021). Replications can add value in detecting and eradicating threats. In the line manner, probability sampling method lends a hand in counter sampling biases that oftentimes take place during selection ( Chen et al., 2022)..

[Audio] When building a research framework, one has to be keenly aware of the internal and external validity. The goal of the research is that both internal and external validity are high. ( Jimenez-Buedo & Russo, 2021). Controlled settings that have the lowest, if any, impact from the outside variation is what internal validity seeks to attain ( Salkind, 2012b). Highly regulatory environment, tight metrics and procedures, randomization of sampling, and treatment centralization can rule out multivariate outcomes and yield high level confidence and trust in causation and validity ( Lin et al., 2021; Salkind, 2012a). On the external validity side of the spectrum, a researcher intends to derive realistic results that can be applied not only in small population, but transplanted to larger research terrains in order to be global or generalized ( Bracht & Glass, 2011). Research permits confounders and secondary and tertiary reasons of discrepant data ( Slocum et al., 2022). When this happens, replication and duplication can take place in order to study the threats and come up with mitigation tactics (Lin et al., 2021). A symbiotic fidelity to both internal and external validity takes place when internal validity can answer questions without bias, while the external validity can answer questions that relate to its ability to be applied to other generalized contexts(Jimenez-Buedo & Russo, 2021)..

References (part 1). Bo, H., & Galiani , S. (2021). Assessing external validity. Research in Economics, 75(3), 274–285. Retrieved from: https://doi.org/10.1016/j.rie.2021.06.005 Bracht , G. H., & Glass, G. v. (2011). The external validity of experiments. SAGE Quantitative Research Methods, 438–474. Retrieved from https://doi.org/10.4135/9780857028228 Chen, S.-W., Keglovits , M., Devine, M., & Stark, S. (2022). Sociodemographic differences in respondent preferences for survey formats: Sampling bias and potential threats to external validity. Archives of Rehabilitation Research and Clinical Translation, 4(1), 100175. Retrieved from : https://doi.org/10.1016/j.ARRCT.2021.100.175.

References (part 2). Dehejia , R., Pop- Eleches , C., & Samii , C. (2021). From local to global: External validity in a fertility natural experiment . Journal of Business & Economic Statistics , 39(1), 217-243. Jiménez- Buedo , M., & Russo, F. (2021). Experimental practices and objectivity in the social sciences: re-embedding construct validity in the internal-external validity distinction. Synthese : An International Journal for Epistemology, Methodology, and Philosophy of Science, 199(3–4), 9549–9579. https://doi.org/10.1007/S11229-021-03215-3/FIGURES/1 Lin, H., Werner, K. M., & Inzlicht , M. (2021). Promises and perils of experimentation: The mutual internal-validity problem. Perspectives on Psychological Science, 16(4), 854–863. Retrieved from: https://doi.org/10.1177/1745691620974773.

References (part 3). Salkind , N. ( 2012a ). External validity. In Encyclopedia of Research Design (Vols. 1-0). SAGE Publications, Inc. Retrieved from: https://doi.org/10.4135/9781412961288 Salkind , N. ( 2012b ). Internal validity. In Encyclopedia of Research Design (Vols. 1-0). SAGE Publications, Inc. Retrieved from: https://doi.org/10.4135/9781412961288 Slocum, T. A., Pinkelman , S. E., Joslyn, P. R., & Nichols, B. (2022). Threats to internal validity in multiple-baseline design variations. Perspectives on Behavior Science, 1–20. Retrieved from: https://doi.org/10.1007/S40614-022-00326-1/TABLES/1.