Vignette Method in Online Research: Measuring Perceptions of Fairness

The article covers the use of the vignette method in online research. The possibilities of the method based on the measurement of the perception of justice are demonstrated and the tools developed for these purposes are evaluated. The analysis of the quality of the proposed scales is carried out using quantitative assessments of the main psychometric indicators: constructive and criterion validity. The substantiation and calculation of these indicators are described in detail, as well as the interpretation of the obtained values. A moderate indicator of construct validity and a satisfactory indicator of criterion validity indicate the need for further optimisation of the scales.

Special attention is paid to the peculiarities of using this method online with the help of participants recruited from a commercial panel of test subjects. The paradata are analysed and the dependence of the meaningful results of the study on the non-semantic aspects of the tools is traced. In particular, the influence of the interaction between the order of presentation of stimuli and the type of devices on judgments about fairness is demonstrated. It is shown that participants who completed tasks using mobile devices tend to consider the conditions that they were shown first to be more fair. In contrast to participants who performed tasks on desktop devices, where this effect is practically not observed.

Recruiting participants from an online panel for a study using the vignette method demonstrated a number of limitations that should be considered when designing a study. Given the demonstrated inattention of participants when completing tasks from mobile devices, it is recommended to limit the completion of questionnaires to the use of desktop devices - computers and laptops. Also, unlike previous studies in this area, when using the vignette method, it is recommended to limit the number of levels of factors in the experimental plan to reduce the cognitive complexity of tasks in the conditions of the inability to control the attentiveness of their performance.

The results obtained are analysed in the context of methodological research. Methodological solutions are proposed for using the online vignette method, current and possible sources of data distortion are indicated, and recommendations are given for the further use of the developed tools.

 vignette method, factorial survey, experiment, construct validity, criterion validity, distributive justice, belief in a just world

1.  Bykov A. V. Factorial Vignettes Method and Self-Report Altruism Scale: Comparison of Online and Offline Surveys. Sotsiologiya: metodologiya, metody, matematicheskoye modelirovaniye, 2014: 39: 62–98 (in Russ.).
2. Vlasov M. S., Protasova I. N. Vignettes as a Measure of Moral Foundations. Vestnik Altayskogo Gosudarstvennogo Pedagogicheskogo Universiteta, 2018: 37: 71–75 (in Russ.).
3. Goodwin С. J. Research in Psychology: Methods and Design.  St. Petersburg, Piter, 2004: 558 (in Russ.).
4. Kalinin R. G. Distributive Justice Research in Social Sciences: A Review of Conceptualizations and Methodological Approaches. Sotsiologiya: metodologiya, metody, matematicheskoye modelirovaniye, 2019: 49: 7–56 (in Russ.).
5. Puzanova Zh. V., Tertyshnikova A. G. Vignette Method in Sociological Research: Methodological Principles and Techniques. Vestnik Rossiyskogo universiteta druzhby narodov. Ser.: Sotsiologiya, 2015: 4: 44–56 (in Russ.).
6. Terentiev E. A., Nefedova A. I., Gruzdev I. A. Visualization of Questionnaire in Online Surveys: How Different Design Features Impact the Data Quality. Monitoring obshchestvennogo mneniya: Ekonomicheskiye i sotsial'nyye peremeny, 2016: 5: 1–15 (in Russ.).
7. Alves W. M., Rossi P. H. Who Should Get What? Fairness Judgments of the Distribution of Earnings. American Journal of Sociology, 1978: 84: 3: 541–564.
8. Antoun C., Couper M. P., Conrad F. G. Effects of Mobile versus PC Web on Survey Response QualityA Crossover Experiment in a Probability Web Panel. Public Opinion Quarterly, 2017: 81: S1: 280–306. DOI: 10.1093/poq/nfw088
9. Atzmüller C., Steiner P. M. Experimental Vignette Studies in Survey Research. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2010: 6: 3: 128–138.
10. Auspurg K., Hinz T. Factorial Survey Experiments: 175. Los Angeles, SAGE, 2015: 168.
11. Auspurg K., Hinz T., Liebig S., Sauer C. The Factorial Survey as a Method for Measuring Sensitive Issues. In Improving Survey Methods. Routledge, 2014: 137–149.
12. Auspurg K., Jäckle A. First Equals Most Important? Order Effects in Vignette–Based Measurement. Sociological Methods and Research, 2017: 46: 3: 490–539. DOI: 10.1177/0049124115591016
13. Bakdash J. Z., Marusich L. R. Repeated Measures Correlation. Frontiers in Psychology, 2017: 8: 456. DOI: 10.3389/fpsyg.2017.00456
14. Carmines E. G., Woods J. Validity. The SAGE Encyclopedia of Social Science Research Methods. Thousand Oaks, CA, Sage Publications Inc., 2004: 1171–1172.
15. Couper M. P., Conrad F. G., Tourangeau R. Visual Context Effects in Web Surveys. Public Opinion Quarterly, 2007: 71: 4: 623–634. DOI: 10.1093/poq/nfm044
16. Cullen S. Survey-Driven Romanticism. Review of Philosophy and Psychology, 2010: 1: 2: 275–296. DOI: 10.1007/s13164-009-0016-1
17. Dalbert C., Montada L., Schmitt M. Belief in a just world: Validation correlates of two scales. Psychologische Beitrage, 1987: 29: 4: 596–615 (in Germ.).
18. Eifler S., Petzold K. Validity Aspects of Vignette Experiments: Expected “What-If” Differences Between Reports of Behavioral Intentions and Actual Behavior. In Experimental Methods in Survey Research: Techniques That Combine Random Sampling with Random Assignment. Wiley Online Books, 2019: Validity Aspects of Vignette Experiments: 393–416.
19. Hafer C. L., Sutton R. Belief in a Just World. In Handbook of Social Justice Theory and Research. New York, NY, Springer, 2016: 145–160. DOI: 10.1007/978-1-4939-3216-0_8
20. Hainmueller J., Hangartner D., Yamamoto T. Validating vignette and conjoint survey experiments against real-world behavior. Proceedings of the National Academy of Sciences of the United States of America, 2015: 112: 8: 2395–2400. DOI: 10.1073/pnas.1416587112
21. Hermkens P. L. J., Boerman F. A. Consensus with respect to the fairness of incomes: Differences between social groups. Social Justice Research, 1989: 3: 3: 201–215.
22. Jasso G. Factorial Survey Method (Rossi’s Method). In The SAGE Encyclopedia of Social Science Research Methods. Thousand Oaks, CA, Sage Publications Inc., 2004: 374–376.
23. Jasso G. Factorial Survey Methods for Studying Beliefs and Judgments. Sociological Methods & Research, 2006: 34: 3: 334–423. DOI: 10.1177/0049124105283121
24. Jasso G. Safeguarding Justice Research. Sociological Methods & Research, 2012: 41: 1: 217–239. DOI: 10.2458/v7i2.20321
25. Jasso G., Rossi P. H. Distributive Justice and Earned Income. American Sociological Review, 1977: 42: 4: 639–651.
26. Krosnick J. A. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 1991: 5: 3: 213–236.
27. Lang V. Scaling Sensitive Factorial Survey Analysis. Sociological Methods & Research, 2018: 649–682. DOI: 10.1177/0049124118799382
28. Liebig S., Sauer C. Sociology of Justice. In Handbook of Social Justice Theory and Research. Springer, New York, 2016: 37–59.
29. Liebig S., Sauer C., Friedhoff S. Using Factorial Surveys to Study Justice Perceptions: Five Methodological Problems of Attitudinal Justice Research. Social Justice Research, 2015: 28: 4: 415–434.
30. Lucas J. W. Theory-Testing, Generalization, and the Problem of External Validity. Sociological Theory, 2003: 21: 3: 236–253.
31. Markovsky B. Anchoring Justice. Social Psychology Quarterly, 1988: 51: 3: 213–224.
32. Markovsky B., Eriksson K. Comparing Direct and Indirect Measures of Just Rewards. Sociological Methods & Research, 2012: 41: 1: 199–216.
33. Mavletova A. Data Quality in PC and Mobile Web Surveys. Social Science Computer Review, 2013: 31: 6: 725–743. DOI: 10.1177/0894439313485201
34. Messick D. M., Schell T. Evidence for an equality heuristic in social decision making. Acta Psychologica, 1992: 80: 1–3: 311–323.
35. Nartova-Bochaver S., Donat M., Astanina N., Rüprich C. Russian Adaptations of General and Personal Belief in a Just World Scales: Validation and Psychometric Properties. Social Justice Research, 2018: 31: 1: 61–84. DOI: 10.1007/s11211-017-0302-5
36. Peng C. Y. J., Mueller D. J. Construct Validity. In The SAGE Encyclopedia of Social Science Research Methods. Thousand Oaks, CA, Sage Publications Inc., 2004: 182–183.
37. Rossi P. H. Vignette Analysis: Uncovering the Normative Structure of Complex Judgments. In Qualitative and Quantitative Social Research: Papers in Honor of Paul F. Lazarsfeld. New York, Free Press, 1979: Vignette Analysis: 176–186.
38. Rossi P. H., Anderson A. The Factorial Survey Approach: An Introduction. In Measuring Social Judgments: The Factorial Survey Approach. Beverly Hills, Sage Publications, 1982: The Factorial Survey Approach: 15–67.
39. Rossi P. H., Sampson W. A., Bose C. E., Jasso G. Measuring household social standing. Social Science Research, 1974: 3: 3: 169–190.
40. Sauer C., Auspurg K., Hinz T. Designing Multi-Factorial Survey Experiments: Effects of Presentation Style (Text or Table), Answering Scales, and Vignette Order. Methods, data, analyses, 2020: 14: 2: 195–214. DOI: 10.12758/mda.2020.06
41. Sommer J., Diedenhofen B., Musch J. Not to Be Considered Harmful: Mobile–Device Users Do Not Spoil Data Quality in Web Surveys. Social Science Computer Review, 2017: 35: 3: 378–387. DOI: 10.1177/0894439316633452
42. Spalding N. J., Phillips T. Exploring the Use of Vignettes: From Validity to Trustworthiness. Qualitative Health Research, 2007: 17: 7: 954–962. DOI: 10.1177/1049732307306187
43. Wu H., Leung S. O. Can Likert Scales be Treated as Interval Scales? –A Simulation Study. Journal of Social Service Research, 2017: 43: 4: 527–532. DOI: 10.1080/01488376.2017.1329775