Simplicity is the ultimate sophistication.

Leonardo da Vinci

 

Problem

Check-box grid is a common question pattern in modern market research questionnaires. Unfortunately, the results based on check frequencies often show insufficient discrimination between tested objects. The OBIMA - Object Image Analysis, described on the previous page, has been designed with emphasis on reliability, robustness, possibility to evaluate both positive and negative perceptions of object features, and to achieve better discrimination between objects and their perceived properties. To estimate potential of the method, a number of market research clients were invited to take part in a small pilot survey and asked to provide their views on the method.

While the method was labeled "highly sophisticated", some negative points concerning the interviewing alone were appearing regularly:
  • Too long for the purpose.
    • Average time for 4 brands and 12 features was 10 min.
  • Repeating of already answered questions.
    • Respondents did not recognize the idea of selecting items (brands and brand features) in the two dimension.
  • No escape from a question.
    • This is a general problem in effort to obtain enough data. Obligation to answer should be relaxed whenever possible.
  • An IQ test rather than a questionnaire.
    • A subjective view, but - admittedly - relevant.

Due to common availability of Internet panels composed of ever-ready-to-anything respondents and price of interviewing becoming ridiculously low, market research clients are asking for more and more questions. Researches have no other way than to comply and resort to simplifications. Once used batteries of scale-based questions have been turned into grids of check-boxes. Of course, the yes/no answer cannot provide the amount of information as a multiple-point Likert scale, and, consequently, data quality have faded away. In a battle to get at least some amount of data, respondents are often asked to check a predefined number of boxes irrespective from relevance and differing weights of such answers. Thus the data and results get more deteriorated.


Solution

Most shortcomings of a plain checkbox grid can be eliminated by introducing ranking of answers. This will improve sensitivity by giving the checks different weights. Most often, answers in just one dimension will be ranked. In such a case, the method is technically equivalent to the OBIMA method applied in one dimension and omitted questions on negated features.

To simplify terminology, the grid dimensions will be named "brands" and "features" throughout the following text.

As aside

The ranks obtained from a grid are processed as data from several SCE - Sequential  Choice Exercises with no ties in the ranked dimension. A set of interval variable values (the origin of the scale left undetermined) is obtained. To obtain values comparable in the opposite dimension, the checked items in the direction opposite to the ranking can be viewed as sequential choices with ties. This approach was originally used for estimation of part-worths for all brand-feature combinations that could be compared one with another. As a more reliable way appeared addition of the exclusive answer "No other / None of them" to the grid, and using the exclusive answer as a common anchor. Further improvement was achieved by adding the exclusive answer "Don't know / No answer" in order to eliminate forced answers. 

As aside

Hierarchical Bayesian MCMC is used to estimate part-worths of brand-feature combinations. The procedure respects covariances between all combinations thus increasing robustness of estimates.

As aside

Estimated part-worths are quantitative measures. They can be input for additional analyses, such as segmentation, clustering, factor, latent class, correspondence, multidimensional scaling, and other types of classic analyses.

As aside

Check-box ranking is an undemanding task. Respondent is asked to check boxes in one dimension of a two-dimensional grid in order of decreasing significance relevant to the question. Ranking prolongs interviewing time, but the gain in information is worth it.


Properties

Strengths

Weaknesses


Live demo

You may experience the look and feel of  RGA - Ranked Grid Analysis among other live questionnaires.


Examples

Example 1: Tea brands - ranking of features

This example comes from a research project carried out in 2018. Data from 1000 interviews were evaluated.

Questionnaire

The grid had 6 columns for brands from a total of 16 brands selected by respondent. Rows of the grid represented 21 features to be ranked for each of the shown brands. Respondent had to rank exactly 5 features for each brand. The grid question was similar to that of the second example in the demo questionnaire.

Estimation

Part-worths of brand-feature combinations were estimated with the program CBC/HB 5.5.6 (Sawtoothsoftware, Inc.). Brand profiles were evaluated as "feature shares" (in analogy to "market shares") whose sum was 100% for each brand.

For a purpose of comparison with the common method using check-box grid without ranking, ranking was neglected and the data were transformed to the format equivalent to the brand profiles.

Results

The brand profiles for results from both ways of estimation (without and with ranking) are shown in the pictures below.

No ranking
Ranking respected
Comparison of brand profiles obtained from a check-box grid question without (left) and with ranking (right)

The overall pattern of results from both methods is very similar. No systematic bias is observed. Ranking increases sensitivity about five fold, as based on ratios of the maximal and minimal shares of features in brand profiles. Inside a brand profile, differentiation is improved mainly between features with the highest shares which is typical for discrete choice experiments. This facilitates finding the most distinctive features.

Distinctive features, and their differences from similarly perceived brands, is important for the brand communication. This is namely a case when brands are understood mostly as commodities or simple substitutes. A check-box grid question without ranking often gives several top features insufficiently distinguished one from another, such as 5 features of the Fytofarma and Megafyt brands. Higher sensitivity of the method with ranking can reveal the dominant feature for a brand and discriminate it from other brands.

An as-if analysis of ranks in opposite dimension

In this study, respondents ranked features for the brands known to them. The number of brands chosen ranged from 2 to 6. Besides brand profiles, an interest is in distribution of individual features among brands as if the brands were standing one next to other (irrespective of frequency of their choice in the study or on the market). This approach will compare how the brands are perceived or, possibly, preferred from each other for each of the features. For this purpose, the ranks should be obtained in the dimension of brands rather than features but we did not have such data at the time. For purpose of a pure demonstration, available ranks in the dimension of features are used. Clearly, the same rank values across different brands may have different weights and there are many ties that may not be real ties at all. The results are provided just to show the potential of the method. The data were analyzed in two ways, one taking checks as having the same weights as if from a simple check-box grid, and the other respecting the ranks. The results are in the pictures below.

No ranking
Ranking respected
Comparison of feature profiles obtained from a check-box grid question without (left) and with ranking (right)
As aside

The feature profiles where ranking was neglected (assigned "no ranking") were estimated as unweighted percentages of checked-box frequencies. These profiles are very similar to each other for nearly all features. This could be expected as the most often selected brands are common black teas whose perceptions are similar.

The picture is dramatically changed when ranks are taken in account. Each feature has a distribution of brands evidently different from most of other features. One can clearly see which brand dominates in which feature or features. It is believed feature profiles are better suited for the purpose of brand discrimination than brand profiles mentioned earlier. Bayesian estimation that assigns "reasonable" values to unanswered items (cf. information borrowing) seems to do a good piece of work. 

As aside


Example 2: Telecommunication providers - ranking of brands

This example presents comparison of checking versus ranking. It comes from two research projects carried out in 2018 and 2019. The grid featured the same 17 statements and 3 telecommunication providers. Data from 604 and 870 interviews, resp., were analyzed.

Questions

The question of the 2018 study was designed as a plain check-box grid consisting of 3 columns of provider names. Respondents (N = 604) should select 4 of the 17 statements that best describe each provider.

In the 2019 study, ranking was used. The grid had 3 columns of the 3 provider names completed with 2 exclusive options, i.e. "None of them" and "Don't know / no answer". Respondents (N = 870) had to select and rank up to 3 brands for each of the statements, or to select one of the exclusive options. The first example in the demo questionnaire has a similarly constructed grid except its size.

Estimation

Weights of provider-feature combinations from 2018 study are simple percentages that make total of 100%.

Part-worths of ranked brand-feature combinations from 2019 study were estimated with the program CBC/HB 5.5.6 (Sawtoothsoftware, Inc.). Features assigned as unknown were eliminated from the estimation. As the set of providers was complete both the explicit "None of them" and implicit "None next" for unchecked options were unified in a common threshold for each respondent and used in two-sided soft constraints. Computed odds were rescaled so that their sum would be 100% in order to compare results from the studies.

Results

Providers were nearly indistinguishable one from another based on weights of the features obtained in 2018 study without ranking as can be seen in the upper picture below. This lead to the decision to use a variant of a check-box grid with ranking in the next year wave. The results are in the lower picture below.

No ranking (2018 study, N = 604)
Ranked brands (2019 study, N = 870)

Comparison of brand profiles obtained from a check-box grid question without (2018) and with ranking (2019)

Ranking clearly solved the problem of low resolution. As is usual in DCM experiments, differences are more pronounced in features respondents are more interested in or are better known to them. 


Conclusions

Ranking substantially increases discrimination power compared to the analogous methods without ranking. However, researcher must be aware of interviewing time increased by about 50%. While processing standard grid data (without ranking) can often do with a simple averaging of frequencies, processing of ranks requires more demanding numerical methods.

The RGA - Ranked Grid Analysis method can be recommended when standard check-box grid (without ranking) does not provide satisfactory discrimination. Since its introduction, RGA has been successfully used in several market research projects and seems to become a common research tool.

Ranking in the dimension of features

When a number of checks is forced to be constant, ranking gives an overall picture similar to that from a simple check-box grid, but with strongly enhanced discrimination. Relaxing number of required number of checks and, possibly, adding a threshold choice, the results become more informative. Ranking in the dimension of features might be useful namely when features are closely related to each other.

Ranking in the dimension of brands

In most cases, brands tested in a grid are selected by respondents from a pool of brands. Since analysis of rankings is made by probability maximization using Bayesian principle, the respondents who did not answer a brand are assigned preference values corresponding to those respondents who answered the brand, and, at the same time, the "borrowed" values are corrected by the sample covariance matrix. Therefore, the assigned values are the best estimates given the way of the selection from a brand pool. If the selection is conditional to brand usage, the results will reflect views of brand users. If the selection is based on brand knowledge, the results will reflect perceptions of the brands claimed as known by respondents. 


References

  1. Chapman, R.G., and Staelin, R. (1982). Exploiting Rank Ordered Choice Set Data Within the Stochastic Utility Model. J. of Marketing research, 19, 288-301
  2. Street, D.J., Burgess, L. and Louviere, J.J. (2005). Quick and easy choice sets: Constructing optimal and nearly optimal stated choice experiments. Intern. J. of Research in Marketing, 22, 459–470 (2005).