> MXD - Maximum Difference Scaling > Concept and Package Tests

Concept and Package Tests

---

In God we trust. All others must bring data.

W. Edwards Deming

Abstract

A method to estimate preferences between product concepts and their aspects is presented. The method is designed for a special case when concepts are fixed and described as a set of aspects. The simplest interview arrangement consists of two blocks.
One block is a best-choice MaxDiff of concepts or packages. Each choice can be completed with a calibration question on purchase intention. The other block is best-worst MaxDiff of aspects the concepts are composed of. The choice preferences obtained from the blocks are combined and quantified as part-worths of the aspects that make utilities of the concepts.
The method has been designed to handle more attributes than a CBC. A non-compensatory model of choice, especially useful for product bundles, is assumed using acceptability thresholds obtained from calibration questions.
The test is equivalent to the usual concept test with the advantage of obtaining the influence of aspects on a common scale.

Problem

The choice-based conjoint (CBC) is a widely used research technique for products that can be described by about 6 or fewer attributes having 2 or more levels each. Actual market products and services have often many more attributes that make use of CBC unfeasible. This concerns namely packages that have aspects either present or absent and are not mutually exclusive. Aspects are often grouped according to the total value of an offer, and relationships between them are complicated. Only some combinations of aspects are allowed. Generating reasonable product profiles in a conjoint study would require numerous cross-attribute prohibitions that are a frequent cause of inefficient or unrealistic product profiles, of a slow or none estimation convergence, and nearly always of estimation bias. The method of product classes has proved useful for products with only a few attributes, typically up to 6 attributes.

The approach of incomplete conjoint profiles is seldom usable. It leads to leveling the importance of attributes in general and to an apparently increased importance of the least often shown attributes. The method is unusable in a branded study since respondents implicitly set the missing attributes to the level values they suppose the brand usually has.

Due to the inconvenience with the application and the results of a conjoint method, the current approach relies on product concepts evaluated in a sequential monadic test usually followed by additional questions on numerous aspects of the concepts. The main disadvantage of this approach is a lack of information on the influence of individual aspects on the acceptability of a product. No simulation of a product set in the frame of the tested aspects is possible.

Solution

Model

Instead of using profiles generated by an algorithm, the method relies on fixed profiles designed in a managerial way. The range of aspects, their settings, and balance between them (typically value and price) are taken as guaranteed since the profiles are provided by a study submitter who is supposed to have good knowledge of the market. By their very nature, the profiles exclude the need to introduce prohibitions between attribute levels. These compensations are not reflected in any known algorithm for generating near-to-orthogonal profiles for a conjoint task.

The method is inspired by the half-forgotten approach of the 1950s, referred to as the SEC - Self Explicated Conjoint (or Self Explanatory Conjoint). The respondent evaluated each attribute level attractiveness and each attribute importance one at a time. The obtained evaluations were transformed into values of an additive variable. The sum of these values characterized a product composed of attribute levels.

As aside to SEC

• SEC is still offered in some interviewing SW, e.g. by Qualtrics. Despite respondents having made no trade-offs between products,  H. Sattler and Susanne H. Börner claim no severe differences between results from SEC and a standard conjoint technique were observed.

In our approach, the attractiveness of both profiles and aspects is estimated in the respective MaxDiff tasks. The scale-based questions on concepts in the traditional concept test are replaced by MaxDiff of profiles. The choices can be completed with questions on (stated) purchase intention. The scale-based questions on attribute levels in SEC are replaced by choices from small MaxDiff choice sets composed of the aspects of which the tested profiles are composed. The influence of aspects is obtained from uniting the results from both groups of tasks using a common behavioral model. The main principles of CSDCA - Common Scale Discrete Choice Analysis are applied.

In a conjoint-type exercise, the number of attribute levels should be about the same, the differences between effects of levels within an attribute equidistant, and the spans of attributes of similar importance. This is required to obtain reliable estimates from responses to profiles generated by an algorithm. The present method does not impose these requirements. The overall scope and implicit (or expected) importance of the aspects is given by the provided concepts.

Choice of the behavioral model is important. In the simplest case, the traditional additive model can be used. If there is a suspicion or assumption of non-compensatory behavior, the interview should be extended by a CSDCA PRIORS block of questions to obtain better estimates of acceptability thresholds.

Improvements in the quality of results is expected to be similar to the transition from scale-based questions to MaxDiff, CSDCA or AGA procedures.

Questionnaire

The actual order of the questionnaire blocks should take account of the learning process of respondents.

As aside to design

• The design of DCM blocks requires a lot of care to avoid overfitting as the model kernel will usually have many more parameters than is the number of choices in a questionnaire. The information theory, Bayesian principles, and (sic!) close cooperation with the study submitter are of help.

Estimation

The mutually exclusive aspects can be grouped into formal attributes. This will allow taking advantage of commercial programs designed to analyze conjoint tasks. Among these are constraints between the values of part-worth of ordinal level values, the use of different types of coding for estimation of thresholds, etc.

Utilities of profiles and part-worths of aspects are estimated by the hierarchical Bayes maximum likelihood estimation. The data set is composed of all the discrete choice exercises each of which has a different weight. The weights are determined from the number of tasks and a number of items in a task using the gain in information obtained as a loss of information entropy (cf. How many CBC tasks).

As aside to estimation

• The estimation process is composed of several implicit steps:
• Formulation of a numerical model of the aspect part-worths satisfying a considered behavioral model.
• When calibration answers (from questions on purchase intention) are available, the respective parameters of profile thresholds are included in the model.
• Mapping choices from MaxDiff of aspects onto the numerical model.
• Mapping choices from MaxDiff of profiles (and from calibration questions, if available) onto the numerical model.
• Model parameters (concept utilities, aspect part-worths, and threshold values) are estimated in a single run.

The effects of aspects are expressed in terms of perceptance, i.e. the difference between the probabilities of accepting and non-accepting an aspect. A profile is characterized by a stated purchase intention computed as its censored perceptance, the censoring being made by considering only non-negative values.

Simulation

A simulation of purchase intentions of untested profiles is possible only for profiles from the close vicinity of the tested profiles. One reason is the prerequisite "considered jointly" is not valid for all combinations of aspects since only a limited number of aspect combinations were tested. Another reason is the number of effects summed in the simulation. The present method is designed to handle many more aspects than a conjoint method is able to. Utility error of untested concepts grows with the number of changed aspects because of the error cumulation.

Properties

Strengths

• The profiles to be tested are fixed and supposed to be balanced from the perspective of the market. The estimation bias resulting from often unbalanced and sometimes unrealistic profiles generated by a computer algorithm is avoided.
• The method is limited only by the feasibility of the MaxDiff tasks of profiles and aspects. It is believed up to 30 profiles and 60 aspects can be tested without a study split design. This is in contrast to the conjoint method not allowing more than about 6 attributes, but with (theoretically) an unlimited number of attribute levels.
• Both the profile and aspect MaxDiffs are much simpler for respondents than a conjoint exercise with the same gain of information.
• Estimated part-worths of aspects can be calibrated to express the stated purchase intentions (SPI) of the profiles. Aspects can be characterized by values of derived variables for attractiveness, typically perceptances, which are more comprehensible than the values of partial utilities expressed in logit units.
  
  As aside to part-worths vs. acceptances and perceptances
  

  • Part-worths are non-linear measures of probabilities and should not be averaged over the sample. Acceptances and perceptances are linear transformations of choice probabilities and, in contrast to part-worths, allow averaging.
• All aspects are comparable to each other. This is in contrast to levels of (different) conjoint attributes that are incomparable.
• Estimated price sensitivity is related to the willingness to spend on a higher level product. The same measure obtained from a standard CBC study is confounded with the willingness to pay for the same product (e.g. from a brand-price CBC).
• Similar to a conjoint study, a share simulator can be developed (cf. weaknesses below).

Weaknesses

• Only the aspects present in the tested concepts are estimable. If the concepts are unbalanced in terms of their aspects the procedure may fail.
• The effect of each aspect is estimated by decomposing the likelihood of choices in the MaxDiff of concepts while utilizing the other DCM blocks as soft constraints. In such an arrangement the systematic error of estimates (bias) strongly increases when the total number of aspects exceeds some limit.
• A simulation of untested profiles is less robust for profiles very dissimilar from the tested profiles. It is recommended the simulated profiles should be created from the tested profiles by changing only a few aspects.
  

Neutrals

• The method is based on and preserves the standard assumptions of DCM modeling and methods. 
  

Future developments

• The method is a hybrid of a classical concept test and a MaxDiff method. It is very flexible and open to various modifications. For example, when the usual price elasticity is of interest, several (or all) profiles may be offered at reasonably different prices.
• In 2022, the method has been reworked by using more efficient estimation procedures that allow designs similar to those used in CBC. The new method has been named CBCT - Choice-based Concept Test.

Examples

Example 1: Bank account

The model was applied on a set of several bank account concepts that could be described by 12 attributes with 2 to 5 levels each. This number of attributes was too high to use a CBC methodology. Instead, the levels were treated as 46 aspects. The questionnaire contained a MaxDiff of the concepts and a MaxDiff of the aspects. A web questionnaire was created in Sawtooth SSI ver. 8.4.8 system. Data were processed using Sawtooth CBC/HB 5.5.4. A shortened and obfuscated version of the live questionnaire in Czech language is available on-line.

The acceptance of each of the concepts was obtained from the MaxDiff of concepts. Each unique concept choice was completed with a calibration question on the intention to switch to the concept so that the raw utilities could be modified to reflect the stated intentions.

The client was interested in which aspects and to what extent they were responsible for an attractiveness or, possibly, a repulsiveness of the concepts. This was achieved by combining the results from both the MaxDiffs. Since perceptance threshold value for aspects was not determined experimentally a common threshold was set arbitrarily to the mean utility value of the price range. The results are in the picture below.

Perceptance of the concept aspects

The interpretation of perceptances is straightforward for most aspects. Nevertheless, it should be remembered a perceptance value is a single number that comprises several attitudes, namely common knowledge, expectation, and satisfaction with the aspect (and its value, if quantifiable).

As aside to perceptance

• The simplest interpretation of perceptance is the percentage of those who take an aspect positively minus those who take it negatively. As all DCM models and methods are relative and all variables are interval measures, the computed values are dependent on less or more arbitrarily determined "neutral" threshold value of 50% positive and 50% negative perceptions in the sample. In this particular case, the zero perceptance value has been assigned to the mean utility of tested monthly fees.

Among the tested aspects, neither a credit card nor airport lounges are usually offered as a part of a bank account. The explicitly stated absence of an airport lounge has higher expectations compared to the absence of a credit card that respondents are more familiar with. However, including even the lowest credit card option in a bank account leads to a higher satisfaction than providing any of the offered airport lounge options. The most important interpretation value is the difference between perceptances of individual aspects. An aspect with substantially low perceptance of both its absence and presence, i.e. an aspect with high expectations that cannot be satisfied, should be omitted from an offer.

Example 2: Current account

The Bank prepared 24 concepts for testing based on a total of 47 aspects. Of these, 2 aspects were optional (insurance with codes 2.2 and 3.2), all others were fixed. Each concept was composed of 11 attributes.

The first block of questions consisted of 9 choice tasks with 4 concepts each. Each choice was completed with a purchase intention question. The next block was a best-worst MaxDiff of 35 aspects in 16 choice sets composed of 4 aspects each. The levels from Condition and Price attributes were not included as they could not be compared with any other aspect, and were treated separately. The obtained perceptance values are shown in the figure below.

Perceptance of aspects

Quite as expected, the most important aspects are conditions and price. The first three conditions practically do not reduce the acceptability of the product. The lowest ATM service level is sufficient for most respondents, and its further improvement has only a small effect.

Four types of insurance were offered. The price was increasing with the index behind the dot of the insurance code. The most attractive insurance was insurance 4.0 without an additional fee. Second in order was the simplest 1.1 insurance for the lowest price. Optional insurance contributes more to the attractiveness of the profile than the same fixed insurance. Eligibility, as concluded, contributes to the overall acceptability of the offer.

The offer of financial reserve and bonus proved to be ineffective. The discount program is better, but without it, the offer will still be acceptable. Belief in discount programs is generally low. Specific welcome packages have been welcomed, but it seems the responses were exaggerated. Omitting welcome packages would not fundamentally harm the acceptability of the product.

Gifts were generally welcomed. On the other hand, an explicit statement about the absence of a gift, when offered elsewhere, has a mild negative effect. The effect of the statement "no insurance offered" was even more pronounced, as there was the possibility of optional insurance in other offers. It is believed that not only the context between the aspects in the profile, but also between the profiles, namely those offered by the competition, is important.

Conclusions

The method has proven to be a stable and practical solution for estimating the effects of a large number of attributes with levels understood as separate aspects of the product. It is considered useful for testing concepts or packages with many limitations between aspects. It is less prone to estimation bias due to collinearity of aspects than the conjoint-like approach. Collinearity of aspects is ubiquitous in market products, namely when aspects are grouped according to the product price classes.

During its use, the method has been enhanced with possibilities to randomly modify the default concepts, use optional levels of attributes, create a histogram of acceptable prices, and refine the estimation of thresholds by using a reworked estimation method. The new version has been named CBCT - Choice Based Concept Test and made available for deployment.