How the MaxDiff experimental design works

A MaxDiff question presents participants with a series of attribute sets, and asks participants to select their most and least preferred attributes from each set. Generally speaking:

• Fewer sets, and fewer attributes per set, are better for participant experience. This minimizes decision fatigue.
• An attribute needs to appear frequently enough to calculate or infer a comparison of it against other attributes.
• If an attribute appears too few or too many times, the experimental design of the MaxDiff question is considered imbalanced. Therefore, the data could be unreliable.

The system-generated responding options take the trade-off between thorough data collection and participant fatigue into account:

• An attribute appears at least 3 times across different attribute sets.
• At the same time, the number of sets is minimized to prevent participant fatigue.
• When participants reach the final attribute set, the MaxDiff question has collected enough data to calculate a comparison and ranking of all attributes.
• In cases where an attribute could not be compared to another attribute directly, there is enough data to infer a comparison.

The application uses the following formula to calculate Attributes per set /sets combinations:

nK/k = s

• n: The number of times each attribute is displayed across all sets.
• K: The overall number of attributes.
• k: The number of attributes per set.
• s: Sets per respondent.

The application then gives you three responding options.

• The first option uses the formula 4K/k = s. Each attribute appears 4 times, ensuring thorough data collection and a robust experimental design.
• For the second and third options, attributes may appear fewer than 4 times. However, if you have a large list of attributes and you want to minimize participant fatigue, these options may be preferable. Each attribute is still shown at least 3 times, and the pair-wise correlation is still balanced (any two attributes are not highly correlated or non-correlated).

Example

The following binary matrix represents an experimental design generated for a MaxDiff question that contains 10 attributes, using the formula nK/k = s. The binary matrix shows whether an attribute is displayed or hidden in a set. The columns represent the attributes (10 in total). The rows represent the sets (8 in total). The 1 and 0 indicate whether the attribute is shown or hidden, respectively. In each set, there are 4 attributes shown. Each attribute is shown at least 3 times; two of the attributes are shown 4 times.

When you select Create a different version for each respondent, the overall experimental design remains the same. However, the application shuffles the attribute columns in the binary matrix before generating the attribute sets. Therefore, each attribute still appears the same number of times, but each participant sees their own version of the attribute sets.

Example

This option is useful if:

• You have many attributes and want to show all of them enough times to generate reliable data.
• At the same time, you want to minimize the number of attributes and attribute sets shown to participants, to minimize participant fatigue.
• You want to minimize the impact on aggregate results of any attribute pairs that are not shown together.