What should a recommendation system optimize?

Start with the experience on a particular screen

Write down what a useful recommendation helps someone do in that moment. Finding an unfamiliar topic, continuing a sequence and selecting an additional purchase are different tasks. Give each surface a clear objective before deciding which behavior the model should predict.

YouTube’s 2021 engineering account describes using viewing behavior alongside satisfaction surveys, including models that predict feedback people have not provided. The example illustrates why an observed action and a judgment about value deserve separate definitions. It describes a historical implementation, rather than a specification for today’s service.

For a hypothetical tutorial library, compare opening a lesson, reaching a useful explanation and reporting that the lesson answered the question. Decide which result matters, when it can be observed and which other outcomes must remain acceptable. Avoid letting the easiest event to count settle that decision.

Give selection, scoring and controls separate responsibilities

Google’s recommendation-system documentation describes candidate generation, scoring and re-ranking as distinct stages. Candidate generation determines what gets considered; scoring orders those candidates; later adjustments can address preferences, freshness or diversity. A strong scoring model cannot recover an item that never entered its candidate set.

Specify eligibility before exposure: availability, access restrictions and a person’s explicit exclusions. Then decide which requirements belong in the score and which must remain enforced constraints. Test the displayed list after all transformations, including any business rules that alter the model’s ordering.

Not every preference is a pass-or-fail test. Keep non-negotiable requirements as constraints, but represent preferences that vary by degree as graded fit. This lets a product distinguish a strong match from an acceptable one without treating both as identical. Explain which preferences affect the ranking and let people adjust them. Label the scale by what it represents, without implying a measured probability of a successful outcome.

Keep a record of why an item was eligible, which stage selected it and which rule changed its placement. When a useful item disappears, that record helps distinguish a candidate-coverage problem from an intentional restriction or a ranking mistake.

Validate signals and the final experience

The YouTube account describes withholding survey responses to check predicted satisfaction. Apply that discipline to the feedback your own system predicts. Check performance on fresh observations and examine people with little history separately; abundant interactions from established users can conceal a weak starting experience.

Ask whether the same event has the same meaning across situations. Repeatedly opening an item might reflect interest, confusion or a failed task. Compare behavioral signals with purpose-specific feedback, and account for what users were shown before interpreting their reactions.

Evaluate the complete recommendation experience against a credible baseline. Review the intended outcome, exclusions, unwanted repetition and coverage together. Set the conditions for changing or reversing the release before a favorable engagement result creates pressure to overlook a material failure.