Implicit Judgements and COEC

Driving the evaluation aspect of the Evaluation Framework is the choice of how we model user clicks to decide what is a good search result and what isn’t. We are starting with Clicks over Expected Clicks (COEC). This is a well understood approach for modeling what users like using the Click Through Rate as the key bit of information.

It is straightforward to calculate the ratio of clicks to impressions. This metric tells us how effective our results are at attracting user interest. For instance, if we track the number of clicks over expected clicks, we can assess which search results are performing well.

However, we have big dreams of moving to using Bayesian Probability Modeling in the future. There fore we’ve made the Implicit Judgement calculator pluggable to allow more sophisticated solutions to be added to our processing pipeline:

At the end of the day, we just need our Calculated Judgements to be a number that is meaningful.

Understanding User Journey Metrics: A Closer Look at Funnels and Events

In the world of digital marketing and user experience, understanding how customers navigate through a journey is crucial for optimizing conversions. This journey is often visualized as a funnel, where users progress through a series of steps until they achieve a desired goal. However, as we know, the user journey is more complex than a simple linear path; users can move forward, backward, or even abandon the process altogether.

The Funnel Concept

The funnel metaphor is essential for understanding user behavior because it illustrates how each step in the process tends to lose a portion of users. For example, in an e-commerce context, the journey might look like this:

1. Search Query: The user initiates a search.

2. Impression of Results: Each result appears (potentially generating an event for each).

3. Click on Result: The user clicks on a product.

4. Add to Cart: They add the item to their cart.

5. Purchase: Finally, the user completes the purchase.

As users navigate this funnel, we can track their behavior at each step using an event-based system, like User Behavior Insights (UBI). This allows us to analyze the efficacy of each stage of the journey.

Key Metrics to Consider

Clicks Over Impressions

One of the most straightforward metrics to calculate is the ratio of clicks to impressions. This metric tells us how effective our results are at attracting user interest. For instance, if we track the number of clicks over impressions, we can assess which search results are performing well.

It’s crucial to clearly define this metric within our tools to avoid confusion with other events that may not be relevant. For instance, differentiating between clicks on actual product links versus other elements can help clarify our insights.

Purchase Rates and Add-to-Cart Rates

Beyond clicks, we should also consider purchase rates (purchases over impressions) and add-to-cart rates (additions to cart over impressions). These metrics provide deeper insight into user intent and conversion efficiency. By analyzing these rates, we can identify potential bottlenecks in the funnel and optimize accordingly.

Configurability of Metrics

Given the varying contexts of user journeys, it’s important that any tracking plugin we implement is configurable. This flexibility allows marketers to focus on the metrics that matter most to their specific campaigns and goals.

Other Contexts: Video and Newsletters

Video Engagement

For video content, the user journey might resemble the following:

1. Search Query

2. Impression

3. Click

4. Watch

In this case, we could track a "watch" event if the user watches for a specified duration (e.g., X seconds). Similar to our previous example, we can start by calculating clicks over impressions and later dive into watch rates (watches over impressions).

Newsletter Engagement

In the realm of email marketing, the journey might look like this:

1. Email Sent: One event per user.

2. Open Email: The user opens the email.

3. Click on Email Link: The user clicks a link within the email.

Here, we can measure click rates (opens over emails sent) and click-through rates (clicks on email links over emails sent). These metrics provide insights into email effectiveness and user engagement levels.

Conclusion

While I don’t have all the answers yet, the exploration of these metrics opens up fascinating discussions about user behavior and marketing effectiveness. By focusing on clicks, impressions, and configuring our tracking tools to capture the right data, we can better understand and optimize our user journeys.

I’d love to hear your thoughts on these ideas and any insights you might have on implementing these metrics effectively!