Hotness is Back! Read the complete explanation.

You’ve been asking for it, and now it’s here –  hotness is back!  We’ve reimplemented Yesware’s email hotness measure to be more quickly computed on our end and more intelligently calculated for your needs.  To help you better use this new implementation, we’ll go over what it means for a message to be hot.

Our previous scheme assigned a hotness score for a message based on how long it took to open the message, how many times the message was opened, and how many different people opened the message.  In each case we assigned a weight to each property to combine them into a single score.  For example, the open portion of a hotness score is computed as (# times a message is opened) * (weight given to opens). We wanted to preserve this idea of combining various properties into a single score, but to better tie it to what makes a message important.

Yesware’s Email Hotness Measure

Our goal with hotness has been to assign a numerical score to how wide reaching a message is. This means that we need to analyze the relative importance of the various properties. Our previous scheme roughly ranked our properties in the following order from least important to most important: opens, time to open, and openers. In order to re-implement hotness, we needed to include a new property, number of clicks, and possibly reorder the importance of the various properties.

Before doing so, we performed an analysis of the past scores of hotness, generating a histogram of the various 7-day hotness scores sorted into buckets of 5, which is displayed below. (Note this ignores all messages with a hotness of 0 as they are most definitely not hot):

This shows an extremely long tail with some messages generating hotness around 10,000.  While these messages are definitely “hot”, they also appear to be outliers. We replotted the data, zooming in on the interesting data:

This shows again a long tail, but makes clear there are 3 initial peaks and thus 3 initial clustering, which falls in line with our initial scheme of generating 3 levels of hotness (one flame, two flames, and three flames).  We then sought to generate a new system of weights that clusters results well.

The Techniques

We next discussed the importance of the various techniques, giving them a new ordering for importance: time to open, opens, openers, and clicks. The reasoning for this is as follows: You will never have a non-zero time to open score with a zero open score, thus a time to open score adds to an open score, but does not overtake it. A higher number of openers is more important than a higher number of opens because it implies that a higher number of people read the message instead of possibly people rereading the same message, and thus you are reaching more people. Finally clicks are more important than opens, because while the message may have reached many people, the people have been so engaged in your message, that they have sought out more information.

Using this updated importance ranking, we have updated the weights accordingly, which generates the histogram below:

Again we drop off the long tail, but we also drop off the initially large peak at our 0 bucket to view our results in greater detail.  The result is the following:

 

 We again see peaks, but they continue throughout the graph and decrease in size as the hotness grows. We feel this better fits with hotness as it shows that higher hotness in messages are harder to achieve because they occur less frequently. However, in keeping with our old scheme we will only have three levels of hotness in our current implementation, thus only setting levels of hotness based on the first three distinct peaks, and thus the first three distinct clusters.

The first level (one flame) will be any message with hotness above 0, meaning that someone interacted with your message through an open or a click. The second level (two flames) is a hotness higher than 30, which will correspond to at minimum a link was clicked on your message 5 times, your message was opened 5 times by more than one person, or some combination of opens and clicks. The final and third level (three flames) will be any hotness higher than 50, which corresponds to over 8 clicks or over 7 different people opening your message.

Your team can use these updated hotness values to compare the effectiveness of your team’s emails and to set goals for your team. As with any of our features, we have plans for making hotness even better and applying it in all sorts of ways, but we’d also like to hear what you’re thinking. You can leave feedback in the comments below or at support@yesware.com

About the Author: Jacob Bartel is a summer software engineering intern at Yesware and a Ph.D. candidate in the Computer Science Department at UNC – Chapel Hill.  In his academic work, he has been working to make information sharing a more accurate and intuitive process through the designing and analyzing of recommender systems for access control systems.