Daniel Semy Profile picture
May 24, 2022 12 tweets 5 min read Read on X
Want to create a Graph Neural Network based recommendation engine? You first have to get intimate with its performance metrics: recall@k and precision@k. Here are a few helpful illustrations I made that help visualize what you are measuring:
First, the definition of precision and recall.
- Precision: "What ratio of the top k elements that we recommended are relevant?"
- Recall: "What ratio of the existing pool of relevant items did we get right in the top k elements?"
In mathematical terms: Image
Let's take an example to illustrate this, with simple tabular data first. We have a list of item ratings that we take as the baseline (targets) and we have the recommendation that our model spat out. ImageImage
- Scores are between 0.0 - 10.0.
- We define a relevant match if it is above 4.0 (arbitrary, depends on the case). In our case items 1,2,3 and 5 were relevant.

The first diagram shows the items sorted by the actual scores; the second by the predicted scores given by our model. ImageImage
So how well did our model do?
* For k=3 we have *
- Precision: 2 relevant recommended items and 3 recommended items => 0.66..
- Recall: We have 2 relevant recommended items from a pool of 4 => 0.50 Image
* For k=1 and k=4 we have * Image
Now that we have established it for tabular data, let's look at how this looks for #graphs. Below is a simple graph showing user purchases. Green lines are existing purchases, red lines are user seen items that they didn't purchase. Image
Here are the edges that our model thinks should be connected to this user (our prediction): Image
We got c, e and d right! All edges have a prediction (confidence value) attached to it. Similarly to our tabular data we can list these in a table and sort them by score. ImageImage
Let's calculate k=3. Image
Finally, k=1, and k=4 Image
If you want to dive deeper into these metrics, why they can't be used to calculate loss and generally Graph Neural Network based recommendations (link prediction) then I heavily recommend Jure Leskovec's (@jure) course at Stanford: CS224W web.stanford.edu/class/cs224w/. It's fantastic.

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Daniel Semy

Daniel Semy Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(