Thread by @ro_laguna_

And finally, after finding many different ways on how not to do this.... some characters start to appear! Much more things to prove yet but at least there are some of them :D #Khipu2019 @DelCruzar

https://twitter.com/ro_laguna_/status/1193986963747098624

@DelCruzar Well, I could submit my first (baseline?) model and it scores 1.0. What does it exactly means? not really sure, 0 is a perfect match, so the lower, the better. The score is the average *error* among each of the evaluation samples...

@DelCruzar ... where *error* is the levenshtein distance between both strings (predicted and real) divided by the number of chars in the real string.
But, what is levenshtein distance?

@DelCruzar [1] Levenshtein distance is the minimal number of chars you need to modify (remove, add or replace) to convert the predicted string into the real one.

[1] en.wikipedia.org/wiki/Levenshte…

@DelCruzar Since this distance is divided by the length of the real string, we can think it as the proportion of chars you need to modify in your predicted string to get the real one. And this is taken on average to compute the final score. So, what exactly means that my model get 1.0?

@DelCruzar Well, you may guess it: on average I need to replace/change/modify every single predicted char to get the expected string. Can it be worse than 1?

@DelCruzar The answer (fortunately) is yes. Let's see some extreme cases:
1) predict only one char which is not in the expected char. Then your edit distance is exactly the length of the real string, the portion is 1. What is more: if the character IS on the real string, the portion is less

than one. So my model is worse (from this metric point of view) than a model that only outputs 1 random character.
2) you predict exactly the same number of characters as the real string but none of them is in the real string. Then you need to change every single char in the ...

sequence. Again: this leads to 1, and even less if some chars are actually present on the real string.
So, if the predicted string is less or equal than the expected, then your score is at most 1.

3) what if you predict a string longer than the real one? Well, if none of those are part of the real string, then you'll get a score greater than one: you need to replace every single char to get the real string, and then remove the extra chars. With a real string of 10 chars...

and a predicted string of 15 under the previous hypothesis, will lead to 1.5 points.
To finish those experiments I decided to submit randomly generated strings, trying to follow the distribution of chars on the training set and randomly pick the expected length of string

I only used 35k training samples on both: training the baseline and approximating the characters and string length distribution, out of 6.95M available, that is 0.5%.
This random submit achieved 1.06

Next steps? Try to include a LSTM or something like that to add knowledge from the sequence, and include more samples from the training set.