This is a really great question without a really concise answer. But I'll try to answer the best I can without getting overly technical.
The foundation of historical linguistics is the comparative method. You take languages that are thought to be related, look at their differences, and try to work out what their ancestor might have looked like.
A great example of this is Grimm's law, one of the best-known sound changes. Early linguists noted that words from Germanic languages often resembled words in Latin or Greek.
For example, "foot" looks a lot like "pedis"/"podos", "father" looks a lot like "pater", "fire" looks a lot like "pyro", and so on. And it's not just /f/ and /p/ but a whole host of correspondences: en.wikipedia.org/wiki/Grimm%27s…
And all of these correspondences look phonologically related too: /f/ and /p/ are both pronounced with the lips or with the bottom lip against the teeth, /t/ and /d/ are both pronounced behind the teeth, and on and on.
So it seems likely that either an ancestral /p/ became /f/ in the Germanic languages, or an ancestral /f/ became /p/ in Latin and Greek and the rest.
It's more likely that one subfamily will change than that all the other subfamilies will change in the same way, so we can assume that /p/ became /f/ in Germanic and the rest just kept /p/.
You also have to look at what's phonologically more likely. Either /p/ could become /f/ or /f/ could become /p/, but I believe the former is more likely when you look across languages.
You really have to look at the preponderance of evidence, though. When you take the phonological possibilities and combine them with what we observe across different languages in the family, it points to an ancestral /p/ that became /f/ in Germanic.
And like I said, that's just one example. Grimm's law alone includes 14 different sound changes.
When you put together enough of these sound changes, you can begin to reconstruct the proto-language (that is, the ancestral language that gave rise to all the other ones, just as Latin gave rise to French, Spanish, and so on.)
It's important to note that all the changes proposed by Grimm's law happened before the Germanic languages were ever written down, but we can be very confident in them because they appear to be very systematic.
That is, sounds generally don't change randomly from one word to another. Sound changes usually affect every instance of that sound in the language (for a given phonetic environment and yada yada yada).
By looking at the modern sound correspondences and figuring out what sound changes might have led to the modern sounds, we can trace those changes backwards.
So basically, reconstructing a proto-language is a very laborious process of connecting dots and following them back to a common origin.
It seems like it should be impossible to figure out how people spoke even a thousand years ago, let alone two or five or ten thousand years ago, but the surprising thing is that it seems to work pretty well.
Of course, we can never know *exactly* what people from thousands of years ago sounded like, but we can get close enough.
Close enough, anyway, to know that at some point the speakers of a language subfamily that included Latin started pronouncing /θ/ (the initial sound in "thin") as /f/.
And I wasn't kidding when I said that it's laborious. Just scroll through this page on the phonological history of English and get a load of all the sound changes that happened in English and its immediate ancestors in the last 2,000 years. en.wikipedia.org/wiki/Phonologi…
To be a real etymologist or historical linguist, of course, you need to know not only phonology and morphology and syntax but also a ton of dead languages.
I know enough about phonology and the rest to be able to read and explain an etymology, but that's about it.
Anyway, I hope that explanation was at least a little bit helpful. If not, let me know, and I'll try to clarify. /fin
