Tweet

More from @fchollet

François Chollet

@fchollet

3 Mar

New code walkthrough on keras.io: speech recognition with Transformer. Very readable and concise demonstration of how to build and train a speech recognition model on the LJSpeech dataset.
keras.io/examples/audio…

@NandanApoorv

This example was implemented by @NandanApoorv. Let's take a look at the model architecture.

It starts by defining two embedding layers: a positional embedding for text tokens, and an embedding for speech features, that uses 1D convolutions with strides for downsampling.

Then it defines a Transformer encoder, which is your usual Transformer block, as well as a Transformer decoder, which is also your usual Transformer block, but with causal attention to prevent later timesteps to influence the decoding of earlier timesteps.

Read 4 tweets

François Chollet

@fchollet

22 Feb

https://twitter.com/moms4nuclear/status/1304473231743574016

Seeing lots of takes about nuclear power and its opponents. Yes, nuclear power could be an important element of a climate solution. Yes, the world needs to build more nuclear power plants. But it's absurd to blame environmental activists for the fact that it hasn't happened yet.

https://twitter.com/moms4nuclear/status/1304473231743574016

The primary reason why countries with large CO2 emissions haven't gone nuclear is economic: the upfront cost of a nuclear plant is a large multiple of that of a coal plant. That's why coal is king in India, for instance. Nothing to do with activists.

Or consider China, the largest emitter of CO2 today. You think environmental activism is why China hasn't built more nuclear plants? Lol. Economically, coal has been "good enough" -- assuming we ignore its health costs and long-term environmental costs.

Read 6 tweets

François Chollet

@fchollet

19 Feb

@mhmazur

Interesting analysis by @mhmazur. Human work is driven by clear goals and is informed by task-specific context. A model that is optimized for generating plausible-sounding text, ignoring goals and context, virtually never produces any useful answer (unless by random chance).

https://twitter.com/mrpatto/status/1362844477157560320

Reminder: language serves a variety of purposes -- transmit information, act on the world to achieve specific goals, serve as a social lubricant, etc. Language cannot be modeled as a statistical distribution independent of these purposes.

This is akin to modeling the appearance of animals as a statistical distribution while ignoring the environment in which they live. You could use such a model to generate plausible-looking animals, but don't expect them to be able to survive in the wild (environmental fitness)

Read 4 tweets

François Chollet

@fchollet

14 Feb

There's a pretty strong relationship between one's self-image as a dispassionate rational thinker and the degree to which one is susceptible to fall for utterly irrational beliefs that are presented with some sort of scientific veneer

The belief in recursive intelligence explosion is a good example: only someone who thinks of themselves as a very-high-IQ hyper-rationalist could be susceptible to buy into it

If you want to fool a nerd, make long, complex, overly abstract arguments, free from the shackles of reality. Throw equations in there. Use physics analogies. Maybe a few greek words

Read 5 tweets

François Chollet

@fchollet

14 Feb

An event that only happens once can have a probability (before it happens): this probability represents the uncertainty present in your model of why that event may happen. It's really a property of your model of reality, not a property of the event itself.

Of course, if the event has never happened before, that implies that your model of how it happens has never been validated in practice. You can model the uncertainty present in what you know you don't know, but you'll miss what you don't know you don't know.

But that doesn't mean your model is worthless. Surely we all have the experience of writing a large piece of code and having it work on first try.

Read 4 tweets

François Chollet

@fchollet

14 Feb

An under-appreciated feature of our present is how we record almost everything -- far more data than we can analyze. Future historians will be able to reconstruct and understand our time far better than we perceive and understand it right now.

Consider the events of January 6. Future historians will likely know who was there, who said what to whom, who did what, minute by minute. The amount of information you can recover from even a single video is enormous, and we have hundreds of them.

We're recording all of the dots -- our successors will have currently-unimaginable technology to connect them.

Read 4 tweets

Share this page!

François Chollet

Try unrolling a thread yourself!

More from @fchollet

François Chollet

François Chollet

François Chollet

François Chollet

François Chollet

François Chollet

Did Thread Reader help you today?

Like this author's thread?