Thread by @f1l1b0x

Deep Neural Networks coming to livepeer thread:
#livepeer $LPT livepeer.com

During the pandemic livepeer was concentrating on 3 areas. Adoption Scalability and value adding services. /2

To minimize integration efforts for live content platforms we created media plugins for the popular deployable media servers Mist and Wowza (more soon). /3

User of these can outsource transcoding jobs from their server to the livepeer network. This minimizes their hardware footprint and allows them to scale without preplanning. /4

Livepeer comes with some unique advantages solving problems that are hard to overcome with traditional cloud computing providers and where livepeer really makes a huge difference for them: /5

Price:
We are transcoding at about 1% of the price of a AWS encoding offering. If you pay 2$ per hour for your transcoding currently you will end up paying only 0,02$ with livepeer. /6

Availability:
Instead of waiting 30 seconds for a new transcoder or server to become available you will have instant starting times with livepeer. /7

There is no need for pre notice or reserve transcoders.
Livepeer has access to over 50.000 privately run GPUs and could handle the transcoding volumes of both twitch and youtube combined with ease. /8

This is a result of our efforts to get large scale GPU miners to start offering video services.
To my knowledge there is not a single comparable offering with similar flexibility /9

Reliability:
Livepeer Protocol was design to ensure reliability even in unreliable public p2p networks. /10

Any processing job is constantly being tracked and validated for quality and performance and our nodes do instant failover to different transcoders and different locations in cases of downtime. /11

With all these benefits and unique value props we are finally prepared to go to market but we did not stop there. /12

A big opportunity for us when we started with livepeer´s streamflow planning was to be able to run transcoding next to crypto hashing in the same GPU without interference. /13

This offers miners great side by side revenue opportunity, a second ground for their business. But mining profits went down constantly over the last two years and with Ethereum about to switch to POS millions of GPUs will become unemployed soon.
/14

While we expect many miners to go out of business we expect those heavily invested and well equipped to be able to sell it to alternative markets. Specifically those with Nvidia Rigs have a great chance on additional revenues. /15

About half a year ago, our transcoding pipeline ready for release, we started to explore ways to replace the hashing on the cuda cores with useful work that fits to video and audio processing and also find a job for the tensor flow cores you find in nvidia´s latest GPUs. /16

Since livepeer would carry and decode any video it processes it can do smart AV tasks next to it. Artificial intelligence automatically looking into the video for you and doing tasks you could never perform manually.
/17

We ended up building a framework in which a livepeer user could choose to use an own or public AI models next to the video transcoding. /18

This opens up endless use cases that make livepeer a one stop shop for media processing /19

Here an overview of the demos/pocs we created so far and what they are good for: /20

Scene classification allows you to filter content automatically.
Search ability of live and on demand content is key for a good user experience but most challengers in the user generated content market lack exactly those tools. /21

This is caused by the complexity of a DIY solution for this and expensive cloud offerings as an alternative. Scene classification by itself can be more resource hungry than transcoding is, depending on the amount of classifiers you want to run on it. /22

The most common filter is a „not save for work“ filter that is looking for all kinds of nudity violence or abuse. These can be applied very cost effectively. /23

We saw a lot of abuse on big social media platforms like rape or terrorist attacks. Those could have been automatically flagged if using scene classification would be a everyday practice at social media companies. /24

Livepeer allows you to know the content type while you encode it so you can block content in a live workflow before it airs! Users will be able to provide their own filter and search for anything they’d like. /25

Deep Speech is giving us a powerful automated subtitle creation engine. /26

Our poc will be available for English and Chinese and other languages will soon follow. Similar as with scene classification this can be a non compute trivial task. Our GPU architecture fits it perfectly. /27

Speech to text in general will improve search ability within the live content. /28

Object detection for realtime interactivity

The most promising addition to our smart AV value adds is object detection. /29

Since many years people asked me how they could improve their live audience engagement and the answer always was interactivity. The more interactive content is the more bonding the experience for the user is. /30

Ways to do this were very limited up until now. /31

Twitch had major success as being the first to enable ways how users could become a part of the content creation itself. Donations triggered twitch api events that fired a graphic played into the content creators video mixing app. /32

They just recently released an api for realtime annotation that could be applied to a livestream manually. While this is a great addition in making live content more interactive with livepeer we want to emphasize automated ways to enable interactivity. /33

Object Detection can be used in many ways. The ability to detect a object and track it in a moving scene allows deterministic rules to be applied to the objects automatically. /34

The oldest case I know for it is advertising. I remember doing it for fashion shows 12 years ago. Designers we’re asking to have the web player hyperlink the dresses while they appear on stage. /35

We did this with manual triggers that would end up being rtmp cues and flash would link to the specific store page of the designer. Doing this manually was a nightmare since you would need to make the triggers part of the video mixing itself. /36

Whenever an camera angle changed the trigger would need to be applied again. You would only be able to link one dress at a time and chances on linking to wrong dress was high. /37

The user experience was not good enough in the end to justify the amount of work to do this in a live show. With automation of the detection this is changing now. You could show several dresses in one picture and automatically mask, track and link to them. /38

It seems plausible that e-commerce is going to love this. But it is just one example of many that were impractical before and become easy now thanks to Deep Neural Networks. /39

I hope for new ways to entertain us during this pandemic. One could build a point and click adventure game that plays in real life and uses object detection to allow the audience to interact with the objects in sight of the webcaster like in Monkey Island. /40

And of course the surveillance industry is in favor of such a network. They struggle with over the air updates and onboard hw limitations. /41

Doing cloud based image recognition would be what solves their issues but since there are so many security cams in a low cost market it exceeds their budget to do it on a public cloud provider at their current cost. /42

I hope I could give you a little look out where we are heading towards. Please check out livepeer.com and sign up for a free test account. DNN features are not yet available to the public so please ping me when you want to test them.

I expected a debate on us adding censorship tools but nobody reacted so far...