Thread by @DataScienceFest

1 PM today #DSFthegreatindoors presents

Lunch & Learn: Thinking in Frameworks for Data Engineers with @Facebook
A talk by Shachar Meir, Data Engineering Manager, Facebook

tickets.datasciencefestival.com/talks/workshop…

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols
ND

ABOUT THIS TALK
Frameworks are one of the best ways for Data Engineers to accelerate their work

Thinking in Frameworks for Data Engineers
A talk by Shachar Meir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#ML
@DataIdols
#DSFthegreatindoors

We are off

Thinking in Frameworks for Data Engineers with @Facebook
A talk by Shachar Meir, Data Engineering Manager, Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning
#ShacharMeir

@DataIdols

#DSFthegreatindoors
ND

Welcome Shachar!
Pls ask questions on the chat.

Thinking in Frameworks for Data Engineers with @Facebook
A talk by Shachar Meir, Data Engineering Manager, Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning
#ShacharMeir

@DataIdols

#DSFthegreatindoors
ND

Accelerate your work with frameworks
Basic concepts will be covered

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Say we work in a money transfer org and have this fact table

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning
#ShacharMeir

@DataIdols

#DSFthegreatindoors
ND

We might create aggergation tables like these
See the pattern?

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning
#ShacharMeir

@DataIdols

#DSFthegreatindoors
ND

Can we create a template, a simple aggregation class?

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning
#ShacharMeir

@DataIdols

#DSFthegreatindoors
ND

We need to get the parameters for this function
Define in __init__ but not best solution

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#ML

@DataIdols

#DSFthegreatindoors
ND

You need to know how to use kwargs, see python docs

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

See line 101

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Now define another class SimpleAggregationMetric line 91

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Expanded line 91 onwards

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Same output as before but with better structure
Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

List comprehensions are another powerful construct in Python

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Line 110 for string concatenation

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

We OO inheritance in Python

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

We can now implement design patterns for common tasks line 112

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Added to line 112 onwards

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

We want to avoid repetition....

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

See line 123 onwards, another list comprehension

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

The output of above but breaks the code!

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Expect a string input but get a list, error message....

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Have you used bucketing? Circa line 127

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

Output shown uses 10 lines of code highlighted

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

This is very powerful, few lines of code to produces lots of aggregation

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

What an awesome demo of superb live coding by Shachar! Thanks!

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND

At Facebook many #DataEngineers contribute to each #framework
Optimisation of design patterns saves folks time
Thinking in Frameworks for Data Engineers
By #ShacharMeir, Data Engineering Manager, @Facebook
#DataEngineering
#AI
#DataScience
#ML

@DataIdols
#DSFthegreatindoors
ND

Hey #DSFthegreatindoors community

Please RETWEET, share and mention 🙏
Please RETWEET, share and mention 🙏
Please RETWEET, share and mention 🙏
Please RETWEET, share and mention 🙏
Let's create some noise on #Social

Thinking in Frameworks for Data Engineers by #ShacharMeir
ND

MASSIVE THANKS TO Shachar & FB

Thinking in Frameworks for Data Engineers
A talk by #ShacharMeir, Data Engineering Manager, @Facebook

#DataEngineering
#Frameworks
#AI
#DataScience
#MachineLearning

@DataIdols

#DSFthegreatindoors
ND