1/ Last week, @s_ryz published a blog about how you can use Dagster to dramatically improve the developer lifecycle and increase productivity for pyspark developers creating production workflows. dagster.io/blog/pyspark 2/ Amongst PySpark developers it’s quite common to develop directly against their deployment environment: Databricks, EMR, Dataproc, etc with little tooling support. The resulting workflow can be very painful.

1/ We had our first @dagsterio community meeting on Tuesday which will now be monthly. Video here:

Special thanks to @treff7es from @prezi for talking through process of moving their entire system to Dagster

https://twitter.com/dagsterio/status/1304492670530850816

2/ @gasnerpants discussed recap of our latest major release docs.google.com/presentation/d…

1/ Dagster has been public over a year. Last week we pushed out a new version that marks a new level of maturity for the project. We now call Dagster a data orchestrator. Here is a post about what we’ve built, learned, and principles we've developed:

medium.com/dagster-io/dag… 2/ Over the past decade, there have been huge advances in data technology. Advanced computational runtimes and cloud data warehouses built on infinite, cheap storage and elastic compute are available to any organization with the right tools and sufficient resources.

1/ elementl.com/2020-2021-engi…

This year students are facing the prospect of returning to a degraded college experience while still paying full tuition. We at Elementl suggest an alternative: a well-paid, year-long fellowship where you work on an open source project, Dagster. 2/ The model already works really well. Just look at Waterloo, one of the best engineering schools in the world. With this you can make your own Waterloo experience. You can work from anywhere and would be treated as a full-time employee with commensurate pay and benefits.

1/ First community meeting for Dagster users going through our biggest release yet, 0.8.0. Fantastic meeting with really great questions at end. This was for existing users so we jumped right into advanced stuff. 2/ Will be posting in detail about the release next week. But there are huge internal changes in addition to user-facing features. A little preview of some of the interesting bits below.

Happy to announced that the @dagsterio team has pushed out our latest major rev, 0.7.0. In the last six months we've moved from a tool suitable for local development, to a hostable one for smaller pipelines, to one for large scale pipelines in modern infra.

https://twitter.com/dagsterio/status/1233156761571319808

First: a reskin and new navigation scheme for our frontend, Dagit. We think that Dagit sets a new standard for frontend in data tools. We also have dramatically improved rendering perf for large pipelines (1000s of nodes) along with the ability to subselect with a condense syntax

California is under siege by horrific fires. Both my wife @lesliejz and I have felt helpless witnessing heroic firefighters fight this battle. They deserve our support. We're starting a @gofundme to raise critical funds and will be matching donations. gf.me/u/wdzjam Sonoma County continues to be ground zero for many of these catastrophic fires, such as the 2017 Tubbs Fire and 2019 Kincaid Fire. And the firefighters in those very areas are critically underfunded.

1/ Yesterday @software_daily published a podcast with @floydophone and I as part of the series on FB eng. What started as a "bear case" ended up being more of a holistic assessment. It's a distillation of things we've learned from the whole series.

softwareengineeringdaily.com/2019/10/30/fac… 2/ I'm really proud of this interview and touches on a lot of themes/issues I care about. In previous episodes, I've tried to do huge one day tweetstorms about these podcasts but that's really chaotic. So I'm going to break it up over a few days.

1/ Today we at Elementl is happy to announce our latest version of @dagsterio, 0.6.0 (code name: Impossible Princess)

We are shipping a 1) built-in scheduler 2) a multi-process execution engine 3) hosted monitoring and 4) one click deployment to AWS

medium.com/dagster-io/dag… 2/ We were thrilled by the interest in our July announcement (medium.com/dagster-io/int…). But the feedback from our early users was clear: They wanted to use Dagit, our local dev environment, to monitor production pipelines. They also wanted execution and scheduling out-of-the-box.

1/ Another (belated) tweetstorm on the @software_daily FB engineering podcast series. This time starring my friend @tomocchino. It aired about a month ago. See original here: softwareengineeringdaily.com/2019/07/18/fac… 2/ Tom is the director of the React, Relay, web core teams, among others, and is an extraordinarily talented manager. The output and happiness of his teams speaks for itself. @the_prion covered a bunch of topics in interview. I'll select a few to comment on.

1/ Yesterday I tweeted about this paper: www-cs.stanford.edu/~matei/papers/… I want to explain why I think it’s so interesting. It’s re: gg, a framework for executing “everyday apps” (e.g. distributed compilation, video encoding) on FaaS platforms (e.g. aws lambda). 2/ They lead with well-known workloads amenable to distributed computation like build systems and video encoding. They are familiar and good for benchmarking. But this is a much more generalizable system/technique that should be applicable across a bunch of different domains.

1/ Today we at Elementl are excited to launch an early release of Dagster, an open-source Python library for building data applications. Here's a post about what Dagster is, why I moved to data infra, why data is hard, and why we need a new system. medium.com/p/dbd28442b2b7 2/ We also did a talk a Data Council a couple of months ago. The code samples are out-of-date in the talk (We’ve been working hard!) but the core themes and tooling demos are still relevant.

1/ Really proud of my father today, who just released this video announcing the creation of his nonprofit, Lp(a) care. A huge, underreported issue (esp. in the US) around cardiac health.

2/ After my older brother's near death in a heart attack a few years ago that came out of nowhere, my father took it upon himself to figure out what happened.

1/ There's been an interesting resurgence recently of criticizing DRY (Don't Repeat Yourself) and embracing copying and pasting of code in order to avoid the costs of dependency and coupling. There are good arguments for this. See theregister.co.uk/2016/03/23/npm… 2/ Like a lot of things in software, engineers respond to experiencing pain by wildly overreacting. Copying and pasting code should still give you pause. Like anything else there is a cost benefit tradeoff.

💯💯💯 When you see a legacy media institution touting a "flat IPO" as failure and an "IPO pop" as success, remember that all they are doing is broadcasting their utter and complete financial innumeracy.

https://twitter.com/bgurley/status/1138880860361449472

The other interpretation is that since an IPO pop is a direct transfer of wealth from the companies that actually build things to the banks that manage the transaction, the media understands an IPO pop as success because they are more aligned with the banks than the companies.

3.1/ Mike on Facebook Engineering: “I will say a thing about this two, an interesting anecdote about Facebook culture is that I think many of the very best engineers in Facebook's history were also the cause of catastrophic engineering failures.” 3.2/ Harkens back to MS: “I remember there was this element of lore. I don't know if it's still true … Excel was written in hand-tuned assembly. There was something like 91 global variables in Excel that were protected by one spin lock. No one could unravel the hairball.”

2.1/ (Old) Microsoft vs Facebook. Vernal hot take alert 🚨🚨🚨: “(At Microsoft) I actually sat on the SOAP standards body and I thought about this thing that hopefully no one on the podcast remember is called the WS star protocols. It's like CORBA, but worse.” 😂😂😂 2.2/ Point of talking about this: “One of the lessons I mislearned when I first joined Microsoft was the way you got things done was you allocated 200 people to a project and told them to go and start doing some stuff.“

1.1/ Today’s installment from the @software_daily /Facebook podcast series: Mike Vernal (@mvernal). softwareengineeringdaily.com/2019/05/17/fac… As a reminder it is part of ongoing series interviews about the Facebook’s engineering culture and practices.

https://twitter.com/schrockn/status/1130503625518616578

1.2/ First some context on Mike. Mike doesn’t brag, so I’ll do it it for him: He was one of *the* most influential execs at the company. Ran huge swaths of the co., and he reported to Mark directly for years. He was on M-team, the “strategic high command” of FB so to speak.

3.1/ FB v GOOG:
Jeff asks about the FB/GOOG culture comparison which is a common theme in the whole series. My/Pete’s thesis: Facebook’s engineering values are better suited for the vast majority (but not all) of startups building cloud-based consumer- and business-facing apps 3.2/ First of all downstream from founder personalities/experiences. Google founded by academics/PhD’s. Facebook founded by college dropouts. Early team and early hires also in the same cohort for the most part.

2.1/ Technical Strategy: “Evolutionary Means For Revolutionary Ends: The Art of Changing Large Systems” Bunch of my career was spent building/tech-leading the abstraction stack that led to GraphQL. Operated mostly on instinct but formed a philosophy around what worked. 2.2/ So this means having revolutionary ambitions, but having incremental process in order to get there. It might seem like a from-scratch rewrite in retrospect; but more Ship-of-Theseus-style transformation. It was done piece by piece on a live system.

1.1/ For today’s installment podcast summary/hot takes, I’m doing my own podcast.: softwareengineeringdaily.com/2019/05/14/fac…

This episode is mostly about evolving large systems and codebase. I call this: “Evolution Means for Revolutionary Ends: The Art of Changing Systems in Place” 1.2/ Original post about series is here:

https://twitter.com/schrockn/status/1130503625518616578

(Note also doing new threading approach. Twitter’s threading seems to totally break down past 25 tweets, so doing this in tranches)