For my first tweet storm, I wanted to share a bit about the @sourcedtech open-source stack and how recent releases and announcements fit into the bigger picture.
In the coming years we believe #MLonCode will start to drastically change developer tooling in the following areas:
In the coming years we believe #MLonCode will start to drastically change developer tooling in the following areas:
* Security & Compliance
* QA & Testing
* API Understanding
* Code Review/Quality
For #MLonCode to make an impact in these areas we believe the following ingredients are needed:
* QA & Testing
* API Understanding
* Code Review/Quality
For #MLonCode to make an impact in these areas we believe the following ingredients are needed:
1. Large datasets of millions of repositories (thank you, @github)
To make this accessible to others, we released Public Git Archive:
To make this accessible to others, we released Public Git Archive:
2. A language-agnostic representation of code
* Universal Abstract Syntax Trees
* Abstractions for high-level concepts (functions, imports etc.)
* Ability to resolve cross-references
Which is why we're working on Babelfish, check it out here: dashboard.bblf.sh
* Universal Abstract Syntax Trees
* Abstractions for high-level concepts (functions, imports etc.)
* Ability to resolve cross-references
Which is why we're working on Babelfish, check it out here: dashboard.bblf.sh
3. Ability to query the history of millions of repositories, their source code, and the language agnostic representations of it
Which is why we're working on Gitbase as a SQL layer on top of Git: github.com/src-d/gitbase & github.com/src-d/go-git
Which is why we're working on Gitbase as a SQL layer on top of Git: github.com/src-d/gitbase & github.com/src-d/go-git
4. Ability to learn from source code, which means fast scalable distributed processing of billions of Universal ASTs & their diffs
Which is why we're working on the engine (extending Apache Spark for #MLonCode): github.com/src-d/engine
Which is why we're working on the engine (extending Apache Spark for #MLonCode): github.com/src-d/engine
5. An understanding of natural language in code since language is intent i.e. naturalness of code
Which is why we are training large-scale identifier embedding models on top of 10s of millions of repositories: github.com/src-d/models
Which is why we are training large-scale identifier embedding models on top of 10s of millions of repositories: github.com/src-d/models
6. An understanding of structure in code at UAST level, at the project level and at global dependency level
So far we have tackled structural embeddings on top of UAST's: github.com/src-d/models (combined with identifier embeddings, these are very powerful).
So far we have tackled structural embeddings on top of UAST's: github.com/src-d/models (combined with identifier embeddings, these are very powerful).
7. Shareable, versionable datasets & models for the community to be able to use and improve upon
Which is why we have github.com/src-d/modelfor… & github.com/src-d/datasets but also tools like code annotation: github.com/src-d/code-ann…
Which is why we have github.com/src-d/modelfor… & github.com/src-d/datasets but also tools like code annotation: github.com/src-d/code-ann…
8. And the most important: an #MLonCode community of ML researchers, PL enthusiasts, dev. tooling engineers, data engineers, dev. advocates, PM's, designers and many other profiles that believe in the future of a language-agnostic ML powered development experience.
We believe these are the fundamental components for building #MLonCode applications (and here we'll have some exciting announcements as well coming up later this summer).
