Thread by @ireneista

So we should probably explain the ideas we're gonna put in our shell project, now that we've just spent five hours debugging it in public, heh :)

https://twitter.com/ireneista/status/1353151142763622406

So first off, let us talk about why we're doing this. We've been using Unix command lines for... a bit over 25 years at this point, we think. In all that time we've developed some opinions.

We don't expect this to change the world. We expect it to be a moderately useful tool. We're well-known enough at this point that it's quite possible people besides ourselves will use it, and if so, we'll be happy. We'll also be happy if we're the only ones who do.

It doesn't have a name yet, by the way. The perfect name will come to us at some point, but for now the repo is just called "shell".

We're bipolar, and we have historically had this whole thing where our recreational software projects get names before they get anything else, and it's... not been a great pattern for us, we're trying to break it. So we're not worrying about a name just yet.

So first and foremost, we're writing this because there's a convenience feature we really want. It turns out we have a lot of small ideas that are also going into it, because they all fit together pretty well, but the reason we're bothering is this one feature.

The feature is something we're calling "workflows".

A "workflow", in our shell, is a first-class object representing a sequence of commands to be run one after the other.

This probably doesn't sound revolutionary, in that you can do the same thing informally in any shell. For example, in bash you might write:

$ git commit -m "missing semicolon"; git push; cargo build; cargo run

But, see, bash doesn't have "sequence of commands" as a data type that you can inspect and programmatically modify from within the shell. You can concatenate strings and even shell out to another instance of bash to run them, but that's not the same thing.

When we say "first-class object" we mean it in the same sense that functions are first-class objects in Lisp and derivatives: They exist in the type system and you have a suite of functions for working with them conveniently.

We plan to have key bindings and builtins that do small useful things like: run this workflow; pause it where it is; re-run from the last step that succeeded.

Over the years, virtually every interesting use of the shell as an *interactive* tool that we can remember inventing, has been something that could be captured as a workflow.

We also plan to have features around monitoring the progress of an ongoing workflow. For a command that creates a file, things might be configured so that asking to show the progress gives you the equivalent of "$ watch du -h output" - telling you how big it is.

We might literally use the existing "watch" command to implement it; what we're interested in is mostly the UI glue so you don't have to type it out every time.

Pretty much every software project we've worked on, there has been some small set of commands that we basically needed to re-run in order any time we changed things.

Wrapping things up in build tools doesn't change that because not all commands are build-related. For example, we emphatically do not want the build system to know about our version control, let alone depend on its details.

The shell is the abstraction layer at which these high-level integrations are appropriate.

So. How will users actually define and invoke workflows, in practice? To answer that, we need to talk about the other major concept we're putting in this. It's a UI concept. Sort-of a GUI thing.

Many of you are likely familiar with the idea of "worksheets", as seen in Smalltalk and MPW - windows which blur the line between text file and command prompt.

Basically, a worksheet is a text file but if you hit, like, control-return or whatever, instead of inserting a newline the editor/shell evaluates the line you're on and prints the result on the line below.

We imagine they've probably been used in software that was written more recently than the 1980s, too. We look forward to some of you telling us all about this newfangled stuff in the replies. ;)

So. When you're running commands interactively, you don't actually want to be creating a big text file for every single thing. Or at least, it is our opinion that even if you do want that, it probably will not make you happy.

So to be super clear, our shell will have two modes of operation. When you first start it up, it will look just like any ordinary Unix shell, you'll have a command prompt and if you type things, they run.

We'll be doing a tmux-like paned window thing, GUI within the terminal, whereby you can *also* have worksheets, and both a worksheet and the command prompt can be visible at the same time.

The reason we dug up the old idea of worksheets for this is that we think it dovetails particularly well with our "workflows" concept.

See, there's going to be a distinguished filename, maybe something like WORKFLOWS, maybe something less obtrusive, which if you cd into a directory that has a file by that name, by default it pops up as a split pane at the top of the terminal.

There will be collapsing sections in the worksheet, similar to Emacs org-mode, so you can get a really quick overview of what's in it without having to use very many lines of the display.

Having it open like that will also (by default, but you can configure it) bring its named workflows into scope. We're hopeful this won't be a security issue, because you'll still have to type their names to invoke them.

Workflows will still function fine without the worksheet functionality, this is just a convenient way to learn about them and be reminded of their details.

Ultimately, we hope, this feature will make it a little more friendly to explore a strange software project's source tree, because in each part of the tree you'll have easy reference to the things that people have found it useful to run in that directory.

We also intend to have one of these files in our home directory for some system maintenance tasks that we wind up running frequently. We could use shell aliases for that, but what we really want is to be *reminded* of them, not to have to memorize something.

We must be doing this exposition correctly, because @d_christiansen anticipated our next topic ;)

https://twitter.com/d_christiansen/status/1353251820374786048

The first iteration will have the terminal-GUI be part of the same program as the shell components. (Sure, we can call it a TUI.) Long-term, we would like to factor that out.

We want to get something in front of people as quickly as possible, but also we think that monolithic systems are bad and it would be preferable to build this as smaller components that users can combine in novel ways.

So long-term there will be a separate program which just provides paned windows, and programs that run under it can do whatever they want with the panes.

We thought a lot about the best way to connect those two parts, and we regret to inform you that we intend to do it by inventing a bunch of terminal escape sequences that control window panes. Sorry about that. We wish there were a better way.

Unix is really heavily designed around streams of bytes as the most important way to connect software tools to each other. That was a very deliberate decision from early on; there's a paper about it somewhere.

If we didn't do the input/output via a stream, we'd break tools such as ssh, mosh, screen, tmux...

We encourage you not to think about the messiness of parsing and generating terminal escape sequences, and instead to focus on the API that they'll be implementing. If it's any consolation, these ones will at least be well documented and easy to use in your own scripts.

As far as the actual API, the interactive fiction community has thought a LOT about the problem of, basically, going one step up from terminals. See, for example, the functionality of glk: eblong.com/zarf/glk/Glk-S…

We don't intend to adopt glk wholesale; we have no interest in turning this into a sounds-and-multimedia thing in the short or medium term. If that ever happens it will only be because people actually use it heavily and the user community wants those features.

We also see a need for a few small things that glk doesn't provide, such as points of integration for client programs to do tab completion and customize key bindings. Basically the stuff you'd see in GNU readline, but through a formally defined interface.

As far as the opening and adjusting and measuring of window panes, though, glk is a very thoughtful set of primitives and we see no need to reinvent those parts.

For example, it has separate concepts of window panes that represent a long history of stuff you did interactively, vs. window panes that represent grids of characters.

Making those concepts explicit at this layer allows the UI to be specialized, for example with scroll functionality.

We don't intend to write any proper GUI apps for this right away, but we want to design it in such a way that somebody else could. There's some prior art; iTerm for OS X has a special tmux integration which uses native GUI for tabs.

By the same token, anyone who really wants to do this within Emacs will be able to write a wrapper for that.

We note that there are security implications to co-mingling window panes from different layers of processes. We will definitely make sure that the ability to manipulate panes is NOT inherited by subprocesses by default, and that the UI clearly distinguishes where panes come from.

Anyway, that's all version 2 stuff. Version 1 will just have the UI features be part of the main process. We'll be keeping it as cleanly separated as we can though, in anticipate of the later abstraction.

There's one more topic we should address, and that's the actual shell syntax.

It's going to be radically simple.

Bash has about fifty zillion weird syntaxes for extremely specific types of string manipulation. We are not doing that.

We'll just have ordinary data types like an ordinary programming language, and maybe some implicit promotion rules that keep things terse for the core use-case of invoking commands.

We have seen a lot of shell syntaxes and we've never seen one that didn't feel like a mistake.

You may ask: Why not just use an existing general-purpose programming language?

If you want to do that, we encourage you to! We've tried that a few times in the murky past.

First, using an existing language would encourage people to, if you'll pardon the pun, type-cast the shell - perceive it as only a tool for fans of that specific programming language.

Second, we don't want to encourage general-purpose programming in our shell. We want it to be possible, but if you are writing a program that has actual data structures, you should use a real language.

We'll make the data types and control flow as elegant and thoughtful as we can manage, based on our many years of experience using a wide variety of programming languages. Then we'll probably never use them, and we hope you won't, either.

After all, you don't need to use bash as your shell to run a bash script. The schebang line takes care of that.

We're under no illusion that bash will cease to exist in favor of our shell, and that's fine. There can be more than one tool.

People who want to write shell scripts in traditional shell languages can, and should, continue to do that. It's a totally separate decision from what shell they use for interactive purposes.

You may say that it would be simpler for those users to only need to learn one syntax. Fine. That's part of why we're including general-purpose facilities. We seriously doubt, though, that simplicity is the real motivation for writing traditional shell scripts.

Traditional shell scripts were "simple" by the standards of the 1970s, but nobody today has that excuse. Admit it, you're writing them because you enjoy a challenge. ;)

There's nothing wrong with that, as long as you don't push it at other people. Keep doing it. We're not going to make our syntax overly complicated for your sake, though. ;)

Way, way down the line we have ideas that might go in a version 3, but ... two things at a time.

So! That's it! If you read this far, thank you and we're flattered. Again, we're doing this for our own use, but we very much do hope somebody out there will wind up enjoying it.

We fondly remember the 80s and 90s, when it seemed like nearly all software was written out of a spirit of "hey, somebody else might enjoy it, let's share!"

App stores really changed that atmosphere. We want it back, so we're trying to be the change we want to see, you know?