Marcel the Shell
- Jack Orenstein
- Jun 16, 2020
- 2 min read
Welcome to marcel! Marcel is yet another shell that pipes objects between commands, instead of strings. What is different about marcel is it's reliance on Python. First, it is implemented in Python.
Second, marcel exposes Python. If you have an input stream of Files and you want to output a stream of (file name, file size) tuples, then you can use the map operator, which applies a function to input values to generate output values. The function used by the map operator is an ordinary Python function. So for example:
ls -fr | map (lambda file: (file.path, file.size))ls -fr visits the current directly recursively (-r) and outputs files only (-f), not directories or symlinks. The function is in parentheses, (you could omit the lambda keyword if you'd like).
Third, marcel uses Python to take a different approach to scripting. Shell commands are augmented by control structures and rudimentary datatypes to provide a scripting language. These languages are usually pretty poorly designed, (e.g. bash). And of course, it's yet another language to learn. Furthermore, individual commands have extensive sub-languages, and to do anything sophisticated requires the reading of man pages documenting these sub-languages.
The marcel approach is to recognize that Python is a good programming language, but is bad at scripting because of the API to the host OS. You can't just run the ls command from Python, for example, you have to use something like subprocess.Popen. Piping requires chaining the stdin/stdout arguments together from successive Popen calls. Marcel offers an API module, marcel.api, which provides much cleaner integration. For example, the above example, done as Python code:
from marcel.api import *
for path, size in (ls(file=True, recursive=True)
| map(lambda file: (file.path, file.size))):
print(f'{file.path}: {file.size})
The ls -fr command, which was used in the shell, turns into ls(file=True, recursive=True) in the API. The pipe symbol again pipes the ls output to the map input. The command output is made available as a Python iterator, so that a for loop can iterate over the results.
I'm sorry, the example was only meant to show syntax, not to be a correct statement of any operation. Thank you for the detailed info on conversion between text and object streams for Marcel vs OS pipeline elements. Armed w/ that understanding I'll have another go at using Marcel.
I understood that you are using {::} instead of []. I'm wondering what the pipeline is meant to accomplish. If the goal is to extract the first field of each /etc/fstab line, I would do this:
read /etc/fstab | map (line: line.split()[0])
- read is the marcel operator for reading text files. (Or you could use cat; or your cat | lines idea, if we wanted binary streams.)
- The map operator's function splits on whitespace, and then extracts the first item of the resulting sequence. (You could leave out "map", since a function by itself assumes that operator.)
About [] and (): Yes, [] and () are overloaded, so while marcel's use of top-level parens may be unambiguous, it…
Thank you for taking the time to write such a detailed response to my comment. The cryptic text "{: l: x[0] for x in l :}" in my last post certainly did need an explanation. What I intended was to suggest an alternative syntax of {: .... :} rather than [ .... ] to delimit pipeine definitions that are stored as lambdas rather than immediatly executed, as in your example for the definiton of recent in your page at https://www.marceltheshell.org/pipelines-1
Both bash and python are already somewhat challenged in their use of [] and (). In python (1) == 1, (1,) == tuple([1]), and () is the empty tuple, so the meaning of (x) varies greatly depending on what x…
Hi eleog, thank you for your comments. I have spent a lot of time thinking about the issues you've raised, and I'm not attached to my current design, partly for the reasons that you discuss.
There are marcel/bash conflicts and confusions, which I'll go through, below. As for marcel/Python: I believe there are no actual conflicts. In marcel, Python expressions are always delimited by top level parens. So this marcel statement:
x = 1
assigns the string '1' to x, while this marcel statement:
x = (1)
evaluates everything between the parens, yielding the integer 1, and then assigns that to x. Inside the parens, you can have any valid python expression, including using [] and (). So, for example,
I ran across your shell in HackerNews and was intrigued. I like the concept, so I read the examples and blog posts and started playing with it. I quickly ran into some trouble, and I think that the source of it comes down to the way that operators have been overloaded in marcel. The use of [], (), >, and < causes problems IMHO. Python programmers expect to be able to use [] and () in the usual way. More importantly, < & > are core shell operators for file access. Have you considered creating new syntax that doesn't conflict with that of the base languages Python and Bash? Something like
saved_pipeline = {: x: cmd | cmd2 (x) :}