Saving and recalling streams
Streams carry objects between operators. You can save and recall streams using files or environment variables. For example, to store a listing of processes whose command line contains marcel:
M 0.18.3 jao@loon ~/git/marcel$ ps -c marcel > marcel_processes.txt
To read that file and write its contents to the console:
M 0.18.3 jao@loon ~/git/marcel$ marcel_processes.txt <
10207 9110 jao sleeping /bin/bash /usr/local/bin/marcel
10208 10207 jao sleeping python3 -m marcel.main
36782 10208 jao running python3 -m marcel.main
If you want to use an environment variable instead:
M 0.18.3 jao@loon ~/git/marcel$ ps -c marcel >$ marcel_processes
M 0.18.3 jao@loon ~/git/marcel$ marcel_processes <$
10207 9110 jao sleeping /bin/bash /usr/local/bin/marcel
10208 10207 jao sleeping python3 -m marcel.main
36816 10208 jao running python3 -m marcel.main
You can also append to a file or environment variable by using >> and >>$, respectively.
The behavior of the symbols <, >, >> was designed to be familiar to users of Linux and UNIX shells, going back decades. In marcel, these symbols are "syntactic sugar", which translate to operators and pipelines:
-
... > filename: translates to ... | write filename
-
... >> filename: translates to ... | write --append filename
-
filename < ...: translates to read filename | ...
This Linux/UNIX compatibility leads to the decision to read and write text files. So while the ps operator delivers a stream of Process objects to the write operator, what is stored is a sequence of strings, obtained by applying str() to each Process. You could use the write operator to store other formats, by specifying flags, e.g. ... | write --pickle filename, or ... | write --csv filename, but those options are not available for use with > and >>. (The read operator also has --pickle and --csv options.)
While storing and reading text is familiar to users of other shells, and allows for access by text editors, and other text-oriented tools, text does have its drawbacks. In particular, the rendering as text is not easily reversible. For example, ls > ls.txt stores a description of files, but you would have to parse the content of ls.txt and do more work to be able to access full file metadata, or file content.
The symbols <$, >$, >>$ operate on variables, not files, and have no counterpart in other shells. So there is more freedom in designing their behavior. Marcel implements the behavior of these symbols in terms of operators that work on variables:
-
... >$ var: translates to ... | store var
-
... >>$ var: translates to ... | store --append var
-
var <$ | ...: translates to load var | ...
store saves an entire stream in the variable, and load reads it back. So, for example, this command visits all .py files recursively inside the current directory, and stores the stream of those File objects in a variable named pyfiles:
M 0.18.3 jao@loon ~/git/marcel$ ls -fr | select (f: f.suffix == '.py') >$ pyfiles
This command reads back that stream, and locates the files that contain import readline:
M 0.18.3 jao@loon ~/git/marcel$ pyfiles <$ select (f: 'import readline' in f.read())
-rw-rw-r-- jao jao 10895 2023 Sep 10 21:43:21 build/lib/marcel/main.py
-rw-rw-r-- jao jao 6700 2023 Sep 07 20:53:42 build/lib/marcel/multilinereader.py
-rw-rw-r-- jao jao 3616 2023 Sep 07 20:53:42 build/lib/marcel/op/edit.py
-rw-rw-r-- jao jao 2678 2023 Sep 07 20:53:42 build/lib/marcel/op/run.py
-rw-rw-r-- jao jao 7966 2023 Sep 10 12:59:47 build/lib/marcel/tabcompleter.py
-rw-r--r-- jao jao 81 2020 Mar 04 11:17:43 experiments/multiline.py
-rw-r--r-- jao jao 171 2020 Feb 21 13:11:41 experiments/tryreadline.py
-rw-rw-r-- jao jao 10895 2023 Oct 27 11:54:55 marcel/main.py
-rw-rw-r-- jao jao 6700 2023 Sep 07 20:53:42 marcel/multilinereader.py
-rw-rw-r-- jao jao 3616 2023 Sep 07 20:53:42 marcel/op/edit.py
-rw-rw-r-- jao jao 2678 2023 Sep 07 20:53:42 marcel/op/run.py
-rw-rw-r-- jao jao 8101 2023 Sep 11 23:00:28 marcel/tabcompleter.py
-rw-r--r-- jao jao 44906 2019 Sep 28 11:38:53 venv/lib/python3.7/site-packages/dill/source.py