top of page
Saving and recalling streams

Streams carry objects between operators. You can save and recall streams using files or environment variables. For example, to store a listing of processes whose command line contains marcel:

M 0.18.3 jao@loon ~/git/marcel$ ps -c marcel > marcel_processes.txt

To read that file and write its contents to the console:

M 0.18.3 jao@loon ~/git/marcel$ marcel_processes.txt <
10207    9110  jao       sleeping    /bin/bash /usr/local/bin/marcel
10208   10207  jao       sleeping    python3 -m marcel.main
36782   10208  jao       running     python3 -m marcel.main

If you want to use an environment variable instead:

M 0.18.3 jao@loon ~/git/marcel$ ps -c marcel >$ marcel_processes
M 0.18.3 jao@loon ~/git/marcel$ marcel_processes <$
10207    9110  jao       sleeping    /bin/bash /usr/local/bin/marcel
10208   10207  jao       sleeping    python3 -m marcel.main
36816   10208  jao       running     python3 -m marcel.main

You can also append to a file or environment variable by using >> and >>$, respectively.

The behavior of the symbols <, >, >> was designed to be familiar to users of Linux and UNIX shells, going back decades. In marcel, these symbols are "syntactic sugar", which translate to operators and pipelines:

  • ... > filename: translates to ... | write filename

  • ... >> filename: translates to ... | write --append filename

  • filename < ...: translates to read filename | ...

This Linux/UNIX compatibility leads to the decision to read and write text files. So while the ps operator delivers a stream of Process objects to the write operator, what is stored is a sequence of strings, obtained by applying str() to each Process. You could use the write operator to store other formats, by specifying flags, e.g. ... | write --pickle filename, or ... | write --csv filename, but those options are not available for use with > and >>. (The read operator also has --pickle and --csv options.)

While storing and reading text is familiar to users of other shells, and allows for access by text editors, and other text-oriented tools, text does have its drawbacks. In particular, the rendering as text is not easily reversible. For example, ls > ls.txt stores a description of files, but you would have to parse the content of ls.txt and do more work to be able to access full file metadata, or file content.

The symbols <$, >$, >>$ operate on variables, not files, and have no counterpart in other shells. So there is more freedom in designing their behavior. Marcel implements the behavior of these symbols in terms of operators that work on variables:

  • ... >$ var: translates to ... | store var

  • ... >>$ var: translates to ... | store --append var

  • var <$ | ...: translates to load var | ...

store saves an entire stream in the variable, and load reads it back. So, for example, this command visits all .py files recursively inside the current directory, and stores the stream of those File objects in a variable named pyfiles:

 

M 0.18.3 jao@loon ~/git/marcel$ ls -fr | select (f: f.suffix == '.py') >$ pyfiles

This command reads back that stream, and locates the files that contain import readline:

M 0.18.3 jao@loon ~/git/marcel$ pyfiles <$ select (f: 'import readline' in f.read())
-rw-rw-r--   jao    jao   10895   2023 Sep 10 21:43:21   build/lib/marcel/main.py
-rw-rw-r--   jao    jao    6700   2023 Sep 07 20:53:42   build/lib/marcel/multilinereader.py
-rw-rw-r--   jao    jao    3616   2023 Sep 07 20:53:42   build/lib/marcel/op/edit.py
-rw-rw-r--   jao    jao    2678   2023 Sep 07 20:53:42   build/lib/marcel/op/run.py
-rw-rw-r--   jao    jao    7966   2023 Sep 10 12:59:47   build/lib/marcel/tabcompleter.py
-rw-r--r--   jao    jao      81   2020 Mar 04 11:17:43   experiments/multiline.py
-rw-r--r--   jao    jao     171   2020 Feb 21 13:11:41   experiments/tryreadline.py
-rw-rw-r--   jao    jao   10895   2023 Oct 27 11:54:55   marcel/main.py
-rw-rw-r--   jao    jao    6700   2023 Sep 07 20:53:42   marcel/multilinereader.py
-rw-rw-r--   jao    jao    3616   2023 Sep 07 20:53:42   marcel/op/edit.py
-rw-rw-r--   jao    jao    2678   2023 Sep 07 20:53:42   marcel/op/run.py
-rw-rw-r--   jao    jao    8101   2023 Sep 11 23:00:28   marcel/tabcompleter.py
-rw-r--r--   jao    jao   44906   2019 Sep 28 11:38:53   venv/lib/python3.7/site-packages/dill/source.py

bottom of page