Streams and Variables

Marcel pipes carry streams of data between operators. In the same way that you find it useful to store lists or arrays or sets of objects in programming languages, it is often useful to store marcel streams. For example, you might want to explore a directory recursively, and find the .py files. You might need to do a variety of things with this information, and it would be handy to hold onto this information, instead of rerunning the command repeatedly. To store a stream, use the > operator as follows:

ls -fr | select (f: f.suffix == '.py') >$ pyfiles

The ls command produces a stream of Files. Files whose suffix is .py are stored in the pyfiles variable. (You could also append to this variable by using >>$ instead of >$.) Note that pyfiles is a variable, not a file.

Later, you can turn the stored Files back into a stream of Files, e.g.

pyfiles <$ map (f: f.size) | red +

This pipes the previously stored .py Files to the map operator, which obtains the size of each. red + sums the file sizes.

About the syntax: >$, >>$, and <$ are inspired by Linux/UNIX shell notation for redirecting streams into and out of files. Because that notation is so ingrained, it is kept in marcel. So for example, ls *.py > pyfiles writes the textual listing of files produced by ls into the file pyfiles. pyfiles then stores strings, not File objects, and reading it back you would see text. The $ parts of the symbols are meant to evoke the idea of environment variables, which these symbols operate on.

It's all pipes and operators under the covers. So ls ... >$ pyfiles is equivalent to ls ... | store pyfiles, while ls ... >>$ pyfiles is equivalent to ls ... | store --append pyfiles. Both of these operate on the environment variable named pyfiles. But ls ... > pyfiles is equivalent to ls ... | write pyfiles, creating (or replacing) a file named pyfiles.