Streams and Variables

Marcel pipes carry streams of data between operators. In the same way that you find it useful to store lists or arrays or sets of objects in programming languages, it is often useful to store marcel streams. For example, you might want to explore a directory recursively, and find the .py files. You might need to do a variety of things with this information, and it would be handy to hold onto this information, instead of rerunning the command repeatedly. To store a stream, use the > operator as follows:

ls -fr | select (f: f.suffix == '.py) > pyfiles

The ls command produces a stream of Files. Files whose suffix is .py are stored in the pyfiles variable. (You could also append to this variable by using >> instead of >.) Note that pyfiles is a variable, not a file.

Later, you can turn the stored Files back into a stream of Files, e.g.

pyfiles > map (f: f.size) | red +

This pipes the previously stored .py Files to the map operator, which obtains the size of each. red + sums the file sizes.

Note that > is context-sensitive. At the end of a pipeline it means that the stream should be stored in a variable. At the beginning of a pipeline, it means that the stored stream should be loaded into the pipeline. In fact, > is just syntactic sugar for the load and store operators. So the previous two commands could be rewritten as follows:

ls -fr | select (f: f.suffix == '.py') | store pyfiles
load pyfiles | map (f: f.size) | red +

(And yes, x > y does work, it means that the contents of x should be copied to y. x >> y would append to y instead.)