Saving and recalling streams
Streams carry objects between operators. You can also save and recall streams. For example, to store the stream of Files created by ls:
This will replace the value of pyfiles, if there is one. Or, to append to pyfiles: ls *.py >> pyfiles.
You can recall a stored stream by placing the variable before the > symbol. E.g., to find Files of size >= 15000, from the previously saved result:
You can use > to copy stored streams from one variable to another, e.g. pyfiles > pyfiles2.
Pipelines can also be used as arguments to operators. This is especially useful when combined with pipelines that store streams in variables. Example:
The ps operator generates a stream of Process objects, each representing one current process. The ifelse operator evaluates a predicate for each input, in this case, checking whether the Process, p, is owned by root. (username() is a marcel builtin function). If the predicate evaluates to true for Process p, then p is passed to the bracketed pipeline immediately following the predicate, [> root]. This pipeline stores its input in the variable root. If the predicate evaluates to false, then p is passed downstream, to be stored in the variable other. I.e., we have split the stream of Processes into two streams, and stored each.
There is also an ifthen operator. It is just like ifelse except that all inputs are passed downstream. I.e., an item causing the predicate to evaluate to true will be passed to both pipelines, the pipeline argument, and downstream.
Pipelines storing and loading variables are particularly useful with marcel's set operators. For example, suppose that we explore a directory recursively, once to find recently modified files, and once to find .py files:
We can now use set operators with these variables as inputs. For example, to find recently updated .py files:
This passes the recent files to the intersect operator. The second input to intersect comes from the pipeline argument, which loads the .py files. Marcel also has the other set operators, difference and union. There is also a join operator, inspired by relational algebra.