Marcel runs on MacOS!

Yes, marcel now runs on MacOS. I've been working on this for months, and it's finally done.

This was a major project because:

• Most marcel commands run in separate processes. This enables job control, and clean cancellation of commands.

• Marcel relies on the Python multiprocessing module to manage these processes.

• The multiprocessing module can start the processes by forking or spawning.

• Forking is easy, the child process starts out with a copy of the parent's memory.

• But Python forking combined with threading doesn't work on MacOS, and some Mac libraries rely on threading. Furthermore, the default behavior of multiprocessing is currently forking on Linux, but this will change to spawning. So I was motivated to switch to spawning.

• But spawning means that any data passed from the parent to child is pickled/unpickled, not just copied by the magic of virtual memory management.

• And pickling pipelines and the marcel namespace was a disaster!

• So I had to make pipelines and marcel namespaces safe for pickling. And that was a major project.

Namespaces were difficult to pickle because previously, I had no reason to be careful about what was in them. So if the marcel configuration file (which is Python code) imports a module, then that module was in the namespace, and modules can't be pickled. The namespace could also contain arbitrary functions and methods, which can present difficulties. Pipelines sometimes state with references to methods in other classes, which is incompatible with pickling. Finally, a pipeline could contain an operator that relies on a function, the compiled version of which might not survive pickling. To deal with this, functions specified on the command line needed to shed their compiled form before pickling, and then recompile after unpickling.

So there was a lot to fix! But that's all done now. I completely overhauled the NestedNamespace, the object that holds the marcel namespace, and (as the name suggests) takes care of the scoping required by commands containing pipelines with parameters. Pipelines were heavily refactored, to rely more on an abstraction that works equally well for CLI and API usage.

Anyways, it's done. You can now run marcel on MacOS. Let me know if you find bugs.