Luigi (not Mario's brother)

Python processing framework built by Spotify that I found while trying figure out a reliable distributed task orchestrator system.  Once you understand that the central scheduler is where the various agents dump their work to do and then it builds a plan everything comes together smoothly and the built in web administration tool saves so much time.

Major points

Sample Central Scheduler configuration

scheduler_host=<IP of host>

customizable_resource=<put limit here>


Agent configuration to Central Scheduler

scheduler_host=<IP of above scheduler>

Calling luigi task from bash command line example

luigi --module python_module.python_class LUIGI_TASK_NAME --param1 ${param1} --param2 ${param2} > local_logfile.log &