Luigi (not Mario's brother)

Python processing framework built by Spotify that I found while trying figure out a reliable distributed task orchestrator system.  Once you understand that the central scheduler is where the various agents dump their work to do and then it builds a plan everything comes together smoothly and the built in web administration tool saves so much time.

Major points

• Construct tasks with parameters as inputs
• Keep the tasks small and build them up like a pyramid
• Setup a central scheduler and have your luigi agents point to it

Sample Central Scheduler configuration

[core]
scheduler_host=<IP of host>
parallel_scheduling=true
log_level=INFO

[resources]
customizable_resource=<put limit here>

[scheduler]
retry_delay=86400

Agent configuration to Central Scheduler

[core]
scheduler_host=<IP of above scheduler>

Calling luigi task from bash command line example

luigi --module python_module.python_class LUIGI_TASK_NAME --param1 ${param1} --param2 ${param2} > local_logfile.log &