TaskDL

Task Farming for IDL

Learn more about TaskDL:

How Does TaskDL Work?
TaskDL is an IDL task farming system that uses a central task server to distribute tasks amongst multiple worker nodes running IDL on local or remote machines. Each worker node operates independently, contributing its maximum computing power without being held back by high loads or problems with other worker nodes.

TaskDL sessions can be created from within any IDL program or interactive IDL session, where users can start the task server on a specified host machine, launch or terminate IDL workers on remote hosts, or query the server for task lists and task status. IDL tasks, which may be any executable IDL command or procedure with its associated arguments, may be sent to the server via an easy-to-use object interface. TaskDL offers mechanisms to define task priorities and task dependencies.

Example: Wavelet Analysis of NASA's SOHO Images
NASA's Solar and Heliospheric Observatory mission generates a wealth of image data of the Sun and it's environs. Analysis of sequences of images can be time consuming and computationally intensive, although the image analysis routines may be applied to each image independently. To demonstrate the advantage of task-farming for this type of problem, we have written a simple analysis algorithm which we apply to SOHO images in parallel.

View a larger, mpeg version
of the Sun Image Animation

For each image in the series (314 1024x1024 pixel images of the solar disk measured at 195 Angstroms) we calculate the discrete wavelet transform coefficients, subtract the coefficients from a baseline image, and reconstruct a difference image based on the residual wavelet coefficients. This analysis is simple and is just representative of the types of analysis which could be performed on time series' of image data. While the algorithm is simple, the process is time consuming, taking over 7 minutes when run in serial.

View a larger, mpeg version
of the Processed Sun Image Animation

By using TaskDL we achieved better than 7x speedup on a cluster of 12 processors. Below is a plot which compares the speedup as a function of number of working processors. The data point for zero processors is the time for analysis running in serial.

Because each image is processed independently from the other images, speedup is nearly linear with number of working processors. The roll off from linearity is due to increasing network congestion from remote file writing.