RAVE
Lib.algorithm_runner.algorithm_runner Class Reference

Wrapper around the multiprocessing pool used for generating products Since we want to avoid processing old jobs before newer jobs we have to prioritize the jobs and make sure that we are not draining computer resources on irrelevant jobs. More...

Inheritance diagram for Lib.algorithm_runner.algorithm_runner:

Public Member Functions

 __init__ (self, nrprocesses)
 add (self, func, jobid, algorithm, files, arguments, jobdone=None)
 Adds one job to be processed.
 async_callback (self, arg)
 Invoked by the async pool on answer.
 async_error_callback (self, exc)
 Invoked by the async pool if the job could not be run.
 join (self)
 Shutsdown and waits for pool to terminate.
 terminate (self)
 Terminates the pool.

Public Attributes

 lock = threading.Lock()
 queue = queue.PriorityQueue()
 pool = RavePool(nrprocesses, initializer=worker_initfunction)
 nrprocesses = nrprocesses
int running_jobs = 0

Static Public Attributes

 Process = NonDaemonProcess

Protected Member Functions

 _handle_queue (self)
 Runs one job from the queue.

Detailed Description

Wrapper around the multiprocessing pool used for generating products Since we want to avoid processing old jobs before newer jobs we have to prioritize the jobs and make sure that we are not draining computer resources on irrelevant jobs.

Also, if a job with same algorithm_id, date and time is added when another job is in the queue, the queued job will be replaced with the new jobs jobid, arguments and files.

Member Function Documentation

◆ _handle_queue()

Lib.algorithm_runner.algorithm_runner._handle_queue ( self)
protected

Runs one job from the queue.

Synchronization must be performed before entering this method. Will increase running jobs with 1 if job added async to pool Called from add(), async_callback() and async_error_callback() - only the latter two guard their caller against an exception escaping here, so this method must never let one through on its own.

◆ add()

Lib.algorithm_runner.algorithm_runner.add ( self,
func,
jobid,
algorithm,
files,
arguments,
jobdone = None )

Adds one job to be processed.

If arguments contains an algorithm_id, the priority of this job will be increased. Also, if the arguments contains date & time, this will also increase priority. If queue already contains an algorithm that is matching a previously added algorithm, its contant will be replaced (files and arguments).

◆ async_callback()

Lib.algorithm_runner.algorithm_runner.async_callback ( self,
arg )

Invoked by the async pool on answer.

Wrap everything within a exception block since there is a risc that an exception will cause the pool to hang. See issue gh-83371: Fix deadlock when a Pool callback raises an exception (GH-155777) Might be the culprit for a really strange random bug that occurs once a year or something.

◆ async_error_callback()

Lib.algorithm_runner.algorithm_runner.async_error_callback ( self,
exc )

Invoked by the async pool if the job could not be run.

Wrap everything within try/except since we don't want callback to cause pool failures.

◆ join()

Lib.algorithm_runner.algorithm_runner.join ( self)

Shutsdown and waits for pool to terminate.

◆ terminate()

Lib.algorithm_runner.algorithm_runner.terminate ( self)

Terminates the pool.


The documentation for this class was generated from the following file: