|
RAVE
|
Wrapper around the multiprocessing pool used for generating products Since we want to avoid processing old jobs before newer jobs we have to prioritize the jobs and make sure that we are not draining computer resources on irrelevant jobs. More...
Public Member Functions | |
| __init__ (self, nrprocesses) | |
| add (self, func, jobid, algorithm, files, arguments, jobdone=None) | |
| Adds one job to be processed. | |
| async_callback (self, arg) | |
| Invoked by the async pool on answer. | |
| async_error_callback (self, exc) | |
| Invoked by the async pool if the job could not be run. | |
| join (self) | |
| Shutsdown and waits for pool to terminate. | |
| terminate (self) | |
| Terminates the pool. | |
Public Attributes | |
| lock = threading.Lock() | |
| queue = queue.PriorityQueue() | |
| pool = RavePool(nrprocesses, initializer=worker_initfunction) | |
| nrprocesses = nrprocesses | |
| int | running_jobs = 0 |
Static Public Attributes | |
| Process = NonDaemonProcess | |
Protected Member Functions | |
| _handle_queue (self) | |
| Runs one job from the queue. | |
Wrapper around the multiprocessing pool used for generating products Since we want to avoid processing old jobs before newer jobs we have to prioritize the jobs and make sure that we are not draining computer resources on irrelevant jobs.
Also, if a job with same algorithm_id, date and time is added when another job is in the queue, the queued job will be replaced with the new jobs jobid, arguments and files.
|
protected |
Runs one job from the queue.
Synchronization must be performed before entering this method. Will increase running jobs with 1 if job added async to pool Called from add(), async_callback() and async_error_callback() - only the latter two guard their caller against an exception escaping here, so this method must never let one through on its own.
| Lib.algorithm_runner.algorithm_runner.add | ( | self, | |
| func, | |||
| jobid, | |||
| algorithm, | |||
| files, | |||
| arguments, | |||
| jobdone = None ) |
Adds one job to be processed.
If arguments contains an algorithm_id, the priority of this job will be increased. Also, if the arguments contains date & time, this will also increase priority. If queue already contains an algorithm that is matching a previously added algorithm, its contant will be replaced (files and arguments).
| Lib.algorithm_runner.algorithm_runner.async_callback | ( | self, | |
| arg ) |
Invoked by the async pool on answer.
Wrap everything within a exception block since there is a risc that an exception will cause the pool to hang. See issue gh-83371: Fix deadlock when a Pool callback raises an exception (GH-155777) Might be the culprit for a really strange random bug that occurs once a year or something.
| Lib.algorithm_runner.algorithm_runner.async_error_callback | ( | self, | |
| exc ) |
Invoked by the async pool if the job could not be run.
Wrap everything within try/except since we don't want callback to cause pool failures.
| Lib.algorithm_runner.algorithm_runner.join | ( | self | ) |
Shutsdown and waits for pool to terminate.
| Lib.algorithm_runner.algorithm_runner.terminate | ( | self | ) |
Terminates the pool.