Help Page Target Processor
The Target Processor can be found at:
The following help is available;
Below is a screenshot with the starting page of the targe processor. The target processor can be
used to process data and to give an overview of what has been and has not been processed. First choose
a project and instrument, not supported projects and instruments are greyed out.
After selecting a project and instrument click on the preselect target link, depending on the instrument
different targets can be shown. After selecting a target the target processor will step by step
guide you to selecting your targets and submitting these to a cluster. Below the image of the start
screen are the different links explained. The left part of the screen is always visible, the
rest of the screenshots of these help pages will not show this part.
- (Contact) My email adress, mail me (Willem-Jan Vriend) with any question or comments you have about the Target Processor.
- (DB User) The (astro-wise) username that is used for interaction with the database. When you visit
this page for the first time this will be the anonymous user awanonymous. You can click on it
to log in with your own astro-wise username / password. In the login screen you can specify
to 'remember' your username and password for future sessions. Then your username and password
will encrypted and stored in a cookie.
- (Help) The help pages, will open the current page.
- (Project) The available projects for the current user.
- (Instrument) The available instrument(s) for the chosen project.
- (Preferences) This will open the preferences screen.
- (Process Parameters) Here you can view and edit the process parameters of the selected target.
- (Upload Code) With the Upload Code link you can upload your own code tree. Check the box to use the uploaded code. See the howto how to generate a zipped codefile.
- (Job overview) This link will open a window with the queued, running and finished jobs of all the dpu's. Every job can be canceled and for finished jobs the logging can be retrieved.
2. Preselect Target
This screen gives the possible targets for processing for the given instrument. The
example screen below gives the possible targets for the WFI instrument, with on the
left the image pipeline products and on the right the calibration.
3. Specify Target
This screen gives an overview of the possible targets grouped per night and filter or
Observing Block. Specify a period and optinal a filter and click the Show button. Then
click on one of the blocks to see the targets. The example screen shows the groups of
targets for november 2001. Only targets with a DATE_OBS have this selection screen.
- To specify the target with manually selected input parameters click the
manually link. In the preferences can be selected which screen is shown by default, this
or the manually input screen.
- For some targets it is possible to specify the input. For CoaddedRegriddedFrame's it
is for example possible to specify whether RegriddedFrame's or RawScienceFrame's
should be as input.
- The numbers (x/y) in the legenda below the graph specify how many targets can be
processed (x) and how many are already processed (y).
4. Select Target
This screen gives the possible targets which were selected in the previous screen.
The user can check the targets and request these to be processed or queried. When
querying it is possible to specify the query depth. When a target uses a list of
objects as input the user can specify which to use. For example; for a BiasFrame the
user can specify which RawBiasFrame's to use.
- Targets with a ccd attribute are plotted on one line for all ccd's. The ccd name
links to a viewable version of the frame.
- Most targets have the processed column. This column shows the processed state
of the target(s), hover over the value to see when the targets were processed. When
a targets is processed the ccd will be printed in bold.
- When the process button is clicked the Submit screen will be shown.
- When the query button is clicked the target will be recursivly queried with
the given query depth. The result will be shown as a tree strcture, giving the target
and its dependencies. For larger depths this can take some time.
The submit screen, giving an overview of what is going to be submitted to the dpu.
The user can change the following settings :
- The dpu for single and parallel jobs can be chosen. For the experimental
target processor also the test dpu(s) will be available.
- The processing depth can be changed.
- Multiple jobs can be grouped in different ways on the available processing nodes.
When grouped per chip all jobs for the same chip will be run on the same node.
When maximized the jobs will run independent on a maximum number of nodes.
Or specify the number of nodes self.
- Force the short queue option will always queue the jobs in the short queue, independent
of the expected job time.
- For CoaddedRegriddedFrame's the Global Astrometry (GAS) can be turned on and off.
The GAS can also chosen to be processed on the single or parallel dpu.
- With the Submit button the jobs will be send to the dpu together with a zip file
with all the python code. See the Upload Code section
howto provide your own zipped code file.
This screen shows the result of a query for a RegriddedFrame that has not
been processed. See the text on screen capture for more information.
The preferences screen can be opened any time and gives the following options :
- The query and process depth determine how deep the querying or processing
will be done. A depth of 1 means that only the target will be processed. A
depth of 2 will also query or process the dependencies of the target. In
determining only calibration objects are taken into account. So Coadded-,
Regridded-, Reduced- and RawScienceFrames all count as the same depth.
- Two dpu's can be chosen. One for the parallel jobs and one for the single
jobs. The parallel dpu's are clusters of processing nodes, meant for
(embarassingly) parallel processing. The single dpu's are single node
machines optimized for a specific task, for example Coadding.
- Show process options will show a fold-out menu with options in the
Query view for all dependencies.
- Show popup info will show a popup window in the Query view for all targets
in the tree with information of the target.
- When set 'Start with graphical input' will show the graphical screen when
specifying the target, otherwise a manually input screen will be shown.
- The set privileges determine which data will be shown and what privileges will
be given to new created objects. All data with privileges equal to or higher
then the set privileges will be shown. The default privileges are 1 and the maximum
privileges are set by the project.
7. Process Parameters
For the selected target the process parameters will be shown and can be set.
In the python classes these are known as process_parameters.
- Not all process parameters can be changed; lists and class hardcoded parameters.
These are shown with a grey background.
- The recursion depth, how deep the dependencies are shown, depends on the query depth.
- To globally set a process parameter, check the option 'Change parameters globally'
on top of the screen.
- The ReducedScienceFrame dependency of RegriddedFrame is not shown, because
this dependency is the same as the ReducedScienceFrame dependency of
8. Upload Code
Every time a job is submitted to a dpu all the python code is shipped also in
zipped file. Normally the code from cvs is used but you can also use your own code.
9. Job Overview