Difference: AnalysisScheme (1 vs. 7)

Revision 72011-04-21 - PeterWinter

Line: 1 to 1
 
META TOPICPARENT name="AnalysisInformation"
Changed:
<
<
-- BrendanKiburg - 28 Aug 2006

Production Passes

PSI: Pseudo Online Analysis During Run10

ANA root files

The first sort of output file generated by the UIUC analysis software is a raw analysis. At the "raw" data level, some plots are constructed and stored in files of the format ana_#####.root . These root files contain some information about the raw detector efficiencies as well as statistics about the type of events that were inserted into the trees.

TREE root files

The second sort of output file is the tree root file. This file has the format tree_#####.root and contains only 1 object, a TTree called "EventTree" . This tree now contains mustops and electron tracks and can be analyzed much more quickly than the raw data.

MTA root files

An additional program, mta, takes the tree file as input and creates a file with the format mta_#####.root as output. This file contains plots that have cuts based on parameters that are stored in the event tree. If additional plots are desired, rerunning the mta analysis over stored tree files is very fast compared with the time associated with creating the tree root files.

NCSA : Offline Analysis program

There are several components to the analysis at NCSA. We use the mass storage system to store the dozens of Terabytes of raw data, as well as to temporarily store the intermediate tree files. We use the tungsten cluster to process the raw files as well as the tree files, to produce the fittable physics plots.

Analysis related macros

  • ncsa_compute.pl [run #] [ dataset #] : This perl script launches a mu analysis on a specific run in a private directory and stores the output files in the appropriate subdirectories within ds_dataset#
  • ncsa_compute.mta [run #] [dataset #] : This perl script launches the mta analyses on a specific run which is found in the dataset# subdirectory and creates the processed tree files
  • ncsa_jobscript.pl [runlist] [dataset #] : This perl script launches a series of ncsa_compute analyses, one for each element of the runlist
  • ncsa_jobscript_mta.pl [runlist][dataset #] : This perl script launches a series of tree level analyses, based on the runlist
  • generate_sublists.pl [runlist][dataset #][subset_size] : This takes a list of runs and breaks it up into multiple shorter lists of size subset_size . The name will become ds_dataset#_subset . It will also produce two files called "Proddataset#.exe" and "MTAdataset#.exe" which may be submitted to tungsten for processing.
  • get_files_from_ncsa.pl [runlist] : This script takes a list of files and stages them locally for use with analysis. It continually probes mss until all of the files are transferred
  • analyze.pl : Runs the above scripts in an appropriate order to produce the analysis for several files simply by requesting that a runlist is analyzed

-- BrendanKiburg - 10 Oct 2006

Analysis Schematic

analysis_process.png

Useful NCSA info

Here are some parts of our NCSA analysis:
  • Project names: myb for run8 analysis (9 kSU) , lqo for the run9/10 analysis (100 kSU) .
  • Project disks: /usr/projects/smclayto for run8 , /usr/projects/bkiburg for run9/10
  • usage : a command for checking the projection allocation and usage
  • _bsub _ : a command for submitting jobs to a queue (man for info) My default is : "bsub -q normal -J ProdPass4 -o ProdRun10Imp_4 -N -Plqo -n2 -W 3:30 "ncsa_jobscript_prod.pl prod_4"";

  • bjobs - a : a command for checking the status of the currently submitted job

NPL Analysis Cluster

There are several machines available for analysis usage at the NPL cluster, summarized below

Machine Name Scratch Space [GB] Mem [GB] Processors Speed [GHz]
up 64 3.1 4 2.40
down 64 3.1 4 2.40
charm 64 3.1 4 2.40
strange 64 3.1 4 2.40
bottom 64 3.1 4 2.40
gravity 27 4.2 4 3.20
electricity 27 3.5 4 3.20
strong 27 3.5 4 3.20
weak 27 4.2 4 3.20
<-- /editTable -->
>
>
Please go to https://muon.npl.washington.edu/twiki/bin/view/Main/AnalysisScheme
 
META FILEATTACHMENT attachment="analysis_process.png" attr="" comment="Analysis diagram" date="1156794861" name="analysis_process.png" path="analysis_process.png" size="94887" stream="analysis_process.png" user="Main.BrendanKiburg" version="1"

Revision 62007-02-08 - BrendanKiburg

Line: 1 to 1
 
META TOPICPARENT name="AnalysisInformation"
-- BrendanKiburg - 28 Aug 2006
Added:
>
>

Production Passes

 

PSI: Pseudo Online Analysis During Run10

ANA root files

The first sort of output file generated by the UIUC analysis software is a raw analysis. At the "raw" data level, some plots are constructed and stored in files of the format ana_#####.root . These root files contain some information about the raw detector efficiencies as well as statistics about the type of events that were inserted into the trees.

Revision 52007-02-06 - BrendanKiburg

Line: 1 to 1
 
META TOPICPARENT name="AnalysisInformation"
-- BrendanKiburg - 28 Aug 2006
Line: 40 to 40
 
  • bjobs - a : a command for checking the status of the currently submitted job
Added:
>
>

NPL Analysis Cluster

There are several machines available for analysis usage at the NPL cluster, summarized below

Machine Name Scratch Space [GB] Mem [GB] Processors Speed [GHz]
up 64 3.1 4 2.40
down 64 3.1 4 2.40
charm 64 3.1 4 2.40
strange 64 3.1 4 2.40
bottom 64 3.1 4 2.40
gravity 27 4.2 4 3.20
electricity 27 3.5 4 3.20
strong 27 3.5 4 3.20
weak 27 4.2 4 3.20
<-- /editTable -->
 
META FILEATTACHMENT attachment="analysis_process.png" attr="" comment="Analysis diagram" date="1156794861" name="analysis_process.png" path="analysis_process.png" size="94887" stream="analysis_process.png" user="Main.BrendanKiburg" version="1"

Revision 42006-10-12 - BrendanKiburg

Line: 1 to 1
 
META TOPICPARENT name="AnalysisInformation"
-- BrendanKiburg - 28 Aug 2006
Changed:
<
<
>
>
 

PSI: Pseudo Online Analysis During Run10

ANA root files

The first sort of output file generated by the UIUC analysis software is a raw analysis. At the "raw" data level, some plots are constructed and stored in files of the format ana_#####.root . These root files contain some information about the raw detector efficiencies as well as statistics about the type of events that were inserted into the trees.

Revision 32006-10-10 - BrendanKiburg

Line: 1 to 1
 
META TOPICPARENT name="AnalysisInformation"
-- BrendanKiburg - 28 Aug 2006
Line: 14 to 14
 

NCSA : Offline Analysis program

There are several components to the analysis at NCSA. We use the mass storage system to store the dozens of Terabytes of raw data, as well as to temporarily store the intermediate tree files. We use the tungsten cluster to process the raw files as well as the tree files, to produce the fittable physics plots.
Added:
>
>

Analysis related macros

  • ncsa_compute.pl [run #] [ dataset #] : This perl script launches a mu analysis on a specific run in a private directory and stores the output files in the appropriate subdirectories within ds_dataset#
  • ncsa_compute.mta [run #] [dataset #] : This perl script launches the mta analyses on a specific run which is found in the dataset# subdirectory and creates the processed tree files
  • ncsa_jobscript.pl [runlist] [dataset #] : This perl script launches a series of ncsa_compute analyses, one for each element of the runlist
  • ncsa_jobscript_mta.pl [runlist][dataset #] : This perl script launches a series of tree level analyses, based on the runlist
  • generate_sublists.pl [runlist][dataset #][subset_size] : This takes a list of runs and breaks it up into multiple shorter lists of size subset_size . The name will become ds_dataset#_subset . It will also produce two files called "Proddataset#.exe" and "MTAdataset#.exe" which may be submitted to tungsten for processing.
  • get_files_from_ncsa.pl [runlist] : This script takes a list of files and stages them locally for use with analysis. It continually probes mss until all of the files are transferred
  • analyze.pl : Runs the above scripts in an appropriate order to produce the analysis for several files simply by requesting that a runlist is analyzed

-- BrendanKiburg - 10 Oct 2006

 

Analysis Schematic

analysis_process.png

Useful NCSA info

Revision 22006-08-28 - BrendanKiburg

Line: 1 to 1
 
META TOPICPARENT name="AnalysisInformation"
-- BrendanKiburg - 28 Aug 2006
Changed:
<
<

Analysis processing diagram

>
>

PSI: Pseudo Online Analysis During Run10

ANA root files

The first sort of output file generated by the UIUC analysis software is a raw analysis. At the "raw" data level, some plots are constructed and stored in files of the format ana_#####.root . These root files contain some information about the raw detector efficiencies as well as statistics about the type of events that were inserted into the trees.

TREE root files

The second sort of output file is the tree root file. This file has the format tree_#####.root and contains only 1 object, a TTree called "EventTree" . This tree now contains mustops and electron tracks and can be analyzed much more quickly than the raw data.

MTA root files

An additional program, mta, takes the tree file as input and creates a file with the format mta_#####.root as output. This file contains plots that have cuts based on parameters that are stored in the event tree. If additional plots are desired, rerunning the mta analysis over stored tree files is very fast compared with the time associated with creating the tree root files.

NCSA : Offline Analysis program

There are several components to the analysis at NCSA. We use the mass storage system to store the dozens of Terabytes of raw data, as well as to temporarily store the intermediate tree files. We use the tungsten cluster to process the raw files as well as the tree files, to produce the fittable physics plots.

Analysis Schematic

 analysis_process.png
Added:
>
>

Useful NCSA info

Here are some parts of our NCSA analysis:
  • Project names: myb for run8 analysis (9 kSU) , lqo for the run9/10 analysis (100 kSU) .
  • Project disks: /usr/projects/smclayto for run8 , /usr/projects/bkiburg for run9/10
  • usage : a command for checking the projection allocation and usage
  • _bsub _ : a command for submitting jobs to a queue (man for info) My default is : "bsub -q normal -J ProdPass4 -o ProdRun10Imp_4 -N -Plqo -n2 -W 3:30 "ncsa_jobscript_prod.pl prod_4"";

  • bjobs - a : a command for checking the status of the currently submitted job
 
META FILEATTACHMENT attachment="analysis_process.png" attr="" comment="Analysis diagram" date="1156794861" name="analysis_process.png" path="analysis_process.png" size="94887" stream="analysis_process.png" user="Main.BrendanKiburg" version="1"

Revision 12006-08-28 - BrendanKiburg

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="AnalysisInformation"
-- BrendanKiburg - 28 Aug 2006

Analysis processing diagram

analysis_process.png

META FILEATTACHMENT attachment="analysis_process.png" attr="" comment="Analysis diagram" date="1156794861" name="analysis_process.png" path="analysis_process.png" size="94887" stream="analysis_process.png" user="Main.BrendanKiburg" version="1"
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback