Differences between revisions 26 and 50 (spanning 24 versions)

Task Manager

document

Installation

Yuzurihara prepared the virtual environment o4_dqr_proto_3 on k1det1.
DQR tasks are supported by python 3.8
This procedure will require the authentication of git.ligo.org.

To read KAGRA data by using NDS, We need to edit gwpy code (see instruction)

--- /home/controls/bin/miniconda2/envs/o4_dqr_proto_3/lib/python3.8/site-packages/gwpy/io/nds2.py_bak   2021-11-05 19:58:53.198393415 +0900
+++ /home/controls/bin/miniconda2/envs/o4_dqr_proto_3/lib/python3.8/site-packages/gwpy/io/nds2.py       2021-11-05 19:38:57.644634458 +0900
@@ -47,6 +47,7 @@
     ('V1', ('nds.ligo.caltech.edu', 31200)),
     ('C1', ('nds40.ligo.caltech.edu', 31200)),
     ('C0', ('nds40.ligo.caltech.edu', 31200)),
+    ('K1', ('10.68.10.123', 8088)),
 ])

conda create --name o4_dqr_proto_3 python=3.8 -y
conda activate o4_dqr_prot_3
conda install gwpy
conda install m2crypto ciecplib python-nds2-client cryptography==3.3.1 -y
conda install igwn-alert -y
conda install tree markuppy -y
conda activate o4_dqr_proto_3

pip install git+https://git.ligo.org/o4-dqr/o4-dqr-configuration.git
pip install git+https://github.com/gwdetchar/gwdetchar.git
pip install git+https://git.ligo.org/detchar/dqrtasks.git

python -m pip install gwpy[astro]

install ligo-proxy-init, from the LDG Client package

wget http://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/ldg-client.sh -O /tmp/ldg-client.sh && sudo bash /tmp/ldg-client.sh

X.509 certification
- For the additional permission to access GraceDB, we need to ask LIGO+Virgo colleague.
- There are two options for KAGRA account. Here Yuzurihara used ligo-proxy-init.
```
ligo-proxy-init -i shibbi.pki.itc.u-tokyo.ac.jp albert.einstein   
```
- To test the certification,
```
gracedb ping
```
```
grid-proxy-info
```
- The usage of ecp-get-cert
```
ecp-get-cert --identity-provider KAGRA --username albert.einstein --verbose
```

Added RETRY option for DAG submission

Edit the code (o4-dqr-configuration/dqr_config/DQRdag.py)

 #self.retry = 0
 # (2022/07/21 yuzu added)
 self.retry = 5

Generate DAG files

By following instruction, prepare ini files for DQR task manager.
- This procedure will NOT require the access GraceDB, because the public alert will be used.
- For test code, see /users/yuzu/work/DQR/Script/o4-dqr-configuration:k1det1
```
mkdir ~/public_html/event
dqr-create-dag -g S190924h -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity1.ini ex_configs/omega1.ini
```
The generated DAG files are located in /home/controls/public_html/events/

Because we don't store the O3 data in KAGRA site, we need to run this job with gps option.

mkdir ~/public_html/event
dqr-create-dag -g 1318260000 -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity1.ini ex_configs/omega1.ini

Submit DAG jobs

For example on k1det1

cd /home/controls/public_html/events/
condor_submit_dag S190924h/data_quality_report_S190924h.dag

Result plots and html are ~/public_html/eventsS190924h/omegascan/plots/ and ~/public_html/eventsS190924h/omegascan/about/index.html

KAGRA/Subgroups/DET/DQR/TaskManager (last edited 2022-12-21 15:09:51 by HirotakaYuzurihara)

-  ⇤ ← Revision 26 as of 2021-10-06 22:23:37 → 
  Size: 2593
  Editor: HirotakaYuzurihara
  Comment:
+   ← Revision 50 as of 2022-12-21 15:09:51 → ⇥
  Size: 3741
  Editor: HirotakaYuzurihara
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 6:
- * Yuzurihara prepared the virtual environment {{{o4_dqr_proto}}} on k1det1.
+ * Yuzurihara prepared the virtual environment {{{o4_dqr_proto_3}}} on k1det1.
 Line 9:
+ * To read KAGRA data by using NDS, We need to edit `gwpy` code (see [[http://gwwiki.icrr.u-tokyo.ac.jp/JGWwiki/KAGRA/Subgroups/DET/OmegaScan|instruction]])
-Line 11:
+Line 12:
-conda create --name o4_dqr_proto python=3.8 -y
conda activate o4_dqr_proto -y
conda install M2Crypto -y
+--- /home/controls/bin/miniconda2/envs/o4_dqr_proto_3/lib/python3.8/site-packages/gwpy/io/nds2.py_bak   2021-11-05 19:58:53.198393415 +0900
+++ /home/controls/bin/miniconda2/envs/o4_dqr_proto_3/lib/python3.8/site-packages/gwpy/io/nds2.py       2021-11-05 19:38:57.644634458 +0900
@@ -47,6 +47,7 @@
     ('V1', ('nds.ligo.caltech.edu', 31200)),
     ('C1', ('nds40.ligo.caltech.edu', 31200)),
     ('C0', ('nds40.ligo.caltech.edu', 31200)),
+    ('K1', ('10.68.10.123', 8088)),
 ])
  }}}


  {{{
conda create --name o4_dqr_proto_3 python=3.8 -y
conda activate o4_dqr_prot_3
conda install gwpy
conda install m2crypto ciecplib python-nds2-client cryptography==3.3.1 -y
conda install igwn-alert -y
conda install tree markuppy -y
conda activate o4_dqr_proto_3
-Line 17:
+Line 35:
-conda install ciecplib -y
+python -m pip install gwpy[astro]
-Line 43:
+Line 62:
+  * The usage of {{{ecp-get-cert}}} 
  {{{
ecp-get-cert --identity-provider KAGRA --username albert.einstein --verbose
   }}}

== Added RETRY option for DAG submission ==
 * Edit the code (`o4-dqr-configuration/dqr_config/DQRdag.py`)

  {{{
 #self.retry = 0
 # (2022/07/21 yuzu added)
 self.retry = 5
  }}}
-Line 51:
+Line 83:
-dqr-create-dag S190924h -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity.ini ex_configs/omega.ini
   }}}

 * This procedure will require the access GraceDB.
  * ini files were edited to fit KAGRA environment.
   {{{
dqr-create-dag -vvv --config-files O4dev/ja-ub-tst.ini --config-dir /users/yuzu/work/DQR/Script/o4-dqr-configuration/etc S190930ak S190427ba S190816v
+dqr-create-dag -g S190924h -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity1.ini ex_configs/omega1.ini
-Line 61:
+Line 87:
-   {{{
ls /home/controls/public_html/events/
+ * Because we don't store the O3 data in KAGRA site, we need to run this job with gps option.

  {{{
mkdir ~/public_html/event
dqr-create-dag -g 1318260000 -v -c ex_configs/defaults.ini ex_configs/condor.ini ex_configs/stationarity1.ini ex_configs/omega1.ini
-Line 64:
+Line 94:
-Line 75:
+Line 104:
- * Result plots and html are {{{S190924h/omegascan/plots/}}} and {{{S190924h/omegascan/about/index.html}}}
+ * Result plots and html are {{{~/public_html/eventsS190924h/omegascan/plots/}}} and {{{~/public_html/eventsS190924h/omegascan/about/index.html}}}