= Omicron =
* The information for pyOmicron is summarized in DAC wiki. See [[https://dac.icrr.u-tokyo.ac.jp/KAGRA/DAWG/Detchar/Omicron|DAC wiki]].
== O3GK log ==
Some report on O3GK data processing in detail.
[[KAGRA/Subgroups/DET/Omicron/O3GKlog|O3GKlog]]
== Installation ==
* The Omicron code is managed on the Virgo SVN and it should be able to be installed by CMT system.
* But when we use the CMT in order to install Omicron, dependency conflict is occurred and we cannot install it correctly.
* F. Robinet helped us to install it on the {{{k1sum0}}} and we can use copy of it on the another Debian 9 systems.
* T. Yamamoto had already copied Omicron and the related packages on {{{k1nfs0:/export/virgo/virgosoft}}}.
* So Omicron can be used by mounting NFS area on the Debian 9 systems.
{{{
> grep virgsoft /etc/fstab
10.68.10.10:/export/virgo/virgosoft /home/controls/opt/virgosoft nfs rw,bf,soft 0 0
}}}
== How to use on Kamioka server ==
* Before using omicron, we must set the environmental variables by
{{{
> source ~controls/opt/virgosoft/environment.sh
}}}
* Omicron is available as a command line tool on k1sum0, k1sum1, and another test system of Debian 9.
* There are three ways to use omicron
{{{
case1> omicron start_gps stop_gps option_file
case2> omicron segment_file option_file ### TY don't know much about this way.
case3> omicron central_gps option_file
}}}
* Some examples of a form of {{{option_file}}} are shown in [1] and [2].
* Omicron is available as a GUI launched from MEDM on the client workstations.
* {{{Sitemap -> Commissioning -> Glitch pipeline -> Omicron}}}
== How to use on Kashiwa server ==
* Before using omicron, we must set the environmental variables by
{{{
> source /gpfs/ligo/sw/conda/etc/profile.d/conda.sh
> conda activate igwn-py38
}}}
* If you want to use this environment on new machine, [[https://computing.docs.ligo.org/conda/]] is helpful.
* Omicron is available as a command line tool on m31-01, m31-02.
* There are three ways to use omicron
{{{
case1> omicron start_gps stop_gps option_file
case2> omicron segment_file option_file ### TY don't know much about this way.
case3> omicron central_gps option_file
}}}
* Some examples of a form of {{{option_file}}} are shown in [1] and [2].
== Trigger generation for DailySummary ==
* [[http://gwwiki.icrr.u-tokyo.ac.jp/JGWwiki/KAGRA/Subgroups/PEM/SummaryPage|DailySummaryPage]] uses the trigger files using {{{gwtrigfind}}} for constructing the Omicron summary page.
* T. Yamamoto modified {{{gwtrigfind}}} because our computer environment is slightly different from LIGO's one.
* In LIGO convention [3], trigger files should be generated in the {{{/home/detchar/triggers}}}
* We have only {{{controls}}} user on {{{k1sum0}}} and another computers.
{{{
--- /home/controls/opt/summary-2.7/lib/python2.7/site-packages/gwtrigfind/core.py.bak 2018-10-19 09:12:45.375654000 +0900
+++ /home/controls/opt/summary-2.7/lib/python2.7/site-packages/gwtrigfind/core.py 2019-01-23 18:29:06.224172322 +0900
@@ -161,12 +161,12 @@ def find_detchar_files(channel, start, e
ifo, name = _format_channel_name(channel).split('-', 1)
# find base path relative to O1 or O2 formatting
if start >= OMICRON_O2_EPOCH:
- base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers')
+ base = os.path.join(os.path.sep, 'home', 'controls', 'triggers')
tag = etg.upper()
dirtag = '%s_%s' % (name, tag)
else:
epoch = '*'
- base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers', '*')
+ base = os.path.join(os.path.sep, 'home', 'controls', 'triggers', '*')
tag = etg.title()
dirtag = '%s_%s' % (str(channel).split(':', 1)[1], tag)
}}}
* (2021/06/17 yuzu) Because the conda environment was updated, I applied this patch for `k1sum0` again. The difference is here.
{{{
--- /home/controls/bin/miniconda2/envs/ligo-summary-3.7/lib/python3.7/site-packages/gwtrigfind/core.py 2021-06-17 16:15:21.718561137 +0900
+++ /home/controls/bin/miniconda2/envs/ligo-summary-3.7/lib/python3.7/site-packages/gwtrigfind/core.py_bak 2021-06-17 16:12:06.961886182 +0900
@@ -161,12 +161,12 @@ def find_detchar_files(channel, start, e
ifo, name = _format_channel_name(channel).split('-', 1)
# find base path relative to O1 or O2 formatting
if start >= OMICRON_O2_EPOCH:
- base = os.path.join(os.path.sep, 'home', 'controls', 'triggers')
+ base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers')
tag = etg.upper()
dirtag = '%s_%s' % (name, tag)
else:
epoch = '*'
- base = os.path.join(os.path.sep, 'home', 'controls', 'triggers', '*')
+ base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers', '*')
tag = etg.title()
dirtag = '%s_%s' % (str(channel).split(':', 1)[1], tag)
}}}
* The Omicron job is automated using {{{crontab}}} on {{{k1sum1}}}. summary_*.txt is for * Hz sampling rate channels.
{{{
> crontab -l | grep Omicron
7-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_2048.txt > /tmp/auto-omicron-2048_segment.log 2>&1
9-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_1024.txt > /tmp/auto-omicron-1024_segment.log 2>&1
11-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_512.txt > /tmp/auto-omicron-512_segment.log 2>&1
13-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_256.txt > /tmp/auto-omicron-256_segment.log 2>&1
3-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_4096.txt > /tmp/auto-omicron-4096_segment.log 2>&1
}}}
* When you want to add some channels for the Omicron search,
1) add channel name to {{{DATA CHANNELS}}} line of {{{/users/DET/tools/Omicron/Parameter/summary_*.txt}}}. Please choose appropriate * referring sampling rate.<
>
or <
>
2) add new job to crontab
{{{
> crontab -e
5-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation.sh /users/DET/tools/Omicron/Parameter/summary_test.txt > /tmp/auto-omicron.log 2>&1
5-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation.sh /path/to/your_parameter.txt > /tmp/auto-omicron2.log 2>&1
}}}
* In order to use trigger results on [[http://gwwiki.icrr.u-tokyo.ac.jp/JGWwiki/KAGRA/Subgroups/PEM/SummaryPage|DailySummaryPage]], use same {{{OUTPUT}}} parameters as
{{{
OUTPUT DIRECTORY /home/controls/triggers/tmp
OUTPUT PRODUCTS triggers
OUTPUT FORMAT xml
OUTPUT VERBOSITY 0
}}}
* The output of Omicron is temporary stored in {{{k1sum1:/mnt/triggers/K1/}}}. By crontab every 30 minutes, this directory is copied in {{{k1detfs0/mnt/detfs0/triggers/K1/}}}, which is mounted at k1sum0 as {{{~/triggers}}}.
* Finally, we must modify the INI file of [[http://gwwiki.icrr.u-tokyo.ac.jp/JGWwiki/KAGRA/Subgroups/PEM/SummaryPage|DailySummaryPage]].
== Trigger generation on Kashiwa server ==
It is not yet automated, only manual script is available. Command example:
{{{
>./manual_segment_kashiwa.sh /home/detchar/git/kagra-detchar/tools/Omicron/Parameter/O3_C20.txt /home/detchar/Segments/K1-DET_FOR_GRB200415A/2020/K1-DET_FOR_GRB200415A_UTC_2020-04-15.txt > log.dat 2>&1 }}}
Example of submission file to condor is Omicron/Script/condor.sdf
For automation, cache file in parameter file (DATA FFL) has to be modified.
Now directory structure modification command is commented out. mvtoK1.sh (mvtoK1.sdf for condor)
== Failure check ==
1. Running omicron job as above
2. checkOmicron.py (checkOmicron.sdf for condor): please check the file name of {{{L32 locked, L89 f}}}. The output goes in Omicron/Script/checkLog/. succeeded* is OK, failed is due to too high trigger rate, unknown is failure by other reason.
3. retry.sh (retry.sdf for condor): It will reprocess according to checkLog/failed* segment files. Now it runs for all the failed* files.
4. Go back to 2. and check the result.
== Note for developers ==
* Bug in automation_segment.sh should be fixed.
* Treatment of 0:00:00 : the segment including 0:00:00 maybe separated in segment files. If one of them is shorter than 64 sec, not processed. manual_segment.sh has a hint for this modification.
* MARGIN is different from the O3GK configuration. In O3GK configuration, GPS_END is set earlier by 30 sec. 30 sec is determined by burst analysis margin.
* Please modify {{{segment}}} if needed.
* To share the trigger files with other machines,
{{{rsync -av /mnt/triggers/K1/ /mnt/detfs0/triggers/K1/ > /tmp/cp-omicron.log 2>&1}}}
is required on k1sum1. It is temporary commented out in crontab.
* If something went wrong, please check OUTPUT DIRECTORY in the parameter file. It should be /home/controls/triggers/tmp on Kamioka server and /data/detchar/triggers/tmp on Kashiwa server.
* If the following error message occurred, the cause might be list of ffl file. Check whether file list in cache is ordered or duplicated. [[https://klog.icrr.u-tokyo.ac.jp/osl/?r=18336|see k-log]]
{{{
Segments::CheckConsistency: segments should be time sorted (GWOLLUM convention) -- 1315847008.00000>1315846912.0000
}}}
* [example] make correct ffl file for 13158 directory : {{{/users/DET/tools/Cache/Script/makeCache.sh 13158}}}
== Reference ==
* [1] [[https://tds.virgo-gw.eu/?content=3&r=11553|VIR-0135A-15]], Omicron: a tool for detector characterization
* [2] [[https://tds.virgo-gw.eu/?content=3&r=14693|VIR-0545C-14]], Omicron: an algorithm to detect and characterize transient events in gravitational-wave detectors
* [3] [[https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?.submit=Identifier&docid=T1300468&version=|LIGO-T1300468]], DetChar Trigger Handling Conventions