Omicron

O3GK log

Some report on O3GK data processing in detail.

O3GKlog

Installation

• The Omicron code is managed on the Virgo SVN and it should be able to be installed by CMT system.
  • But when we use the CMT in order to install Omicron, dependency conflict is occurred and we cannot install it correctly.

• F. Robinet helped us to install it on the k1sum0 and we can use copy of it on the another Debian 9 systems.

• T. Yamamoto had already copied Omicron and the related packages on k1nfs0:/export/virgo/virgosoft.

  • So Omicron can be used by mounting NFS area on the Debian 9 systems.

    > grep virgsoft /etc/fstab
    10.68.10.10:/export/virgo/virgosoft /home/controls/opt/virgosoft nfs rw,bf,soft 0 0

How to use on Kamioka server

• Before using omicron, we must set the environmental variables by

  • > source ~controls/opt/virgosoft/environment.sh

• Omicron is available as a command line tool on k1sum0, k1sum1, and another test system of Debian 9.
  • There are three ways to use omicron

    case1> omicron  start_gps  stop_gps  option_file
    case2> omicron  segment_file  option_file         ### TY don't know much about this way.
    case3> omicron  central_gps  option_file

  • Some examples of a form of option_file are shown in [1] and [2].

• Omicron is available as a GUI launched from MEDM on the client workstations.

  • Sitemap -> Commissioning -> Glitch pipeline -> Omicron

How to use on Kashiwa server

• Before using omicron, we must set the environmental variables by

  • > source /gpfs/ligo/sw/conda/etc/profile.d/conda.sh
    > conda activate igwn-py38

  • If you want to use this environment on new machine, https://computing.docs.ligo.org/conda/ is helpful.

• Omicron is available as a command line tool on m31-01, m31-02.
  • There are three ways to use omicron

    case1> omicron  start_gps  stop_gps  option_file
    case2> omicron  segment_file  option_file         ### TY don't know much about this way.
    case3> omicron  central_gps  option_file

  • Some examples of a form of option_file are shown in [1] and [2].

Trigger generation for DailySummary

• DailySummaryPage uses the trigger files using gwtrigfind for constructing the Omicron summary page.

• T. Yamamoto modified gwtrigfind because our computer environment is slightly different from LIGO's one.

  • In LIGO convention [3], trigger files should be generated in the /home/detchar/triggers

  • We have only controls user on k1sum0 and another computers.

  --- /home/controls/opt/summary-2.7/lib/python2.7/site-packages/gwtrigfind/core.py.bak   2018-10-19 09:12:45.375654000 +0900
  +++ /home/controls/opt/summary-2.7/lib/python2.7/site-packages/gwtrigfind/core.py       2019-01-23 18:29:06.224172322 +0900
  @@ -161,12 +161,12 @@ def find_detchar_files(channel, start, e
       ifo, name = _format_channel_name(channel).split('-', 1)
       # find base path relative to O1 or O2 formatting
       if start >= OMICRON_O2_EPOCH:
  -        base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers')
  +        base = os.path.join(os.path.sep, 'home', 'controls', 'triggers')
           tag = etg.upper()
           dirtag = '%s_%s' % (name, tag)
       else:
           epoch = '*'
  -        base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers', '*')
  +        base = os.path.join(os.path.sep, 'home', 'controls', 'triggers', '*')
           tag = etg.title()
           dirtag = '%s_%s' % (str(channel).split(':', 1)[1], tag)

• (2021/06/17 yuzu) Because the conda environment was updated, I applied this patch for k1sum0 again. The difference is here.

  --- /home/controls/bin/miniconda2/envs/ligo-summary-3.7/lib/python3.7/site-packages/gwtrigfind/core.py  2021-06-17 16:15:21.718561137 +0900
  +++ /home/controls/bin/miniconda2/envs/ligo-summary-3.7/lib/python3.7/site-packages/gwtrigfind/core.py_bak      2021-06-17 16:12:06.961886182 +0900
  @@ -161,12 +161,12 @@ def find_detchar_files(channel, start, e
       ifo, name = _format_channel_name(channel).split('-', 1)
       # find base path relative to O1 or O2 formatting
       if start >= OMICRON_O2_EPOCH:
  -        base = os.path.join(os.path.sep, 'home', 'controls', 'triggers')
  +        base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers')
           tag = etg.upper()
           dirtag = '%s_%s' % (name, tag)
       else:
           epoch = '*'
  -        base = os.path.join(os.path.sep, 'home', 'controls', 'triggers', '*')
  +        base = os.path.join(os.path.sep, 'home', 'detchar', 'triggers', '*')
           tag = etg.title()
           dirtag = '%s_%s' % (str(channel).split(':', 1)[1], tag)

• The Omicron job is automated using crontab on k1sum1. summary_*.txt is for * Hz sampling rate channels.

  > crontab -l | grep Omicron
  7-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_2048.txt > /tmp/auto-omicron-2048_segment.log 2>&1
  9-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_1024.txt > /tmp/auto-omicron-1024_segment.log 2>&1
  11-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_512.txt > /tmp/auto-omicron-512_segment.log 2>&1
  13-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_256.txt > /tmp/auto-omicron-256_segment.log 2>&1
  3-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation_segment.sh /users/DET/tools/Omicron/Parameter/summary_4096.txt > /tmp/auto-omicron-4096_segment.log 2>&1

• When you want to add some channels for the Omicron search,

  • 1) add channel name to DATA CHANNELS line of /users/DET/tools/Omicron/Parameter/summary_*.txt. Please choose appropriate * referring sampling rate.
    or
    2) add new job to crontab

    > crontab -e
    5-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation.sh /users/DET/tools/Omicron/Parameter/summary_test.txt > /tmp/auto-omicron.log 2>&1
    5-59/15 * * * * env - `cat /users/DET/tools/Omicron/etc/sum1_cron-omicron.env` /users/DET/tools/Omicron/Script/automation.sh /path/to/your_parameter.txt > /tmp/auto-omicron2.log 2>&1

  • In order to use trigger results on DailySummaryPage, use same OUTPUT parameters as

    • OUTPUT DIRECTORY /home/controls/triggers/tmp
      OUTPUT PRODUCTS triggers
      OUTPUT FORMAT xml
      OUTPUT VERBOSITY 0

• Finally, we must modify the INI file of DailySummaryPage.

Trigger generation on Kashiwa server

It is not yet automated, only manual script is available. Command example:

>./manual_segment_kashiwa.sh /home/detchar/git/kagra-detchar/tools/Omicron/Parameter/O3_C20.txt /home/detchar/Segments/K1-DET_FOR_GRB200415A/2020/K1-DET_FOR_GRB200415A_UTC_2020-04-15.txt > log.dat 2>&1 

Example of submission file to condor is Omicron/Script/condor.sdf For automation, cache file in parameter file (DATA FFL) has to be modified.

Now directory structure modification command is commented out. mvtoK1.sh (mvtoK1.sdf for condor)

Failure check

1. Running omicron job as above

2. checkOmicron.py (checkOmicron.sdf for condor): please check the file name of L32 locked, L89 f. The output goes in Omicron/Script/checkLog/. succeeded* is OK, failed is due to too high trigger rate, unknown is failure by other reason.

3. retry.sh (retry.sdf for condor): It will reprocess according to checkLog/failed* segment files. Now it runs for all the failed* files.
4. Go back to 2. and check the result.

Note for developers

• Bug in automation_segment.sh should be fixed.
  • Treatment of 0:00:00 : the segment including 0:00:00 maybe separated in segment files. If one of them is shorter than 64 sec, not processed. manual_segment.sh has a hint for this modification.
  • MARGIN is different from the O3GK configuration. In O3GK configuration, GPS_END is set earlier by 30 sec. 30 sec is determined by burst analysis margin.

  • Please modify segment if needed.

• To share the trigger files with other machines,

  rsync -av /mnt/triggers/K1/ /mnt/detfs0/triggers/K1/ > /tmp/cp-omicron.log 2>&1 is required on k1sum1. It is temporary commented out in crontab.

• If something went wrong, please check OUTPUT DIRECTORY in the parameter file. It should be /home/controls/triggers/tmp on Kamioka server and /data/detchar/triggers/tmp on Kashiwa server.

Reference

• [1] VIR-0135A-15, Omicron: a tool for detector characterization

• [2] VIR-0545C-14, Omicron: an algorithm to detect and characterize transient events in gravitational-wave detectors

• [3] LIGO-T1300468, DetChar Trigger Handling Conventions