Differences between revisions 29 and 85 (spanning 56 versions)
Revision 29 as of 2019-07-05 16:46:58
Size: 4114
Comment:
Revision 85 as of 2020-12-16 18:00:59
Size: 7846
Comment:
Deletions are marked like this. Additions are marked like this.
Line 6: Line 6:
This project is for glitch study. You can see the result on [[https://www.icrr.u-tokyo.ac.jp/~yuzu/bKAGRA_summary/html/|Yuzu Summary]]. This project is for glitch and lock loss study. Also applicable to event validation of GW candidate. You can see the result [[https://gwdet.icrr.u-tokyo.ac.jp/~controls/GlitchPlot/index.html|here]].
Line 8: Line 8:
== Recent update ==
 *7/5 SNR threshold is lowered during 0am-8am. Instead, daytime events are all rejected.
Original developer: Chihiro Kozakai (production of plots), Hirotaka Yuzurihara (web page).
Line 11: Line 10:
 *7/5 Missing plot for PEM trigger channel is fixed.
Line 13: Line 11:
 *7/4 ER events are reprocessed.

Old information is more below.
Line 18: Line 13:
7/5~
Line 20: Line 14:
||<:>'''Channel'''||'''SNR threshold (8:00~24:00)'''||'''SNR threshold (0:00~8:00)'''||
||IMC-CAV_TRANS_OUT_DQ||<:> veto ||<:> 10 ||
||PSL-PMC_TRANS_DC_OUT_DQ||<:>veto ||<:> 30 ||
||PEM-ACC_MCF_TABLE_REFL_Z_OUT_DQ||<:>veto ||<:> 30 ||
||PEM-ACC_PSL_PERI_PSL1_Y_OUT_DQ||<:>veto ||<:> 7 ||
||PEM-MIC_PSL_TABLE_PSL4_Z_OUT_DQ||<:>veto ||<:> 7 ||
||<:>'''Lock state : IMC'''||
4/7~4/20 O3GK
||<style="text-align:center">'''Channel''' ||'''SNR threshold''' ||
||K1:CAL-CS_PROC_DARM_DISPLACEMENT_DQ ||<style="text-align:center">100 ||
||<style="text-align:center">'''Lock state : Observation mode (K1:GRD-IFO_STATE_N == 1000)''' ||
Line 28: Line 19:
6/8 during ER

||<:>'''Channel'''||'''SNR threshold'''||
||LSC-CARM_SERVO_MIXER_DAQ_OUT_DQ||<:> 10 ||
||AOS-TMSX_IR_PD_OUT_DQ||<:> 10 ||
||IMC-MCL_SERVO_OUT_DQ||<:> 30 ||
||IMC-SERVO_SLOW_DAQ_OUT_DQ||<:> 7 ||
||IMC-CAV_TRANS_OUT_DQ||<:> 10 ||
||IMC-CAV_REFL_OUT_DQ||<:> 10 ||
||PSL-PMC_TRANS_DC_OUT_DQ||<:> 30 ||
||PSL-PMC_MIXER_MON_OUT_DQ||<:> 10 ||
||<:>'''Lock state : IMC'''||

Old information is more below.
Line 44: Line 21:
The main purpose is to study glitches and lock loss by visual inspection. Also, collecting information for glitch database is done through voting form for each glitch event. Using the result, we aim to do noise hunting, glitch characterization, and labeling for machine learning application. Here is slides describing how to use: [[https://gwdoc.icrr.u-tokyo.ac.jp/cgi-bin/private/DocDB/ShowDocument?docid=10371|slides]]
Line 45: Line 23:
The main purpose is to study glitches by visual inspection. Also, collecting information for glitch database is done through voting form for each glitch event.
Using the result, we aim to do noise hunting, glitch characterization, and labeling for machine learning application.
GlitchPlot uses trigger information provided by Omicron. GlitchPlot does clustering of the triggers and summarize the trigger information for each events based on the Omicron output.
Line 48: Line 25:
GlitchPlot uses trigger information provided by Omicron. GlitchPlot does clustering of the triggers and summarize the trigger information for each events. Also, GlitchPlot uses lock loss information based on lock state information of guardian.
Line 50: Line 27:
Plots for triggered channel and relevant channels are provided on [[https://www.icrr.u-tokyo.ac.jp/~yuzu/bKAGRA_summary/html/|Yuzu Summary]].
The relevant channels are basically chosen from
For each events, plots for triggered channel and relevant channels are provided [[https://gwdet.icrr.u-tokyo.ac.jp/~controls/GlitchPlot/index.html|here]]. The relevant channels are basically chosen from
Line 58: Line 35:
Using [[https://github.com/gw-detchar/Kozapy|Kozapy]] batch codes, following plots are provided:

 * Time series
 * Spectrum before trigger and after trigger
 * Spectrogram
 * Coherencegram with trigger channel
 * Q-transform (Under testing)

The parameter for plots are determined based on trigger information. Also, lock state summary around the trigger time is provided.

There is channel list of 2 types of suggestion. The intent of this list is to pick up suspicious channels which is related to noise source.
 * Suggestion 1: picked up by higher coherence than usual
 * Suggestion 2: picked up by high SNR in Q-transform

The event web page shows the result in the following order.
1. Lock state
2. Trigger channel plots
3. Suggestion 1 channel plots
4. Suggestion 2 channel plots
5. Channels affected by DARM DoF
6. Other channels
Line 60: Line 58:
=== Trigger clustering ===
For the trigger clustering method, please refer p3 of [[https://gwdoc.icrr.u-tokyo.ac.jp/cgi-bin/private/DocDB/ShowDocument?docid=10368|slides]].
Line 61: Line 61:
=== Plot channel list ===
Line 63: Line 64:
#{{{
#xxx
#}}}
=== Plot parameter ===
For the exact method for plot parameter determination, please refer plotter.py and condor_jobfile_plotter.sh in [[https://github.com/gw-detchar/tools/tree/kashiwa/GlitchPlot/Script|GlitchPlot github]]. The intent and the variable name in the shell script is as follows.
Line 67: Line 67:
= Info log =  * Time series
  * GPS start time ($gpsstart), end time ($gpsend)
   * The center is the trigger time.
   * Time span is roughly larger one of [4sec, trigger duration * 10]. (Adjusted to fit the spectrogram requirement)
  * Output directory ($outdir)
  * Channel ($chlist[@])
  * Lock state segment bar: flag($lock), guardian channel($lchannel), guardian value($lnumber), segment bar title($llabel)
  * Data file type ($data)
   * Full data is used.
  * If time span is too long, down sample is applied.
Line 69: Line 78:
== Trigger channel list log ==  * Spectrum
  * List of GPS start time ($gpsstarts30[@]), end time ($gpsend30[@])
   * Trigger time and 2sec before and after the trigger is avoided.
   * About 30 sec is used each before trigger and after trigger spectrum.
   * The length is adusted to the multiple of the FFT length.
  * Output directory ($outdir)
  * Channel ($chlist[@])
  * FFT length ($fft30)
   * 1/band width (Adjusted to the nearest 2^n sec.)
Line 71: Line 88:
7/5~  * Spectrogram, coherencegram
  * GPS start time ($gpsstart), end time ($gpsend)
   * The center is the trigger time.
   * Time span is roughly larger one of [4sec, trigger duration * 10].
   * The length is adjusted to be Stride * n
  * Output directory ($outdir)
  * Channel ($chlist[@])
  * Lock state segment bar: flag($lock), guardian channel($lchannel), guardian value($lnumber), segment bar title($llabel)
  * Stride ($stride)
   * To have enough time resolution to see the glitch.
  * FFT length ($fft)
   * stride = FFT length * n
  * With and without whitening (Spectrogram)
Line 73: Line 102:
||<:>'''Channel'''||'''SNR threshold (8:00~24:00)'''||'''SNR threshold (0:00~8:00)'''||
||IMC-CAV_TRANS_OUT_DQ||<:> veto ||<:> 10 ||
||PSL-PMC_TRANS_DC_OUT_DQ||<:>veto ||<:> 30 ||
||PEM-ACC_MCF_TABLE_REFL_Z_OUT_DQ||<:>veto ||<:> 30 ||
||PEM-ACC_PSL_PERI_PSL1_Y_OUT_DQ||<:>veto ||<:> 7 ||
||PEM-MIC_PSL_TABLE_PSL4_Z_OUT_DQ||<:>veto ||<:> 7 ||
||<:>'''Lock state : IMC'''||
 * Q-transform (Under testing. GWpy default parameter is used.)
  * GPS start time ($gpsstart), end time ($gpsend)
  * Output directory ($outdir)
  * Channel ($chlist[@])
  * Lock state segment bar: flag($lock), guardian channel($lchannel), guardian value($lnumber), segment bar title($llabel)
Line 81: Line 108:
6/8 during ER  * Lock segment summary
  * GPS start time ($gpsstart), end time ($gpsend)
   * Same as time series
  * Output directory ($outdir)
  * Channel ($channel)
  * Trigger time ($gpstime) and duration ($max_duration)
   * Taken from Omicron information
Line 83: Line 116:
||<:>'''Channel'''||'''SNR threshold'''||
||LSC-CARM_SERVO_MIXER_DAQ_OUT_DQ||<:> 10 ||
||AOS-TMSX_IR_PD_OUT_DQ||<:> 10 ||
||IMC-MCL_SERVO_OUT_DQ||<:> 30 ||
||IMC-SERVO_SLOW_DAQ_OUT_DQ||<:> 7 ||
||IMC-CAV_TRANS_OUT_DQ||<:> 10 ||
||IMC-CAV_REFL_OUT_DQ||<:> 10 ||
||PSL-PMC_TRANS_DC_OUT_DQ||<:> 30 ||
||PSL-PMC_MIXER_MON_OUT_DQ||<:> 10 ||
||<:>'''Lock state : IMC'''||
=== Suggestion channel list ===
It is generated event by event. {{{GlitchPlot/Script/suggestion.py}}} is used. Please refer p3,4 in [[https://gwdoc.icrr.u-tokyo.ac.jp/cgi-bin/private/DocDB/ShowDocument?docid=10987|slides]].
Line 94: Line 119:
6/12~7/5 === Source code ===
[[https://github.com/gw-detchar/tools/tree/kashiwa/GlitchPlot/Script|GlitchPlot github]].
Line 96: Line 122:
||<:>'''Channel'''||'''SNR threshold'''||
||IMC-CAV_TRANS_OUT_DQ||<:> 100 ||
||PSL-PMC_TRANS_DC_OUT_DQ||<:> 100 ||
||PEM-ACC_MCF_TABLE_REFL_Z_OUT_DQ||<:> 100 ||
||PEM-ACC_PSL_PERI_PSL1_Y_OUT_DQ||<:> 100 ||
||PEM-MIC_PSL_TABLE_PSL4_Z_OUT_DQ||<:> 100 ||
||<:>'''Lock state : IMC'''||
Master branch: On Kamioka server, plotter.sh runs by crontab every 15 min. plotter.py is called. it gives clustered Omicron trigger information as a text file. The text files are sent to Kashiwa server by rsync.
Line 104: Line 124:
6/7~6/12 kashiwa branch: On Kashiwa server, GlitchPlot.sh runs by crontab every 15 min. Using text file sent from Kamioka, condor_jobfile_plotter.sh runs to throw jobs into condor to make plots.
Line 106: Line 126:
||<:>'''Channel'''||'''SNR threshold'''||
||LSC-CARM_SERVO_MIXER_DAQ_OUT_DQ||<:> 100 ||
||AOS-TMSX_IR_PD_OUT_DQ||<:> 100 ||
||IMC-MCL_SERVO_OUT_DQ||<:> 30 ||
||IMC-SERVO_SLOW_DAQ_OUT_DQ||<:> 7 ||
||IMC-CAV_TRANS_OUT_DQ||<:> 10 ||
||IMC-CAV_REFL_OUT_DQ||<:> 10 ||
||PSL-PMC_TRANS_DC_OUT_DQ||<:> 30 ||
||PSL-PMC_MIXER_MON_OUT_DQ||<:> 10 ||
||<:>'''Lock state : X-arm'''||
=== Job structure ===
O3GK configuration (2020/12/16: It is stopped.)
Line 117: Line 129:
== Update log ==
 *7/5 SNR threshold is lowered during 0am-8am. Instead, daytime events are all rejected.
1. On k1sum1: From the Omicron result, triggered event and the parameters are summarized as text files. The files are rsync-ed to Kashiwa server.
Line 120: Line 131:
 *7/5 Missing plot for PEM trigger channel is fixed. By crontab, {{{4-59/15 * * * * /users/DET/tools/GlitchPlot/Script/plotter.sh > /tmp/GlitchPlot.log 2>&1}}}
Line 122: Line 133:
 *7/4 ER events are reprocessed. 2. On chihiro.kozakai@m31-02: Based on text file sent from k1sum1, plots are produced using condor.
Line 124: Line 135:
By crontab,
{{{
4-59/15 * * * * /home/chihiro.kozakai/detchar/KamiokaTool/tools/GlitchPlot/Script/GlitchPlot.sh > /tmp/GlitchPlot.log 2>&1
10 */3 * * * /home/chihiro.kozakai/detchar/KamiokaTool/tools/GlitchPlot/Script/plotter_lockloss.sh > /tmp/GP_lockloss.log 2>&1
}}}
Line 125: Line 141:
#{{{#!wiki
#xxx
#}}}
3. On k1det0: Plots on Kashiwa server is rsync-ed. Then html for the web page is generated and accessible on GWDET server.

By crontab, {{{30 */3 * * * /users/DET/tools/GlitchPlot/Script/GlitchPlot_html.sh > /tmp/GlitchPlot.log}}}

=== Note for developer ===
 * Please modify output directory.
 * Job structure should be modified. Step 2 should be moved to detchar account. If web server is implemented on Kashiwa server, Step 1 and 3 also have to be moved to Kashiwa server.
 * It is better to use low-latency pipeline result for RRT. CBC and burst pipeline configuration is under constructing.
  * For cWB offline result: {{{GlitchPlot/Script/urst_automation}}}
 * Please reconsider channel list. At least, SRCL related channels will be required. ASC maybe also important.
 * Top page maybe better to start from the latest date.
 * manual code introduction (they are under GlitchPlot/Script/)
  * checkOmicron.py: To check if Omicron

GlitchPlot

This project is for glitch and lock loss study. Also applicable to event validation of GW candidate. You can see the result here.

Original developer: Chihiro Kozakai (production of plots), Hirotaka Yuzurihara (web page).

The latest trigger channel list

4/7~4/20 O3GK

Channel

SNR threshold

K1:CAL-CS_PROC_DARM_DISPLACEMENT_DQ

100

Lock state : Observation mode (K1:GRD-IFO_STATE_N == 1000)

Overview

The main purpose is to study glitches and lock loss by visual inspection. Also, collecting information for glitch database is done through voting form for each glitch event. Using the result, we aim to do noise hunting, glitch characterization, and labeling for machine learning application. Here is slides describing how to use: slides

GlitchPlot uses trigger information provided by Omicron. GlitchPlot does clustering of the triggers and summarize the trigger information for each events based on the Omicron output.

Also, GlitchPlot uses lock loss information based on lock state information of guardian.

For each events, plots for triggered channel and relevant channels are provided here. The relevant channels are basically chosen from

  • Trigger channel
  • Unsafe channels
  • Important upstream channels
  • Relevant VIS channels
  • PEM near the trigger channel sensor

Using Kozapy batch codes, following plots are provided:

  • Time series
  • Spectrum before trigger and after trigger
  • Spectrogram
  • Coherencegram with trigger channel
  • Q-transform (Under testing)

The parameter for plots are determined based on trigger information. Also, lock state summary around the trigger time is provided.

There is channel list of 2 types of suggestion. The intent of this list is to pick up suspicious channels which is related to noise source.

  • Suggestion 1: picked up by higher coherence than usual
  • Suggestion 2: picked up by high SNR in Q-transform

The event web page shows the result in the following order. 1. Lock state 2. Trigger channel plots 3. Suggestion 1 channel plots 4. Suggestion 2 channel plots 5. Channels affected by DARM DoF 6. Other channels

Specification detail

Trigger clustering

For the trigger clustering method, please refer p3 of slides.

Plot channel list

For the exact channel list for each trigger, please refer $channelname.dat in GlitchPlot github.

Plot parameter

For the exact method for plot parameter determination, please refer plotter.py and condor_jobfile_plotter.sh in GlitchPlot github. The intent and the variable name in the shell script is as follows.

  • Time series
    • GPS start time ($gpsstart), end time ($gpsend)
      • The center is the trigger time.
      • Time span is roughly larger one of [4sec, trigger duration * 10]. (Adjusted to fit the spectrogram requirement)
    • Output directory ($outdir)
    • Channel ($chlist[@])
    • Lock state segment bar: flag($lock), guardian channel($lchannel), guardian value($lnumber), segment bar title($llabel)
    • Data file type ($data)
      • Full data is used.
    • If time span is too long, down sample is applied.
  • Spectrum
    • List of GPS start time ($gpsstarts30[@]), end time ($gpsend30[@])
      • Trigger time and 2sec before and after the trigger is avoided.
      • About 30 sec is used each before trigger and after trigger spectrum.
      • The length is adusted to the multiple of the FFT length.
    • Output directory ($outdir)
    • Channel ($chlist[@])
    • FFT length ($fft30)
      • 1/band width (Adjusted to the nearest 2^n sec.)
  • Spectrogram, coherencegram
    • GPS start time ($gpsstart), end time ($gpsend)
      • The center is the trigger time.
      • Time span is roughly larger one of [4sec, trigger duration * 10].
      • The length is adjusted to be Stride * n
    • Output directory ($outdir)
    • Channel ($chlist[@])
    • Lock state segment bar: flag($lock), guardian channel($lchannel), guardian value($lnumber), segment bar title($llabel)
    • Stride ($stride)
      • To have enough time resolution to see the glitch.
    • FFT length ($fft)
      • stride = FFT length * n
    • With and without whitening (Spectrogram)
  • Q-transform (Under testing. GWpy default parameter is used.)
    • GPS start time ($gpsstart), end time ($gpsend)
    • Output directory ($outdir)
    • Channel ($chlist[@])
    • Lock state segment bar: flag($lock), guardian channel($lchannel), guardian value($lnumber), segment bar title($llabel)
  • Lock segment summary
    • GPS start time ($gpsstart), end time ($gpsend)
      • Same as time series
    • Output directory ($outdir)
    • Channel ($channel)
    • Trigger time ($gpstime) and duration ($max_duration)
      • Taken from Omicron information

Suggestion channel list

It is generated event by event. GlitchPlot/Script/suggestion.py is used. Please refer p3,4 in slides.

Source code

GlitchPlot github.

Master branch: On Kamioka server, plotter.sh runs by crontab every 15 min. plotter.py is called. it gives clustered Omicron trigger information as a text file. The text files are sent to Kashiwa server by rsync.

kashiwa branch: On Kashiwa server, GlitchPlot.sh runs by crontab every 15 min. Using text file sent from Kamioka, condor_jobfile_plotter.sh runs to throw jobs into condor to make plots.

Job structure

O3GK configuration (2020/12/16: It is stopped.)

1. On k1sum1: From the Omicron result, triggered event and the parameters are summarized as text files. The files are rsync-ed to Kashiwa server.

By crontab, 4-59/15 * * * * /users/DET/tools/GlitchPlot/Script/plotter.sh > /tmp/GlitchPlot.log 2>&1

2. On chihiro.kozakai@m31-02: Based on text file sent from k1sum1, plots are produced using condor.

By crontab,

4-59/15 * * * * /home/chihiro.kozakai/detchar/KamiokaTool/tools/GlitchPlot/Script/GlitchPlot.sh > /tmp/GlitchPlot.log 2>&1
10 */3 * * * /home/chihiro.kozakai/detchar/KamiokaTool/tools/GlitchPlot/Script/plotter_lockloss.sh > /tmp/GP_lockloss.log 2>&1

3. On k1det0: Plots on Kashiwa server is rsync-ed. Then html for the web page is generated and accessible on GWDET server.

By crontab, 30 */3 * * * /users/DET/tools/GlitchPlot/Script/GlitchPlot_html.sh > /tmp/GlitchPlot.log

Note for developer

  • Please modify output directory.
  • Job structure should be modified. Step 2 should be moved to detchar account. If web server is implemented on Kashiwa server, Step 1 and 3 also have to be moved to Kashiwa server.
  • It is better to use low-latency pipeline result for RRT. CBC and burst pipeline configuration is under constructing.
    • For cWB offline result: GlitchPlot/Script/urst_automation

  • Please reconsider channel list. At least, SRCL related channels will be required. ASC maybe also important.
  • Top page maybe better to start from the latest date.
  • manual code introduction (they are under GlitchPlot/Script/)
    • checkOmicron.py: To check if Omicron

KAGRA/Subgroups/DET/GlitchPlot (last edited 2020-12-16 18:00:59 by chihiro.kozakai)