Diff for "PJJung/CAGMonEtude"

Differences between revisions 1 and 71 (spanning 70 versions)

 ,-----.  ,---.   ,----.   ,--.   ,--.                            ,--.             ,--.        
'  .--./ /  O  \ '  .-./   |   `.'   | ,---. ,--,--,      ,---. ,-'  '-.,--.,--. ,-|  | ,---.  
|  |    |  .-.  ||  | .---.|  |'.'|  || .-. ||      \    | .-. :'-.  .-'|  ||  |' .-. || .-. : 
'  '--'\|  | |  |'  '--'  ||  |   |  |' '-' '|  ||  |    \   --.  |  |  '  ''  '\ `-' |\   --. 
 `-----'`--' `--' `------' `--'   `--' `---' `--''--'     `----'  `--'   `----'  `---'  `----'

Description

The CAGMon etude is a study version of CAGMon that evaluates the dependence between the primary and auxiliary channels.

Project goal

The goal of this project is to find a systematic way of identifying the abnormal glitches in the gravitational-wave data using various methods of correlation analysis. Usually, the community such as LIGO, Virgo, and KAGRA uses a conventional way of finding glitches in auxiliary channels of the detector - Klein-Welle, Omicron, Ordered Veto Lists, etc. However, some different ways can be possible to find and monitor them in a (quasi-) realtime. Also, the method can point out which channel is responsible for the found glitch. In this project, we study its possible to apply three different correlation methods - maximal information coefficient, Pearson's correlation coefficient, and Kendall's tau coefficient - in the gravitational wave data from the KAGRA detector.

Participants

John.J Oh (NIMS)
Young-Min Kim (UNIST)
Pil-Jong Jung (NIMS)

Methods and Frameworks

Maximal Information Coefficient (MIC)

the Maximal Information coefficient(MIC) of a set D of two-variable data with sample size n and the grid less than B(n) is given by

\[ MIC(D)=\underset{xy<B(n)}{\max}{\left\{ \frac{I^{*}(D,x,y)}{\log \min \left\{x,y \right\}} \right \}} \],

where \[\omega(1)<B(n)\le O(n^{1-\epsilon}) \] for some \[ 0<\epsilon<1 \]

Pearson's Correlation Coefficient (PCC)

Pearson Correlation Coefficient(PCC) is a statistic that explains the amount of variance accounted for in the relationship between two (or more) variables by \[ R=} \],

where \[ \overline{X} \] and \[ \overline{Y} \] are the mean of X and Y, respectively

Kendall's tau Coefficient

Kendall’s tau with a random samples n of observations from two variables measures the strength of the relationship between two ordinal level variables by

\[ \tau =\frac{c-d} \],

where c is the number of concordant pairs, and d is the number of discordant pairs

Flow chart

Code development

GitHub

TBA

Code versions

CAGMon Etude Alpha
- for the basic test and evaluation of the LASSO regression method developed by LIGO
- reproduced original CAGMon methods and idea
CAGMon Etude Beta
- added coefficient trend plots with LASSO beta, coherence, MIC, PCC, and Kendall's tau
CAGMon Etude Delta
- fixed a critical problem that sucked enormous memory when it used the matplotlib module
CAGMon Etude Eta
- fixed minor issues
- added the range limitation of stride
CAGMon Etude Flat
- fixed minor issues and optimized scripts
- added the script of HTML summary page
- added coefficient distribution plots
CAGMon Etude Octave (current version)
- remove some processes that make Time-series and Scatter plots. Even though it required tremendous memory, this information is not useful
- adjust HTML code
- fixed minor issues and optimized scripts
- added the analysis option whether or not the algorithm proceeds in the active segment only
- improve script efficiency
- added the process to make scatter and OmegaScan plots in detail boxes of the summary page
CAGMon Etude Rhapsody (development version)
- Require auto-selection of CAGMon parameters
- Require pre-estimation process to check intrinsic sample rate of each channel
- Improve script efficiency and completeness

Series of scripts

Agrement.py
- the script gathered functions the model required
Melody.py
- the script to calcutate each coefficient and to save trend data as csv
Conchord.py
- the script to make plots
Echo.py
- the script to save the result as HTML web page
CAGMonEtude{Version}.py
- the script to run each script

User guide

How to use CAGMon

Needs of code development

Fundamental criteria or guideline of CAGMon parameters, such as the stride, the sample rate, and its data-size
Daily running on KAGRA

Empirical study (No free lunch)

Apply to glitch data on KAGRA during O3GK
- Glitch information
  - KAGRA glitch catalog
- Purpose
  - To decide on appropriate parameters when we run CAGMon for searching glitches and correlation
  - To make recommended parameters in the short-range analysis
- Result
  - Glitch Catalog for the empirical study of CAGMon parameters
- Conclusion
Apply to the glitch data of GravitySpy on LIGO
- Data
  - TBD
Apply to the mid-range data
- Data
  - TBD
Apply to the long-range data
- Data
  - TBD

Exemplary results

1. Earthquake effects during O3GK (with CAGMon Etude Flat)

Datetime: 19 April 2020 20:39 UTC
Purpose
- Test to run CAGMon algorithm with a remarkable event
- To figure out the cause of lock-loss in KAGRA
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 128GB
Results
- stride 5 seconds Summary page
- stride 20 seconds Summary page
- stride 30 seconds Summary page

2. With iKAGRA hardware injection data (with CAGMon Etude Flat)

Event
- Phenomenon: the strain channel and seismometer channels in iKAGRA had a high correlation during the hardware injection test
- Cause: still unknown
- Hypothesis: the glitches have relatively the same behavior as the vacuum rotary pump
- More detail analysis: h-veto brief Report for K1 and KGWG Face-to-Face Meeting
Purpose
- To verify whether this model senses injected signals and abnormal glitches
- To test noise resistance and data-size limitation
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 64GB

Results

Stride	Sample sata	Data size	Dada length	Summary page link
10s	512Hz	about 5,000	about 12m	summary page
10s	1024Hz	about 10,000	about 12m	summary page
10s	2048Hz	about 20,000	about 12m	summary page
10s	3072Hz	about 30,000	about 12m	summary page
10s	4096Hz	about 40,000	about 12m	summary page
2s	4096Hz	about 8,000	about 12m	summary page
5s	4096Hz	about 20,000	about 12m	summary page
60s	128Hz	about 7,500	whole iKAGRA data	summary page
150s	64Hz	about 10,000	whole iKAGRA data	summary page
300s	64Hz	about 20,000	whole iKAGRA data	summary page
600s	16Hz	about 10,000	whole iKAGRA data	summary page

3. Skim through some obs-segments of O3GK (with CAGMon Etude Octave)

Purpose
- Test for calculation time and required resources with all observation segments during O3GK
- To figure out trigger events or abnormal behaviors
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 128GB