Diff for "PJJung/CAGMonEtude"

Differences between revisions 22 and 102 (spanning 80 versions)

 ,-----.  ,---.   ,----.   ,--.   ,--.                            ,--.             ,--.        
'  .--./ /  O  \ '  .-./   |   `.'   | ,---. ,--,--,      ,---. ,-'  '-.,--.,--. ,-|  | ,---.  
|  |    |  .-.  ||  | .---.|  |'.'|  || .-. ||      \    | .-. :'-.  .-'|  ||  |' .-. || .-. : 
'  '--'\|  | |  |'  '--'  ||  |   |  |' '-' '|  ||  |    \   --.  |  |  '  ''  '\ `-' |\   --. 
 `-----'`--' `--' `------' `--'   `--' `---' `--''--'     `----'  `--'   `----'  `---'  `----'

Description

The CAGMon etude is a study version of CAGMon that evaluates the dependence between the primary and auxiliary channels.

Project goal

The goal of this project is to find a systematic way of identifying the abnormal glitches in the gravitational-wave data using various methods of correlation analysis. Usually, the community such as LIGO, Virgo, and KAGRA uses a conventional way of finding glitches in auxiliary channels of the detector - Klein-Welle, Omicron, Ordered Veto Lists, etc. However, some different ways can be possible to find and monitor them in a (quasi-) realtime. Also, the method can point out which channel is responsible for the found glitch. In this project, we study its possible to apply three different correlation methods - maximal information coefficient, Pearson's correlation coefficient, and Kendall's tau coefficient - in the gravitational wave data from the KAGRA detector.

Participants

John.J Oh (NIMS)
Young-Min Kim (UNIST)
Pil-Jong Jung (NIMS)

Methods and Frameworks

Maximal Information Coefficient (MIC)

the Maximal Information coefficient(MIC) of a set D of two-variable data with sample size n and the grid less than B(n) is given by

\[ MIC(D)=\underset{xy<B(n)}{\max}{\left\{ \frac{I^{*}(D,x,y)}{\log \min \left\{x,y \right\}} \right \}} \],

where \[\omega(1)<B(n)\le O(n^{1-\epsilon}) \] for some \[ 0<\epsilon<1 \]

Pearson's Correlation Coefficient (PCC)

Pearson Correlation Coefficient(PCC) is a statistic that explains the amount of variance accounted for in the relationship between two (or more) variables by \[ R=} \],

where \[ \overline{X} \] and \[ \overline{Y} \] are the mean of X and Y, respectively

Kendall's tau Coefficient

Kendall’s tau with a random samples n of observations from two variables measures the strength of the relationship between two ordinal level variables by

\[ \tau =\frac{c-d} \],

where c is the number of concordant pairs, and d is the number of discordant pairs

Flow chart

Code development

GitHub

TBA

Code versions

CAGMon Etude Alpha
- for the basic test and evaluation of the LASSO regression method developed by LIGO
- reproduced original CAGMon methods and idea
CAGMon Etude Beta
- added coefficient trend plots with LASSO beta, coherence, MIC, PCC, and Kendall's tau
CAGMon Etude Delta
- fixed a critical problem that sucked enormous memory when it used the matplotlib module
CAGMon Etude Eta
- fixed minor issues
- added the range limitation of stride
CAGMon Etude Flat
- fixed minor issues and optimized scripts
- added the script of HTML summary page
- added coefficient distribution plots
CAGMon Etude Octave (current version)
- remove some processes that make Time-series and Scatter plots. Even though it required tremendous memory, this information is not useful
- adjust HTML code
- fixed minor issues and optimized scripts
- added the analysis option whether or not the algorithm proceeds in the active segment only
- improve script efficiency
- added the process to make scatter and OmegaScan plots in detail boxes of the summary page
CAGMon Etude Rhapsody (development version)
- utilize MICe with the auto-selection method of Alpha
- improve script efficiency and completeness
- publish the script on Github

Series of scripts

Agrement.py
- the script gathered functions the model required
Melody.py
- the script to calcutate each coefficient and to save trend data as csv
Conchord.py
- the script to make plots
Echo.py
- the script to save the result as HTML web page
CAGMonEtude{Version}.py
- the script to run each script

User guide

How to use CAGMon

Needs of code development

Fundamental criteria or guideline of CAGMon parameters, such as the stride, the sample rate, and its data-size
Daily running on KAGRA

Empirical study (No free lunch)

Apply to glitch data on KAGRA during O3GK
- Glitch information
  - KAGRA glitch catalog
- Purpose
  - To decide on appropriate parameters when we run CAGMon for searching glitches and correlation
  - To make recommended parameters in the short-range analysis
- Result
  - Glitch Catalog for the empirical study of CAGMon parameters
- Appropriate parameters of CAGMon for the glitch search
  - Data-size: 8,192 or 16,384
  - Stride: 0.5 or 1.0 seconds (the glitch duration users want to find)
Apply to the glitch data of GravitySpy on LIGO
- Data
  - TBD
Apply to the mid-range data
- Data
  - TBD
Apply to the long-range data
- Data
  - TBD

Exemplary results

1. Earthquake effects during O3GK (with CAGMon Etude Flat)

Datetime: 19 April 2020 20:39 UTC
Purpose
- Test to run CAGMon algorithm with a remarkable event
- To figure out the cause of lock-loss in KAGRA
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 128GB
Results
- stride 5 seconds Summary page
- stride 20 seconds Summary page
- stride 30 seconds Summary page

2. With iKAGRA hardware injection data (with CAGMon Etude Flat)

Event
- Phenomenon: the strain channel and seismometer channels in iKAGRA had a high correlation during the hardware injection test
- Cause: still unknown
- Hypothesis: the glitches have relatively the same behavior as the vacuum rotary pump
- More detail analysis: h-veto brief Report for K1 and KGWG Face-to-Face Meeting
Purpose
- To verify whether this model senses injected signals and abnormal glitches
- To test noise resistance and data-size limitation
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 64GB

Results

Stride	Sample sata	Data size	Dada length	Summary page link
10s	512Hz	about 5,000	about 12m	summary page
10s	1024Hz	about 10,000	about 12m	summary page
10s	2048Hz	about 20,000	about 12m	summary page
10s	3072Hz	about 30,000	about 12m	summary page
10s	4096Hz	about 40,000	about 12m	summary page
2s	4096Hz	about 8,000	about 12m	summary page
5s	4096Hz	about 20,000	about 12m	summary page
60s	128Hz	about 7,500	whole iKAGRA data	summary page
150s	64Hz	about 10,000	whole iKAGRA data	summary page
300s	64Hz	about 20,000	whole iKAGRA data	summary page
600s	16Hz	about 10,000	whole iKAGRA data	summary page

3. Skim through some obs-segments of O3GK (with CAGMon Etude Octave)

Purpose
- Test for calculation time and required resources with all observation segments during O3GK
- To figure out trigger events or abnormal behaviors
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 128GB

Results

Date	GPS time	Data length	Stride	Sample rate	Data size	Summary page link	Remarks
April 7	1270287158 - 1270328032	11h	500s	16Hz	about 8,000	summary page	processing time: 4h12m / memory usage: 42GB
			240s	32Hz	about 8,000	summary page	processing time: 5h21m / memory usage: 23GB
			120s	64Hz	about 8,000	summary page	processing time: 17h10m / memory usage: 41.9GB
			60s	128Hz	about 8,000	summary page	processing time: 23h03m / memory usage: 28.8GB
			30s	256Hz	about 8,000	summary page	processing time: 1d23h / memory usage: 24GB
			15s	512Hz	about 8,000	summary page	processing time: > 3 days => killed
			8s	1024Hz	about 8,000	summary page	processing time: > 3 days => killed
			4s	2048Hz	about 8,000	summary page	processing time: > 3 days => killed
			2s	4096Hz	about 8,000	summary page	processing time: > 3 days => killed
			1s	8192Hz	about 8,000	summary page	processing time: > 3 days => killed
April 14	1270909686 - 1270937768	7h	500s	16Hz	about 8,000	summary page	processing time: 50m / memory usage: 1.9GB
			240s	32Hz	about 8,000	summary page	processing time: 2h8m / memory usage: 2.4GB
			120s	64Hz	about 8,000	summary page	processing time: 4h40m / memory usage: 3.6GB
			60s	128Hz	about 8,000	summary page	processing time: 8h30m / memory usage: 2.0GB
			30s	256Hz	about 8,000	summary page	processing time: 15h30m / memory usage: 2.2GB
			500s	32Hz	about 16,000	summary page	processing time: 50m / memory usage: 3.2GB
			240s	64Hz	about 16,000	summary page	processing time: 5h25m / memory usage: 3.3GB
			120s	128Hz	about 16,000	summary page	processing time: 10h10m / memory usage: 3.4GB
			60s	256Hz	about 16,000	summary page	processing time: 18h50m / memory usage: 3.4GB
			30s	512Hz	about 16,000	summary page	processing time: 17h / memory usage: 3.7GB
			500s	64Hz	about 36,000	summary page	processing time: 6h30m / memory usage: 6.7GB
			240s	128Hz	about 36,000	summary page	processing time: 10h57m / memory usage: 6.7GB
			120s	256Hz	about 36,000	summary page	processing time: 17h3m / memory usage: 7.0GB
			60s	512Hz	about 36,000	summary page	processing time: 1d6h / memory usage: 7.1GB
			30s	1024Hz	about 36,000	summary page	processing time: 2d22h30m / memory usage: 7.0GB
			500s	128Hz	about 64,000	summary page	processing time: 12h10m / memory usage: 14.6GB
			240s	256Hz	about 64,000	summary page	processing time: 1d3h40m / memory usage: 14.6GB
			120s	512Hz	about 64,000	summary page	processing time: 2d6h / memory usage: 14.7GB
			60s	1024Hz	about 64,000	summary page	processing time: > 4 days => kille
			30s	2048Hz	about 64,000	summary page	processing time: > 4 days => kille

4. Glitch analysis and channel correlation study during O3GK (with CAGMon Etude Octave)

Purpose
- To identify channel correlation and the cause of glitches
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 64GB
CAGMon parameters
- MIC Alpha: 0.6
- MIC c: 15
- Data-size: 8192
- Stride: 512 seconds for the long-term search and 1 seconds for the short-term search
- Aux-channels: channels used daily running h-veto on KAGRA

Results with the daily data (without Scatter and Spectrogram plots)

Date	GPS time	Data length	Stride	Sample rate	Data size	Summary page	Resource	Remarks
April 7	1270287158 - 1270328032	11.35h	512s	16Hz	8192	link	4h / 4GB
April 8	1270339218 - 1270425618	24h	512s	16Hz	8192			N/A in KISTI
April 9	1270425618 - 1270510167	23.45h	512s	16Hz	8192	link	6h57m / 12.2GB
April 10	1270513160 - 1270596544	23.16h	512s	16Hz	8192	link	8h30m / 5.6GB
April 11	1270598418 - 1270683904	23.75h	512s	16Hz	8192	link	6h59m / 13.8GB
April 12	1270684818 - 1270762046	21.45h	512s	16Hz	8192	link	6h4m / 7.6GB
April 14	1270909686 - 1270937768	7.8h	512s	16Hz	8192	link	2h40m / 3.2GB
April 15	1270945288 - 1271017582	20.08h	512s	16Hz	8192			N/A in KISTI
April 16	1271030433 - 1271112809	22.88h	512s	16Hz	8192	link	8h10m / 8.0GB
April 17	1271119833 - 1271186507	18.52h	512s	16Hz	8192	link	7h / 6.4GB
April 18	1271227441 - 1271288128	16.86h	512s	16Hz	8192			N/A in KISTI
April 19	1271289618 - 1271364033	20.67h	512s	16Hz	8192	link	8h15m / 8.8GB
April 20	1271377409 - 1271460608	23.11h	512s	16Hz	8192			N/A in KISTI

Results with the borders of obs-segments (with Scatter and OmegaScan plots)

Date	GPS time	Data length	Stride	Sample rate	Data size	Summary page	Resource	Remarks
April 7	1270287723	±120s	1s	8192Hz	8192	link	4h23m / 22.4GB
	1270289095	±120s	1s	8192Hz	8192	link	4h20m / 21.3GB
	1270291338	±120s	1s	8192Hz	8192	link	4h29m / 23.5GB
	1270299544	±120s	1s	8192Hz	8192	link	4h17m / 21.8GB
	1270302108	±120s	1s	8192Hz	8192	link	4h20m / 22.1GB
	1270312047	±120s	1s	8192Hz	8192	link	4h24m / 21.7GB
	1270320096	±120s	1s	8192Hz	8192	link	4h20m / 19.9GB
	1270320127	±120s	1s	8192Hz	8192	link	4h20m / 19.4GB
	1270325304	±120s	1s	8192Hz	8192	link	4h20m / 19.6GB
	1270328032	±120s	1s	8192Hz	8192	link	4h20m / 19.3GB
April 9	1270427598	±120s	1s	8192Hz	8192	link
	1270433380	±120s	1s	8192Hz	8192	link
	1270452484	±120s	1s	8192Hz	8192	link
	1270453899	±120s	1s	8192Hz	8192	link
	1270456187	±120s	1s	8192Hz	8192	link
	1270458641	±120s	1s	8192Hz	8192	link
	1270459231	±120s	1s	8192Hz	8192	link
	1270459613	±120s	1s	8192Hz	8192	link
	1270460499	±120s	1s	8192Hz	8192	link
	1270460578	±120s	1s	8192Hz	8192	link
	1270460855	±120s	1s	8192Hz	8192	link
	1270467750	±120s	1s	8192Hz	8192	link
	1270477524	±120s	1s	8192Hz	8192	link
	1270489046	±120s	1s	8192Hz	8192	link
	1270495548	±120s	1s	8192Hz	8192	link
	1270496325	±120s	1s	8192Hz	8192	link
	1270497144	±120s	1s	8192Hz	8192	link
	1270500085	±120s	1s	8192Hz	8192	link
	1270502766	±120s	1s	8192Hz	8192	link
	1270508915	±120s	1s	8192Hz	8192	link
	1270510167	±120s	1s	8192Hz	8192	link
April 10	1270513361	±120s	1s	8192Hz	8192	link	4h20m / 21.7GB
	1270519811	±120s	1s	8192Hz	8192	link	4h15m / 20.2GB
	1270544263	±120s	1s	8192Hz	8192	link	4h20m / 19.7GB
	1270546596	±120s	1s	8192Hz	8192	link	4h20m / 19.2GB
	1270547109	±120s	1s	8192Hz	8192	link	4h19m / 22.6GB
	1270549273	±120s	1s	8192Hz	8192	link	4h18m / 21.6GB
	1270550718	±120s	1s	8192Hz	8192	link	4h17m / 21.6GB
	1270551599	±120s	1s	8192Hz	8192	link	4h19m / 22.8GB
	1270552685	±120s	1s	8192Hz	8192	link	4h20m / 19.2GB
	1270555167	±120s	1s	8192Hz	8192	link	4h16m / 24.9GB
	1270572585	±120s	1s	8192Hz	8192	link	4h13m / 21.5GB
	1270580988	±120s	1s	8192Hz	8192	link	4h23m / 21.5GB
	1270585866	±120s	1s	8192Hz	8192	link	4h16m / 27.1GB
	1270596544	±120s	1s	8192Hz	8192	link	4h20m / 19.4GB
April 11	1270609879	±120s	1s	8192Hz	8192	link
	1270610651	±120s	1s	8192Hz	8192	link
	1270631460	±120s	1s	8192Hz	8192	link
	1270636472	±120s	1s	8192Hz	8192	link
	1270637495	±120s	1s	8192Hz	8192	link
	1270641494	±120s	1s	8192Hz	8192	link
	1270655146	±120s	1s	8192Hz	8192	link
	1270669014	±120s	1s	8192Hz	8192	link
	1270669700	±120s	1s	8192Hz	8192	link
	1270683904	±120s	1s	8192Hz	8192	link
April 12	1270696199	±120s	1s	8192Hz	8192	link
	1270713927	±120s	1s	8192Hz	8192	link
	1270715460	±120s	1s	8192Hz	8192	link
	1270720692	±120s	1s	8192Hz	8192	link
	1270721256	±120s	1s	8192Hz	8192	link
	1270726572	±120s	1s	8192Hz	8192	link
	1270746411	±120s	1s	8192Hz	8192	link
	1270748613	±120s	1s	8192Hz	8192	link
	1270759925	±120s	1s	8192Hz	8192	link
	1270762046	±120s	1s	8192Hz	8192	link
April 14	1270909778	±120s	1s	8192Hz	8192	link
	1270909999	±120s	1s	8192Hz	8192	link
	1270911048	±120s	1s	8192Hz	8192	link
	1270911986	±120s	1s	8192Hz	8192	link
	1270912899	±120s	1s	8192Hz	8192	link
	1270913182	±120s	1s	8192Hz	8192	link
	1270924011	±120s	1s	8192Hz	8192	link
	1270924958	±120s	1s	8192Hz	8192	link
	1270925716	±120s	1s	8192Hz	8192	link
	1270925944	±120s	1s	8192Hz	8192	link
	1270927783	±120s	1s	8192Hz	8192	link
	1270928712	±120s	1s	8192Hz	8192	link
	1270930893	±120s	1s	8192Hz	8192	link
	1270932636	±120s	1s	8192Hz	8192	link
	1270933841	±120s	1s	8192Hz	8192	link
	1270935617	±120s	1s	8192Hz	8192	link
	1270936532	±120s	1s	8192Hz	8192	link
	1270937055	±120s	1s	8192Hz	8192	link
	1270937768	±120s	1s	8192Hz	8192	link
April 16	1271032316	±120s	1s	8192Hz	8192	link
	1271038814	±120s	1s	8192Hz	8192	link
	1271040295	±120s	1s	8192Hz	8192	link
	1271040412	±120s	1s	8192Hz	8192	link
	1271040672	±120s	1s	8192Hz	8192	link
	1271040766	±120s	1s	8192Hz	8192	link
	1271041322	±120s	1s	8192Hz	8192	link
	1271043681	±120s	1s	8192Hz	8192	link
	1271046678	±120s	1s	8192Hz	8192	link
	1271052148	±120s	1s	8192Hz	8192	link
	1271054241	±120s	1s	8192Hz	8192	link
	1271055150	±120s	1s	8192Hz	8192	link
	1271059886	±120s	1s	8192Hz	8192	link
	1271073604	±120s	1s	8192Hz	8192	link
	1271091615	±120s	1s	8192Hz	8192	link
	1271103700	±120s	1s	8192Hz	8192	link
	1271107810	±120s	1s	8192Hz	8192	link
	1271112809	±120s	1s	8192Hz	8192	link
April 17	1271125030	±120s	1s	8192Hz	8192	link
	1271127419	±120s	1s	8192Hz	8192	link
	1271142158	±120s	1s	8192Hz	8192	link
	1271145974	±120s	1s	8192Hz	8192	link
	1271160724	±120s	1s	8192Hz	8192	link
	1271178914	±120s	1s	8192Hz	8192	link
	1271186507	±120s	1s	8192Hz	8192	link
April 19	1271300385	±120s	1s	8192Hz	8192	link
	1271325984	±120s	1s	8192Hz	8192	link
	1271328149	±120s	1s	8192Hz	8192	link
	1271329025	±120s	1s	8192Hz	8192	link
	1271330275	±120s	1s	8192Hz	8192	link
	1271340023	±120s	1s	8192Hz	8192	link
	1271354879	±120s	1s	8192Hz	8192	link
	1271361736	±120s	1s	8192Hz	8192	link
	1271364033	±120s	1s	8192Hz	8192	link

5. Glitch analysis and channel correlation study during O3GK (with CAGMon Etude Rhapsody)

Purpose
- Test for the new version with MICe and its reliability
Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 32GB

CAGMon parameters

Data-size: 8192 (default value is 16384)
Stride: 512 seconds for the long-term search and 1 second for the short-term search

MICe parameters from the empirical study

Data size n	Alpha	c
n >= 1	0.85	5
n >= 25	0.80	5
n >= 50	0.75	5
n >= 250	0.70	5
n >= 500	0.65	4
n >= 1,000	0.60	3
n >= 2,500	0.55	1
n >= 10,000	0.45	1
n >= 40,000	0.40	1

Aux-channels: channels used daily running h-veto on KAGRA