,-----.  ,---.   ,----.   ,--.   ,--.                            ,--.             ,--.        
'  .--./ /  O  \ '  .-./   |   `.'   | ,---. ,--,--,      ,---. ,-'  '-.,--.,--. ,-|  | ,---.  
|  |    |  .-.  ||  | .---.|  |'.'|  || .-. ||      \    | .-. :'-.  .-'|  ||  |' .-. || .-. : 
'  '--'\|  | |  |'  '--'  ||  |   |  |' '-' '|  ||  |    \   --.  |  |  '  ''  '\ `-' |\   --. 
 `-----'`--' `--' `------' `--'   `--' `---' `--''--'     `----'  `--'   `----'  `---'  `----'                                                                                             

Description

The CAGMon etude is a study version of CAGMon that evaluates the dependence between the primary and auxiliary channels.

• KGWG wiki link

• KAGRA wiki link

Project goal

The goal of this project is to find a systematic way of identifying the abnormal glitches in the gravitational-wave data using various methods of correlation analysis. Usually, the community such as LIGO, Virgo, and KAGRA uses a conventional way of finding glitches in auxiliary channels of the detector - Klein-Welle, Omicron, Ordered Veto Lists, etc. However, some different ways can be possible to find and monitor them in a (quasi-) realtime. Also, the method can point out which channel is responsible for the found glitch. In this project, we study its possible to apply three different correlation methods - maximal information coefficient, Pearson's correlation coefficient, and Kendall's tau coefficient - in the gravitational wave data from the KAGRA detector.

Participants

• John.J Oh (NIMS)
• Young-Min Kim (UNIST)
• Pil-Jong Jung (NIMS)

Methods and Frameworks

Maximal Information Coefficient (MIC)

the Maximal Information coefficient(MIC) of a set D of two-variable data with sample size n and the grid less than B(n) is given by

\[ MIC(D)=\underset{xy<B(n)}{\max}{\left\{ \frac{I^{*}(D,x,y)}{\log \min \left\{x,y \right\}} \right \}} \],

where \[\omega(1)<B(n)\le O(n^{1-\epsilon}) \] for some \[ 0<\epsilon<1 \]

Pearson's Correlation Coefficient (PCC)

Pearson Correlation Coefficient(PCC) is a statistic that explains the amount of variance accounted for in the relationship between two (or more) variables by \[ R=} \],

where \[ \overline{X} \] and \[ \overline{Y} \] are the mean of X and Y, respectively

Kendall's tau Coefficient

Kendall’s tau with a random samples n of observations from two variables measures the strength of the relationship between two ordinal level variables by

\[ \tau =\frac{c-d} \],

where c is the number of concordant pairs, and d is the number of discordant pairs

Flow chart

Code development

GitHub

TBA

Code versions

1. CAGMon Etude Alpha
   • for the basic test and evaluation of the LASSO regression method developed by LIGO
   • reproduced original CAGMon methods and idea
2. CAGMon Etude Beta
   • added coefficient trend plots with LASSO beta, coherence, MIC, PCC, and Kendall's tau
3. CAGMon Etude Delta
   • fixed a critical problem that sucked enormous memory when it used the matplotlib module
4. CAGMon Etude Eta
   • fixed minor issues
   • added the range limitation of stride
5. CAGMon Etude Flat
   • fixed minor issues and optimized scripts
   • added the script of HTML summary page
   • added coefficient distribution plots
6. CAGMon Etude Octave (current version)
   • remove some processes that make Time-series and Scatter plots. Even though it required tremendous memory, this information is not useful
   • adjust HTML code
   • fixed minor issues and optimized scripts
   • added the analysis option whether or not the algorithm proceeds in the active segment only
   • improve script efficiency

   • added the process to make scatter and OmegaScan plots in detail boxes of the summary page

7. CAGMon Etude Rhapsody (development version)
   • Require auto-selection of CAGMon parameters
   • Require pre-estimation process to check intrinsic sample rate of each channel
   • Improve script efficiency and completeness

Series of scripts

• Agrement.py
  • the script gathered functions the model required
• Melody.py
  • the script to calcutate each coefficient and to save trend data as csv
• Conchord.py
  • the script to make plots
• Echo.py
  • the script to save the result as HTML web page
• CAGMonEtude{Version}.py
  • the script to run each script

User guide

• How to use CAGMon

Needs of code development

• Fundamental criteria or guideline of CAGMon parameters, such as the stride, the sample rate, and its data-size
• Daily running on KAGRA

Empirical study (No free lunch)

1. Apply to glitch data on KAGRA during O3GK
   • Glitch information

     • KAGRA glitch catalog

   • Purpose
     • To decide on appropriate parameters when we run CAGMon for searching glitches and correlation
     • To make recommended parameters in the short-range analysis
   • Result

     • Glitch Catalog for the empirical study of CAGMon parameters

   • Appropriate parameters of CAGMon for the glitch search
     • Data-size: 8,192 or 16,384
     • Stride: 0.5 or 1.0 seconds (the glitch duration users want to find)

2. Apply to the glitch data of GravitySpy on LIGO

   • Data
     • TBD
3. Apply to the mid-range data
   • Data
     • TBD
4. Apply to the long-range data
   • Data
     • TBD

Exemplary results

1. Earthquake effects during O3GK (with CAGMon Etude Flat)

• Datetime: 19 April 2020 20:39 UTC
• Purpose
  • Test to run CAGMon algorithm with a remarkable event
  • To figure out the cause of lock-loss in KAGRA
• Computing resource
  • KISTI-LDG
  • Requested CPUS: 32cores
  • Requested memory: 128GB
• Results

  • stride 5 seconds Summary page

  • stride 20 seconds Summary page

  • stride 30 seconds Summary page

2. With iKAGRA hardware injection data (with CAGMon Etude Flat)

• Event
  • Phenomenon: the strain channel and seismometer channels in iKAGRA had a high correlation during the hardware injection test
  • Cause: still unknown
  • Hypothesis: the glitches have relatively the same behavior as the vacuum rotary pump

  • More detail analysis: h-veto brief Report for K1 and KGWG Face-to-Face Meeting

• Purpose
  • To verify whether this model senses injected signals and abnormal glitches
  • To test noise resistance and data-size limitation
• Computing resource
  • KISTI-LDG
  • Requested CPUS: 32cores
  • Requested memory: 64GB
• Results

  Stride	Sample sata	Data size	Dada length	Summary page link
  10s	512Hz	about 5,000	about 12m	summary page
  10s	1024Hz	about 10,000	about 12m	summary page
  10s	2048Hz	about 20,000	about 12m	summary page
  10s	3072Hz	about 30,000	about 12m	summary page
  10s	4096Hz	about 40,000	about 12m	summary page
  2s	4096Hz	about 8,000	about 12m	summary page
  5s	4096Hz	about 20,000	about 12m	summary page
  60s	128Hz	about 7,500	whole iKAGRA data	summary page
  150s	64Hz	about 10,000	whole iKAGRA data	summary page
  300s	64Hz	about 20,000	whole iKAGRA data	summary page
  600s	16Hz	about 10,000	whole iKAGRA data	summary page

3. Skim through some obs-segments of O3GK (with CAGMon Etude Octave)

• Purpose
  • Test for calculation time and required resources with all observation segments during O3GK
  • To figure out trigger events or abnormal behaviors
• Computing resource
  • KISTI-LDG
  • Requested CPUS: 32cores
  • Requested memory: 128GB
• Results

  Date	GPS time	Data length	Stride	Sample rate	Data size	Summary page link	Remarks
  April 7	1270287158 - 1270328032	11h	500s	16Hz	about 8,000	summary page	processing time: 4h12m / memory usage: 42GB
  			240s	32Hz	about 8,000	summary page	processing time: 5h21m / memory usage: 23GB
  			120s	64Hz	about 8,000	summary page	processing time: 17h10m / memory usage: 41.9GB
  			60s	128Hz	about 8,000	summary page	processing time: 23h03m / memory usage: 28.8GB
  			30s	256Hz	about 8,000	summary page	processing time: 1d23h / memory usage: 24GB
  			15s	512Hz	about 8,000	summary page	processing time: > 3 days => killed
  			8s	1024Hz	about 8,000	summary page	processing time: > 3 days => killed
  			4s	2048Hz	about 8,000	summary page	processing time: > 3 days => killed
  			2s	4096Hz	about 8,000	summary page	processing time: > 3 days => killed
  			1s	8192Hz	about 8,000	summary page	processing time: > 3 days => killed
  April 14	1270909686 - 1270937768	7h	500s	16Hz	about 8,000	summary page	processing time: 50m / memory usage: 1.9GB
  			240s	32Hz	about 8,000	summary page	processing time: 2h8m / memory usage: 2.4GB
  			120s	64Hz	about 8,000	summary page	processing time: 4h40m / memory usage: 3.6GB
  			60s	128Hz	about 8,000	summary page	processing time: 8h30m / memory usage: 2.0GB
  			30s	256Hz	about 8,000	summary page	processing time: 15h30m / memory usage: 2.2GB
  			500s	32Hz	about 16,000	summary page	processing time: 50m / memory usage: 3.2GB
  			240s	64Hz	about 16,000	summary page	processing time: 5h25m / memory usage: 3.3GB
  			120s	128Hz	about 16,000	summary page	processing time: 10h10m / memory usage: 3.4GB
  			60s	256Hz	about 16,000	summary page	processing time: 18h50m / memory usage: 3.4GB
  			30s	512Hz	about 16,000	summary page	processing time: 17h / memory usage: 3.7GB
  			500s	64Hz	about 36,000	summary page	processing time: 6h30m / memory usage: 6.7GB
  			240s	128Hz	about 36,000	summary page	processing time: 10h57m / memory usage: 6.7GB
  			120s	256Hz	about 36,000	summary page	processing time: 17h3m / memory usage: 7.0GB
  			60s	512Hz	about 36,000	summary page	processing time: 1d6h / memory usage: 7.1GB
  			30s	1024Hz	about 36,000	summary page	processing time: 2d22h30m / memory usage: 7.0GB
  			500s	128Hz	about 64,000	summary page	processing time: 12h10m / memory usage: 14.6GB
  			240s	256Hz	about 64,000	summary page	processing time: 1d3h40m / memory usage: 14.6GB
  			120s	512Hz	about 64,000	summary page	processing time: 2d6h / memory usage: 14.7GB
  			60s	1024Hz	about 64,000	summary page	processing time: > 4 days => kille
  			30s	2048Hz	about 64,000	summary page	processing time: > 4 days => kille

4. Glitch analysis and channel correlation study during O3GK (with CAGMon Etude Octave)

• Purpose
  • To identify channel correlation and the cause of glitches
• Computing resource
  • KISTI-LDG
  • Requested CPUS: 32cores
  • Requested memory: 64GB
• CAGMon parameters
  • MIC Alpha: 0.6
  • MIC c: 15
  • Data-size: 8192
  • Stride: 512 seconds for the long-term search and 0.5 seconds for the short-term search
  • Aux-channels: channels used daily running h-veto on KAGRA
• Results with the daily data (without Scatter and Spectrogram plots)

  Date	GPS time	Data length	Stride	Sample rate	Data size	Summary page	Running time and memory usage	Remarks
  April 7	1270287158 - 1270328032	11.35h	512s	16Hz	8192	link	4h / 4GB
  April 8	1270339218 - 1270425618	24h	512s	16Hz	8192			N/A in KISTI
  April 9	1270425618 - 1270510167	23.45h	512s	16Hz	8192			N/A in KISTI
  April 10	1270513160 - 1270596544	23.16h	512s	16Hz	8192	link	8h30m / 5.6GB
  April 11	1270598418 - 1270683904	23.75h	512s	16Hz	8192			N/A in KISTI
  April 12	1270684818 - 1270762046	21.45h	512s	16Hz	8192	link	6h4m / 7.6GB
  April 14	1270909686 - 1270937768	7.8h	512s	16Hz	8192	link	2h40m / 3.2GB
  April 15	1270945288 - 1271017582	20.08h	512s	16Hz	8192			N/A in KISTI
  April 16	1271030433 - 1271112809	22.88h	512s	16Hz	8192	link	8h10m / 8.0GB
  April 17	1271119833 - 1271186507	18.52h	512s	16Hz	8192	link	7h / 6.4GB
  April 18	1271227441 - 1271288128	16.86h	512s	16Hz	8192			N/A in KISTI
  April 19	1271289618 - 1271364033	20.67h	512s	16Hz	8192	link	8h15m / 8.8GB
  April 20	1271377409 - 1271460608	23.11h	512s	16Hz	8192			N/A in KISTI

• Results with the borders of obs-segments (with Scatter and OmegaScan plots)

  Date	GPS time	Data length	Stride	Sample rate	Data size	Summary page	Running time and memory usage	Remarks
  April 7	1270287723	±120s	1s	8192Hz	8192	link	4h23m / 22.4GB
  	1270289095	±120s	1s	8192Hz	8192	link	4h20m / 21.3GB
  	1270291338	±120s	1s	8192Hz	8192	link	4h29m / 23.5GB
  	1270299544	±120s	1s	8192Hz	8192	link	4h17m / 21.8GB
  	1270302108	±120s	1s	8192Hz	8192	link	4h20m / 22.1GB
  	1270312047	±120s	1s	8192Hz	8192	link	4h24m / 21.7GB
  	1270320096	±120s	1s	8192Hz	8192	link	4h20m / 19.9GB
  	1270320127	±120s	1s	8192Hz	8192	link	4h20m / 19.4GB
  	1270325304	±120s	1s	8192Hz	8192	link	4h20m / 19.6GB
  	1270328032	±120s	1s	8192Hz	8192	link	4h20m / 19.3GB
  April 10	1270513361	±120s	1s	8192Hz	8192	link	4h20m / 21.7GB
  	1270519811	±120s	1s	8192Hz	8192	link	4h15m / 20.2GB
  	1270544263	±120s	1s	8192Hz	8192	link	4h20m / 19.7GB
  	1270546596	±120s	1s	8192Hz	8192	link	4h20m / 19.2GB
  	1270547109	±120s	1s	8192Hz	8192	link	4h19m / 22.6GB
  	1270549273	±120s	1s	8192Hz	8192	link	4h18m / 21.6GB
  	1270550718	±120s	1s	8192Hz	8192	link	4h17m / 21.6GB
  	1270551599	±120s	1s	8192Hz	8192	link	4h19m / 22.8GB
  	1270552685	±120s	1s	8192Hz	8192	link	4h20m / 19.2GB
  	1270555167	±120s	1s	8192Hz	8192	link	4h16m / 24.9GB
  	1270572585	±120s	1s	8192Hz	8192	link	4h13m / 21.5GB
  	1270580988	±120s	1s	8192Hz	8192	link	4h23m / 21.5GB
  	1270585866	±120s	1s	8192Hz	8192	link	4h16m / 27.1GB
  	1270596544	±120s	1s	8192Hz	8192	link	4h20m / 19.4GB
  April 12	1270696199	±120s	1s	8192Hz	8192	link
  	1270713927	±120s	1s	8192Hz	8192	link
  	1270715460	±120s	1s	8192Hz	8192	link
  	1270720692	±120s	1s	8192Hz	8192	link
  	1270721256	±120s	1s	8192Hz	8192	link
  	1270726572	±120s	1s	8192Hz	8192	link
  	1270746411	±120s	1s	8192Hz	8192	link
  	1270748613	±120s	1s	8192Hz	8192	link
  	1270759925	±120s	1s	8192Hz	8192	link
  	1270762046	±120s	1s	8192Hz	8192	link
  April 14		±120s	1s	8192Hz	8192	link

Cross-validation

Beyond

References

Presentation materials

JGW-G2112481-v1

Papers

Science.1518; Detecting Novel Associations in Large Data Sets