Differences between revisions 5 and 51 (spanning 46 versions)
Revision 5 as of 2021-01-20 14:55:06
Size: 2149
Editor: PJJung
Comment:
Revision 51 as of 2021-01-29 16:05:56
Size: 25693
Editor: PJJung
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= CAGMon etude =
== Descripstion ==
{{{
 ,-----. ,---. ,----. ,--. ,--. ,--. ,--.
' .--./ / O \ ' .-./ | `.' | ,---. ,--,--, ,---. ,-' '-.,--.,--. ,-| | ,---.
| | | .-. || | .---.| |'.'| || .-. || \ | .-. :'-. .-'| || |' .-. || .-. :
' '--'\| | | |' '--' || | | |' '-' '| || | \ --. | | ' '' '\ `-' |\ --.
 `-----'`--' `--' `------' `--' `--' `---' `--''--' `----' `--' `----' `---' `----'
}}}


== Description ==
Line 5: Line 13:
 * [[https://kgwg.nims.re.kr/cbcwiki/CAGMon | KGWG wiki link]]
 * [[http://gwwiki.icrr.u-tokyo.ac.jp/JGWwiki/CAGMon| KAGRA wiki link]]
Line 22: Line 31:
the Maximal Information coefficient(MIC) of a set D of two-variable data with sample size n and the grid less than B(n) is given by
Line 24: Line 34:
r=\frac{\sum_{i=1}^{n} (x_i - \bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n} (x_i-\bar{x})^2} \sqrt{\sum_{i=1}^{n} (y_i -\bar{y})^2}}
\]

MIC(D)=\underset{xy<B(n)}{\max}{\left\{ \frac{I^{*}(D,x,y)}{\log \min \left\{x,y \right\}} \right \}}
\],

where \[\omega(1)<B(n)\le O(n^{1-\epsilon}) \] for some \[ 0<\epsilon<1 \]
Line 30: Line 40:
Pearson Correlation Coefficient(PCC) is a statistic that explains the amount of variance accounted for in the relationship between two (or more) variables by
\[
R={{\sum_{i=1}^{n} (X_i - \overline{X})(Y_i - \overline{Y})} \over {\sqrt{\sum_{i=1}^{n} (X_i - \overline{X}) \sum_{i=1}^{n} (Y_i - \overline{Y})}}}
\],

where \[ \overline{X} \] and \[ \overline{Y} \] are the mean of X and Y, respectively

Line 32: Line 50:
Kendall’s tau with a random samples n of observations from two variables measures the strength of the relationship between two ordinal level variables by

\[
\tau =\frac{c-d}{{n \choose 2}}
\],

where c is the number of concordant pairs, and d is the number of discordant pairs

==== Flow chart ====


Line 34: Line 64:
==== Basic structure ====

==== Code version ====
 1. CAGmon Etude Alpha
   * for the basic test and evaluation of the LASSO regression method in LIGO
==== GitHub ====
[[T
BA]]

==== Code versions ====
 1. CAGMon Etude Alpha
   * for the basic test and evaluation of the LASSO regression method developed by LIGO
   * reproduced original CAGMon methods and idea
Line 43: Line 75:
 4. CAGmon Etude Eta
  * fixed memory issues
  * fixed minor bugs
 4. CAGMon Etude Eta
  * fixed minor issues
Line 47: Line 78:
 5. CAGmon Etude Flat (latest version)  5. CAGMon Etude Flat (current version)
  * fixed minor issues and optimized scripts
Line 50: Line 82:
 6. CAGMon Etude Octave (development version)
  * remove some processes that make Time-series and Scatter plots. Even though it required tremendous memory, this information is not useful
  * adjust HTML code
  * fixed minor issues and optimized scripts
  * added the analysis option whether or not the algorithm proceeds in the active segment only
  * improve script efficiency
 
==== Series of scripts ====
 * Agrement.py
  * the script gathered functions the model required
 * Melody.py
  * the script to calcutate each coefficient and to save trend data as csv
 * Conchord.py
  * the script to make plots
 * Echo.py
  * the script to save the result as HTML web page
 * CAGMonEtude{Version}.py
  * the script to run each script

==== User guide ====
 * [[/userguide | How to use CAGMon]]

==== Needs of code development ====
 * Fundamental critarian or guideline of CAGMon parameters, such as the stride, the sample rate, and its data-size
 * Daily running on KAGRA
Line 53: Line 111:
1. Earthquake effects during O3GK
 * Datetime: 19 April 2020 20:39 UTC
 * Purpose
  * Test to run CAGMon algorithm with a remarkable event
  * To figure out the cause of lock-loss in KAGRA
 * Computing resource
  * KISTI-LDG
  * Requested CPUS: 32cores
  * Requested memory: 128GB
 * Results
  * stride 5 seconds [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/2020-04-19_K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ_1271363358-1271364078(5)/ | Summary page]]
  * stride 20 seconds [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/2020-04-19_K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ_1271363358-1271364078/ | Summary page]]
  * stride 30 seconds [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/2020-04-19_K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ_1271363358-1271364078(30)/ | Summary page]]

2. With iKAGRA hardware injection data
 * Event
  * Phenomenon: the strain channel and seismometer channels in iKAGRA had a high correlation during the hardware injection test
  * Cause: still unknown
  * Hypothesis: the glitches have relatively the same behavior as the vacuum rotary pump
  * More detail analysis: [[https://www.dropbox.com/s/950vjc807sgz24u/hveto%20brief%20Report%20for%20K1.pdf?dl=0 | h-veto brief Report for K1]] and [[https://www.dropbox.com/s/hb7rx93an8yluiq/PilJong%2C%20KGWG%20Face-to-Face%20Meeting.pdf?dl=0 | KGWG Face-to-Face Meeting]]
 * Purpose
  * To verify whether this model senses injected signals and abnormal glitches
  * To test noise resistance and data-size limitation
 * Computing resource
  * KISTI-LDG
  * Requested CPUS: 32cores
  * Requested memory: 64GB
 *Results
 || Stride || Sample sata || Data size || Dada length || Summary page link ||
 || 10s || 512Hz || about 5,000 || about 12m || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b5000%5d/ | summary page]] ||
 || 10s || 1024Hz || about 10,000 || about 12m || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b10000%5d/ | summary page]] ||
 || 10s || 2048Hz || about 20,000 || about 12m || [[ https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b20000%5d/| summary page]] ||
 || 10s || 3072Hz || about 30,000 || about 12m || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b30000%5d/ | summary page]] ||
 || 10s || 4096Hz || about 40,000 || about 12m || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b40000%5d/ | summary page]] ||
 || 2s || 4096Hz || about 8,000 || about 12m || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b2s%5d/| summary page]] ||
 || 5s || 4096Hz || about 20,000 || about 12m || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145624200-1145624936%5b5s%5d/| summary page]] ||
 || 60s || 128Hz || about 7,500 || whole iKAGRA data || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145621548-1145670954%5b60s%5d/ | summary page]] ||
 || 150s || 64Hz || about 10,000 || whole iKAGRA data || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145621548-1145670954%5b150s%5d/ | summary page]] ||
 || 300s || 64Hz || about 20,000 || whole iKAGRA data || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145621548-1145670954%5b300s%5d/ | summary page]] ||
 || 600s || 16Hz || about 10,000 || whole iKAGRA data || [[https://ldas-jobs.ligo.caltech.edu/~pil-jong.jung/CAGMon/iKAGRA/2016-04-25_K1:LSC-MICH_CTRL_CAL_OUT_DQ_1145621548-1145670954%5b600s%5d/ | summary page]] ||


3. Skim through all obs-segments of O3GK
 * Purpose
  * Test for calculation time and required resources with all observation segments during O3GK
  * To figure out trigger events or abnormal behaviors
 * Computing resource
  * KISTI-LDG
  * Requested CPUS: 32cores
  * Requested memory: 128GB
 * Results
 || Date || GPS time || Data length || Stride || Sample rate || Data size || Summary page link || Remarks ||
 || April 7 || 1270287158 - 1270328032 || 11h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: 4h12m / memory usage: 42GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: 5h21m / memory usage: 23GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: 17h10m / memory usage: 41.9GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: 23h03m / memory usage: 28.8GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: 1d23h / memory usage: 24GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 8 || 1270339218 - 1270425618 || 24h || || || || || Full Data is unavailable in the KISTI cluster ||
 || April 9 || 1270425618 - 1270510167 || 23h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 10 || 1270513160 - 1270596544 || 23h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 11 || 1270598418 - 1270683904 || 23h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 12 || 1270684818 - 1270762046 || 21h || 500s || 16Hz || about 8,000 || [[ | summary page]] || ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 14 || 1270909686 - 1270937768 || 7h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: 50m / memory usage: 1.9GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: 2h8m / memory usage: 2.4GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 500s || 32Hz || about 16,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 240s || 64Hz || about 16,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 128Hz || about 16,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 256Hz || about 16,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 512Hz || about 16,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 500s || 64Hz || about 36,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 240s || 128Hz || about 36,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 256Hz || about 36,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 512Hz || about 36,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 1024Hz || about 36,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 500s || 128Hz || about 64,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 240s || 256Hz || about 64,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 512Hz || about 64,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 1024Hz || about 64,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 2048Hz || about 64,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 15 || 1270945288 - 1271017582 || 20h || || || || || GRB200415 (08:48:05 UTC) / Full Data is unavailable in the KISTI cluster ||
 || April 16 || 1271030433 - 1271112809 || 22h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 17 || 1271119833 - 1271186507 || 18h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 18 || 1271227441 - 1271288128 || 16h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 19 || 1271289618 - 1271364033 || 20h || 500s || 16Hz || about 8,000 || [[ | summary page]] || processing time: h / memory usage: GB ||
 || || || || 240s || 32Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 120s || 64Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 60s || 128Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 30s || 256Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 15s || 512Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 8s || 1024Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 4s || 2048Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 2s || 4096Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || || || || 1s || 8192Hz || about 8,000 || [[ | summary page]] || processing time: / memory usage: GB ||
 || April 20 || 1271377409 - 1271460608 || 23h || || || || || GRB200420A (2:32:58 UTC) / Full Data is unavailable in the KISTI cluster ||


Line 58: Line 284:
[[https://gwdoc.icrr.u-tokyo.ac.jp/cgi-bin/private/DocDB/ShowDocument?docid=12481|JGW-G2112481-v1]]
Line 60: Line 287:
[[https://science.sciencemag.org/content/334/6062/1518 | Science.1518; Detecting Novel Associations in Large Data Sets]]

 ,-----.  ,---.   ,----.   ,--.   ,--.                            ,--.             ,--.        
'  .--./ /  O  \ '  .-./   |   `.'   | ,---. ,--,--,      ,---. ,-'  '-.,--.,--. ,-|  | ,---.  
|  |    |  .-.  ||  | .---.|  |'.'|  || .-. ||      \    | .-. :'-.  .-'|  ||  |' .-. || .-. : 
'  '--'\|  | |  |'  '--'  ||  |   |  |' '-' '|  ||  |    \   --.  |  |  '  ''  '\ `-' |\   --. 
 `-----'`--' `--' `------' `--'   `--' `---' `--''--'     `----'  `--'   `----'  `---'  `----'                                                                                             

Description

The CAGMon etude is a study version of CAGMon that evaluates the dependence between the primary and auxiliary channels.

Project goal

The goal of this project is to find a systematic way of identifying the abnormal glitches in the gravitational-wave data using various methods of correlation analysis. Usually, the community such as LIGO, Virgo, and KAGRA uses a conventional way of finding glitches in auxiliary channels of the detector - Klein-Welle, Omicron, Ordered Veto Lists, etc. However, some different ways can be possible to find and monitor them in a (quasi-) realtime. Also, the method can point out which channel is responsible for the found glitch. In this project, we study its possible to apply three different correlation methods - maximal information coefficient, Pearson's correlation coefficient, and Kendall's tau coefficient - in the gravitational wave data from the KAGRA detector.

Participants

  • John.J Oh (NIMS)
  • Young-Min Kim (UNIST)
  • Pil-Jong Jung (NIMS)

Methods and Frameworks

Maximal Information Coefficient (MIC)

the Maximal Information coefficient(MIC) of a set D of two-variable data with sample size n and the grid less than B(n) is given by

\[ MIC(D)=\underset{xy<B(n)}{\max}{\left\{ \frac{I^{*}(D,x,y)}{\log \min \left\{x,y \right\}} \right \}} \],

where \[\omega(1)<B(n)\le O(n^{1-\epsilon}) \] for some \[ 0<\epsilon<1 \]

Pearson's Correlation Coefficient (PCC)

Pearson Correlation Coefficient(PCC) is a statistic that explains the amount of variance accounted for in the relationship between two (or more) variables by \[ R=\sum_{i=1}^{n} (X_i - \overline{X})(Y_i - \overline{Y})} \over {\sqrt{\sum_{i=1}^{n} (X_i - \overline{X}) \sum_{i=1}^{n} (Y_i - \overline{Y})} \],

where \[ \overline{X} \] and \[ \overline{Y} \] are the mean of X and Y, respectively

Kendall's tau Coefficient

Kendall’s tau with a random samples n of observations from two variables measures the strength of the relationship between two ordinal level variables by

\[ \tau =\frac{c-d}n \choose 2 \],

where c is the number of concordant pairs, and d is the number of discordant pairs

Flow chart

Code development

GitHub

TBA

Code versions

  1. CAGMon Etude Alpha
    • for the basic test and evaluation of the LASSO regression method developed by LIGO
    • reproduced original CAGMon methods and idea
  2. CAGMon Etude Beta
    • added coefficient trend plots with LASSO beta, coherence, MIC, PCC, and Kendall's tau
  3. CAGMon Etude Delta
    • fixed a critical problem that sucked enormous memory when it used the matplotlib module
  4. CAGMon Etude Eta
    • fixed minor issues
    • added the range limitation of stride
  5. CAGMon Etude Flat (current version)
    • fixed minor issues and optimized scripts
    • added the script of HTML summary page
    • added coefficient distribution plots
  6. CAGMon Etude Octave (development version)
    • remove some processes that make Time-series and Scatter plots. Even though it required tremendous memory, this information is not useful
    • adjust HTML code
    • fixed minor issues and optimized scripts
    • added the analysis option whether or not the algorithm proceeds in the active segment only
    • improve script efficiency

Series of scripts

  • Agrement.py
    • the script gathered functions the model required
  • Melody.py
    • the script to calcutate each coefficient and to save trend data as csv
  • Conchord.py
    • the script to make plots
  • Echo.py
    • the script to save the result as HTML web page
  • CAGMonEtude{Version}.py
    • the script to run each script

User guide

Needs of code development

  • Fundamental critarian or guideline of CAGMon parameters, such as the stride, the sample rate, and its data-size
  • Daily running on KAGRA

Exemplary results

1. Earthquake effects during O3GK

  • Datetime: 19 April 2020 20:39 UTC
  • Purpose
    • Test to run CAGMon algorithm with a remarkable event
    • To figure out the cause of lock-loss in KAGRA
  • Computing resource
    • KISTI-LDG
    • Requested CPUS: 32cores
    • Requested memory: 128GB
  • Results

2. With iKAGRA hardware injection data

  • Event
    • Phenomenon: the strain channel and seismometer channels in iKAGRA had a high correlation during the hardware injection test
    • Cause: still unknown
    • Hypothesis: the glitches have relatively the same behavior as the vacuum rotary pump
    • More detail analysis: h-veto brief Report for K1 and KGWG Face-to-Face Meeting

  • Purpose
    • To verify whether this model senses injected signals and abnormal glitches
    • To test noise resistance and data-size limitation
  • Computing resource
    • KISTI-LDG
    • Requested CPUS: 32cores
    • Requested memory: 64GB
  • Results

    Stride

    Sample sata

    Data size

    Dada length

    Summary page link

    10s

    512Hz

    about 5,000

    about 12m

    summary page

    10s

    1024Hz

    about 10,000

    about 12m

    summary page

    10s

    2048Hz

    about 20,000

    about 12m

    summary page

    10s

    3072Hz

    about 30,000

    about 12m

    summary page

    10s

    4096Hz

    about 40,000

    about 12m

    summary page

    2s

    4096Hz

    about 8,000

    about 12m

    summary page

    5s

    4096Hz

    about 20,000

    about 12m

    summary page

    60s

    128Hz

    about 7,500

    whole iKAGRA data

    summary page

    150s

    64Hz

    about 10,000

    whole iKAGRA data

    summary page

    300s

    64Hz

    about 20,000

    whole iKAGRA data

    summary page

    600s

    16Hz

    about 10,000

    whole iKAGRA data

    summary page

3. Skim through all obs-segments of O3GK

  • Purpose
    • Test for calculation time and required resources with all observation segments during O3GK
    • To figure out trigger events or abnormal behaviors
  • Computing resource
    • KISTI-LDG
    • Requested CPUS: 32cores
    • Requested memory: 128GB
  • Results

    Date

    GPS time

    Data length

    Stride

    Sample rate

    Data size

    Summary page link

    Remarks

    April 7

    1270287158 - 1270328032

    11h

    500s

    16Hz

    about 8,000

    summary page

    processing time: 4h12m / memory usage: 42GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: 5h21m / memory usage: 23GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: 17h10m / memory usage: 41.9GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: 23h03m / memory usage: 28.8GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: 1d23h / memory usage: 24GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 8

    1270339218 - 1270425618

    24h

    Full Data is unavailable in the KISTI cluster

    April 9

    1270425618 - 1270510167

    23h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 10

    1270513160 - 1270596544

    23h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 11

    1270598418 - 1270683904

    23h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 12

    1270684818 - 1270762046

    21h

    500s

    16Hz

    about 8,000

    summary page

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 14

    1270909686 - 1270937768

    7h

    500s

    16Hz

    about 8,000

    summary page

    processing time: 50m / memory usage: 1.9GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: 2h8m / memory usage: 2.4GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    500s

    32Hz

    about 16,000

    summary page

    processing time: / memory usage: GB

    240s

    64Hz

    about 16,000

    summary page

    processing time: / memory usage: GB

    120s

    128Hz

    about 16,000

    summary page

    processing time: / memory usage: GB

    60s

    256Hz

    about 16,000

    summary page

    processing time: / memory usage: GB

    30s

    512Hz

    about 16,000

    summary page

    processing time: / memory usage: GB

    500s

    64Hz

    about 36,000

    summary page

    processing time: / memory usage: GB

    240s

    128Hz

    about 36,000

    summary page

    processing time: / memory usage: GB

    120s

    256Hz

    about 36,000

    summary page

    processing time: / memory usage: GB

    60s

    512Hz

    about 36,000

    summary page

    processing time: / memory usage: GB

    30s

    1024Hz

    about 36,000

    summary page

    processing time: / memory usage: GB

    500s

    128Hz

    about 64,000

    summary page

    processing time: / memory usage: GB

    240s

    256Hz

    about 64,000

    summary page

    processing time: / memory usage: GB

    120s

    512Hz

    about 64,000

    summary page

    processing time: / memory usage: GB

    60s

    1024Hz

    about 64,000

    summary page

    processing time: / memory usage: GB

    30s

    2048Hz

    about 64,000

    summary page

    processing time: / memory usage: GB

    April 15

    1270945288 - 1271017582

    20h

    GRB200415 (08:48:05 UTC) / Full Data is unavailable in the KISTI cluster

    April 16

    1271030433 - 1271112809

    22h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 17

    1271119833 - 1271186507

    18h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 18

    1271227441 - 1271288128

    16h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 19

    1271289618 - 1271364033

    20h

    500s

    16Hz

    about 8,000

    summary page

    processing time: h / memory usage: GB

    240s

    32Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    120s

    64Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    60s

    128Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    30s

    256Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    15s

    512Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    8s

    1024Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    4s

    2048Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    2s

    4096Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    1s

    8192Hz

    about 8,000

    summary page

    processing time: / memory usage: GB

    April 20

    1271377409 - 1271460608

    23h

    GRB200420A (2:32:58 UTC) / Full Data is unavailable in the KISTI cluster

Beyond

References

Presentation materials

JGW-G2112481-v1

Papers

Science.1518; Detecting Novel Associations in Large Data Sets

PJJung/CAGMonEtude (last edited 2021-07-28 08:43:57 by PJJung)