|
Size: 55121
Comment:
|
Size: 57054
Comment:
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 390: | Line 390: |
| * Data-size: 8192 (default value is 16384) | * Data-size: 8192 (default value is 8192) |
| Line 417: | Line 417: |
| * Results of PEM injection analysis during hammering test (without Scatter and Spectrogram plots) * [[ https://klog.icrr.u-tokyo.ac.jp/osl/?c=13138 | KAGRA Logbook 13138 ]] || GPS time || Hammering point || Analysis duration || Stride || Sample rate || Data size || Summary page || Remarks || || 1266530332 || between PR2 and BS, PR2 side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || || 1266530502 || between PR2 and BS, BS side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || || 1266530698 || between BS and SR2, BS side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || || 1266530857 || between BS and SR2, SR2 side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || || 1268280351 || between SR2 and SR3, SR2 side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || || 1266531140 || between SR2 and SR3, SR3 side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || || 1266531247 || between SR3 and SRM, SR3 side || +-600s || 2s || 4086Hz || 8192 || [[ | link ]] || ERR || || || || || || || || [[ | link ]] || BNS || * note that ERR means K1:CAL-CS_PROC_DARM_ERR_DBL_DQ, and BNS means K1:DAS-RANGE_BNS |
,-----. ,---. ,----. ,--. ,--. ,--. ,--. ' .--./ / O \ ' .-./ | `.' | ,---. ,--,--, ,---. ,-' '-.,--.,--. ,-| | ,---. | | | .-. || | .---.| |'.'| || .-. || \ | .-. :'-. .-'| || |' .-. || .-. : ' '--'\| | | |' '--' || | | |' '-' '| || | \ --. | | ' '' '\ `-' |\ --. `-----'`--' `--' `------' `--' `--' `---' `--''--' `----' `--' `----' `---' `----'
Description
The CAGMon etude is a study version of CAGMon that evaluates the dependence between the primary and auxiliary channels.
Project goal
The goal of this project is to find a systematic way of identifying the abnormal glitches in the gravitational-wave data using various methods of correlation analysis. Usually, the community such as LIGO, Virgo, and KAGRA uses a conventional way of finding glitches in auxiliary channels of the detector - Klein-Welle, Omicron, Ordered Veto Lists, etc. However, some different ways can be possible to find and monitor them in a (quasi-) realtime. Also, the method can point out which channel is responsible for the found glitch. In this project, we study its possible to apply three different correlation methods - maximal information coefficient, Pearson's correlation coefficient, and Kendall's tau coefficient - in the gravitational wave data from the KAGRA detector.
Participants
- John.J Oh (NIMS)
- Young-Min Kim (UNIST)
- Pil-Jong Jung (NIMS)
Methods and Frameworks
Maximal Information Coefficient (MIC)
the Maximal Information coefficient(MIC) of a set D of two-variable data with sample size n and the grid less than B(n) is given by
\[ MIC(D)=\underset{xy<B(n)}{\max}{\left\{ \frac{I^{*}(D,x,y)}{\log \min \left\{x,y \right\}} \right \}} \],
where \[\omega(1)<B(n)\le O(n^{1-\epsilon}) \] for some \[ 0<\epsilon<1 \]
Pearson's Correlation Coefficient (PCC)
Pearson Correlation Coefficient(PCC) is a statistic that explains the amount of variance accounted for in the relationship between two (or more) variables by \[ R=} \],
where \[ \overline{X} \] and \[ \overline{Y} \] are the mean of X and Y, respectively
Kendall's tau Coefficient
Kendall’s tau with a random samples n of observations from two variables measures the strength of the relationship between two ordinal level variables by
\[ \tau =\frac{c-d} \],
where c is the number of concordant pairs, and d is the number of discordant pairs
Flow chart
Code development
GitHub
Code versions
- CAGMon Etude Alpha (v0.1.0)
- for the basic test and evaluation of the LASSO regression method developed by LIGO
- reproduced original CAGMon methods and idea
- CAGMon Etude Beta (v0.2.0)
- added coefficient trend plots with LASSO beta, coherence, MIC, PCC, and Kendall's tau
- CAGMon Etude Delta (v0.3.0)
- fixed a critical problem that sucked enormous memory when it used the matplotlib module
- CAGMon Etude Eta (v0.4.0)
- fixed minor issues
- added the range limitation of stride
- CAGMon Etude Flat (v0.5.0)
- fixed minor issues and optimized scripts
- added the script of HTML summary page
- added coefficient distribution plots
- CAGMon Etude Octave (v0.6.0)
- remove some processes that make Time-series and Scatter plots. Even though it required tremendous memory, this information is not useful
- adjust HTML code
- fixed minor issues and optimized scripts
- added the analysis option whether or not the algorithm proceeds in the active segment only
- improve script efficiency
added the process to make scatter and OmegaScan plots in detail boxes of the summary page
- CAGMon Etude Rhapsody (v0.7.0) [current version]
- utilize MICe with the auto-selection method of Alpha, c
- improve script efficiency and completeness
- minor update for summary page layout and plot design
- CAGMon Etude Sinfornietta (v0.8.0) [development version]
support for installation through PyPl
- support for both Python 3 and Python2.7
- improve code stability
Series of scripts
version <= Rhapsody
- Agrement.py
- the script gathered functions the model required
- Melody.py
- the script to calcutate each coefficient and to save trend data as csv
- Conchord.py
- the script to make plots
- Echo.py
- the script to save the result as HTML web page
- CAGMonEtude{Version}.py
- the script to run each script
- Agrement.py
version >= Sinfornietta
- cagmon
init.py
- agrement.py
- melody,py
- conchord.py
- echo.py
- main.py
- setup.py
- cagmon
User guide
version <= Rhapsody (v0.7.0)
version >= Sinfornietta (v0.8.0)
Needs of code development
- Fundamental criteria or guideline of CAGMon parameters, such as the stride, the sample rate, and its data-size
- Daily running on KAGRA
Empirical study (No free lunch)
- Apply to glitch data on KAGRA during O3GK
- Glitch information
- Purpose
- To decide on appropriate parameters when we run CAGMon for searching glitches and correlation
- To make recommended parameters in the short-range analysis
- Result
- Appropriate parameters of CAGMon for the glitch search
- Data-size: 8,192 or 16,384
- Stride: 0.5 or 1.0 seconds (the glitch duration users want to find)
- Statistical power test
- prepare for the paper of parameter selection
Exemplary results
1. Earthquake effects during O3GK (with CAGMon Etude Flat)
- Datetime: 19 April 2020 20:39 UTC
- Purpose
- Test to run CAGMon algorithm with a remarkable event
- To figure out the cause of lock-loss in KAGRA
- Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 128GB
- Results
stride 5 seconds Summary page
stride 20 seconds Summary page
stride 30 seconds Summary page
2. With iKAGRA hardware injection data (with CAGMon Etude Flat)
- Event
- Phenomenon: the strain channel and seismometer channels in iKAGRA had a high correlation during the hardware injection test
- Cause: still unknown
- Hypothesis: the glitches have relatively the same behavior as the vacuum rotary pump
More detail analysis: h-veto brief Report for K1 and KGWG Face-to-Face Meeting
- Purpose
- To verify whether this model senses injected signals and abnormal glitches
- To test noise resistance and data-size limitation
- Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 64GB
- Results
Stride
Sample sata
Data size
Dada length
Summary page link
10s
512Hz
about 5,000
about 12m
10s
1024Hz
about 10,000
about 12m
10s
2048Hz
about 20,000
about 12m
10s
3072Hz
about 30,000
about 12m
10s
4096Hz
about 40,000
about 12m
2s
4096Hz
about 8,000
about 12m
5s
4096Hz
about 20,000
about 12m
60s
128Hz
about 7,500
whole iKAGRA data
150s
64Hz
about 10,000
whole iKAGRA data
300s
64Hz
about 20,000
whole iKAGRA data
600s
16Hz
about 10,000
whole iKAGRA data
3. Skim through some obs-segments of O3GK (with CAGMon Etude Octave)
- Purpose
- Test for calculation time and required resources with all observation segments during O3GK
- To figure out trigger events or abnormal behaviors
- Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 128GB
- Results
Date
GPS time
Data length
Stride
Sample rate
Data size
Summary page link
Remarks
April 7
1270287158 - 1270328032
11h
500s
16Hz
about 8,000
processing time: 4h12m / memory usage: 42GB
240s
32Hz
about 8,000
processing time: 5h21m / memory usage: 23GB
120s
64Hz
about 8,000
processing time: 17h10m / memory usage: 41.9GB
60s
128Hz
about 8,000
processing time: 23h03m / memory usage: 28.8GB
30s
256Hz
about 8,000
processing time: 1d23h / memory usage: 24GB
15s
512Hz
about 8,000
processing time: > 3 days => killed
8s
1024Hz
about 8,000
processing time: > 3 days => killed
4s
2048Hz
about 8,000
processing time: > 3 days => killed
2s
4096Hz
about 8,000
processing time: > 3 days => killed
1s
8192Hz
about 8,000
processing time: > 3 days => killed
April 14
1270909686 - 1270937768
7h
500s
16Hz
about 8,000
processing time: 50m / memory usage: 1.9GB
240s
32Hz
about 8,000
processing time: 2h8m / memory usage: 2.4GB
120s
64Hz
about 8,000
processing time: 4h40m / memory usage: 3.6GB
60s
128Hz
about 8,000
processing time: 8h30m / memory usage: 2.0GB
30s
256Hz
about 8,000
processing time: 15h30m / memory usage: 2.2GB
500s
32Hz
about 16,000
processing time: 50m / memory usage: 3.2GB
240s
64Hz
about 16,000
processing time: 5h25m / memory usage: 3.3GB
120s
128Hz
about 16,000
processing time: 10h10m / memory usage: 3.4GB
60s
256Hz
about 16,000
processing time: 18h50m / memory usage: 3.4GB
30s
512Hz
about 16,000
processing time: 17h / memory usage: 3.7GB
500s
64Hz
about 36,000
processing time: 6h30m / memory usage: 6.7GB
240s
128Hz
about 36,000
processing time: 10h57m / memory usage: 6.7GB
120s
256Hz
about 36,000
processing time: 17h3m / memory usage: 7.0GB
60s
512Hz
about 36,000
processing time: 1d6h / memory usage: 7.1GB
30s
1024Hz
about 36,000
processing time: 2d22h30m / memory usage: 7.0GB
500s
128Hz
about 64,000
processing time: 12h10m / memory usage: 14.6GB
240s
256Hz
about 64,000
processing time: 1d3h40m / memory usage: 14.6GB
120s
512Hz
about 64,000
processing time: 2d6h / memory usage: 14.7GB
60s
1024Hz
about 64,000
processing time: > 4 days => kille
30s
2048Hz
about 64,000
processing time: > 4 days => kille
4. Glitch analysis and channel correlation study during O3GK (with CAGMon Etude Octave)
- Purpose
- To identify channel correlation and the cause of glitches
- Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 64GB
- CAGMon parameters
- MIC Alpha: 0.6
- MIC c: 15
- Data-size: 8192
- Stride: 512 seconds for the long-term search and 1 seconds for the short-term search
- Aux-channels: channels used daily running h-veto on KAGRA
- Results with the daily data (without Scatter and Spectrogram plots)
Date
GPS time
Data length
Stride
Sample rate
Data size
Summary page
Resource
Remarks
April 7
1270287158 - 1270328032
11.35h
512s
16Hz
8192
4h / 4GB
April 8
1270339218 - 1270425618
24h
512s
16Hz
8192
N/A in KISTI
April 9
1270425618 - 1270510167
23.45h
512s
16Hz
8192
6h57m / 12.2GB
April 10
1270513160 - 1270596544
23.16h
512s
16Hz
8192
8h30m / 5.6GB
April 11
1270598418 - 1270683904
23.75h
512s
16Hz
8192
6h59m / 13.8GB
April 12
1270684818 - 1270762046
21.45h
512s
16Hz
8192
6h4m / 7.6GB
April 14
1270909686 - 1270937768
7.8h
512s
16Hz
8192
2h40m / 3.2GB
April 15
1270945288 - 1271017582
20.08h
512s
16Hz
8192
N/A in KISTI
April 16
1271030433 - 1271112809
22.88h
512s
16Hz
8192
8h10m / 8.0GB
April 17
1271119833 - 1271186507
18.52h
512s
16Hz
8192
7h / 6.4GB
April 18
1271227441 - 1271288128
16.86h
512s
16Hz
8192
N/A in KISTI
April 19
1271289618 - 1271364033
20.67h
512s
16Hz
8192
8h15m / 8.8GB
April 20
1271377409 - 1271460608
23.11h
512s
16Hz
8192
N/A in KISTI
Results with the borders of obs-segments (with Scatter and OmegaScan plots)
Date
GPS time
Data length
Stride
Sample rate
Data size
Summary page
Resource
Remarks
April 7
1270287723
±120s
1s
8192Hz
8192
4h23m / 22.4GB
1270289095
±120s
1s
8192Hz
8192
4h20m / 21.3GB
1270291338
±120s
1s
8192Hz
8192
4h29m / 23.5GB
1270299544
±120s
1s
8192Hz
8192
4h17m / 21.8GB
1270302108
±120s
1s
8192Hz
8192
4h20m / 22.1GB
1270312047
±120s
1s
8192Hz
8192
4h24m / 21.7GB
1270320096
±120s
1s
8192Hz
8192
4h20m / 19.9GB
1270320127
±120s
1s
8192Hz
8192
4h20m / 19.4GB
1270325304
±120s
1s
8192Hz
8192
4h20m / 19.6GB
1270328032
±120s
1s
8192Hz
8192
4h20m / 19.3GB
April 9
1270427598
±120s
1s
8192Hz
8192
1270433380
±120s
1s
8192Hz
8192
1270452484
±120s
1s
8192Hz
8192
1270453899
±120s
1s
8192Hz
8192
1270456187
±120s
1s
8192Hz
8192
1270458641
±120s
1s
8192Hz
8192
1270459231
±120s
1s
8192Hz
8192
1270459613
±120s
1s
8192Hz
8192
1270460499
±120s
1s
8192Hz
8192
1270460578
±120s
1s
8192Hz
8192
1270460855
±120s
1s
8192Hz
8192
1270467750
±120s
1s
8192Hz
8192
1270477524
±120s
1s
8192Hz
8192
1270489046
±120s
1s
8192Hz
8192
1270495548
±120s
1s
8192Hz
8192
1270496325
±120s
1s
8192Hz
8192
1270497144
±120s
1s
8192Hz
8192
1270500085
±120s
1s
8192Hz
8192
1270502766
±120s
1s
8192Hz
8192
1270508915
±120s
1s
8192Hz
8192
1270510167
±120s
1s
8192Hz
8192
April 10
1270513361
±120s
1s
8192Hz
8192
4h20m / 21.7GB
1270519811
±120s
1s
8192Hz
8192
4h15m / 20.2GB
1270544263
±120s
1s
8192Hz
8192
4h20m / 19.7GB
1270546596
±120s
1s
8192Hz
8192
4h20m / 19.2GB
1270547109
±120s
1s
8192Hz
8192
4h19m / 22.6GB
1270549273
±120s
1s
8192Hz
8192
4h18m / 21.6GB
1270550718
±120s
1s
8192Hz
8192
4h17m / 21.6GB
1270551599
±120s
1s
8192Hz
8192
4h19m / 22.8GB
1270552685
±120s
1s
8192Hz
8192
4h20m / 19.2GB
1270555167
±120s
1s
8192Hz
8192
4h16m / 24.9GB
1270572585
±120s
1s
8192Hz
8192
4h13m / 21.5GB
1270580988
±120s
1s
8192Hz
8192
4h23m / 21.5GB
1270585866
±120s
1s
8192Hz
8192
4h16m / 27.1GB
1270596544
±120s
1s
8192Hz
8192
4h20m / 19.4GB
April 11
1270609879
±120s
1s
8192Hz
8192
1270610651
±120s
1s
8192Hz
8192
1270631460
±120s
1s
8192Hz
8192
1270636472
±120s
1s
8192Hz
8192
1270637495
±120s
1s
8192Hz
8192
1270641494
±120s
1s
8192Hz
8192
1270655146
±120s
1s
8192Hz
8192
1270669014
±120s
1s
8192Hz
8192
1270669700
±120s
1s
8192Hz
8192
1270683904
±120s
1s
8192Hz
8192
April 12
1270696199
±120s
1s
8192Hz
8192
1270713927
±120s
1s
8192Hz
8192
1270715460
±120s
1s
8192Hz
8192
1270720692
±120s
1s
8192Hz
8192
1270721256
±120s
1s
8192Hz
8192
1270726572
±120s
1s
8192Hz
8192
1270746411
±120s
1s
8192Hz
8192
1270748613
±120s
1s
8192Hz
8192
1270759925
±120s
1s
8192Hz
8192
1270762046
±120s
1s
8192Hz
8192
April 14
1270909778
±120s
1s
8192Hz
8192
1270909999
±120s
1s
8192Hz
8192
1270911048
±120s
1s
8192Hz
8192
1270911986
±120s
1s
8192Hz
8192
1270912899
±120s
1s
8192Hz
8192
1270913182
±120s
1s
8192Hz
8192
1270924011
±120s
1s
8192Hz
8192
1270924958
±120s
1s
8192Hz
8192
1270925716
±120s
1s
8192Hz
8192
1270925944
±120s
1s
8192Hz
8192
1270927783
±120s
1s
8192Hz
8192
1270928712
±120s
1s
8192Hz
8192
1270930893
±120s
1s
8192Hz
8192
1270932636
±120s
1s
8192Hz
8192
1270933841
±120s
1s
8192Hz
8192
1270935617
±120s
1s
8192Hz
8192
1270936532
±120s
1s
8192Hz
8192
1270937055
±120s
1s
8192Hz
8192
1270937768
±120s
1s
8192Hz
8192
April 16
1271032316
±120s
1s
8192Hz
8192
1271038814
±120s
1s
8192Hz
8192
1271040295
±120s
1s
8192Hz
8192
1271040412
±120s
1s
8192Hz
8192
1271040672
±120s
1s
8192Hz
8192
1271040766
±120s
1s
8192Hz
8192
1271041322
±120s
1s
8192Hz
8192
1271043681
±120s
1s
8192Hz
8192
1271046678
±120s
1s
8192Hz
8192
1271052148
±120s
1s
8192Hz
8192
1271054241
±120s
1s
8192Hz
8192
1271055150
±120s
1s
8192Hz
8192
1271059886
±120s
1s
8192Hz
8192
1271073604
±120s
1s
8192Hz
8192
1271091615
±120s
1s
8192Hz
8192
1271103700
±120s
1s
8192Hz
8192
1271107810
±120s
1s
8192Hz
8192
1271112809
±120s
1s
8192Hz
8192
April 17
1271125030
±120s
1s
8192Hz
8192
1271127419
±120s
1s
8192Hz
8192
1271142158
±120s
1s
8192Hz
8192
1271145974
±120s
1s
8192Hz
8192
1271160724
±120s
1s
8192Hz
8192
1271178914
±120s
1s
8192Hz
8192
1271186507
±120s
1s
8192Hz
8192
April 19
1271300385
±120s
1s
8192Hz
8192
1271325984
±120s
1s
8192Hz
8192
1271328149
±120s
1s
8192Hz
8192
1271329025
±120s
1s
8192Hz
8192
1271330275
±120s
1s
8192Hz
8192
1271340023
±120s
1s
8192Hz
8192
1271354879
±120s
1s
8192Hz
8192
1271361736
±120s
1s
8192Hz
8192
1271364033
±120s
1s
8192Hz
8192
5. Glitch analysis and channel correlation study during O3GK (with CAGMon Etude Rhapsody)
- Purpose
- Test for the new version with MICe and its reliability
- Computing resource
- KISTI-LDG
- Requested CPUS: 32cores
- Requested memory: 32GB
- CAGMon parameters
- Data-size: 8192 (default value is 8192)
- Stride: 512 seconds for the long-term search and 1 second for the short-term search
- MICe parameters from the empirical study
Data size n
Alpha
c
n >= 1
0.85
5
n >= 25
0.80
5
n >= 50
0.75
5
n >= 250
0.70
5
n >= 500
0.65
4
n >= 1,000
0.60
3
n >= 2,500
0.55
1
n >= 10,000
0.45
1
n >= 40,000
0.40
1
- Aux-channels: channels used daily running h-veto on KAGRA + PEM-SEIS/ACC/MIC/MAG channels
- Results with the daily data (without Scatter and Spectrogram plots)
- Main channel: K1:DAS-RANGE_BNS
Date
GPS time
Data length
Stride
Sample rate
Data size
Summary page
Resource
Remarks
April 7
1270287158 - 1270328032
11.35h
512s
16Hz
8192
April 9
1270425618 - 1270510167
23.45h
512s
16Hz
8192
April 10
1270513160 - 1270596544
23.16h
512s
16Hz
8192
April 11
1270598418 - 1270683904
23.75h
512s
16Hz
8192
April 12
1270684818 - 1270762046
21.45h
512s
16Hz
8192
April 14
1270909686 - 1270937768
7.8h
512s
16Hz
8192
April 16
1271030433 - 1271112809
22.88h
512s
16Hz
8192
April 17
1271119833 - 1271186507
18.52h
512s
16Hz
8192
April 19
1271289618 - 1271364033
20.67h
512s
16Hz
8192
- Results of PEM injection analysis during hammering test (without Scatter and Spectrogram plots)
GPS time
Hammering point
Analysis duration
Stride
Sample rate
Data size
Summary page
Remarks
1266530332
between PR2 and BS, PR2 side
+-600s
2s
4086Hz
8192
ERR
BNS
1266530502
between PR2 and BS, BS side
+-600s
2s
4086Hz
8192
ERR
BNS
1266530698
between BS and SR2, BS side
+-600s
2s
4086Hz
8192
ERR
BNS
1266530857
between BS and SR2, SR2 side
+-600s
2s
4086Hz
8192
ERR
BNS
1268280351
between SR2 and SR3, SR2 side
+-600s
2s
4086Hz
8192
ERR
BNS
1266531140
between SR2 and SR3, SR3 side
+-600s
2s
4086Hz
8192
ERR
BNS
1266531247
between SR3 and SRM, SR3 side
+-600s
2s
4086Hz
8192
ERR
BNS
- note that ERR means K1:CAL-CS_PROC_DARM_ERR_DBL_DQ, and BNS means K1:DAS-RANGE_BNS
Cross-validation
Beyond
References
Presentation materials
JGW-G2112481-v1 JGW-G2112750-v1 JGW-G2112703-v1 JGW-G2112763-v1
Papers
Science.1518; Detecting Novel Associations in Large Data Sets