Differences between revisions 13 and 22 (spanning 9 versions)

Prepare h(t) reconstruction review

LIGO time-domain calibration review wiki: https://wiki.ligo.org/Calibration/TDCalibReview

Calibration State Vector (K1:DCS-CALIB_STATE_VECTOR_C10)

(Under preparation. It's planned to be used in O3b)

bit	Identifier	Description
0	`HOFT_OK`	$h(t)$ was successfully computed (logical AND of bits ...)
1	`OBS_INTENT`	The observation intent is set by the operator
2	`OBS_READY_OR_OBSERVING`	Interferometer is in observation ready mode: the LSC_LOCK+ guardian is in the nominal (observation) state.
3	`NO_STOCHASTIC_HW_INJ`	No stochastic hardware injections present
4	`NO_CBC_HW_INJ`	No compact binary coalescence hardware injections present
5	`NO_BURST_HW_INJ`	No burst hardware injections present
6	`NO_DETCHAR_HW_INJ`	No detector characterization hardware injections present
7	`NO_CW_HW_INJ`	No continuous wave characterization hardware injections present
8	`TM_LINE_COH_OK`	TM line coherence OK
9	`IM_LINE_COH_OK`	IM line coherence OK
10	`MN_LINE_COH_OK`	MN line coherence OK
11	`PCAL1_LINE_COH_OK`	PCAL1 line coherence OK
12	`PCAL2_LINE_COH_OK`	PCAL2 line coherence OK
13	`PCAL3_LINE_COH_OK`	PCAL3 line coherence OK
14	`PCAL4_LINE_COH_OK`	PCAL4 line coherence OK
15	`PCAL5_LINE_COH_OK`	PCAL5 line coherence OK
16	`SMOOTH_KAPPA_TM_OK`	TDCF smoothed $\kappa_\mathrm{TM}$ values are in the expected range
17	`SMOOTH_KAPPA_IM_OK`	TDCF smoothed $\kappa_\mathrm{IM}$ values are in the expected range
18	`SMOOTH_KAPPA_MN_OK`	TDCF smoothed $\kappa_\mathrm{MN}$ values are in the expected range
19	`SMOOTH_KAPPA_C_OK`	TDCF smoothed $\kappa_\mathrm{C}$ value is in the expected range
20	`SMOOTH_F_CC_OK`	TDCF smoothed $f_\mathrm{cc}$ value is in the expected range
21	`SMOOTH_F_S_OK`	(reserved) TDCF smoothed $f_\mathrm{S}$ value is in the expected range
22	`SMOOTH_Q_S_OK`	(reserved) TDCF smoothed $Q_\mathrm{S}$ value is in the expected range
23	`TDCFS_APPLIED`	Time-dependent calibration factors are being applied
24	`LINE_SUBTRACTION_APPLIED`	(reserved) Line subtraction are being applied
25	`NO_GAP`	(reserved) Input data has no gaps, underflow, or overflow values
26	(not in use)	(not in use)
27	(not in use)	(not in use)
28	(not in use)	(not in use)
29	(not in use)	(not in use)
30	(not in use)	(not in use)
31	(not in use)	(not in use)

Some channels were copied from DET channels

K1:DET-DQ_STATE_VECTOR -> CAL state vector

# bit 2 : SDF flag (= K1:GRD-SYS_SDF_OK) -> # bit 2
# bit 7 : NO Stoch injection -> # bit 3
# bit 8 : NO CBC injection -> # bit 4
# bit 9 : NO Burst injection -> # bit 5
# bit 10 : NO DET injection -> # bit 6
# bit 11 : NO CW injection -> # bit 7

K1:DET-OBS_INTENT -> CAL state vector: # bit 1

Low-latency and offline

Channels used for low-latency and offline calibration (July 12 - ...)

Channel Name	Rate	Format	Description
`K1:CAL-CS_PROC_MICH_RESIDUAL_DBL_DQ`	16384	F64LE	MICH error
`K1:CAL-CS_PROC_MICH_DELTA_CTRL_TM_DBL_DQ`	4096	F64LE	MICH TM stage control
`K1:CAL-CS_PROC_MICH_DELTA_CTRL_IM_DBL_DQ`	4096	F64LE	MICH IM stage control
`K1:CAL-CS_PROC_MICH_DISPLACEMENT_DQ`	16384	F32LE	MICH displacement (optional)
`K1:CAL-CS_PROC_IMC_RESIDUAL_DQ`	16384	F32LE	IMC error
`K1:CAL-CS_PROC_IMC_DELTA_CTRL_AOM_DQ`	4096	F32LE	IMC AOM frequency control
`K1:CAL-CS_PROC_IMC_DELTA_CTRL_TM_DQ`	4096	F32LE	IMC TM stage control
`K1:CAL-CS_PROC_IMC_FREQUENCY_DQ`	16384	F32LE	Laser frequency (optional)
`K1:GRD-LSC_LOCK_OK`	16	F32LE	LSC Guardian OK status
`K1:GRD-LSC_LOCK_STATE_N`	16	F32LE	LSC lock state
`K1:GRD-IO_OK`	16	F32LE	IMC Guardian OK status
`K1:GRD-IO_STATE_N`	16	F32LE	IMC state

Where F32LE and F64LE means: floating-point 32-bit and 64-bit numbers in Little-Endian format.

Channels marked "optional" will not be used in the low-latency and offline computations, K1:CAL-CS_PROC_MICH_DISPLACEMENT_DQ and K1:CAL-CS_PROC_IMC_FREQUENCY_DQ in particular are the channels calibrated with the online system.

Low-latency h(t) reconstruction server (Aug - Oct/31: DT,SH,TY,NK,...)

We already have three computing nodes to test the offline h(t) reconstruction via frame file but we need to install another for the low-latency h(t) reconstruction via shared memory inside the tunnel
(Aug/20-) Connection and software test between a prototype PC and a test bench
- Prototype PC is just a standard desktop PC and will be prepared by SH
(Aug-Sep) Procurement of server (2U height, at least two 10G network card, standard CPU,memory and disk)
- According to Darkhan’s test with LIGO data and discussion with LIGO people, standard spec should be OK.
- DT is working to figure out the spec and model.
- Academia Sinica will make the procurement
(Mid.Sep) Connection test between the server and the test bench
- We also need the collaboration with DMG people for the data transfer
(Sep-Oct) Connection test between the server and the KAGRA online system
- We also need the collaboration with DMG people for the data transfer

Current tasks (May 26 - July 15):

Ask to LIGO CAL team for the detailed information of gstlal-calibration (DT) [status: ongoing | expected finish: N/A]
Make gstlal-calibration running on a machine at KAGRA (DT, SH) [status: ongoing | expected finish: Jul 15]
- Find names and versions of all prerequisite libraries
- Install prerequisites and gstlal-cal package on a machine at AS
- Test gstlal-cal pipeline on an AS machine
- Install prerequisites and gstlal-cal package on a machine at KAGRA
- Test the pipeline on a KAGRA machine

Progress so far (June 21):

Ask LIGO CAL team for the detailed information of gstlal-calibration
- (ongoing) gstlal-calibration maintainers in LIGO are helping to resolve some installation and testing issues.
Find names and versions of all prerequisite libraries,
- (done, entry from June 7) SH made progress on this, see gstlal installation
Install required packages and the gstlal-cal package on a machine at AS
- DT has written scripts for installing lalsuite and gstlal software on a clean CentOS7 distro based on SH's instructions. Scripts were uploaded to KAGRA-AS group's Gitlab repositories:
  - containerized "cold" version of lalsuite containerized "cold" version of gstlal-calibration
- CI pipelines that build docker-lalsuite-cold and docker-gstlal-cold containers were set up to run on AS IoP machine.
Test gstlal-cal pipeline on an AS machine
- As a prerequisite to this task, DT reproduced a portion of LIGO h(t) from raw data with gstlal-calibration-v1.1.8 (w/ a "docker-gstlal-cold" image). This test was done on a private server (2-cores on Intel Xeon 2.4GHz, 8GB RAM).
- Downloaded a few hours of bKAGRA Phase-1 data on AS machine.
- (doing) Will try to reproduce C00 with gstlal-calibration. Details in
  - gstlal-calibration testing on AS machine (cal-iop)
- (to do) Verify the output of gstlal-calibration.
Install prerequisites and gstlal-cal package on a machine at KAGRA
- (done, entry from June 7) SH made progress on this, see gstlal installation.
Test the pipeline on a KAGRA machine gstlal tests
- to do
Documentation
- (doing) Started documenting the work in KAGRA-AS group's Gitlab repositories:
  - containerized "cold" version of lalsuite containerized "cold" version of gstlal-calibration gstlal-calibration testing on AS machine (cal-iop)

Progress so far (June 7):

Ask LIGO CAL team for the detailed information of gstlal-calibration
- Communication with people working on LIGO gstlal has been initiated.
Find names and versions of all prerequisite libraries,
- SH made progress on this, see gstlal installation
Install prerequisites and gstlal-cal package on a machine at AS
- As a prerequisite to this task, SH worked on installation process for CentOS, DT for Debian.
- The goal is to have a better understanding of what is happening under the hood when the gstlal-calibration package is compiled.
Test gstlal-cal pipeline on an AS machine
- As a prerequisite to this task, DT is currently trying to reproduce LIGO h(t) from raw data.
- A sub-goal of this task is to come up with a gstlal code testing procedure.
- DT addressed some questions to LIGO gstlal people.
Install prerequisites and gstlal-cal package on a machine at KAGRA
- SH made progress on this, see gstlal installation
Test the pipeline on a KAGRA machine gstlal tests
- to do

New tasks:

Put more effort on documentation.

KAGRA/Subgroups/CAL/Low-latency_and_offline (last edited 2020-07-25 03:17:08 by darkhan.tuyenbayev)

-  ⇤ ← Revision 13 as of 2019-07-12 17:16:36 → 
  Size: 6377
  Editor: darkhan.tuyenbayev
  Comment:
+   ← Revision 22 as of 2020-03-16 19:34:52 → ⇥
  Size: 10107
  Editor: takahiro.sawada
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 1:
+= Prepare h(t) reconstruction review =

LIGO time-domain calibration review wiki: https://wiki.ligo.org/Calibration/TDCalibReview

= Calibration State Vector (K1:DCS-CALIB_STATE_VECTOR_C10) =
(Under preparation. It's planned to be used in O3b)
||<tablewidth="786px">'''bit<<BR>>'''||'''Identifier<<BR>>'''||'''Description<<BR>>'''||
||  0 || `HOFT_OK` || $h(t)$ was successfully computed (logical AND of bits ...)||
||  1 || `OBS_INTENT` || The ``observation intent'' is set by the operator ||
||  2 || `OBS_READY_OR_OBSERVING` || Interferometer is in ``observation ready'' mode: the LSC_LOCK+ guardian is in the nominal (observation) state. ||
||  3 || `NO_STOCHASTIC_HW_INJ` || No stochastic hardware injections present ||
||  4 || `NO_CBC_HW_INJ` || No compact binary coalescence hardware injections present ||
||  5 || `NO_BURST_HW_INJ` || No burst hardware injections present ||
||  6 || `NO_DETCHAR_HW_INJ` || No detector characterization hardware injections present ||
||  7 || `NO_CW_HW_INJ` || No continuous wave characterization hardware injections present ||
||  8 || `TM_LINE_COH_OK` || TM line coherence OK ||
||  9 || `IM_LINE_COH_OK` || IM line coherence OK ||
||  10 || `MN_LINE_COH_OK` || MN line coherence OK ||
||  11 || `PCAL1_LINE_COH_OK` || PCAL1 line coherence OK ||
||  12 || `PCAL2_LINE_COH_OK` || PCAL2 line coherence OK ||
||  13 || `PCAL3_LINE_COH_OK` || PCAL3 line coherence OK ||
||  14 || `PCAL4_LINE_COH_OK` || PCAL4 line coherence OK ||
||  15 || `PCAL5_LINE_COH_OK` || PCAL5 line coherence OK ||
||  16 || `SMOOTH_KAPPA_TM_OK` || TDCF smoothed $\kappa_\mathrm{TM}$ values are in the expected range ||
||  17 || `SMOOTH_KAPPA_IM_OK` || TDCF smoothed $\kappa_\mathrm{IM}$ values are in the expected range ||
||  18 || `SMOOTH_KAPPA_MN_OK` || TDCF smoothed $\kappa_\mathrm{MN}$ values are in the expected range ||
||  19 || `SMOOTH_KAPPA_C_OK` || TDCF smoothed $\kappa_\mathrm{C}$ value is in the expected range ||
||  20 || `SMOOTH_F_CC_OK` || TDCF smoothed $f_\mathrm{cc}$ value is in the expected range ||
||  21 || `SMOOTH_F_S_OK` || (reserved) TDCF smoothed $f_\mathrm{S}$ value is in the expected range ||
||  22 || `SMOOTH_Q_S_OK` || (reserved) TDCF smoothed $Q_\mathrm{S}$ value is in the expected range ||
||  23 || `TDCFS_APPLIED` || Time-dependent calibration factors are being applied ||
||  24 || `LINE_SUBTRACTION_APPLIED` || (reserved) Line subtraction are being applied ||
||  25 || `NO_GAP` || (reserved) Input data has no gaps, underflow, or overflow values ||
||  26 || (not in use) || (not in use) ||
||  27 || (not in use) || (not in use) ||
||  28 || (not in use) || (not in use) ||
||  29 || (not in use) || (not in use) ||
||  30 || (not in use) || (not in use) ||
||  31 || (not in use) || (not in use) ||


Some channels were copied from DET channels

K1:DET-DQ_STATE_VECTOR -> CAL state vector

   # bit  2 : SDF flag (= K1:GRD-SYS_SDF_OK) ->  # bit 2

   # bit  7 : NO Stoch injection ->  # bit 3 

   # bit  8 : NO CBC injection ->  # bit 4

   # bit  9 : NO Burst injection ->  # bit 5

   # bit 10 : NO DET injection ->  # bit 6

   # bit 11 : NO CW injection ->  # bit 7

K1:DET-OBS_INTENT -> CAL state vector: # bit 1
-Line 3:
+Line 65:
-||<tablewidth="786px">Channel Name||Rate||Format||Description||

||<tablewidth="786px">K1:CAL-CS_PROC_MICH_RESIDUAL_DBL_DQ||16384||F64LE||MICH error||
||K1:CAL-CS_PROC_MICH_DELTA_CTRL_TM_DBL_DQ||4096||F64LE||MICH TM stage control||
||K1:CAL-CS_PROC_MICH_DELTA_CTRL_IM_DBL_DQ||4096||F64LE||MICH IM stage control||
||K1:CAL-CS_PROC_MICH_DISPLACEMENT_DQ||16384||F32LE||MICH displacement (optional)||
||K1:CAL-CS_PROC_IMC_RESIDUAL_DQ||16384||F32LE||IMC error||
||K1:CAL-CS_PROC_IMC_DELTA_CTRL_AOM_DQ||4096||F32LE||IMC AOM frequency control||
||K1:CAL-CS_PROC_IMC_DELTA_CTRL_TM_DQ||4096||F32LE||IMC TM stage control||
||K1:CAL-CS_PROC_IMC_FREQUENCY_DQ||16384||F32LE||Laser frequency (optional)||
||GRD-LSC_LOCK_OK||16||F32LE||Guardian OK status||
||GRD-IO_OK||16||F32LE||Guardian OK status||
+||<tablewidth="786px">'''Channel Name<<BR>>'''||'''Rate<<BR>>'''||'''Format<<BR>>'''||'''Description<<BR>>'''||
||`K1:CAL-CS_PROC_MICH_RESIDUAL_DBL_DQ` ||16384 ||F64LE ||MICH error ||
||`K1:CAL-CS_PROC_MICH_DELTA_CTRL_TM_DBL_DQ` ||4096 ||F64LE ||MICH TM stage control ||
||`K1:CAL-CS_PROC_MICH_DELTA_CTRL_IM_DBL_DQ` ||4096 ||F64LE ||MICH IM stage control ||
||`K1:CAL-CS_PROC_MICH_DISPLACEMENT_DQ` ||16384 ||F32LE ||MICH displacement (optional) ||
||`K1:CAL-CS_PROC_IMC_RESIDUAL_DQ` ||16384 ||F32LE ||IMC error ||
||`K1:CAL-CS_PROC_IMC_DELTA_CTRL_AOM_DQ` ||4096 ||F32LE ||IMC AOM frequency control ||
||`K1:CAL-CS_PROC_IMC_DELTA_CTRL_TM_DQ` ||4096 ||F32LE ||IMC TM stage control ||
||`K1:CAL-CS_PROC_IMC_FREQUENCY_DQ` ||16384 ||F32LE ||Laser frequency (optional) ||
||`K1:GRD-LSC_LOCK_OK` ||16 ||F32LE ||LSC Guardian OK status ||
||`K1:GRD-LSC_LOCK_STATE_N` ||16 ||F32LE ||LSC lock state ||
||`K1:GRD-IO_OK` ||16 ||F32LE ||IMC Guardian OK status ||
||`K1:GRD-IO_STATE_N` ||16 ||F32LE ||IMC state ||
-Line 18:
+Line 80:
-where F32LE and F64LE means: floating-point 32-bit and 64-bit numbers in Little-Endian format.
+Where F32LE and F64LE means: floating-point 32-bit and 64-bit numbers in Little-Endian format.

Channels marked "optional" will not be used in the low-latency and offline computations, `K1:CAL-CS_PROC_MICH_DISPLACEMENT_DQ` and `K1:CAL-CS_PROC_IMC_FREQUENCY_DQ` in particular are the channels calibrated with the online system.

----------------
-Line 56:
+Line 124:
-   . [[https://gitlab.com/kagra-as/docker-lalsuite-cold|containerized "cold" version of lalsuite]]
   [[https://gitlab.com/kagra-as/docker-gstlal-cold|containerized "cold" version of gstlal-calibration]]
+   . [[https://gitlab.com/kagra-as/docker-lalsuite-cold|containerized "cold" version of lalsuite]] [[https://gitlab.com/kagra-as/docker-gstlal-cold|containerized "cold" version of gstlal-calibration]]
-Line 71:
+Line 138:
-   . [[https://gitlab.com/kagra-as/docker-lalsuite-cold|containerized "cold" version of lalsuite]]
   [[https://gitlab.com/kagra-as/docker-gstlal-cold|containerized "cold" version of gstlal-calibration]]
   [[https://gitlab.com/kagra-as/low-latency-cal/test-gstlal-on-cal-iop|gstlal-calibration testing on AS machine (cal-iop)]]
+   . [[https://gitlab.com/kagra-as/docker-lalsuite-cold|containerized "cold" version of lalsuite]] [[https://gitlab.com/kagra-as/docker-gstlal-cold|containerized "cold" version of gstlal-calibration]] [[https://gitlab.com/kagra-as/low-latency-cal/test-gstlal-on-cal-iop|gstlal-calibration testing on AS machine (cal-iop)]]