Differences between revisions 16 and 17
Revision 16 as of 2019-11-01 10:25:23
Size: 7845
Comment:
Revision 17 as of 2019-11-01 10:26:25
Size: 7892
Comment:
Deletions are marked like this. Additions are marked like this.
Line 118: Line 118:
You can launch NoEMi with the command python noemi followed by some arguments: You can launch NoEMi with the command "python noemi" followed by some arguments. All the possible arguments are listed below:

NoEMi

The Noise Frequency Event Miner (NoEMi) is a Virgo software tool dedicated to the monitoring and identification of spectral noise lines.

It uses codes for CW analysis to find and track spectral lines found in a given list of channels. It also looks for spectral features in coincidence between the "main channel" and the "auxiliary channels".

NoEMi documentation: https://apps.virgo-gw.eu/noemi/docs/index.html

This version, has 3 main components:

  • the NoEMi events database (DB);
  • the Line-Finder script (LF);
  • the web user interface (WUI).

In addition to NoEMi the LinesDB will be installed soon.

Requirements

Main requirements:

HTCondor
MySQL
Python
Apache HTTP server
PHP

The following Python libraries can be installed with pip

pip install htcondor
pip install pymysql
pip install numpy

The PHP Data Objects module could be necessary.

MySQL has been installed from scratch, the admin user credentials are (this is the root user for any MySQL operation):

Username: root
Password: user_admin_pass

Get NoEMi

The full NoEMi package is available in the Virgo SVN software repository

Note that, to access the repository, you will need to use your EGO Active Directory credentials. More information on obtaining these is available here.

  • Download the full package

svn co https://svn.ego-gw.it/svn/advsw/Noemi/branches/v2/trunk \
          NoEMi
  • The folder will contain the three components of NoEMi and the documentation

     NoEMi/
          /db      --> MySQL Database
          /doc     --> Documentation
          /noemi   --> Line finder python scripts
          /wui     --> Web user interface

Host machines and Users

The 3 components of NoEMi: the DB, the Line finder and the WUI are installed on the following machines

 DB host: localhost
 LF hosts: localhost
 WUI host: localhost --> web page work in progress

The following MySQL users have been created:

1. The web user interface (WUI) MySQL user

Username: noemi_wui
Password: noemi_wui_pass

2. The Line-Finder (LF) MySQL user

Username: noemi_user
Password: noemi_user_pass

Changes for KAGRA

The "K1" detector case has been added to the main SFDB (Short Fourier Transform DataBase) executable. The SFDBs are used also for CW analysis in Virgo.

The list of channels/sampling frequency of the channel used is:

K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ 16384
K1:PEM-ACC_PSL_TABLE_PSL1_Z_OUT_DQ 2048
K1:PEM-ACC_PSL_TABLE_PSL2_Z_OUT_DQ 2048
K1:PEM-ACC_PSL_TABLE_PSL3_Z_OUT_DQ 2048
K1:PEM-MIC_PSL_TABLE_PSL4_Z_OUT_DQ 16384
K1:PEM-ACC_MCF_TABLE_REFL_Z_OUT_DQ 2048
K1:PEM-MIC_MCF_TABLE_REFL_Z_OUT_DQ 16384
K1:PEM-MIC_EXC_BOOTH_EXC_Z_OUT_DQ 2048
K1:PEM-MIC_EYC_BOOTH_EYC_Z_OUT_DQ 2048
K1:PEM-ACC_OMC_TABLE_AS_Z_OUT_DQ 2048
K1:PEM-MIC_OMC_TABLE_AS_Z_OUT_DQ 2048
K1:PEM-MIC_BS_TABLE_POP_Z_OUT_DQ 16384
K1:PEM-ACC_BS_TABLE_POP_Z_OUT_DQ 2048
K1:PEM-MAG_BS_BOOTH_BS_X_OUT_DQ 16384
K1:PEM-MAG_BS_BOOTH_BS_Y_OUT_DQ 16384
K1:PEM-MAG_BS_BOOTH_BS_Z_OUT_DQ 16384
K1:PEM-MAG_EXC_BOOTH_EXC_X_OUT_DQ 16384
K1:PEM-MAG_EXC_BOOTH_EXC_Y_OUT_DQ 16384
K1:PEM-MAG_EXC_BOOTH_EXC_Z_OUT_DQ 16384
K1:PEM-MAG_EYC_BOOTH_EYC_X_OUT_DQ 16384
K1:PEM-MAG_EYC_BOOTH_EYC_Y_OUT_DQ 16384
K1:PEM-MAG_EYC_BOOTH_EYC_Z_OUT_DQ 16384

Use NoEMi

You can launch NoEMi with the command "python noemi" followed by some arguments. All the possible arguments are listed below:

python noemi --source_ffl_full_path SOURCE_FFL_FULL_PATH --detector
             {H1,L1,V1}
             [--analysis_period_start_date ANALYSIS_PERIOD_START_DATE]
             [--analysis_period_end_date ANALYSIS_PERIOD_END_DATE]
             [--channel_list_full_path CHANNEL_LIST_FULL_PATH]
             [--copy_frames_locally] [--prevent_tear_down] [--create_ligo_ffl]
             [--sfdb_resource_creation_level {0,1,2,3}]
             [--sfdb_multiplication_factor SFDB_MULTIPLICATION_FACTOR]
             [--science_segment_list_full_path SCIENCE_SEGMENT_LIST_FULL_PATH]
             [--resampling_factor RESAMPLING_FACTOR]
             [--sfdb_verbosity_level {0,1,2,3}] [--fft_interlaced]
             [--max_chunks MAX_CHUNKS] [--fft_avg_spectrum FFT_AVG_SPECTRUM]
             [--data_window_type {0,1,2,3,4,5}]
             [--high_pass_filter_cut_freq HIGH_PASS_FILTER_CUT_FREQ]
             [--sub_band_veto_subsampling_factor SUB_BAND_VETO_SUBSAMPLING_FACTOR]
             [--max_fft_in_sfdb MAX_FFT_IN_SFDB]
             [--distribution_min_frequency DISTRIBUTION_MIN_FREQUENCY]
             [--distribution_max_frequency DISTRIBUTION_MAX_FREQUENCY]
             [--fft_resolution FFT_RESOLUTION] [--n_sigma_cut N_SIGMA_CUT]
             [--prevent_line_coalescence]
             [--persistence_line_cut_threshold PERSISTENCE_LINE_CUT_THRESHOLD]
             [--critical_ratio_line_cut_threshold CRITICAL_RATIO_LINE_CUT_THRESHOLD]
             [--prevent_f_loop]

Not all the arguments are compulsory, see https://apps.virgo-gw.eu/noemi/docs/usage/running.html for argument description.

In the example below we are analyzing data from channels in "noemi/utils/default_channel_list.txt", using data from 2019-10-27 to 2019-10-28. Check that the .gwf frames files in "noemi/utils/ffl_file.ffl" are within the desired dates:

python noemi \
    --channel_list_full_path noemi/utils/default_channel_list.txt \
    --source_ffl_full_path noemi/utils/ffl_file.ffl --detector K1 \
    --sub_band_veto_subsampling_factor 1 \
    --sfdb_resource_creation_level 1 \
    --analysis_period_start_date 2019-10-27 \
    --analysis_period_end_date 2019-10-28 --max_fft_in_sfdb 100 \
    --fft_resolution 1 --n_sigma_cut 3 \
    --persistence_line_cut_threshold 0.1 \
    --critical_ratio_line_cut_threshold 3.5 --prevent_f_loop;

An ffl file is simply the list of .gwf frame files where in the first column you have the path to the .gwf file and as a second column the starting GPS time of the frame as:

...
frame1/full/12562/K-K1_C-1256200448-32.gwf 1256200448
frame1/full/12562/K-K1_C-1256200480-32.gwf 1256200480
frame1/full/12562/K-K1_C-1256200512-32.gwf 1256200512
frame1/full/12562/K-K1_C-1256200544-32.gwf 1256200544
frame1/full/12562/K-K1_C-1256200576-32.gwf 1256200576
frame1/full/12562/K-K1_C-1256200608-32.gwf 1256200608
frame1/full/12562/K-K1_C-1256200640-32.gwf 1256200640
frame1/full/12562/K-K1_C-1256200672-32.gwf 1256200672
frame1/full/12562/K-K1_C-1256200704-32.gwf 1256200704
frame1/full/12562/K-K1_C-1256200736-32.gwf 1256200736
frame1/full/12562/K-K1_C-1256200768-32.gwf 1256200768
frame1/full/12562/K-K1_C-1256200800-32.gwf 1256200800
frame1/full/12562/K-K1_C-1256200832-32.gwf 1256200832
frame1/full/12562/K-K1_C-1256200864-32.gwf 1256200864
...

Staus

  • The NoEMi events database is installed
  • The Line finder scripts are installed
  • The WUI output pages are not pubicly visible (coordination with Gary Hemming ongoing)
  • Communication between the LF scripts and Condor BQS is working
  • Communication between the LF scripts and the NoEMi events MySQL DB is working

What's missing

  • The main executable for the creation of SFDBs needs minor modification this is a C compilable code, test ongoing
  • Automatic creation of ffl files (). A script already exist, needs to be adapted.
  • Made the WUI page visible to users (we need to find the best authentication system, and choose the web page name)
  • The last item will be the installation of the LineDB

KAGRA/Subgroups/DET/NoEMi (last edited 2019-11-03 15:00:56 by ornella.piccinni)