##master-page:
##master-date:Unknown-Date
#format wiki
#language en
= NoEMi =
The Noise Frequency Event Miner (NoEMi) is a Virgo software tool dedicated to the monitoring and identification of spectral noise lines.
It uses codes for CW analysis to find and track spectral lines found in a given list of channels. It also looks for spectral features in coincidence between the "main channel" and the "auxiliary channels".
NoEMi documentation: https://apps.virgo-gw.eu/noemi/docs/index.html
This version, has 3 main components:
* the NoEMi events database (DB);
* the Line-Finder script (LF);
* the web user interface (WUI).
In addition to NoEMi the LinesDB will be installed soon.
== Requirements ==
Main requirements:
{{{
HTCondor
MySQL
Python
Apache HTTP server
PHP
}}}
The following Python libraries can be installed with [[https://pypi.org/|pip]]
{{{
pip install htcondor
pip install pymysql
pip install numpy
}}}
The [[https://www.php.net/manual/en/book.pdo.php|PHP Data Objects module]] could be necessary.
MySQL has been installed from scratch, the admin user credentials are (this is the root user for any MySQL operation):
{{{
Username: root
Password: user_admin_pass
}}}
== Get NoEMi ==
{{{#!wiki
The full NoEMi package is available in the [[https://apps.virgo-gw.eu/noemi/docs/usage/getting.html|Virgo SVN software repository]]
Note that, to access the repository, you will need to use your EGO Active Directory credentials. More information on obtaining these is available [[https://wiki.virgo-gw.eu/IT/Cascina_EGO-Virgo_Accounts|here]].
}}}
* Download the full package
{{{
svn co https://svn.ego-gw.it/svn/advsw/Noemi/branches/v2/trunk \
NoEMi
}}}
* The folder will contain the three components of NoEMi and the documentation
{{{
NoEMi/
/db --> MySQL Database
/doc --> Documentation
/noemi --> Line finder python scripts
/wui --> Web user interface
}}}
== Host machines and Users ==
The 3 components of NoEMi: the DB, the Line finder and the WUI are installed on the following machines
{{{
DB host: localhost
LF hosts: localhost
WUI host: localhost --> web page work in progress
}}}
The following MySQL users have been created:
1. The web user interface (WUI) MySQL user
{{{
Username: noemi_wui
Password: noemi_wui_pass
}}}
2. The Line-Finder (LF) MySQL user
{{{
Username: noemi_user
Password: noemi_user_pass
}}}
== Changes for KAGRA ==
The "K1" detector case has been added to the main SFDB (Short Fourier Transform DataBase) executable. The SFDBs are used also for CW analysis in Virgo.
The list of channels/sampling frequency of the channel used is:
{{{
K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ 16384
K1:PEM-ACC_PSL_TABLE_PSL1_Z_OUT_DQ 2048
K1:PEM-ACC_PSL_TABLE_PSL2_Z_OUT_DQ 2048
K1:PEM-ACC_PSL_TABLE_PSL3_Z_OUT_DQ 2048
K1:PEM-MIC_PSL_TABLE_PSL4_Z_OUT_DQ 16384
K1:PEM-ACC_MCF_TABLE_REFL_Z_OUT_DQ 2048
K1:PEM-MIC_MCF_TABLE_REFL_Z_OUT_DQ 16384
K1:PEM-MIC_EXC_BOOTH_EXC_Z_OUT_DQ 2048
K1:PEM-MIC_EYC_BOOTH_EYC_Z_OUT_DQ 2048
K1:PEM-ACC_OMC_TABLE_AS_Z_OUT_DQ 2048
K1:PEM-MIC_OMC_TABLE_AS_Z_OUT_DQ 2048
K1:PEM-MIC_BS_TABLE_POP_Z_OUT_DQ 16384
K1:PEM-ACC_BS_TABLE_POP_Z_OUT_DQ 2048
K1:PEM-MAG_BS_BOOTH_BS_X_OUT_DQ 16384
K1:PEM-MAG_BS_BOOTH_BS_Y_OUT_DQ 16384
K1:PEM-MAG_BS_BOOTH_BS_Z_OUT_DQ 16384
K1:PEM-MAG_EXC_BOOTH_EXC_X_OUT_DQ 16384
K1:PEM-MAG_EXC_BOOTH_EXC_Y_OUT_DQ 16384
K1:PEM-MAG_EXC_BOOTH_EXC_Z_OUT_DQ 16384
K1:PEM-MAG_EYC_BOOTH_EYC_X_OUT_DQ 16384
K1:PEM-MAG_EYC_BOOTH_EYC_Y_OUT_DQ 16384
K1:PEM-MAG_EYC_BOOTH_EYC_Z_OUT_DQ 16384
}}}
== Use NoEMi ==
You can launch NoEMi with the command '''python noemi''' followed by some arguments. All the possible arguments are listed below:
{{{
python noemi --source_ffl_full_path SOURCE_FFL_FULL_PATH --detector
{H1,L1,V1}
[--analysis_period_start_date ANALYSIS_PERIOD_START_DATE]
[--analysis_period_end_date ANALYSIS_PERIOD_END_DATE]
[--channel_list_full_path CHANNEL_LIST_FULL_PATH]
[--copy_frames_locally] [--prevent_tear_down] [--create_ligo_ffl]
[--sfdb_resource_creation_level {0,1,2,3}]
[--sfdb_multiplication_factor SFDB_MULTIPLICATION_FACTOR]
[--science_segment_list_full_path SCIENCE_SEGMENT_LIST_FULL_PATH]
[--resampling_factor RESAMPLING_FACTOR]
[--sfdb_verbosity_level {0,1,2,3}] [--fft_interlaced]
[--max_chunks MAX_CHUNKS] [--fft_avg_spectrum FFT_AVG_SPECTRUM]
[--data_window_type {0,1,2,3,4,5}]
[--high_pass_filter_cut_freq HIGH_PASS_FILTER_CUT_FREQ]
[--sub_band_veto_subsampling_factor SUB_BAND_VETO_SUBSAMPLING_FACTOR]
[--max_fft_in_sfdb MAX_FFT_IN_SFDB]
[--distribution_min_frequency DISTRIBUTION_MIN_FREQUENCY]
[--distribution_max_frequency DISTRIBUTION_MAX_FREQUENCY]
[--fft_resolution FFT_RESOLUTION] [--n_sigma_cut N_SIGMA_CUT]
[--prevent_line_coalescence]
[--persistence_line_cut_threshold PERSISTENCE_LINE_CUT_THRESHOLD]
[--critical_ratio_line_cut_threshold CRITICAL_RATIO_LINE_CUT_THRESHOLD]
[--prevent_f_loop]
}}}
Not all the arguments are compulsory, see https://apps.virgo-gw.eu/noemi/docs/usage/running.html for argument description.
In the example below we are analyzing data from channels in "'''noemi/utils/default_channel_list.txt'''", using data from '''2019-10-27''' to '''2019-10-28'''. Check that the .gwf frames files in "'''noemi/utils/ffl_file.ffl'''" are within the desired dates:
{{{
python noemi \
--channel_list_full_path noemi/utils/default_channel_list.txt \
--source_ffl_full_path noemi/utils/ffl_file.ffl --detector K1 \
--sub_band_veto_subsampling_factor 1 \
--sfdb_resource_creation_level 1 \
--analysis_period_start_date 2019-10-27 \
--analysis_period_end_date 2019-10-28 --max_fft_in_sfdb 100 \
--fft_resolution 1 --n_sigma_cut 3 \
--persistence_line_cut_threshold 0.1 \
--critical_ratio_line_cut_threshold 3.5 --prevent_f_loop;
}}}
An ffl file is simply the list of .gwf frame files where in the first column you have the path to the .gwf file and as a second column the starting GPS time of the frame as:
{{{
...
/frame1/full/12562/K-K1_C-1256200448-32.gwf 1256200448
/frame1/full/12562/K-K1_C-1256200480-32.gwf 1256200480
/frame1/full/12562/K-K1_C-1256200512-32.gwf 1256200512
/frame1/full/12562/K-K1_C-1256200544-32.gwf 1256200544
/frame1/full/12562/K-K1_C-1256200576-32.gwf 1256200576
/frame1/full/12562/K-K1_C-1256200608-32.gwf 1256200608
/frame1/full/12562/K-K1_C-1256200640-32.gwf 1256200640
/frame1/full/12562/K-K1_C-1256200672-32.gwf 1256200672
/frame1/full/12562/K-K1_C-1256200704-32.gwf 1256200704
/frame1/full/12562/K-K1_C-1256200736-32.gwf 1256200736
/frame1/full/12562/K-K1_C-1256200768-32.gwf 1256200768
/frame1/full/12562/K-K1_C-1256200800-32.gwf 1256200800
/frame1/full/12562/K-K1_C-1256200832-32.gwf 1256200832
/frame1/full/12562/K-K1_C-1256200864-32.gwf 1256200864
...
}}}
To run noemi on a longer period use "'''run_noemi_period.sh'''" in noemi/utils/run_noemi_period.sh. You need to provide the list of ffl files divided per day as:
{{{
...
<full_path>/2017-10-01_raw.ffl
<full_path>/2017-10-02_raw.ffl
<full_path>/2017-10-03_raw.ffl
<full_path>/2017-10-04_raw.ffl
<full_path>/2017-10-05_raw.ffl
<full_path>/2017-10-06_raw.ffl
<full_path>/2017-10-07_raw.ffl
...
}}}
== Staus ==
What's working
* The NoEMi events database is installed
* The Line finder scripts are installed
* The WUI output pages are not pubicly visible (coordination with Gary Hemming ongoing)
* Communication between the LF scripts and Condor BQS is working
* Communication between the LF scripts and the NoEMi events MySQL DB is working
* The main executable SFDB has been modified and is working (the detectors coordinates need to be updated, using virgo location for test)
What's missing
* Made the WUI page visible to users (we need to find the best authentication system, and choose the web page name)
* The last item will be the installation of the LineDB