##master-page: ##master-date:Unknown-Date #format wiki #language en = NoEMi = The Noise Frequency Event Miner (NoEMi) is a Virgo software tool dedicated to the monitoring and identification of spectral noise lines. It uses codes for CW analysis to find and track spectral lines found in a given list of channels. It also looks for spectral features in coincidence between the "main channel" and the "auxiliary channels". NoEMi documentation: https://apps.virgo-gw.eu/noemi/docs/index.html This version, has 3 main components: * the NoEMi events database (DB); * the Line-Finder script (LF); * the web user interface (WUI). In addition to NoEMi the LinesDB will be installed soon. == Requirements == Main requirements: {{{ HTCondor MySQL Python Apache HTTP server PHP }}} The following Python libraries can be installed with [[https://pypi.org/|pip]] {{{ pip install htcondor pip install pymysql pip install numpy }}} The [[https://www.php.net/manual/en/book.pdo.php|PHP Data Objects module]] could be necessary. MySQL has been installed from scratch, the admin user credentials are (this is the root user for any MySQL operation): {{{ Username: root Password: user_admin_pass }}} == Get NoEMi == {{{#!wiki The full NoEMi package is available in the [[https://apps.virgo-gw.eu/noemi/docs/usage/getting.html|Virgo SVN software repository]] Note that, to access the repository, you will need to use your EGO Active Directory credentials. More information on obtaining these is available [[https://wiki.virgo-gw.eu/IT/Cascina_EGO-Virgo_Accounts|here]]. }}} * Download the full package {{{ svn co https://svn.ego-gw.it/svn/advsw/Noemi/branches/v2/trunk \ NoEMi }}} * The folder will contain the three components of NoEMi and the documentation {{{ NoEMi/ /db --> MySQL Database /doc --> Documentation /noemi --> Line finder python scripts /wui --> Web user interface }}} == Host machines and Users == The 3 components of NoEMi: the DB, the Line finder and the WUI are installed on the following machines {{{ DB host: localhost LF hosts: localhost WUI host: localhost --> web page work in progress }}} The following MySQL users have been created: 1. The web user interface (WUI) MySQL user {{{ Username: noemi_wui Password: noemi_wui_pass }}} 2. The Line-Finder (LF) MySQL user {{{ Username: noemi_user Password: noemi_user_pass }}} == Changes for KAGRA == The "K1" detector case has been added to the main SFDB (Short Fourier Transform DataBase) executable. The SFDBs are used also for CW analysis in Virgo. The list of channels/sampling frequency of the channel used is: {{{ K1:CAL-CS_PROC_C00_STRAIN_DBL_DQ 16384 K1:PEM-ACC_PSL_TABLE_PSL1_Z_OUT_DQ 2048 K1:PEM-ACC_PSL_TABLE_PSL2_Z_OUT_DQ 2048 K1:PEM-ACC_PSL_TABLE_PSL3_Z_OUT_DQ 2048 K1:PEM-MIC_PSL_TABLE_PSL4_Z_OUT_DQ 16384 K1:PEM-ACC_MCF_TABLE_REFL_Z_OUT_DQ 2048 K1:PEM-MIC_MCF_TABLE_REFL_Z_OUT_DQ 16384 K1:PEM-MIC_EXC_BOOTH_EXC_Z_OUT_DQ 2048 K1:PEM-MIC_EYC_BOOTH_EYC_Z_OUT_DQ 2048 K1:PEM-ACC_OMC_TABLE_AS_Z_OUT_DQ 2048 K1:PEM-MIC_OMC_TABLE_AS_Z_OUT_DQ 2048 K1:PEM-MIC_BS_TABLE_POP_Z_OUT_DQ 16384 K1:PEM-ACC_BS_TABLE_POP_Z_OUT_DQ 2048 K1:PEM-MAG_BS_BOOTH_BS_X_OUT_DQ 16384 K1:PEM-MAG_BS_BOOTH_BS_Y_OUT_DQ 16384 K1:PEM-MAG_BS_BOOTH_BS_Z_OUT_DQ 16384 K1:PEM-MAG_EXC_BOOTH_EXC_X_OUT_DQ 16384 K1:PEM-MAG_EXC_BOOTH_EXC_Y_OUT_DQ 16384 K1:PEM-MAG_EXC_BOOTH_EXC_Z_OUT_DQ 16384 K1:PEM-MAG_EYC_BOOTH_EYC_X_OUT_DQ 16384 K1:PEM-MAG_EYC_BOOTH_EYC_Y_OUT_DQ 16384 K1:PEM-MAG_EYC_BOOTH_EYC_Z_OUT_DQ 16384 }}} == Use NoEMi == You can launch NoEMi with the command '''python noemi''' followed by some arguments. All the possible arguments are listed below: {{{ python noemi --source_ffl_full_path SOURCE_FFL_FULL_PATH --detector {H1,L1,V1} [--analysis_period_start_date ANALYSIS_PERIOD_START_DATE] [--analysis_period_end_date ANALYSIS_PERIOD_END_DATE] [--channel_list_full_path CHANNEL_LIST_FULL_PATH] [--copy_frames_locally] [--prevent_tear_down] [--create_ligo_ffl] [--sfdb_resource_creation_level {0,1,2,3}] [--sfdb_multiplication_factor SFDB_MULTIPLICATION_FACTOR] [--science_segment_list_full_path SCIENCE_SEGMENT_LIST_FULL_PATH] [--resampling_factor RESAMPLING_FACTOR] [--sfdb_verbosity_level {0,1,2,3}] [--fft_interlaced] [--max_chunks MAX_CHUNKS] [--fft_avg_spectrum FFT_AVG_SPECTRUM] [--data_window_type {0,1,2,3,4,5}] [--high_pass_filter_cut_freq HIGH_PASS_FILTER_CUT_FREQ] [--sub_band_veto_subsampling_factor SUB_BAND_VETO_SUBSAMPLING_FACTOR] [--max_fft_in_sfdb MAX_FFT_IN_SFDB] [--distribution_min_frequency DISTRIBUTION_MIN_FREQUENCY] [--distribution_max_frequency DISTRIBUTION_MAX_FREQUENCY] [--fft_resolution FFT_RESOLUTION] [--n_sigma_cut N_SIGMA_CUT] [--prevent_line_coalescence] [--persistence_line_cut_threshold PERSISTENCE_LINE_CUT_THRESHOLD] [--critical_ratio_line_cut_threshold CRITICAL_RATIO_LINE_CUT_THRESHOLD] [--prevent_f_loop] }}} Not all the arguments are compulsory, see https://apps.virgo-gw.eu/noemi/docs/usage/running.html for argument description. In the example below we are analyzing data from channels in "'''noemi/utils/default_channel_list.txt'''", using data from '''2019-10-27''' to '''2019-10-28'''. Check that the .gwf frames files in "'''noemi/utils/ffl_file.ffl'''" are within the desired dates: {{{ python noemi \ --channel_list_full_path noemi/utils/default_channel_list.txt \ --source_ffl_full_path noemi/utils/ffl_file.ffl --detector K1 \ --sub_band_veto_subsampling_factor 1 \ --sfdb_resource_creation_level 1 \ --analysis_period_start_date 2019-10-27 \ --analysis_period_end_date 2019-10-28 --max_fft_in_sfdb 100 \ --fft_resolution 1 --n_sigma_cut 3 \ --persistence_line_cut_threshold 0.1 \ --critical_ratio_line_cut_threshold 3.5 --prevent_f_loop; }}} An ffl file is simply the list of .gwf frame files where in the first column you have the path to the .gwf file and as a second column the starting GPS time of the frame as: {{{ ... /frame1/full/12562/K-K1_C-1256200448-32.gwf 1256200448 /frame1/full/12562/K-K1_C-1256200480-32.gwf 1256200480 /frame1/full/12562/K-K1_C-1256200512-32.gwf 1256200512 /frame1/full/12562/K-K1_C-1256200544-32.gwf 1256200544 /frame1/full/12562/K-K1_C-1256200576-32.gwf 1256200576 /frame1/full/12562/K-K1_C-1256200608-32.gwf 1256200608 /frame1/full/12562/K-K1_C-1256200640-32.gwf 1256200640 /frame1/full/12562/K-K1_C-1256200672-32.gwf 1256200672 /frame1/full/12562/K-K1_C-1256200704-32.gwf 1256200704 /frame1/full/12562/K-K1_C-1256200736-32.gwf 1256200736 /frame1/full/12562/K-K1_C-1256200768-32.gwf 1256200768 /frame1/full/12562/K-K1_C-1256200800-32.gwf 1256200800 /frame1/full/12562/K-K1_C-1256200832-32.gwf 1256200832 /frame1/full/12562/K-K1_C-1256200864-32.gwf 1256200864 ... }}} To run noemi on a longer period use "'''run_noemi_period.sh'''" in noemi/utils/run_noemi_period.sh. You need to provide the list of ffl files divided per day as: {{{ ... /2017-10-01_raw.ffl /2017-10-02_raw.ffl /2017-10-03_raw.ffl /2017-10-04_raw.ffl /2017-10-05_raw.ffl /2017-10-06_raw.ffl /2017-10-07_raw.ffl ... }}} == Staus == What's working * The NoEMi events database is installed * The Line finder scripts are installed * The WUI output pages are not pubicly visible (coordination with Gary Hemming ongoing) * Communication between the LF scripts and Condor BQS is working * Communication between the LF scripts and the NoEMi events MySQL DB is working * The main executable SFDB has been modified and is working (the detectors coordinates need to be updated, using virgo location for test) What's missing * Made the WUI page visible to users (we need to find the best authentication system, and choose the web page name) * The last item will be the installation of the LineDB