[Alex, Sidd]
Conda environments
Siddharth was able to successfully install conda and identical conda environments on the k1sum1, k1det0, and k1det1 machines. (The k1sum0 was already handled by me yesterday.) Siddharth will provide a tutorial and I can provide documentation for how to use and update these environments, and how to maintain package versions in tandem with LIGO and our release schedule.
Datafind
With Joe Areeda’s help, we were able to set up a datafind server to locate KAGRA frame files. The server is hosted on the LDVW machine at Caltech, but it only accesses the shared NFS file paths at KAGRA to get frame locations, it never actually touches the data itself. With this server, we should soon be able to read frame data from the /frame0 or /frame1 archives directly, rather than rely on NDS.
However, I have discovered an issue with KAGRA’s setup that explains at least some of the summary page failures you’ve been seeing, which is described below.
Summary page failures
Many KAGRA summary pages are failing because they run into data access issues caused by the redundancy in frame files. Essentially, the issue is this: say you want to find half an hour of minute trend frames at the start the day today, UTC. With datafind, this query would look like:
In[5]: cache = gwdatafind.find_urls('K', 'K1_M', 1250899218, 1250901018)In[6]: cache Out[6]: ['file://localhost/frame0/trend/minute/12508/K-K1_M-1250899200-3600.gwf', 'file://localhost/frame1/trend/minute/12508/K-K1_M-1250899200-3600.gwf']
Notice that this query returns two files covering the exact same time period, one in /frame0 and the other in /frame1. When you try to pass this to gwpy, you get an error:
In[7]: TimeSeries.read(cache, 'K1:LSC-DARM_OUT_DQ.mean')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
....
ValueError: Cannot append discontiguous TimeSeries
TimeSeries 1 span: [1250899200.0 ... 1250902800.0)
TimeSeries 2 span: [1250899200.0 ... 1250902800.0)If we can get the server to return one and only one set of frames, and fall back to the other set only if there are gaps, then we will be in business. I’ve asked Joe Areeda if he can help with this, and we will likely get a reply overnight after he wakes up and settles in to work.