Switching on the crates and computer

All the modules necessary for the data acquisition are distributed in 5 crates in total. One is the “LUPO crate” where the main clock is located and connected to the others detectors for synchronization. The FASTER are distributed in for other crates, each one containing the modules for a single layer of NEBULA+.

The crate corresponding to the Layer 1 electronics also holds the clock manager, which is connected to the LUPO. The clock manager is connected to the other layer crate and insure that every FASTER modules are synchronized together.

As some times is needed for the modules to be initialized, one should wait a moment before turning a second crate. The crates should be turned on in this order : LUPO CRATE > Layer 1 > Layer 2 > Layer 3,4.

Operating HV

There is three remote interfaces to control the HV. In all cases the communication is performed via the network interface of the SYX527 crates. The first two methods are graphical based, provided by CAEN company. The third one is an in-house text based approach. We strongly suggest to use this last method.

• Access the crates web interface, using their IP addresses
• Access the crates using the GECO2020 software installed on expandacq computer.
• Use the command line hvm_* programs (see below). You need the description of the HV channels within the crate. Presently this file is called nebula_plus.yamland is located in ~/Programs/hvm/

Web interface

Open a web browser and type the IP addresses of the crate

GECO2020

Start GECO2020 and load the configuration

HVM command line

Set of command line programs based on the CAEN libraries. Executable are installed on expandacq, but for information you can find the last source version here:

https://gitlab.in2p3.fr/np/hvm/-/tree/master

• hvm_print nebula_plus.yaml : print the status of the all channels (on/off, Vmon, Vset, Imon)
• hvm_save nebula_plus.yaml <output_file_name.txt> : save the status of the HV into a text file
• hvm_batch nebula_plus.yaml <list_of_Vmon.txt> : change several Vset for a list of given channels. The <list_of_Vmon.txt> is a text file that contains the label of the channel and its Vset value, one channel per line.
• hvm_groupe nebula_plus.yaml <group name> <on|off> : turn on or off a group of channels

Starting a run in standalone

Coming soon

Starting a run in coupled mode

Coming soon

Converting a run to root

Analysis pipeline

The conversion of a .fast (or fast.gz) file to a root file is a process that involve several operation. Some of them are carried away by fasterac utilities and some by nptool v4 . In order for nptool to read .fast file you need to install the faster nptool plugin and the nebula-plus nptool plugin for analysis of nebula-plus. The following section will detailled what each step is doing. However script exist that will run the entire pipeline automatically (see following sections).

Step 1: grouping of SAMURAI trigger and QDC data

The grouping is an offline software trigger applied to a run. A fast file is readout and for every SAMURAI trigger register in faster DAQ (on channel 189), a new event is created. Since the SAMURAI trigger is coming aroung 800ns after a QDC hit in Nebula-plus, a window of 1us is open before the Trigger. All QDC event inside this window are associated with that entry.

This operation is carried out by the faster_file_group binary from fasterac package.

faster_file_group -l -e"189" -b1000 -a0 -f"3000" -g"3001" <input file> <output file>

Here the arguments are:

• -l : lossless mode disable (all followers outside the event are removed from the output file. Single are kept anyway)
• -e”<boolean expression>”: a boolean expression describing the trigger condition, here only a hit in channel 189 is required
• -b<time window in ns>: time window before the trigger
• -a<time window in ns>: time window after the trigger
• -f”<followers>”: List of label to be included in the event. Here we use the label of the groupe 3000, the online trigger.
• -g”<group number>”: group number (label) of the new trigger
• <input_file.fast|.fast.gz> and <output_file.fast>

Step 2: Conversion from .fast to .root

This conversion is handled by the npconversion utility from nptool v4 using the faster nptool plugin for readout of the .fast file and the root nptool plugin for writeout of the .root file. The command is the following:

npconversion --detector <detector.yaml> --input faster,<pid file>,<.fast file> --output root,<TreeName>,<.root file>

Here the arguments are:

• –detector <detector.yaml> : this file describe the detector configuration of the experiment. It will be used to load the plugin associated with each detector used. In our case the presence of the nebula-plus block will trigger the load of the plugin and subsequent call to the necessary method by the framework
• –intput faster,<pid file>,<.fast file> : the faster token will trig the load of the faster plugin and its configuration. The <pid file> is a produce by the faster DAQ and is used to associate a name to the channel ID. Based on this name the correct detector class will be called during convertion.
• –output root,<tree name>,<.root file> : the root token will trig the load of the root plugin and its configuration. The <tree name> will be used to name the tree inside the output root file.

The output root file contains a tree with a Data class per detector. Such class are designed to hold uncalibrated data inside std::vector. At this stage no analysis is performed.

Step 3: Tree Trimming (Merge)

To perform the merge we want to produce an nptool root file that is entry match with the anaroot root file. Since an entry is produce in the nebula plus root file for every received trigger from SAMURAI DAQ, the tree should be already entry match.

However the faster trigger merger has a limitation of 127 channel within a single event. Some events have more than 127 channel (about 1 every 30 Millions during cosmic run). The issue cause faster trigger merger to put the 127 first QDC hit of the event in a special group (group 0), and leave alone the hit in the trigger channel (channel 189). During the merge the group 0 events are kept in the output tree but they do not match any entry in the anaroot file, as they contain no trigger TS. The corresponding trigger is left alone in a separate entry that match the anaroot one.

When this happen the difference in time between entry n of nptool tree and entry n of anaroot tree will have a time difference greater than 1us. If this is the case, the entry is dropped from the Nebula Plus output root file.

The trimmed tree is produced using the nebp_anaroot_merger utility from the nebula-plus nptool plugin.

nebp_anaroot_merger --window 1000 --nptool <nebula plus root file> --anaroot <anaroot root file> --outfile <nptool output root file

Here the arguments are:

• –window <window lenth in ns> : acceptable TS diff between nebula plus and nebula
• –nptool: list of space separated root file containing a nebula-plus branch
• –anaroot: a single root file from anaroot with a TS branch
• –nptool-tree-name <tree name> : optional flag specifying the tree name inside the nebula plus root file. Default is “DataTree”.
• –anaroot-tree-name : optional flag specifying the tree name inside the anaroot root file. Default is “tree”.

The output root file should be of equal length, or smaller than the corresponding anaroot file and could be associated with it using the Friend Tree mechanism.

On expandacq

On expandacq a Snakemake workflow has been implemented to run all of the step mentioned above automatically. Like make, Snakemake has a set of rule to produce necessary file for each steps. If a file already exist it will not be recreated, unless the rule for producing it has been changed.

The workflow is configured using a yaml file. Here is a sample yaml file:

# processing chain for fast file
# fast -> group -> data -> merged -> physics
run:
  start: 220
  stop: 240
  
# main data path
data_path: "data"
# experiment name 
exp_name: "Test_SAMURAI"
# base name of the run
basename: "nebnptest"

# for remote nebula download:
anaroot_basename: "nebula0"

Here is the content of the Snakefile currently used. The first section of the file is written in plain python and use information from analysis.yaml to produce a list of output file, i.e. one merged file per run. The following sections contains rules to make each intermediate file.

import pathlib
import glob
import os
import yaml

with open("analysis.yaml", "r") as f:
    config = yaml.load(f, Loader=yaml.FullLoader)

# list of run to analyse
RUN=list(range(config["run"]["start"],config["run"]["stop"]+1))

basename = config["basename"]
data_path = config["data_path"]
exp_name = config["exp_name"]
anaroot_basename = config["anaroot_basename"]


# list of all output to be produced
nebulaplus_output=[]
nebula_output=[]
merge_output=[]
run_subrun={}# key: run, val: list of subrun
# building the dictionnaries
for r in RUN:
  nebula_output.append(data_path+"/"+exp_name+"_analysis/anaroot/"+anaroot_basename+str(r)+".root")
  SUBRUN=[]
  base=data_path+"/"+exp_name+"/"+basename+"_"+str(r)+"_"
  path=glob.glob(base+"*.fast")
  if path:
    # build date
    DATE=path[0].split(base)
    DATE=DATE[1].split(".fast")
    # build subrun
    fast=glob.glob(base+"*.fast/*.fast")
    for f in fast:
      subrun=f[-9:-5]
      SUBRUN.append(subrun)
      nebulaplus_output.append(data_path+"/"+exp_name+"_analysis/merged/"+basename+"_"+str(r)+"_"+DATE[0]+".root")

  run_subrun[str(r)]=SUBRUN;

############################################################
####################### TRIGGER ############################
############################################################
# trigger the entire analysis chain
rule pipeline:
  threads: 10 # limit the number of concurent convertion
  input: # list of all file to be produced 
    nebulaplus_output

# trigger download of nebulaXXXX.root file frome ribfana04
rule remote:
  input:
    nebula_output

############################################################
######################## RULE ##############################
###########################################################
# group trigger with QDC and produce a groupped fast file
rule group:
  input:
    "{data_path}/{exp_name}/{base}_{run}_{date}_{time}.fast/{base}_{run}_{date}_{time}_{subrun}.fast"  
  output:
    "{data_path}/{exp_name}_analysis/group/{base}_{run}_{date}_{time}_{subrun}_group.fast"   
  shell:
    "./script/fastergroup_ts.sh {input} {output}"

###########################################################
# convert a groupped fast file to root
rule convert:
  input: # a .fast file to be converted
     "{data_path}/{exp_name}_analysis/group/{base}_{run}_{date}_{time}_{subrun}_group.fast"
  output: # a .root file to be produced by npconvertion
     "{data_path}/{exp_name}_analysis/root/{base}_{run}_{date}_{time}_{subrun}.root" 
  shell:
    "npconversion --input faster,sample.pid,{input} --output root,DataTree,{output} --detector detector.yaml > .convert"

##########################################################
# trim a nebula root file based on match with anaroot
rule merge:
  input: # a nptool and anaroot file
     nebp=lambda wildcards: expand("{{data_path}}/{{exp_name}}_analysis/root/{{base}}_{{run}}_{{date}}_{{time}}_{subrun}.root",subrun=run_subrun[wildcards.run]), 
     anaroot="{data_path}/{exp_name}_analysis/anaroot/nebula0{run}.root"
  output: # a .root file to be produced by trimming
     "{data_path}/{exp_name}_analysis/merged/{base}_{run}_{date}_{time}.root" 
  shell:
    "nebp_anaroot_merger --window 1000 --nptool {input.nebp} --anaroot {input.anaroot} --outfile {output}>.merger"

############################################################
# download remotely converted nebula file
rule download_one_anaroot_root:
  output:
    "{output}"
  wildcard_constraints:
    output=".*anaroot.*.root" # any root file containing anaroot in its path
#  shell:
#    "scp ribfana04:~/rootfiles/nebula/"+os.path.basename("{output}")+" {output}"
  run:
    command="scp ribfana04:~/rootfiles/nebula/"+os.path.basename(output[0])+" "+output[0] +"> .download"
    os.system(command)

Quick Summary:

npp commissioning
snakemake pipeline

On ribfana04

On ribfana04 a special environment need to be loaded because nptool require a recent compiler. In addition all the faster lib and binary need to be added to the relevant path. Prior to this analysis, one need to process the ridf file for the run in anaroot.

All nebula-plus related software are installed in the directory :

/home/s053/exp/exp2301_s053/faster

in this directory your will find several component of the conversion and analysis process:

• faster : contain the fasterac lib necessary for reading faster file
• ctm2 : contain the C Trigger Merger of faster acq, this contain the faster_file_group binary necessary to re-run the software trigger
• nptool : contain the nptool framework installation
• plugin: contain the nptool plugin installation
• s053: the nptool project were all the work will take place
• sandbox: an nptool project used for dev and debug.
• nebula_plus.sh : a bash script that prepare the environment to use all the above mentionned tool.

The first step is to source the nebula_plus.sh file.

cd /home/s053/exp/exp2301_s053/faster 
source nebula_plus.sh

## load the correct version of gcc ##
. /opt/rh/devtoolset-9/enable

## load the matching version of root ##
. /opt/cpp17/root-6-24-08-install/bin//thisroot.sh

## for nptool ##
# what env should be used
export NPTOOL_ENV=plugin
# place were the env folder are located/created
export NPTOOL_HOME=/home/s053/exp/exp2301_s053/faster/

# load the nptool config (must happen after previous export of HOME and ENV)
source /home/s053/exp/exp2301_s053/faster/nptool/install/bin/nptool.sh

## for fasterac ##
source /home/s053/exp/exp2301_s053/faster/faster/fasterac-2.18/install/bin/fasterac_config.sh
# this variable allow nptool to find fasterac sources
export FASTERAC=/home/s053/exp/exp2301_s053/faster/faster/fasterac-2.18/install/

## C Trigger Merger from faster ##
export LD_LIBRARY_PATH=/home/s053/exp/exp2301_s053/faster/ctm2/lib:$LD_LIBRARY_PATH
export PATH=/home/s053/exp/exp2301_s053/faster/ctm2/bin:$PATH

## for snakemake ##
conda activate snakemake

Now that the environment is correctly set, we can access the desired nptool project directory using (here for s053 project):

npp s053

the s053 folder is prepared with symbolic link to the raw faster data, the processed faster data and the anaroot data:

• raw_data : /home/s053/rawdata/nebulaplus/
• processed_data : /home/s053/rootfiles/nebulaplus/
• anaroot : /home/s053/rootfiles/nebula/

The first step to perform a conversion is to download the newly acquired .fast file from expandacq to ribfana04 . This is achieve via the rsync script rsync_expand_test.sh :

rsync_expand_test.sh

After rsync is done, and the corresponding anaroot file for this run produced, we can run the snakemake workflow prepared in the Snakefile. This one is using the analysis.yaml file as a configuration :

processing chain for fast file
# fast -> group -> data -> merged -> physics
run:
  start: 220
  stop: 240

# raw data path
raw_data_path: "raw_data"

# processed data path
processed_data_path: "processed_data"

# base name of the run
basename: "nebnptest"

# base name of the anaroot file
anaroot_path: "anaroot"
anaroot_basename: "nebula"

The Snakefile is produced below. The first part contain plain python creating the list of final file to produced (i.e. post merged), and then rules to create all intermediate file.

import pathlib
import glob
import os
import yaml

with open("analysis.yaml", "r") as f:
    config = yaml.load(f, Loader=yaml.FullLoader)

# list of run to analyse
RUN=list(range(config["run"]["start"],config["run"]["stop"]+1))

basename = config["basename"]
raw_data_path = config["raw_data_path"]
processed_data_path = config["processed_data_path"]
anaroot_basename = config["anaroot_basename"]
anaroot_path= config["anaroot_path"]

# list of all output to be produced
nebulaplus_output=[]
nebula_output=[]
merge_output=[]
run_subrun={}# key: run, val: list of subrun
# building the dictionnaries
for r in RUN:
  nebula_output.append(anaroot_path+"/"+anaroot_basename+str(r).zfill(4)+".root")
  SUBRUN=[]
  base=raw_data_path+"/"+basename+"_"+str(r)+"_"
  path=glob.glob(base+"*.fast")
  if path:
    # build date
    DATE=path[0].split(base)
    DATE=DATE[1].split(".fast")
    # build subrun
    fast=glob.glob(base+"*.fast/*.fast")
    for f in fast:
      subrun=f[-9:-5]
      SUBRUN.append(subrun)
      nebulaplus_output.append(processed_data_path+"/merged/"+basename+"_"+str(r)+"_"+DATE[0]+"_merged.root")

  run_subrun[str(r)]=SUBRUN;

############################################################
####################### TRIGGER ############################
############################################################
# trigger the entire analysis chain
rule pipeline:
  threads: 10 # limit the number of concurent convertion
  input: # list of all file to be produced
    nebulaplus_output

############################################################
######################## RULE ##############################
###########################################################
# group trigger with QDC and produce a groupped fast file
rule group:
  input:
    lambda wildcards: expand("{raw_data_path}/{{base}}_{{run}}_{{date}}_{{time}}.fast/{{base}}_{{run}}_{{date}}_{{time}}_{{subrun}}.fast",raw_data_path=raw_data_path)
  output:
    "{processed_data_path}/group/{base}_{run}_{date}_{time}_{subrun}_group.fast"
  shell:
    "faster_file_group -l -e\"189\" -b1000 -a0 -f\"3000\" -g\"3001\" {input} {output}"

###########################################################
# convert a groupped fast file to root
rule convert:
  input: # a .fast file to be converted
     "{processed_data_path}/group/{base}_{run}_{date}_{time}_{subrun}_group.fast"
  output: # a .root file to be produced by npconvertion
     "{processed_data_path}/root/{base}_{run}_{date}_{time}_{subrun}_unmerged.root"
  shell:
    "npconversion --input faster,sample.pid,{input} --output root,DataTree,{output} --detector detector.yaml > .convert"

##########################################################
# trim a nebula root file based on match with anaroot
rule merge:
  input: # a nptool and anaroot file
     nebp=lambda wildcards: expand("{{processed_data_path}}/root/{{base}}_{{run}}_{{date}}_{{time}}_{subrun}_unmerged.root",subrun=run_subrun[wildcards.run]),
     anaroot=lambda wildcards: expand("{anaroot_path}/{anaroot_basename}{{run}}.root",anaroot_path=anaroot_path,anaroot_basename=anaroot_basename)
  output: # a .root file to be produced by trimming
     "{processed_data_path}/merged/{base}_{run}_{date}_{time}_merged.root"
  shell:
    "nebp_ridf_merger --window 50 --nebp {input.nebp} --ridf {input.anaroot} --outfile {output}>.merger"

To run the pipeline use the following command:

snakemake pipeline --cores 10

Be careful about the number of cores required, as it define the number of concurrent process that will be spawn by the snakemake call.

The analysis pipeline will produce file in the process_data directory. To each step of the conversion and analysis is associated a folder:

• group: .fast file produce after the grouping of channel 189 registering SAMURAI trigger.
• root: nptool conversion of the .fast file to root format
• merged: a trimmed nptool tree that could be Friend with the corresponding anaroot file.

Quick Summary:

cd exp/exp2301_s053/faster/
source nebula_plus.sh
npp s053
rsync_expand_test.sh
snakemake pipeline --cores 10