This function is only used by lst1_magic_event_coincidence and at the moment it loads all MAGIC calibrated files from an input directory using as regex f"{input_dir}/dl1_*.h5"
then it proceeds to go through the files with a for cycle, which would be OK if the required operation scope was the cycle itself, but instead it does data_list.append(df_events) which means that it will store in memory all the events of all the list of files
since MAGIC date = LST-1 + 1 day, we know which dates from the MAGIC input list to consider, so it does not make sense to load them all (potentially months)
Unfrotunately it seems that the name of the DL1 files from MAGIC have only the run number, so I suggest to:
- (my recommendation) either modify the upstream script to save those filenames keeping the date and to allow this function to accept a keyword argument
filter_date to add to the regex f"{input_dir}/dl1_{date}_*.h5"
- use the informations from the database to filter only the MAGIC runs effectively taken on the same day of the LST run currently in use
This function is only used by
lst1_magic_event_coincidenceand at the moment it loads all MAGIC calibrated files from an input directory using as regexf"{input_dir}/dl1_*.h5"then it proceeds to go through the files with a for cycle, which would be OK if the required operation scope was the cycle itself, but instead it does
data_list.append(df_events)which means that it will store in memory all the events of all the list of filessince MAGIC date = LST-1 + 1 day, we know which dates from the MAGIC input list to consider, so it does not make sense to load them all (potentially months)
Unfrotunately it seems that the name of the DL1 files from MAGIC have only the run number, so I suggest to:
filter_dateto add to the regexf"{input_dir}/dl1_{date}_*.h5"