1st and 2nd quarter 2021 data

This release generates version 0.7.0 of the ASRC's routine chemistry dataset, which includes all data up to 2021-07-01. The files are available on the atmoschem server here. The repository readme file contains instructions for reproducing the dataset.

Changes

Added

• Add 2021 1st and 2nd quarter WFMS, WFML, and PSP data flags.

Changed

• Rename Whiteface Mountain lodge (WFML) to Whiteface Mountain base (WFMB).

Fixed

• Use correct units for Queens College CO.
• Fix PSP NO2 conversion efficiency calculations.
• Ignore incorrectly set WFMS temperature and relative humidity flags.
• Use outside temperatures for sea level pressure calculations (before
  2018-10-01).

3rd and 4th quarter 2020 data

This release generates version 0.6.0 of the ASRC's routine chemistry dataset, which includes all data up to 2021-01-01. The files are available on the atmoschem server here. The repository readme file contains instructions for reproducing the dataset.

Changes

Added

• Add 2020 3rd and 4th quarter WFMS, WFML, and PSP data flags.
• Add 2018-10-01 to 2021-01-01 Queens College data.
• Add recommended citations to README file.
• Add NARSTO flag definitions.
• Add pkgdown documentation website at
  https://asrcsoft.github.io/atmoschem.process/.
• Add Zenodo integration.

Fixed

• Use outside temperatures for sea level pressure calculations (starts
  2018-10-01).
• Calculate resultant wind speeds instead of mean wind speeds for WFMS
  minute-resolution wind data (starts 2020-06-30).

zenodotest

Use cowplot to organize processing viewer subplots

2020 2nd quarter routine chemistry dataset

This release generates version 0.5.0 of the ASRC's routine chemistry dataset, which includes all data up to 2020-07-01. The files are available on the atmoschem server here. The repository readme file contains instructions for reproducing the dataset.

Changes

Added

• Add 2020 2nd quarter WFMS, WFML, and PSP data flags.
• Add site elevation to sites file.

Changed

• Use empty string instead of "NA" for missing values.
• Add rows for hours with no data.

Fixed

• Include Pinnacle PM2.5 before 2018-10-01.

2020 1st quarter routine chemistry dataset

This release generates version 0.4.0 of the ASRC's routine chemistry dataset, which includes all data up to 2020-05-01. The files are available on the atmoschem server here. The repository readme file contains instructions for reproducing the dataset.

Changes

Added

• Add 2020 1st quarter WFMS, WFML, and PSP data flags.
• Add site information file.
• Add Addison Mesonet winds to PSP data.

Fixed

• Remove incorrect WFMS wind speed maximums.

2019 4th quarter routine chemistry dataset

This release generates version 0.3 of the ASRC's routine chemistry dataset, which includes all data up to 2020-01-01. The files are available on the atmoschem server here. The readme file contains instructions for reproducing the dataset.

Changes

Added

• Add 2019 4th quarter WFMS, WFML, and PSP data flags.
• Add code for cleaning original processed routine data.
• Add Makefile and scripts for reproducing the routine chemistry dataset
  package.

Changed

• Prioritize V4 over V1 flags when both apply.
• Replace NA flag values with M1.
• Flag suspicious 1999-2001 WFMS CO drop as V4.
• Create unified dataset package instead of yearly data files.
• Reprocess all data starting from 2018-10-01.
• Set R package version number to match dataset version.

Fixed

• Flag data and ignore calibrations during WFMS 300EU auto-reference (AREF)
  periods.
• Correct a variety of WFMS and PSP wind direction issues in both the current
  and previously processed data.
• Replace zeros with NA during PSP black carbon tape changes.

2019 1st-3rd quarter Whiteface and Pinnacle data

This release generates v01 of the processed 2019 Whiteface and Pinnacle datasets, which includes data from the first 3 quarters of the year. The files are available on the atmoschem server in the WFM2019 and PSP2019 folders.

Changes

Added

• Add 2019 1st, 2nd, and 3rd quarter data annotations.
• Generate instrument info file with datasets.
• Interpolate calibrator instrument flow values.
• Add PSP aethelometer method detection limits.

Changed

• Simplify and standardize high time resolution dataset files.
• Improve PSP conversion efficiency calculations.
• Improve outlier detection in the presence of missing or flagged data.
• Flag more data at the end of auto-calibrations.
• Adjust WFMS perm tube calibration values based on GPT and IPN tests.
• Use unicode sub- and superscripts in generated dataset column names.
• Apply spike filter to WFMS ultrafine data.

2018 4th quarter Whiteface Mountain lodge data

This release generated the Whiteface Mountain lodge dataset for the 4th quarter of 2018, found in WFML2018_v04.csv (starting in October) and available on the atmoschem server.

Changes

Added

• Add WFML 2018 4th quarter data annotations.
• Add manual flags for calibrations.

Changed

• Apply smoothing to zero and span calibration values.

Fixed

• Convert NYS Mesonet data times to EST.

2018 4th quarter Whiteface Mountain summit data

This release generated the Whiteface Mountain summit dataset for the 4th quarter of 2018, found in the files

• WFMS_1810_minute_campbell.csv, WFMS_1811_minute_campbell.csv, WFMS_1812_minute_campbell.csv
• WFMS_1810_minute_aethelometer.csv, WFMS_1811_minute_aethelometer.csv, WFMS_1812_minute_aethelometer.csv
• WFMS_1810_minute_ultrafine.csv, WFMS_1811_minute_ultrafine.csv, WFMS_1812_minute_ultrafine.csv
• WFMS2018_v04.csv (starting in October)

The un-aggregated data files were split up according to data source. This allows us to keep the original times for each data source, which may not line up between data sources (e.g., "12:05:00" vs. "12:05:25"), and also allows us to include data with different sampling intervals, such as the aethelometer data, which is recorded every 15 minutes.

The new files are currently available on the atmoschem server.

Changes

Added

• Add WFMS 2018 4th quarter data annotations.
• Flag negative WFMS 370nm black carbon and wood smoke values as below
  the method detection limit.
• Treat zero WFMS 370nm and 880nm black carbon as missing data.

Changed

• Create separate minute files for each data source.

2018 4th quarter Pinnacle State Park data

This release generated the Pinnacle State Park atmospheric chemistry dataset for the 4th quarter of 2018, found in the files

• oct18hr.csv, nov18hr.csv, and dec18hr.csv
• oct18db_v01.csv, nov18db_v01.csv, and dec18db_v01.csv
• PSP2018_v05.csv (starting in October)

The un-aggregated data files contain a new 'Status' column that takes the place of the Envidas flag column in older files. The two possible Status values are 0 (measurement was flagged and not used to calculate hourly values) or 1 (not flagged).

The new files are currently available on the atmoschem server.

Changes

Added

• Move raw data into PostgreSQL.
• Automate data processing.
• Add PSP 2018 4th quarter data annotations.
• Automate data formatting for data releases.

Changed

• Correct for clock errors using linear interpolation rather than by
  discarding measurements with overlapping times.
• Ignore ultrafine low pulse height flag.
• Replace standard deviation-based outlier detection with median
  absolute deviation-based outlier detection.
• Use 3-minute moving averages for all calibration estimates.
• Treat zero WindDir_SD1_WVT as an indicator of wind direction
  instrument malfunction.
• Apply M1 flag to hours with >50% missing data.
• Simplify data report format.
• Report binary 'status' value with un-aggregated measurements.

Fixed

• Fix bug ignoring ultrafine 'Nozzle Pressure' flag (400) instead of
  'Service Reminder' flag (4000).
• Account for zero offsets when estimating calibration spans.
• Correct miscalculated WFMS wind speeds from 2016 to 2019.
• Fix PSP NO₂ conversion efficiency calculations.
• Fix PSP solar radiation zero offset.