Merge pull request #5 from Kaszanas/draft_paper

[PR] Re-introduced the DatasetPreparator section, fixing section levels

Author: Kaszanas

paper.md

@@ -74,18 +74,33 @@ Our software consists of multiple modules that the user can match to their speci
 
 ![Simplified full pipeline using SC2Tools to create two datasets, "SC2ReSet" [@Bialecki2022ReSetZenodo] and "SC2EGSet" Dataset [@Bialecki2023EGSetZenodo]. Initially introduced in [@Bialecki2023SC2EGSet]. \label{fig:DatasetPipeline}](./article/SC2Tools_Pipeline_Complete.pdf)
 
+## DatasetPreparator
 
-### SC2InfoExtractorGo
+The "DatasetPreparator" [@Bialecki_2022_SC2DatasetPreparator] submodule is a set of scripts that ease the process of working with major collections of raw data (replaypacks/datasets). A full list of scripts is as follows:
+- "directory_flattener.py"; flattens the nested directory structure of the replaypacks,
+- "directory_packager.py"; packages all of the directories in the specified input directory,
+- "file_renamer.py"; renames the files in the directory to follow a specific naming convention (e.g., to match the dataset schema),
+- "json_merger.py"; merges two JSON files into one,
+- "processed_mapping_copier.py"; copies the auxiliary files generated by directory_flattener.py to matching output directories. Built specifically to prepare the SC2EGSet prior to packaging,
+- "sc2_map_downloader.py"; wraps "SC2InfoExtractorGo" to run the map downloading step,
+- "sc2egset_replaypack_processor.py"; wraps "SC2InfoExtractorGo" to run the replaypack processing step on multiple directories at once,
+- "sc2egset_pipeline.py"; wraps the entire processing pipeline used to obtain the "SC2ReSet" and "SC2EGSet" datasets,
+- "sc2reset_replaypack_downloader.py"; downloads the raw (flattened) replaypacks of "SC2ReSet" [@Bialecki2022ReSetZenodo] for users that wish to use their own tools for data processing.
+
+In the context of our work, this submodule is responsible for preparing directory structure, execution of "SC2InfoExtractorGo" on the data, and packaging the dataset for hosting. Finally, the current capabilities include downloading the raw replaypacks of "SC2ReSet" [@Bialecki2022ReSetZenodo] for ease of "SC2EGSet" Dataset reproduction [@Bialecki2023EGSetZenodo].
+
+
+## SC2InfoExtractorGo
 
 The SC2InfoExtractorGo as a submodule is a tool responsible for extracting the data from SC2Replay files, it depends on previously published open-source lower-level libraries [@URLS2Prot2016;@URLMPQ2017]. The tool is written in Golang and is shipped as a binary file (release), and as a Docker image via DockerHub. A simplified depiction of the data extraction is available on \autoref{fig:file_processing_sc2infoextractorgo}.
 
 ![Pictorial representation of the "SC2InfoExtractorGo" functionality [@Bialecki_2021_SC2InfoExtractorGo]. Replays contain the events which happened during gameplay (blue background), our implementations extracts this data and outputs it for further analysis by the user (orange background). \label{fig:file_processing_sc2infoextractorgo}](./article/server_to_json.pdf)
 
-### SC2AnonServerPy
+## SC2AnonServerPy
 
 In the process of extracting the information from the StarCraft~2 replays, the users have the ability to choose if nicknames of the players should be anonymized with a separate tool "SC2AnonServerPy" [@Bialecki_2021_SC2AnonServerPy], this functionality may be key for laboratories that wish to share their datasets with a wider community. The "SC2AnonServerPy" repository contains a gRPC [@gRPC2014] service that acquires a weakly anonymized unique identifier for the players that participated in the game. Other implementations and tools may be added in the future to further anonymize the data.
 
-### SC2_Datasets
+## SC2_Datasets
 
 One of our solutions, SC2_Datasets [@bialecki_2022_sc2datasets] interfaces with the JSON files produced by the SC2InfoExtractorGo [@Bialecki_2021_SC2InfoExtractorGo]. This includes all of the classes and methods required to load a single JSON, a collection of JSON files (representing a replaypack), and finally a way of loading an entire dataset (a collection of replaypacks). The pictorial representation of the "SC2_Datasets" functionality is presented on \autoref{fig:LoadingDataToPyTorch}.