Skip to content

End-to-End-Example.md

Latest commit

 

History

History
177 lines (144 loc) · 13.4 KB

End-to-End-Example.md

File metadata and controls

177 lines (144 loc) · 13.4 KB
Raw

Minimal End-to-End-Example of Surrogator 🐊

We prefer to use a Linux system. You need around 8 GB disk space.

Installation

  • Install Python 3.11;

  • It is preferred, to use a virtual environment

    • if not installed, install apt install python3.11-venv

    • use a terminal:

      • create virtual environment: python3.11 -m venv venv
      • start virtual environment: source venv/bin/activate

  • Install following packages via Pip, see pyproject.toml or run pip install .

        pandas~=2.2.2
        dkpro-cassis
        pycaprio~=0.3.0
        streamlit
        toml~=0.10.2
        mdutils~=1.6.0
        tabulate~=0.9.0
        streamlit_ext
        python-dateutil~=2.9.0
        requests~=2.32.3
        schwifty
        gender-guesser
        spacy
        joblib
        sentence-transformers
        Levenshtein
        scikit-learn
        openpyxl
        overpy
        anytree
  • Note:
    • If you use the following data and instructions, there is an API call with the OpenStreetMap database by the Overpass API!
    • If you do not want to use a local dump, follow the instructions from Readme.md section

Test Data

Notes about Output of Following Runs

Every run of surrogator.py creates output into the directories

  • private : for every run a private directory is created:

    • the new created UIMA cas files in cas-project_name-timestamp_key
    • a directory with aggregated statistics of quality control output

  • public : for further usage

    • only new generated text files from the projects

Run via Terminal

(1) Assurance Step / Quality Control Step

  • It is preferred an assurance step before replacement by surrogates and look into the data.
  • It is not necessary. Every following surrogation routine run this assurance step as initial step.
  • Run: python surrogator.py -qc -p test_data/projects
  • Example output: test_data/test_output
    • every run fills the directories
      • public : for further usage (research, LLM training, semantic annotation, ...)
      • private : archive @ Data Integration Center for every run a private directory is created, containing
      • with separated directories with a name convention of project_name-timestamp
    • See following description in detail with examples

(2) Run Surrogation