Skip to content

Hosting reports

Jump to bottom
Muluh edited this page May 4, 2026 · 1 revision

Hosting quarto reports on datascience.utu.fi

Purpose

This guide documents the exact production pattern used to host heavy Quarto report outputs, such as Husna and Lotta, on the datascience.utu.fi server.

The important lesson from the original debugging session is:

The live site is served by Apache on the production server, not directly by GitHub Pages. GitHub Actions may run, but copying report output inside GitHub Actions does not update production unless the Apache server also receives those files.

The production site is updated by server-side cron. The main openresearchlabs.github.io repo is pulled and rebuilt by the server at minute 02 each hour, using this pattern:

2 * * * * cd /home/hoster/openresearchlabs.github.io; git reset --hard; git pull; hugo -D; rm -rf /var/www/html/*; cp -r public/* /var/www/html

That means /var/www/html is rebuilt from Hugo output and should not be used as the permanent home for heavy report outputs. The report outputs should instead live in their own report repositories and be served directly by Apache through stable URLs.

Production architecture

Main site

The main website still uses the existing Hugo deployment flow:

openresearchlabs.github.io repo
        |
        | server cron at hh:02
        v
hugo -D
        |
        v
/var/www/html
        |
        v
https://datascience.utu.fi/

This is for the main public site, pages, menus, and links.

Heavy reports

Heavy report outputs are not stored inside openresearchlabs.github.io.

They stay in their own repositories, for example:

/home/hoster/collaboration/reports/fopp-husna
/home/hoster/collaboration/reports/fopp-lotta

Apache serves their rendered output/ folders directly:

https://datascience.utu.fi/reports/husna/
    -> /home/hoster/collaboration/reports/fopp-husna/output/

https://datascience.utu.fi/reports/lotta/
    -> /home/hoster/collaboration/reports/fopp-lotta/output/

This keeps the main Hugo repo lightweight and lets Git LFS remain in the report repositories, where it belongs. The earlier conclusion was that Husna and Lotta should no longer be served through the base Hugo build output; Apache should map stable report URLs directly to the report output folders.

What a new report needs

For a new report, you need five pieces:

  1. A report repository that contains or produces the rendered report output.
  2. The rendered output committed/pushed to that repo, usually with Git LFS if large.
  3. A local clone of that repo on the production server under the hoster account.
  4. A cron job that keeps the server clone updated.
  5. An Apache mapping from a public URL to the report output directory.

Optional but recommended:

  1. Apache Basic Auth if the report is collaborator-only.

Naming convention

Use a short lowercase report key.

Examples:

husna
lotta
newreport

For a new report called newreport, use:

Repo directory:
  /home/hoster/collaboration/reports/fopp-newreport

Rendered output:
  /home/hoster/collaboration/reports/fopp-newreport/output

Public URL:
  https://datascience.utu.fi/reports/newreport/

Password file:
  /home/hoster/.htpasswd-newreport

Cron log:
  /home/hoster/logs/fopp-newreport-cron.log

Step 1: Confirm you are on the production server

SSH into the production server.

ssh <your-user>@<server>

Confirm the site is Apache-served:

curl -I https://datascience.utu.fi

You can also inspect Apache vhosts:

sudo apachectl -S

During the original debugging, the domain was confirmed to be served by Apache on the production server, with local server-side deployment rather than GitHub Pages being authoritative.

Step 2: Use the hoster Account for Report Repos

The report clones should be owned and updated by hoster, because the existing production update flow uses that account.

Do not manually create folders in /home/hoster as another user. If you see this:

mkdir: cannot create directory 'collaboration': Permission denied

that means you are not operating as the right user or need sudo -u hoster.

Create the reports directory like this:

sudo -u hoster mkdir -p /home/hoster/collaboration/reports
sudo -u hoster ls -ld /home/hoster/collaboration/reports

The original session hit exactly this permission issue, and the fix was to create/manage the directory as hoster.

Step 3: Fix GitHub SSH trust for hoster

Before cloning private repos, make sure the hoster account trusts GitHub’s SSH host keys.

Run:

sudo -u hoster mkdir -p /home/hoster/.ssh
sudo -u hoster chmod 700 /home/hoster/.ssh

sudo -u hoster cp /home/hoster/.ssh/known_hosts /home/hoster/.ssh/known_hosts.bak.$(date +%F-%H%M%S) 2>/dev/null || true

sudo -u hoster ssh-keygen -f /home/hoster/.ssh/known_hosts -R github.com

sudo -u hoster bash -c 'ssh-keyscan -t ed25519,ecdsa,rsa github.com >> /home/hoster/.ssh/known_hosts'

sudo -u hoster chmod 600 /home/hoster/.ssh/known_hosts

Test SSH access:

sudo -u hoster ssh -T git@github.com

If GitHub says permission is denied, create a deploy key or user key:

sudo -u hoster ssh-keygen -t ed25519 -C "hoster-datascience-utu" -f /home/hoster/.ssh/id_ed25519 -N ""

sudo -u hoster cat /home/hoster/.ssh/id_ed25519.pub

Add the printed public key to GitHub, either as a deploy key on the report repo or under the GitHub account that owns the repo access.

This exact trust-key repair was needed because strict host checking blocked the clone with an old GitHub host key.

Step 4: Clone the report repository

Clone as hoster.

For Husna and Lotta, the original commands were:

sudo -u hoster git -C /home/hoster/collaboration/reports clone git@github.com:0xMuluh/fopp-husna.git fopp-husna

sudo -u hoster git -C /home/hoster/collaboration/reports clone git@github.com:0xMuluh/fopp-lotta.git fopp-lotta

For a new report, use the same pattern:

sudo -u hoster git -C /home/hoster/collaboration/reports clone git@github.com:<ORG_OR_USER>/<REPO>.git fopp-newreport

Then verify:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport remote -v
sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport branch --show-current
sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport log -1 --date=iso --pretty=fuller

Step 5: Install Git LFS on the server

Heavy report outputs may use Git LFS. If the server does not have Git LFS installed, this command fails:

git: 'lfs' is not a git command

Install Git LFS once:

sudo apt-get update
sudo apt-get install -y git-lfs

sudo git lfs install --system
sudo -u hoster git lfs install --force

Verify:

sudo -u hoster git lfs version

Pull LFS content:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport lfs pull

For Husna and Lotta, the verified pattern was:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-husna lfs pull
sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-lotta lfs pull

Check that LFS files are visible:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport lfs ls-files | head

This was a real issue in the setup: git lfs pull failed until Git LFS was installed and initialized for the server environment.

Step 6: Confirm the report output exists

Apache will serve the rendered output/ folder. Before changing Apache, confirm the report has an index.html.

For a new report:

ls -la /home/hoster/collaboration/reports/fopp-newreport/output/index.html

For Husna and Lotta:

ls -la /home/hoster/collaboration/reports/fopp-husna/output/index.html
ls -la /home/hoster/collaboration/reports/fopp-lotta/output/index.html

If index.html is missing, the report is not ready to be served. Render locally, commit/push the output to the report repo, then pull again on the server.

The original realization was: cron only keeps files updated; to see them in production, Apache must expose those folders as URLs.

Step 7: Add a Cron Job to keep the report updated

Cron goes in the hoster user’s crontab, not root’s crontab unless there is a specific reason.

Create a log directory:

sudo -u hoster mkdir -p /home/hoster/logs

Open the hoster crontab:

sudo -u hoster crontab -e

Add the environment lines if not already present:

SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=""

Add a job for the new report:

9 * * * * cd /home/hoster/collaboration/reports/fopp-newreport && git reset --hard && git pull --ff-only && git lfs pull >> /home/hoster/logs/fopp-newreport-cron.log 2>&1

For Husna and Lotta, the documented jobs were:

7 * * * * cd /home/hoster/collaboration/reports/fopp-husna && git reset --hard && git pull --ff-only && git lfs pull >> /home/hoster/logs/fopp-husna-cron.log 2>&1

8 * * * * cd /home/hoster/collaboration/reports/fopp-lotta && git reset --hard && git pull --ff-only && git lfs pull >> /home/hoster/logs/fopp-lotta-cron.log 2>&1

Save and verify:

sudo -u hoster crontab -l

Test the command immediately instead of waiting for cron:

sudo -u hoster bash -lc 'cd /home/hoster/collaboration/reports/fopp-newreport && git reset --hard && git pull --ff-only && git lfs pull'

Check logs after the scheduled run:

sudo -u hoster tail -n 50 /home/hoster/logs/fopp-newreport-cron.log

The reason for git lfs pull in cron is important: without it, the server may only fetch LFS pointer files instead of the real report assets.

Step 8: Add Apache URL mapping

Edit the active Apache vhost that serves datascience.utu.fi.

In the original server, relevant config was found under Apache site files such as:

/etc/apache2/sites-available/000-default.conf
/etc/apache2/sites-available/default-ssl.conf

The exact active file should be confirmed with:

sudo apachectl -S

Make a backup before editing:

sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/000-default.conf.bak.$(date +%F-%H%M)

Open the file:

sudoedit /etc/apache2/sites-available/000-default.conf

Add this pattern for a public report:

# New report
RedirectMatch 301 ^/reports/newreport$ /reports/newreport/
Alias /reports/newreport/ /home/hoster/collaboration/reports/fopp-newreport/output/

<Directory /home/hoster/collaboration/reports/fopp-newreport/output>
    Options FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

For Husna and Lotta, the same idea was:

# Husna
RedirectMatch 301 ^/reports/husna$ /reports/husna/
Alias /reports/husna/ /home/hoster/collaboration/reports/fopp-husna/output/

<Directory /home/hoster/collaboration/reports/fopp-husna/output>
    Options FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

# Lotta
RedirectMatch 301 ^/reports/lotta$ /reports/lotta/
Alias /reports/lotta/ /home/hoster/collaboration/reports/fopp-lotta/output/

<Directory /home/hoster/collaboration/reports/fopp-lotta/output>
    Options FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

Then validate and reload:

sudo apachectl configtest
sudo systemctl reload apache2

Verify:

curl -I https://datascience.utu.fi/reports/newreport/

Expected result for a public report:

HTTP/2 200

The report URLs do not need to exist in Hugo for Apache to serve them; Hugo links are only for discoverability. Apache can serve the report paths directly if the aliases exist and the folders are updated.

Step 9: Add password protection

Use Apache Basic Auth if the report is collaborator-only.

Install tools:

sudo apt-get update
sudo apt-get install -y apache2-utils

Create the password file:

sudo htpasswd -c /home/hoster/.htpasswd-newreport newreport_user

Do not use -c when adding later users, because -c recreates the file:

sudo htpasswd /home/hoster/.htpasswd-newreport another_user

Set permissions:

sudo chown root:www-data /home/hoster/.htpasswd-newreport
sudo chmod 640 /home/hoster/.htpasswd-newreport

Update the Apache block:

# New protected report
RedirectMatch 301 ^/reports/newreport$ /reports/newreport/
Alias /reports/newreport/ /home/hoster/collaboration/reports/fopp-newreport/output/

<Directory /home/hoster/collaboration/reports/fopp-newreport/output>
    Options FollowSymLinks
    AllowOverride None

    AuthType Basic
    AuthName "New Report Collaboration"
    AuthUserFile /home/hoster/.htpasswd-newreport
    Require valid-user
</Directory>

Validate and reload:

sudo apachectl configtest
sudo systemctl reload apache2

Verify that unauthenticated access is blocked:

curl -I https://datascience.utu.fi/reports/newreport/

Expected result:

HTTP/2 401

Verify credentials work:

curl -I -u newreport_user:YOUR_PASSWORD https://datascience.utu.fi/reports/newreport/

Expected result:

HTTP/2 200

For Husna and Lotta, separate password files were recommended:

sudo htpasswd -c /home/hoster/.htpasswd-husna husna_user
sudo htpasswd -c /home/hoster/.htpasswd-lotta lotta_user

sudo chown root:www-data /home/hoster/.htpasswd-husna /home/hoster/.htpasswd-lotta
sudo chmod 640 /home/hoster/.htpasswd-husna /home/hoster/.htpasswd-lotta

and the corresponding Apache blocks used separate AuthUserFile entries for each report.

Step 10: Add a Link from the Main Site

This is optional.

The report works without a Hugo link if Apache is configured correctly. Add a Hugo link only when users should discover it from the main website.

For collaborator-only reports, it may be better not to list them publicly.

The stable links should be:

https://datascience.utu.fi/reports/newreport/
https://datascience.utu.fi/reports/husna/
https://datascience.utu.fi/reports/lotta/

Day-to-Day publishing workflow

After the one-time server setup, the report owner does this:

1. Render locally.
2. Confirm the local output works.
3. Commit and push the rendered output to the report repo.
4. Wait for the server cron job, or manually run the cron command once.
5. Open the stable production URL.
6. Confirm auth and assets work.

Manual update command for a new report:

sudo -u hoster bash -lc 'cd /home/hoster/collaboration/reports/fopp-newreport && git reset --hard && git pull --ff-only && git lfs pull'

Then verify:

curl -I https://datascience.utu.fi/reports/newreport/

With auth:

curl -I -u newreport_user:YOUR_PASSWORD https://datascience.utu.fi/reports/newreport/

Troubleshooting

Report URL returns 404

Check whether Apache knows the route:

sudo apachectl -S
sudo grep -R "reports/newreport" /etc/apache2/sites-available /etc/apache2/sites-enabled

Check the output exists:

ls -la /home/hoster/collaboration/reports/fopp-newreport/output/index.html

Check Apache config and reload:

sudo apachectl configtest
sudo systemctl reload apache2

A 404 usually means Apache does not have the Alias mapping, the vhost was not reloaded, or the target folder does not contain an index.html.

Report URL returns 403

Check permissions:

namei -l /home/hoster/collaboration/reports/fopp-newreport/output/index.html

Apache needs execute permission on parent directories and read permission on files.

Also check the Apache <Directory> block:

Require all granted

or, for protected reports:

Require valid-user

Report URL returns 401

That is expected if Basic Auth is enabled.

Test with credentials:

curl -I -u newreport_user:YOUR_PASSWORD https://datascience.utu.fi/reports/newreport/

Check the password file:

sudo ls -l /home/hoster/.htpasswd-newreport

Expected ownership/permissions:

root:www-data
640

Assets or figures are missing

Check whether LFS pulled real files:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport lfs pull
sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport lfs ls-files | head

Check the cron log:

sudo -u hoster tail -n 100 /home/hoster/logs/fopp-newreport-cron.log

If LFS was not installed, install it:

sudo apt-get update
sudo apt-get install -y git-lfs
sudo git lfs install --system
sudo -u hoster git lfs install --force

Cron does not update the repo

Check the crontab:

sudo -u hoster crontab -l

Run the command manually:

sudo -u hoster bash -lc 'cd /home/hoster/collaboration/reports/fopp-newreport && git reset --hard && git pull --ff-only && git lfs pull'

Check logs:

sudo -u hoster tail -n 100 /home/hoster/logs/fopp-newreport-cron.log

Git clone fails with GitHub host key error

Refresh GitHub host keys:

sudo -u hoster ssh-keygen -f /home/hoster/.ssh/known_hosts -R github.com
sudo -u hoster bash -c 'ssh-keyscan -t ed25519,ecdsa,rsa github.com >> /home/hoster/.ssh/known_hosts'
sudo -u hoster chmod 600 /home/hoster/.ssh/known_hosts

Test again:

sudo -u hoster ssh -T git@github.com

This exact issue occurred during setup and was caused by stale GitHub host keys in /home/hoster/.ssh/known_hosts.

Git says “dubious ownership”

Run Git as the repository owner instead of globally disabling the protection:

sudo -u hoster git -C /home/hoster/openresearchlabs.github.io remote -v
sudo -u hoster git -C /home/hoster/openresearchlabs.github.io branch --show-current
sudo -u hoster git -C /home/hoster/openresearchlabs.github.io log -1 --date=iso --pretty=fuller

The earlier debug session hit this with /home/hoster/openresearchlabs.github.io, and the recommended approach was to use sudo -u hoster rather than blindly adding safe.directory.

Rollback

Roll back report content

Go to the report repo clone:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport log --oneline -n 10

Reset to a known good commit:

sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport reset --hard <GOOD_COMMIT>
sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport lfs pull

Note: the next cron run will pull the latest remote state again. For a durable rollback, revert or fix the remote repo too.

Roll back Apache config

Restore the backup:

sudo cp /etc/apache2/sites-available/000-default.conf.bak.YYYY-MM-DD-HHMM /etc/apache2/sites-available/000-default.conf
sudo apachectl configtest
sudo systemctl reload apache2

Checklist for Adding a New Report

Copy this section into the wiki as the quick operator checklist.

# 1. Create report parent directory
sudo -u hoster mkdir -p /home/hoster/collaboration/reports

# 2. Make sure GitHub SSH works
sudo -u hoster ssh -T git@github.com

# 3. Clone report repo
sudo -u hoster git -C /home/hoster/collaboration/reports clone git@github.com:<ORG_OR_USER>/<REPO>.git fopp-newreport

# 4. Install Git LFS if missing
sudo apt-get update
sudo apt-get install -y git-lfs
sudo git lfs install --system
sudo -u hoster git lfs install --force

# 5. Pull LFS assets
sudo -u hoster git -C /home/hoster/collaboration/reports/fopp-newreport lfs pull

# 6. Confirm output exists
ls -la /home/hoster/collaboration/reports/fopp-newreport/output/index.html

# 7. Add cron log folder
sudo -u hoster mkdir -p /home/hoster/logs

# 8. Edit hoster cron
sudo -u hoster crontab -e

# Add:
# 9 * * * * cd /home/hoster/collaboration/reports/fopp-newreport && git reset --hard && git pull --ff-only && git lfs pull >> /home/hoster/logs/fopp-newreport-cron.log 2>&1

# 9. Test update manually
sudo -u hoster bash -lc 'cd /home/hoster/collaboration/reports/fopp-newreport && git reset --hard && git pull --ff-only && git lfs pull'

# 10. Add Apache Alias and Directory block
sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/000-default.conf.bak.$(date +%F-%H%M)
sudoedit /etc/apache2/sites-available/000-default.conf

# 11. Validate and reload Apache
sudo apachectl configtest
sudo systemctl reload apache2

# 12. Verify public or protected URL
curl -I https://datascience.utu.fi/reports/newreport/

Final mental model

For future maintainers, the model is:

Main site:
  openresearchlabs.github.io
      -> server cron
      -> Hugo build
      -> /var/www/html
      -> https://datascience.utu.fi/

Heavy reports:
  fopp-husna / fopp-lotta / fopp-newreport
      -> server cron under hoster
      -> git pull + git lfs pull
      -> repo output/ directory
      -> Apache Alias
      -> https://datascience.utu.fi/reports/<report>/

The setup only “just works” after these pieces exist: server clone, GitHub SSH access, Git LFS, cron update, Apache Alias, and optional Basic Auth. This was the path established in the original debugging session.

Clone this wiki locally