Feat: add martin integration

[mashazyu]

This is a proof of concept PR by request of @felixerdy

Type of Change

• Dependency upgrade
• Bug fix (non-breaking change)
• Breaking change
  • e.g. a fixed bug or new feature that may break something else
• New feature
• Code quality improvements
  • e.g. refactoring, documentation, tests, tooling, ...

Implementation

This PR adds Martin tile server integration to enable serving vector tiles from PostGIS database views.

Key Changes

1. Martin Tile Server Setup (docker-compose.yml)

   • Added Martin service running on port 3001 (host) to avoid conflicts with frontend dev server
   • Configured to connect to PostGIS database and automatically discover views/tables with geometry columns (SRID 4326)
   • Added health checks and proper service dependencies

2. Analysis View (drizzle/0023_create_analysis_view.sql)

   • Created denormalized database view analysis_view that combines:
     • Measurement timestamps (createdAt)
     • Device information (boxId, tags)
     • Location geometry (PostGIS Point, SRID 4326)
     • Aggregated sensor measurements as JSONB object
   • Groups measurements by time and device, aggregating all sensor readings into a single JSONB structure
   • Each sensor measurement includes value, unit, and sensor_id metadata

3. API Endpoint (app/routes/api.analysis.ts)

   • New /api/analysis endpoint to query the analysis_view
   • Supports pagination (limit, offset) with max limit of 1000
   • Filtering options:
     • boxId: Filter by specific device ID
     • hasGeometry: Only return rows with non-null geometry
   • Returns paginated JSON response with total count and hasMore flag
   • The markAsRead parameter enables automatic processing workflow - when set to true, the endpoint marks all returned measurement groups as processed in the processed_measurements table and refreshes the materialized view, effectively "clearing" them from subsequent queries while preserving all data in the database

4. Auto-Refresh on New Measurements (app/routes/api.measurements.ts)

   • Automatically refreshes the materialized view after new measurements are inserted
   • Ensures new data is immediately available in the analysis view without manual refresh
   • Uses concurrent refresh when possible to avoid blocking queries

5. Environment Configuration (app/utils/env.server.ts)

   • Added optional MARTIN_URL environment variable (defaults to http://localhost:3001)

6. Documentation (MARTIN_SETUP.md)

   • Comprehensive setup and usage guide
   • Troubleshooting section
   • Examples for integrating Martin tiles in React components with react-map-gl

How It Works

• Martin automatically discovers database views/tables with geometry columns that have SRID 4326
• The analysis_view provides a denormalized structure optimized for data analysis and visualization
• Vector tiles are served on-demand from the PostGIS database, enabling efficient map rendering
• The API endpoint allows programmatic access to the view data with filtering and pagination and automatic data clearing once it is read

Setup and test

1. Run migration

npx tsx ./db/migrate.ts

2. (Re-)start the service

docker-compose up -d

3. Verify Martin is working - navigate one/all of the following links

• http://localhost:3001/analysis_view
• http://localhost:3001/catalog
• http://localhost:3000/api/analysis

4. Testing

• create device with sensors

docker exec frontend-postgres-1 psql -U postgres -d opensensemap -c "
-- Create a test device
INSERT INTO device (id, name, latitude, longitude, user_id, exposure, status, tags)
VALUES ('test-device-001', 'Test SenseBox', 52.5200, 13.4050, 'cleqyv5pi00003uxdszv4mdnk', 'outdoor', 'active', ARRAY['test', 'demo'])
RETURNING id, name;

-- Create 5 sensors for the device
INSERT INTO sensor (id, device_id, title, sensor_type, unit)
VALUES 
  ('test-sensor-001', 'test-device-001', 'Temperature', 'BMP280', '°C'),
  ('test-sensor-002', 'test-device-001', 'Humidity', 'DHT11', '%'),
  ('test-sensor-003', 'test-device-001', 'Pressure', 'BMP280', 'Pa'),
  ('test-sensor-004', 'test-device-001', 'Light', 'TSL45315', 'lux'),
  ('test-sensor-005', 'test-device-001', 'UV', 'VEML6070', 'UV index')
RETURNING id, device_id, title;
"

• post new measurements

curl -X POST http://localhost:3000/api/measurements -H "Content-Type: application/json" -d '[{"sensorId":"test-sensor-001","time":"2024-01-15T10:00:00Z","value":22.5},{"sensorId":"test-sensor-002","time":"2024-01-15T10:00:00Z","value":65.8},{"sensorId":"test-sensor-003","time":"2024-01-15T10:00:00Z","value":101325.0},{"sensorId":"test-sensor-004","time":"2024-01-15T10:00:00Z","value":750.5},{"sensorId":"test-sensor-005","time":"2024-01-15T10:00:00Z","value":5.2}]'

• verify that you see posted measurements at http://localhost:3000/api/analysis

Checklist

• I gave this pull request a meaningful title
• My pull request is targeting the dev branch
• [] I have added documentation to my code
• [] I have deleted code that I have commented out

Additional Information

• This PR adds Martin tile server integration for serving vector tiles from PostGIS
• Related to data visualization and map rendering capabilities
• See MARTIN_SETUP.md for detailed setup and usage instructions

[mashazyu]

issue: I am not sure it needs to be commited

[mashazyu]

@felixerdy

Per our conversation

• I converted sensor data to columns. Downside - if new sensor type is added, we'd need to create and run a migration.
• Api endpoint is not needed, I just added for testing purposes.

Next questions

• if, when and how should data from the view be deleted? currently it's done via api call, but I assume this is not how it's meant to work.