Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
- [#1126](https://github.com/LayerManager/layman/issues/1126) Endpoint [Workspace Map](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#workspace-map) (with methods GET, PATCH, DELETE) was removed and replaced with endpoint [Map](doc/rest.md#map) (with methods GET, PATCH, DELETE) to use UUID-based URL `/rest/maps/{uuid}` instead of workspace&name-based URL.
- [#1126](https://github.com/LayerManager/layman/issues/1126) Endpoint [Workspace Layer](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#workspace-layer) (with methods GET, PATCH, DELETE) was removed and replaced with endpoint [Layer](doc/rest.md#layer) (with methods GET, PATCH, DELETE) to use UUID-based URL `/rest/layers/{uuid}` instead of workspace&name-based URL.
- [#1126](https://github.com/LayerManager/layman/issues/1126) Endpoint [Workspace Layers](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#workspace-layers) (with methods GET, POST, DELETE) was unified into endpoint [Layers](doc/rest.md#get-layers); `workspace` is now supplied via request query/body parameter instead of URL path.
- [#1126](https://github.com/LayerManager/layman/issues/1126) Endpoint [Workspace Layer Chunk](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#workspace-layer-chunk) (with methods GET, POST) was removed and replaced with endpoint [Layer Chunk](doc/rest.md#layer-chunk) to use UUID-based URL `/rest/layers/{uuid}/chunk` instead of workspace&name-based URL.

## v2.1.0
2025-05-02
Expand Down Expand Up @@ -185,7 +186,7 @@
- [#165](https://github.com/LayerManager/layman/issues/165) POST Workspace [Layers](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#post-workspace-layers)/[Maps](doc/rest.md#post-workspace-maps) and PATCH Workspace [Layer](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#patch-workspace-layer)/[Map](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#patch-workspace-map) saves [role names](doc/models.md#role) mentioned in `access_rights.read` and `access_rights.write` parameters into [prime DB schema](doc/data-storage.md#postgresql).
- [#165](https://github.com/LayerManager/layman/issues/165) Many requests respect roles in access rights:
- [GET](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layer)/[PATCH](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#patch-workspace-layer)/[DELETE](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#delete-workspace-layer) Workspace Layer
- GET Workspace Layer [Thumbnail](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layer-thumbnail)/[Style](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layer-style)/[Metadata Comparison](doc/rest.md#get-workspace-layer-metadata-comparison)/[Chunk](doc/rest.md#get-workspace-layer-chunk)
- GET Workspace Layer [Thumbnail](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layer-thumbnail)/[Style](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layer-style)/[Metadata Comparison](doc/rest.md#get-workspace-layer-metadata-comparison)/[Chunk](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layer-chunk)
- [GET](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-map)/[PATCH](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#patch-workspace-map)/[DELETE](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#delete-workspace-map) Workspace Map
- GET Workspace Map [File](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-map-file)/[Thumbnail](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-map-thumbnail)/[Metadata Comparison](doc/rest.md#get-workspace-map-metadata-comparison)
- GET Workspace [Layers](https://github.com/LayerManager/layman/blob/v2.1.0/doc/rest.md#get-workspace-layers)/[Maps](doc/rest.md#get-workspace-maps)
Expand Down
6 changes: 3 additions & 3 deletions doc/async-file-upload.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,12 +104,12 @@ const onFormSubmit = (event) => {
)
);

// find out layer name
const layername = resp_json[0]['name'];
// find out layer uuid
const layer_uuid = resp_json[0]['uuid'];

// set up resumable.js instance
const resumable = new Resumable({
target: `/rest/workspaces/${form_data.get('workspace')}/layers/${layername}/chunk`,
target: `/rest/layers/${layer_uuid}/chunk`,
query: {
'layman_original_parameter': 'file'
},
Expand Down
18 changes: 9 additions & 9 deletions doc/rest.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
|[Layer](models.md#layer)|`/rest/layers/<uuid>`|[GET](#get-layer)| x | [PATCH](#patch-layer) | [DELETE](#delete-layer) |
|Layer Thumbnail|`/rest/layers/<uuid>/thumbnail`|[GET](#get-layer-thumbnail)| x | x | x |
|Layer Style|`/rest/layers/<uuid>/style`|[GET](#get-layer-style)| x | x | x |
|Workspace Layer Chunk|`/rest/workspaces/<workspace_name>/layers/<layername>/chunk`|[GET](#get-workspace-layer-chunk)| [POST](#post-workspace-layer-chunk) | x | x |
|Layer Chunk|`/rest/layers/<uuid>/chunk`|[GET](#get-layer-chunk)| [POST](#post-layer-chunk) | x | x |
|Workspace Layer Metadata Comparison|`/rest/workspaces/<workspace_name>/layers/<layername>/metadata-comparison`|[GET](#get-workspace-layer-metadata-comparison) | x | x | x |
|Maps|`/rest/maps`|[GET](#get-maps)| x | x | x |
|[Map](models.md#map)|`/rest/maps/<uuid>`|[GET](#get-map)| x | [PATCH](#patch-map) | [DELETE](#delete-map) |
Expand Down Expand Up @@ -122,7 +122,7 @@ Response to this request may be returned sooner than the processing chain is fin
It is possible to upload data files asynchronously, which is suitable for large files. This can be done in three steps:
1. Send POST Layers request with **file** parameter filled by file names that you want to upload
2. Read set of files accepted to upload from POST Layers response, **files_to_upload** property. The set of accepted files will be either equal to or subset of file names sent in **file** parameter.
3. Send [POST Workspace Layer Chunk](#post-workspace-layer-chunk) requests using Resumable.js to upload files.
3. Send [POST Layer Chunk](#post-layer-chunk) requests using Resumable.js to upload files.

Check [Asynchronous file upload](async-file-upload.md) example.

Expand All @@ -148,7 +148,7 @@ Body parameters:
- any of above types in single ZIP file (.zip)
- file names, i.e. array of strings
- it is allowed to publish time-series layer by setting time_regex parameter and sending one or more main raster files (compressed in one archive or uncompressed) with the same extension, color interpretation, pixel size, nodata value, mask flags, and data type name. Filename can be at most 210 characters long. Supported characters are 26 Latin letters `a-zA-Z` (with or without diacritics), numbers, underscores, dashes, dots, and spaces. Other Latin characters (e.g. ligatures `ß` or `Æ`) and other than Latin scripts (e.g. Cyrillic or Chinese) are not supported. Files are stored and published with slugified filenames (diacritic is removed from letters, and space ` ` is converted to underscore `_`).
- if file names are provided, files must be uploaded subsequently using [POST Workspace Layer Chunk](#post-workspace-layer-chunk)
- if file names are provided, files must be uploaded subsequently using [POST Layer Chunk](#post-layer-chunk)
- in case of raster data input, following input combinations of bands and color interpretations are supported:
- 1 band: Gray
- 1 band: Palette
Expand Down Expand Up @@ -230,7 +230,7 @@ JSON array of objects representing posted layers with following structure:
- **name**: String. Name of the layer.
- **uuid**: String. UUID of the layer.
- **url**: String. URL of the layer. It points to [GET Layer](#get-layer).
- *files_to_upload*: List of objects. It's present only if **file** parameter contained file names. Each object represents one file that server expects to be subsequently uploaded using [POST Workspace Layer Chunk](#post-workspace-layer-chunk). Each object has following properties:
- *files_to_upload*: List of objects. It's present only if **file** parameter contained file names. Each object represents one file that server expects to be subsequently uploaded using [POST Layer Chunk](#post-layer-chunk). Each object has following properties:
- **file**: name of the file, equal to one of file name from **file** parameter
- **layman_original_parameter**: name of the request parameter that contained the file name; currently, the only possible value is `file`

Expand Down Expand Up @@ -366,7 +366,7 @@ Body parameters:
- If provided, current data file will be deleted and replaced by this file. GeoServer feature types, DB table, normalized raster file, and thumbnail will be deleted and created again using the new file.
- same file types as in [POST Layers](#post-layers) are expected
- only one of `file` or `external_table_uri` can be set
- if file names are provided, files must be uploaded subsequently using [POST Workspace Layer Chunk](#post-workspace-layer-chunk)
- if file names are provided, files must be uploaded subsequently using [POST Layer Chunk](#post-layer-chunk)
- if published file has empty bounding box (i.e. no features), its bounding box on WMS/WFS endpoint is set to the whole World
- if QML style is used (either directly within this request, or indirectly from previous state on server), it must list all attributes contained in given data file
- it is allowed to publish time-series layer - see [POST Layers](#post-layers)
Expand Down Expand Up @@ -461,7 +461,7 @@ Content-Type:
- `application/x-qgis-layer-settings` for QML


## Workspace Layer Chunk
## Layer Chunk
Layer Chunk endpoint enables to upload layer data files asynchronously by splitting them into small parts called *chunks* that are uploaded independently. The endpoint is expected to be operated using [Resumable.js](https://github.com/23/resumable.js/) library. Resumable.js can split and upload files by chunks using [HTML File API](https://developer.mozilla.org/en-US/docs/Web/API/File), widely supported by major browsers.

Check [Asynchronous file upload](async-file-upload.md) example.
Expand All @@ -472,8 +472,8 @@ The endpoint is activated after [POST Layers](#post-layers) or [PATCH Layer](#pa
- layer is deleted

### URL
`/rest/<workspace_name>/layers/<layername>/chunk`
### GET Workspace Layer Chunk
`/rest/layers/<uuid>/chunk`
### GET Layer Chunk
Test if file chunk is already uploaded on the server.

#### Request
Expand All @@ -487,7 +487,7 @@ Content-Type: `application/json`

HTTP status code 200 if chunk is already uploaded on the server, otherwise 404.

### POST Workspace Layer Chunk
### POST Layer Chunk
Upload file chunk to the server.

#### Request
Expand Down
4 changes: 2 additions & 2 deletions src/layman/layer/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ def get_layer_patch_keys():


from ..common import InternalSourceTypeDef
from .rest_workspace_layer_chunk import bp as workspace_layer_chunk_bp
from .rest_layer_chunk import bp as layer_chunk_bp
from .rest_layer_thumbnail import bp as layer_thumbnail_bp
from .rest_layer_style import bp as layer_style_bp
from .rest_workspace_layer_metadata_comparison import bp as workspace_layer_metadata_comparison_bp
Expand All @@ -52,14 +52,14 @@ def get_layer_patch_keys():
'name': PUBLICATION_TYPE_NAME,
'rest_path_name': LAYER_REST_PATH_NAME,
'workspace_blueprints': [ # blueprints to register
workspace_layer_chunk_bp,
workspace_layer_metadata_comparison_bp,
],
'blueprints': [ # blueprints to register
layers_bp,
layer_bp,
layer_thumbnail_bp,
layer_style_bp,
layer_chunk_bp,
],
# see also .util.TASKS_TO_LAYER_INFO_KEYS
'internal_sources': OrderedDict([ # internal sources to process when new source is published
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,25 @@
from flask import Blueprint, jsonify, request, current_app as app, g

from layman import LaymanError
from layman.util import check_workspace_name_decorator, get_publication_uuid
from layman.util import check_uuid_decorator
from layman.authn import authenticate
from layman.authz import authorize_workspace_publications_decorator
from . import util, LAYER_REST_PATH_NAME
from layman.authz import authorize_uuid_publication_decorator
from . import LAYER_REST_PATH_NAME, LAYER_TYPE
from .filesystem import input_chunk
from .filesystem.util import LAYER_TYPE

bp = Blueprint('rest_workspace_layer_chunk', __name__)
bp = Blueprint('rest_layer_chunk', __name__)


@bp.before_request
@check_workspace_name_decorator
@util.check_layername_decorator
@check_uuid_decorator
@authenticate
@authorize_workspace_publications_decorator
@authorize_uuid_publication_decorator(expected_publication_type=LAYER_TYPE)
def before_request():
pass


@bp.route(f"/{LAYER_REST_PATH_NAME}/<layername>/chunk", methods=['POST'])
def post(workspace, layername):
@bp.route(f"/{LAYER_REST_PATH_NAME}/<uuid>/chunk", methods=['POST'])
def post(uuid):
app.logger.info(f"POST Layer Chunk, actor={g.user}")

total_chunks = request.form.get('resumableTotalChunks', type=int)
Expand All @@ -45,19 +43,17 @@ def post(workspace, layername):
chunk.seek(0, os.SEEK_SET)
app.logger.info(f"POST Layer Chunk, size = {chunk_size}")

publ_uuid = get_publication_uuid(workspace, LAYER_TYPE, layername)
input_chunk.save_layer_file_chunk(publ_uuid, parameter_name,
input_chunk.save_layer_file_chunk(uuid, parameter_name,
filename, chunk,
chunk_number, total_chunks)
# time.sleep(5)

return jsonify({
'message': 'Chunk saved.'
}), 200


@bp.route(f"/{LAYER_REST_PATH_NAME}/<layername>/chunk", methods=['GET'])
def get(workspace, layername):
@bp.route(f"/{LAYER_REST_PATH_NAME}/<uuid>/chunk", methods=['GET'])
def get(uuid):
app.logger.info(f"GET Layer Chunk, actor={g.user}")

chunk_number = request.args.get('resumableChunkNumber', default=1,
Expand All @@ -67,8 +63,7 @@ def get(workspace, layername):
parameter_name = request.args.get('layman_original_parameter', default='error',
type=str)

publ_uuid = get_publication_uuid(workspace, LAYER_TYPE, layername)
chunk_exists = input_chunk.layer_file_chunk_exists(publ_uuid, parameter_name, filename, chunk_number)
chunk_exists = input_chunk.layer_file_chunk_exists(uuid, parameter_name, filename, chunk_number)

if chunk_exists:
result = jsonify({
Expand Down
36 changes: 9 additions & 27 deletions test_tools/process_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
layer_keys_to_check,
'sample/layman.layer/small_layer.geojson',
'rest_workspace_layer_metadata_comparison.get',
'rest_workspace_layer_chunk.post',
'rest_layer_chunk.post',
),
None: PublicationTypeDef('publicationname',
'rest_publications.get',
Expand Down Expand Up @@ -152,18 +152,10 @@ def raise_if_not_complete_status(response):
raise LaymanError(55, data=resp_json)


def upload_file_chunks(publication_type,
workspace,
name,
file_paths,
):
publication_type_def = PUBLICATION_TYPES_DEF[publication_type]
def upload_file_chunks(uuid, file_paths):
time.sleep(0.5)
with app.app_context():
chunk_url = url_for(publication_type_def.post_workspace_publication_chunk,
workspace=workspace,
**{publication_type_def.url_param_name: name},
)
chunk_url = url_for('rest_layer_chunk.post', uuid=uuid)

file_chunks = [('file', file_name) for file_name in file_paths]
for file_type, file_name in file_chunks:
Expand Down Expand Up @@ -341,21 +333,19 @@ def publish_publication(publication_type,
timeout=HTTP_TIMEOUT,
)
raise_layman_error(response)
assert response.json()[0]['name'] == name or not name, f'name={name}, response.name={response.json()[0]["name"]}'
name = name or response.json()[0]['name']
result = response.json()[0]
assert result['name'] == name or not name, f'name={name}, response.name={result["name"]}'
name = name or result['name']

if with_chunks and not do_not_upload_chunks:
upload_file_chunks(publication_type,
workspace,
name,
file_paths, )
upload_file_chunks(result['uuid'], file_paths)

if not do_not_upload_chunks:
wait_for_publication_status(workspace, publication_type, name, check_response_fn=check_response_fn,
headers=headers, raise_if_not_complete=raise_if_not_complete)
if temp_dir:
shutil.rmtree(temp_dir)
return response.json()[0]
return result


def publish_workspace_publication(publication_type,
Expand Down Expand Up @@ -900,7 +890,7 @@ def patch_publication_by_uuid(publication_type,
assert all(key in response.json() for key in expected_resp_keys), f'uuid={uuid}, response={response.json()}'

if with_chunks and not do_not_upload_chunks:
upload_file_chunks_by_uuid(publication_type, uuid, file_paths,)
upload_file_chunks(uuid, file_paths)

if not do_not_upload_chunks:
wait_for_publication_status_by_uuid(uuid, publication_type,
Expand All @@ -918,14 +908,6 @@ def patch_publication_by_uuid(publication_type,
patch_layer = partial(patch_publication_by_uuid, LAYER_TYPE)


def upload_file_chunks_by_uuid(publication_type, uuid, file_paths):
with app.app_context():
pub_info = layman_util.get_publication_info_by_uuid(uuid, context={'keys': ['workspace', 'name']})
workspace = pub_info.get('_workspace')
name = pub_info.get('name')
return upload_file_chunks(publication_type, workspace, name, file_paths)


def wait_for_publication_status_by_uuid(uuid, publication_type, *, check_response_fn=None, headers=None,
raise_if_not_complete=True, sleeping_time=0.5):
with app.app_context():
Expand Down
Loading