Skip to content

[Feature] concurrent write of struct sinker #433

Description

@caiq1nyu

Motivations

the current structure's write model only supports serial writes(serial_parallelizer.rs), which can lead to performance bottlenecks when there are many tables.

[extractor]
extract_type=struct

[sinker]
sink_type=struct

[parallelizer]
parallel_type=serial

direct and rough concurrency may cause data problems, such as creating an index first but the table does not exist

Solution

  • provides a new parallelizer implementation that includes data conflict management in concurrent scenarios.
  • mark conflict information in advance in the data fetched in the extractor (xx-struct-extractor.rs)

'tables' depend on 'database';
'indexs' depend on 'tables';
'tables have FK' deplend on 'other tables' and so on

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions