Skip to content

Random splitter in a multicore setting #160

@Ohyoukillkenny

Description

@Ohyoukillkenny

Trill supports using multiple threads to process an input stream by setting Config.StreamScheduler = StreamScheduler.OwnedThreads(n), where n is the number of threads.

Let's consider the number of thread to be 2, and the data item is of shape "(key, val)" (keyset = {1,2,3,...,10}).
According to my observation, when I use MapReduce-pattern to compute the sum of data value over key-based partitions, Trill assigns data items with the same key to the same thread with a certain order, e.g., always assigns key = 1, 3, 5, ... to thread 1 and key = 2, 4, ... to thread 2.

Does Trill provide a random splitter that allows us to do things like: key = 1, 2, 5, 7... to thread 1 and key 3, 4,... to thread 2 (i.e., random key-thread correspondence)? Also, can we implement a customized splitter to split the stream?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions