Coroutines in Python for Data Engineering (0)

Imagine we have several small pieces of data that need to be processed. To minimize the running time, run the data processing job via multiple tasks on a multiple core platform would be the best solution.

There are several ways to achieve the multi-task design, like multi-processing, multi-treading, coroutine …

Multi-processing solutions usually means several machines or clusters, like map-reduce. Because of the natural of process, multi-process architecture could be very complicated and expensive in terms of communication…