Mansheng Yang
ABSTRACT
Task migration happens when distributed data processing systems scale in real-time. To handle the task migration process more gracefully, we propose three task migration methods: (i) worker level migration, (ii) executor level migration, and (iii) executor level migration with reliable messaging. We implement our migration methods on Apache Storm. Our experiments show that, compared with Storm's original migration implementation, our methods significantly reduce the performance degradation and the number of task failures during each migration.
- Apache storm. http://storm.apache.org.Google Scholar
- R. Castro Fernandez, M. Migliavacca, E. Kalyvianaki, and P. Pietzuch. Integrating scale out and fault tolerance in stream processing using operator state management. In SIGMOD 2013, pages 725--736. ACM, 2013. Google ScholarDigital Library
- T. Heinze, Z. Jerzak, G. Hackenbroich, and C. Fetzer. Latency-aware elastic scaling for distributed data stream processing systems. In DEBS 2014, pages 13--22. ACM, 2014. Google ScholarDigital Library
- S. Schneider, H. Andrade, B. Gedik, A. Biem, and K.-L. Wu. Elastic scaling of data parallel operators in stream processing. In IPDPS 2009, pages 1--12, May 2009. Google ScholarDigital Library
Index Terms
- Smooth Task Migration in Apache Storm
Recommendations
A task migration mechanism for distributed many-core operating systems
Spatial locality of task execution is becoming important in future hardware platforms since the number of cores is steadily increasing. The large amount of cores requires an intelligent power manager and the high chip and core density requires increased ...
Predictable task migration for locked caches in multi-core systems
LCTES '10Locking cache lines in hard real-time systems is a common means of achieving predictability of cache access behavior and tightening as well as reducing worst case execution time, especially in a multitasking environment. However, cache locking poses a ...
A system-level simulation framework for evaluating task migration in MPSoCs
CASES '14: Proceedings of the 2014 International Conference on Compilers, Architecture and Synthesis for Embedded SystemsTask migration is the transfer of the execution of a process (task) from one processing element to another. It originates from the massive deployment of distributed systems in the parallel computing field to enable dynamic load distribution, fault ...
Comments