Enhanced Userspace and In-Kernel Trace Filtering for Production Systems

Suchakrapani Datt Sharma; Michel Dagenais

doi:10.1007/s11390-016-1690-y

Volume 31 Issue 6

November 2016

Turn off MathJax

Article Contents

Abstract

References

Journal of Computer Science and Technology > 2016 > 31(6): 1161-1178. > DOI: 10.1007/s11390-016-1690-y

Suchakrapani Datt Sharma, Michel Dagenais. Enhanced Userspace and In-Kernel Trace Filtering for Production Systems[J]. Journal of Computer Science and Technology, 2016, 31(6): 1161-1178. DOI: 10.1007/s11390-016-1690-y

Citation:

Previous Article Next Article

PDF

Enhanced Userspace and In-Kernel Trace Filtering for Production Systems

Suchakrapani Datt Sharma Student Member, IEEE,
Michel Dagenais Senior Member, IEEE

Department of Computer and Software Engineering, École Polytechnique de Montréal, Montréal, H3T 1J4, Canada

Funds: This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant No. CRDPJ424666-11 and the research grants from Ericsson, EfficiOS and PROMPT Québec.

More Information

Received Date: September 16, 2015
Revised Date: May 21, 2016
Published Date: November 04, 2016

Abstract

Abstract

Trace tools like LTTng have a very low impact on the traced software as compared with traditional debuggers.However,for long runs,in resource constrained and high throughput environments,such as embedded network switching nodes and production servers,the collective tracing impact on the target software adds up considerably.The overhead is not just in terms of execution time but also in terms of the huge amount of data to be stored,processed and analyzed offline.This paper presents a novel way of dealing with such huge trace data generation by introducing a Just-In-Time (JIT) filter based tracing system,for sieving through the flood of high frequency events,and recording only those that are relevant,when a specific condition is met.With a tiny filtering cost,the user can filter out most events and focus only on the events of interest.We show that in certain scenarios,the JIT compiled filters prove to be three times more effective than similar interpreted filters.We also show that with the increasing number of filter predicates and context variables,the benefits of JIT compilation increase with some JIT compiled filters being even three times faster than their interpreted counterparts.We further present a new architecture,using our filtering system,which can enable co-operative tracing between kernel and process tracing VMs (virtual machines) that share data efficiently.We demonstrate its use through a tracing scenario where the user can dynamically specify syscall latency through the userspace tracing VM whose effect is reflected in tracing decisions made by the kernel tracing VM.We compare the data access performance on our shared memory system and show an almost 100 times improvement over traditional data sharing for co-operative tracing.
- tracing,
- debugging,
- filtering,
- operating system,
- interpreter

FullText(HTML)

References (40)

References

[1]	Ball T, Burckhardt S, de Halleux J, Musuvathi M, Qadeer S.Deconstructing concurrency heisenbugs. In Proc. the 31st International Conference on Software Engineering, May 2009, pp.403-404.
[2]	Bligh M, Desnoyers M, Schultz R. Linux kernel debugging on Google-sized clusters. In Proc. the Linux Symposium, June 2007, pp.29-40.
[3]	Ezzati Jivan N. Multi-level trace abstraction, linking and display[Ph.D. Thesis]. Ecole Polytechnique de Montréal, 2014.
[4]	Starovoitov, A. Tracing:Accelerate tracing filters with BPF[LWN.net]. http://lwn.net/Articles/598545/, May 2016.
[5]	Goulet D. Unified kernel/user-space efficient Linux tracing architecture[Master Thesis]. Ecole Polytechnique de Montréal, 2012.
[6]	Desnoyers M. Low-impact operating system tracing[Ph.D. Thesis]. Ecole Polytechnique de Montréal, 2009.
[7]	McCanne S, Jacobson V. The BSD packet filter:A new architecture for user-level packet capture. In Proc. the USENIX Winter Conference, January 1993, pp.259-269.
[8]	Mogul J, Rashid R, Accetta M. The packet filter:An efficient mechanism for user-level network code. ACM SIGOPS Operating Systems Review, 1999, 21(5):39-51.
[9]	Bailey M L, Gopal B, Pagels M A, Peterson L L, Sarkar P. Pathfinder:A pattern-based packet classifier. In Proc. the 1st Symposium on Operating Systems Design and Implementation, Nov. 1994, pp.115-123.
[10]	Engler D R, Kaashoek M F. DPF:Fast, flexible message demultiplexing using dynamic code generation. ACM SIGCOMM Computer Communication Review, 1996, 26(4):53-59.
[11]	Begel A, McCanne S, Graham S L. BPF+:Exploiting global data-flow optimization in a generalized packet filter architecture. ACM SIGCOMM Computer Communication Review, 1999, 29(4):123-134.
[12]	Wu Z, Xie M, Wang H. Design and implementation of a fast dynamic packet filter. IEEE/ACM Transactions on Networking, 2011, 19(5):1405-1419.
[13]	Sobel L. Secure input overlays:Increasing security for sensitive data on Android[Master Thesis]. Massachusetts Institute of Technology, 2015.
[14]	Cantrill B M, Shapiro M W, Leventhal A H. Dynamic instrumentation of production systems. In Proc. the USENIX Annual Technical Conference, June 27-July 2, 2004, pp.15-28.
[15]	Jacob B, Larson P, Leitao B H, Silva S A M M. SystemTap:Instrumenting the Linux kernel for analyzing performance and functional problems. IBM Redpaper, 2009. http://www.redbooks.ibm.com/abstracts/redp4469.html,Oct.2016.
[16]	Rostedt S.Using the TRACE EVENT() macro (Part 1)[LWN.net]. http://lwn.net/Articles/379903/,May2016.
[17]	Keniston J, Panchamukhi P S, Hiramatsu M. Kernel probes (Kprobes). https://www.kernel.org/doc/Documentation/kprobes.txt,May2016.
[18]	Hiramatsu M. The Enhancement of kernel probing-Kprobes jump optimization. http://tracingsummit.org/w/images/f/fa/HiramatsuLinuxCon2010.pdf,Oct.2016.
[19]	Brown A, Wilson G. The Architecture of Open Source Applications, Volume Ⅱ. CreativeCommons, 2012.
[20]	Buck B, Hollingsworth J K. An API for runtime code patching. International Journal of High Performance Computing Applications, 2000, 14(4):317-329.
[21]	Reddi V J, Settle A, Connors D A, Cohn R S.PIN:A binary instrumentation tool for computer architecture research and education. In Proc. the Workshop on Computer Architecture Education, June 2004.
[22]	Prasad V, Cohen W, Eigler F C, Hunt M, Keniston J, Chen J. Locating system problems using dynamic instrumentation. In Proc. the Linux Symposium, June 2005, pp.49-64.
[23]	Kim T, Zeldovich N. Practical and effective sandboxing for non-root users. In Proc. USENIX Conference on Annual Technical Conference, June 2013, pp.139-144.
[24]	Corbet J. BPF:The universal in-kernel virtual machine[LWN.net]. http://lwn.net/Articles/599755/,May2016.
[25]	Shi Y, Casey K, Ertl M A, Gregg D. Virtual machine showdown:Stack versus registers. ACM Transactions on Architecture and Code Optimization, 2008, 4(4):2:1-2:36.
[26]	Davis B, Beatty A, Casey K, Gregg D, Waldron J. The case for virtual register machines. In Proc. the Workshop on Interpreters, Virtual Machines and Emulators, June 2003, pp.41-49.
[27]	Gebai M, Giraldeau F, Dagenais M. Fine-grained preemption analysis for latency investigation across virtual machines. Journal of Cloud Computing, 2014, 3(1):Article No. 23.
[28]	McDougall R, Mauro J, Gregg B. Solaris Performance and Tools(c) DTrace and MDB Techniques for Solaris 10 and OpenSolaris. Prentice Hall, 2006.
[29]	Ertl M A, Gregg D. The behavior of efficient virtual machine interpreters on modern architectures. In Proc. the 7th International Euro-Par Conference on Parallel Processing, Aug. 2001, pp.403-412.
[30]	Gagnon E M, Hendren L J. SableVM:A research framework for the efficient execution of java bytecode. In Proc. the 1st USENIX Java Virtual Machine Research and Technology Symposium, April 2001, pp.27-40.
[31]	Schulist J, Borkmann D, Starovoitov A. Linux socket filtering aka Berkeley packet filter (BPF). https://www.kernel.org/doc/Documentation/networking/filter.txt,May 016.
[32]	Desnoyers M, McKenney P E, Stern A S, Dagenais M R, Walpole J. User-level implementations of read-copy update. IEEE Transactions on Parallel and Distributed Systems, 2012, 23(2):375-382.
[33]	Chakraborty S.Efficiency of LTTng as a Kernel and Userspace Tracer on Multicore. VDM Verlag Dr. Müller, Saarbrücken, Germany, 2011.
[34]	Desfossez J. LTTng-UST vs SystemTap userspace tracing benchmarks. https://sourceware.org/ml/systemtap/2011-q1/msg00244.html,May2016.
[35]	Lascu O, Bodily S, Harvala M, Singh A K, Song D, Berg F V D. IBM AIX Continuous Availability Features. IBM Redpaper, 2008. http://www.redbooks.ibm.com/redpapers/pdfs/redp4367.pdf,Oct.2016.
[36]	Beamonte R, Dagenais M R. Linux low-latency tracing for multicore hard real-time systems. Advances in Computer Engineering, 2015, 2015:Article ID 261094.
[37]	Neira-Ayuso P, Gasca R M, Lefevre L. Communicating between the kernel and user-space in Linux using netlink sockets. Software-Practice and Experience, 2010, 40(9):797-810.
[38]	Gregg B. eBPF:One small step. http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html,May2016.
[39]	McKenney P E (ed.). Is parallel programming hard, and, if so, what can you do about it? https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html,May2016.
[40]	Reinders J. Processor tracing. https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing,May2016.

Relative Articles

Supplements (0)

Cited By

Get Citation

PDF

XML

Article views (19) PDF downloads (1098)

Indexed in:

Enhanced Userspace and In-Kernel Trace Filtering for Production Systems

Abstract

References

Catalog

Related

Home

Overview

Resources

Contents

Indexed in:

Enhanced Userspace and In-Kernel Trace Filtering for Production Systems

Abstract

References

Catalog

Related

Home

Overview

Resources

Contents

Export File

Citation

Format

Content