›› 2016, Vol. 31 ›› Issue (6): 1161-1178.doi: 10.1007/s11390-016-1690-y

Special Issue: Computer Architecture and Systems

• Regular Paper • Previous Articles     Next Articles

Enhanced Userspace and In-Kernel Trace Filtering for Production Systems

Suchakrapani Datt Sharma, Student Member, IEEE, Michel Dagenais, Senior Member, IEEE   

  1. Department of Computer and Software Engineering, École Polytechnique de Montréal, Montréal, H3T 1J4, Canada
  • Received:2015-09-17 Revised:2016-05-22 Online:2016-11-05 Published:2016-11-05
  • Supported by:

    This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant No. CRDPJ424666-11 and the research grants from Ericsson, EfficiOS and PROMPT Québec.

Trace tools like LTTng have a very low impact on the traced software as compared with traditional debuggers.However,for long runs,in resource constrained and high throughput environments,such as embedded network switching nodes and production servers,the collective tracing impact on the target software adds up considerably.The overhead is not just in terms of execution time but also in terms of the huge amount of data to be stored,processed and analyzed offline.This paper presents a novel way of dealing with such huge trace data generation by introducing a Just-In-Time (JIT) filter based tracing system,for sieving through the flood of high frequency events,and recording only those that are relevant,when a specific condition is met.With a tiny filtering cost,the user can filter out most events and focus only on the events of interest.We show that in certain scenarios,the JIT compiled filters prove to be three times more effective than similar interpreted filters.We also show that with the increasing number of filter predicates and context variables,the benefits of JIT compilation increase with some JIT compiled filters being even three times faster than their interpreted counterparts.We further present a new architecture,using our filtering system,which can enable co-operative tracing between kernel and process tracing VMs (virtual machines) that share data efficiently.We demonstrate its use through a tracing scenario where the user can dynamically specify syscall latency through the userspace tracing VM whose effect is reflected in tracing decisions made by the kernel tracing VM.We compare the data access performance on our shared memory system and show an almost 100 times improvement over traditional data sharing for co-operative tracing.

[1] Ball T, Burckhardt S, de Halleux J, Musuvathi M, Qadeer S.Deconstructing concurrency heisenbugs. In Proc. the 31st International Conference on Software Engineering, May 2009, pp.403-404.

[2] Bligh M, Desnoyers M, Schultz R. Linux kernel debugging on Google-sized clusters. In Proc. the Linux Symposium, June 2007, pp.29-40.

[3] Ezzati Jivan N. Multi-level trace abstraction, linking and display[Ph.D. Thesis]. Ecole Polytechnique de Montréal, 2014.

[4] Starovoitov, A. Tracing:Accelerate tracing filters with BPF[LWN.net]. http://lwn.net/Articles/598545/, May 2016.

[5] Goulet D. Unified kernel/user-space efficient Linux tracing architecture[Master Thesis]. Ecole Polytechnique de Montréal, 2012.

[6] Desnoyers M. Low-impact operating system tracing[Ph.D. Thesis]. Ecole Polytechnique de Montréal, 2009.

[7] McCanne S, Jacobson V. The BSD packet filter:A new architecture for user-level packet capture. In Proc. the USENIX Winter Conference, January 1993, pp.259-269.

[8] Mogul J, Rashid R, Accetta M. The packet filter:An efficient mechanism for user-level network code. ACM SIGOPS Operating Systems Review, 1999, 21(5):39-51.

[9] Bailey M L, Gopal B, Pagels M A, Peterson L L, Sarkar P. Pathfinder:A pattern-based packet classifier. In Proc. the 1st Symposium on Operating Systems Design and Implementation, Nov. 1994, pp.115-123.

[10] Engler D R, Kaashoek M F. DPF:Fast, flexible message demultiplexing using dynamic code generation. ACM SIGCOMM Computer Communication Review, 1996, 26(4):53-59.

[11] Begel A, McCanne S, Graham S L. BPF+:Exploiting global data-flow optimization in a generalized packet filter architecture. ACM SIGCOMM Computer Communication Review, 1999, 29(4):123-134.

[12] Wu Z, Xie M, Wang H. Design and implementation of a fast dynamic packet filter. IEEE/ACM Transactions on Networking, 2011, 19(5):1405-1419.

[13] Sobel L. Secure input overlays:Increasing security for sensitive data on Android[Master Thesis]. Massachusetts Institute of Technology, 2015.

[14] Cantrill B M, Shapiro M W, Leventhal A H. Dynamic instrumentation of production systems. In Proc. the USENIX Annual Technical Conference, June 27-July 2, 2004, pp.15-28.

[15] Jacob B, Larson P, Leitao B H, Silva S A M M. SystemTap:Instrumenting the Linux kernel for analyzing performance and functional problems. IBM Redpaper, 2009. http://www.redbooks.ibm.com/abstracts/redp4469.html,Oct.2016.

[16] Rostedt S.Using the TRACE EVENT() macro (Part 1)[LWN.net]. http://lwn.net/Articles/379903/,May2016.

[17] Keniston J, Panchamukhi P S, Hiramatsu M. Kernel probes (Kprobes). https://www.kernel.org/doc/Documentation/kprobes.txt,May2016.

[18] Hiramatsu M. The Enhancement of kernel probing-Kprobes jump optimization. http://tracingsummit.org/w/images/f/fa/HiramatsuLinuxCon2010.pdf,Oct.2016.

[19] Brown A, Wilson G. The Architecture of Open Source Applications, Volume Ⅱ. CreativeCommons, 2012.

[20] Buck B, Hollingsworth J K. An API for runtime code patching. International Journal of High Performance Computing Applications, 2000, 14(4):317-329.

[21] Reddi V J, Settle A, Connors D A, Cohn R S.PIN:A binary instrumentation tool for computer architecture research and education. In Proc. the Workshop on Computer Architecture Education, June 2004.

[22] Prasad V, Cohen W, Eigler F C, Hunt M, Keniston J, Chen J. Locating system problems using dynamic instrumentation. In Proc. the Linux Symposium, June 2005, pp.49-64.

[23] Kim T, Zeldovich N. Practical and effective sandboxing for non-root users. In Proc. USENIX Conference on Annual Technical Conference, June 2013, pp.139-144.

[24] Corbet J. BPF:The universal in-kernel virtual machine[LWN.net]. http://lwn.net/Articles/599755/,May2016.

[25] Shi Y, Casey K, Ertl M A, Gregg D. Virtual machine showdown:Stack versus registers. ACM Transactions on Architecture and Code Optimization, 2008, 4(4):2:1-2:36.

[26] Davis B, Beatty A, Casey K, Gregg D, Waldron J. The case for virtual register machines. In Proc. the Workshop on Interpreters, Virtual Machines and Emulators, June 2003, pp.41-49.

[27] Gebai M, Giraldeau F, Dagenais M. Fine-grained preemption analysis for latency investigation across virtual machines. Journal of Cloud Computing, 2014, 3(1):Article No. 23.

[28] McDougall R, Mauro J, Gregg B. Solaris Performance and Tools(c) DTrace and MDB Techniques for Solaris 10 and OpenSolaris. Prentice Hall, 2006.

[29] Ertl M A, Gregg D. The behavior of efficient virtual machine interpreters on modern architectures. In Proc. the 7th International Euro-Par Conference on Parallel Processing, Aug. 2001, pp.403-412.

[30] Gagnon E M, Hendren L J. SableVM:A research framework for the efficient execution of java bytecode. In Proc. the 1st USENIX Java Virtual Machine Research and Technology Symposium, April 2001, pp.27-40.

[31] Schulist J, Borkmann D, Starovoitov A. Linux socket filtering aka Berkeley packet filter (BPF). https://www.kernel.org/doc/Documentation/networking/filter.txt,May 016.

[32] Desnoyers M, McKenney P E, Stern A S, Dagenais M R, Walpole J. User-level implementations of read-copy update. IEEE Transactions on Parallel and Distributed Systems, 2012, 23(2):375-382.

[33] Chakraborty S.Efficiency of LTTng as a Kernel and Userspace Tracer on Multicore. VDM Verlag Dr. Müller, Saarbrücken, Germany, 2011.

[34] Desfossez J. LTTng-UST vs SystemTap userspace tracing benchmarks. https://sourceware.org/ml/systemtap/2011-q1/msg00244.html,May2016.

[35] Lascu O, Bodily S, Harvala M, Singh A K, Song D, Berg F V D. IBM AIX Continuous Availability Features. IBM Redpaper, 2008. http://www.redbooks.ibm.com/redpapers/pdfs/redp4367.pdf,Oct.2016.

[36] Beamonte R, Dagenais M R. Linux low-latency tracing for multicore hard real-time systems. Advances in Computer Engineering, 2015, 2015:Article ID 261094.

[37] Neira-Ayuso P, Gasca R M, Lefevre L. Communicating between the kernel and user-space in Linux using netlink sockets. Software-Practice and Experience, 2010, 40(9):797-810.

[38] Gregg B. eBPF:One small step. http://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html,May2016.

[39] McKenney P E (ed.). Is parallel programming hard, and, if so, what can you do about it? https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html,May2016.

[40] Reinders J. Processor tracing. https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing,May2016.
No related articles found!
Full text



[1] Liu Mingye; Hong Enyu;. Some Covering Problems and Their Solutions in Automatic Logic Synthesis Systems[J]. , 1986, 1(2): 83 -92 .
[2] Chen Shihua;. On the Structure of (Weak) Inverses of an (Weakly) Invertible Finite Automaton[J]. , 1986, 1(3): 92 -100 .
[3] Gao Qingshi; Zhang Xiang; Yang Shufan; Chen Shuqing;. Vector Computer 757[J]. , 1986, 1(3): 1 -14 .
[4] Chen Zhaoxiong; Gao Qingshi;. A Substitution Based Model for the Implementation of PROLOG——The Design and Implementation of LPROLOG[J]. , 1986, 1(4): 17 -26 .
[5] Huang Heyan;. A Parallel Implementation Model of HPARLOG[J]. , 1986, 1(4): 27 -38 .
[6] Min Yinghua; Han Zhide;. A Built-in Test Pattern Generator[J]. , 1986, 1(4): 62 -74 .
[7] Tang Tonggao; Zhao Zhaokeng;. Stack Method in Program Semantics[J]. , 1987, 2(1): 51 -63 .
[8] Min Yinghua;. Easy Test Generation PLAs[J]. , 1987, 2(1): 72 -80 .
[9] Zhu Hong;. Some Mathematical Properties of the Functional Programming Language FP[J]. , 1987, 2(3): 202 -216 .
[10] Li Minghui;. CAD System of Microprogrammed Digital Systems[J]. , 1987, 2(3): 226 -235 .

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
E-mail: jcst@ict.ac.cn
  Copyright ©2015 JCST, All Rights Reserved