Journal of Computer Science and Technology


VTensor: Using Virtual Tensors to Build a Layout-Oblivious AI Programming Framework

Feng Yu1,2 (俞峰), Jia-Cheng Zhao1,2 (赵家程), Member, CCF, Hui-min Cui1,2,* (崔慧敏), Member, CCF, Xiao-bing Feng1,2 (冯晓兵), Member, CCF, and Jingling Xue3 (薛京灵), Senior Member, IEEE   

  1. 1Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
    2School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100080, China
    3School of Computer Science and Engineering, University of New South Wales, Sydney 1466, Australia
  • Received:2021-03-19 Revised:2022-04-07 Accepted:2022-04-12
  • Contact: Hui-min Cui
  • About author:Hui-min Cui received the B.S. and M.S. degrees in computer science from Tsinghua University, Beijing, in 2001 and 2004, respectively, and the Ph.D. degree from the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, in 2012. She is currently a professor at ICT, CAS. Her research interests include compiler optimizations, programming languages, and programming environments.

Tensors are a popular programming interface for developing artificial intelligence (AI) algorithms. Layout refers to the order of placing tensor data in the memory and will affect performance by affecting data locality, therefore the deep neural network library has a convention on the layout. Since AI applications can use arbitrary layouts, and existing AI systems do not provide programming abstractions to shield the layout conventions of libraries, operator developers need to write a lot of layout-related code, which reduces the efficiency of integrating new libraries or developing new operators. Furthermore, the developer assigns the layout conversion operation to the internal operator to deal with the uncertainty of the input layout, thus losing the opportunity for layout optimization. Based on the idea of polymorphism, we propose a layout-agnostic virtual tensor programming interface, namely the VTensor framework, which enables developers to write new operators without caring about the underlying physical layout of tensors. In addition, the VTensor framework performs global layout inference at runtime to transparently resolve the required layout of virtual tensors, and runtime layout-oriented optimizations to globally minimize the number of layout transformation operations. Experimental results demonstrate that with VTensor, developers can avoid writing layout-dependent codes. Compared with TensorFlow, for the 16 operations used in 12 popular networks, VTensor can reduce the lines of code (LOC) of writing a new operation by 47.82% on average, and improve the overall performance by 18.65%, on average.



Key words: artificial intelligence programming; layout-oblivious; tensor processing;

No related articles found!
Full text



No Suggested Reading articles found!

ISSN 1000-9000(Print)

CN 11-2296/TP

Editorial Board
Author Guidelines
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
  Copyright ©2015 JCST, All Rights Reserved