›› 2016, Vol. 31 ›› Issue (6): 1151-1160.doi: 10.1007/s11390-016-1689-4

Special Issue: Artificial Intelligence and Pattern Recognition

• Regular Paper • Previous Articles     Next Articles

A Tensor Neural Network with Layerwise Pretraining: Towards Effective Answer Retrieval

Xin-Qi Bao, Yun-Fang Wu*   

  1. Key Laboratory of Computational Linguistics, Peking University, Beijing 100871, China
  • Received:2015-07-29 Revised:2016-05-27 Online:2016-11-05 Published:2016-11-05
  • Supported by:

    This work is supported by the National High Technology Research and Development 863 Program of China under Grant No. 2015AA015403, the National Natural Science Foundation of China under Grant Nos. 61371129 and 61572245, and the Key Program of Social Science Foundation of China under Grant No. 12&ZD227.

In this paper we address the answer retrieval problem in community-based question answering.To fully capture the interactions between question-answer pairs,we propose an original tensor neural network to model the relevance between them.The question and candidate answers are separately embedded into different latent semantic spaces,and a 3-way tensor is then utilized to model the interactions between latent semantics.To initialize the network layers properly,we propose a novel algorithm called denoising tensor autoencoder (DTAE),and then implement a layerwise pretraining strategy using denoising autoencoders (DAE) on word embedding layers and DTAE on the tensor layer.The experimental results show that our tensor neural network outperforms various baselines with other competitive neural network methods,and our pretraining DTAE strategy improves the system's performance and robustness.

[1] Wang B, Liu B, Wang X, Sun C, Zhang D. Deep learning approaches to semantic relevance modeling for Chinese question-answer pairs. ACM Transactions on Asian Language Information Processing (TALIP), 2011, 10(4):Article No. 21.

[2] Hu H, Liu B, Wang B, Liu M, Wang X. Multimodal DBN for predicting high-quality answers in cQA portals. In Proc. the 51st Annual Meeting of the Association for Computational Linguistics, August 2013, pp.843-847.

[3] Lu Z, Li H. A deep architecture for matching short texts. In Proc. the 27th Advances in Neural Information Processing Systems, December 2013, pp.1367-1375.

[4] Hinton G, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18(7):1527-1554.

[5] Salakhutdinov R, Hinton G. Deep Boltzmann machines. In Proc.the 12th International Conference on Artificial Intelligence and Statistics, April 2009, pp.448-455.

[6] Iyyer M, Boyd-Graber J L, Boyd J, Claudino L, Socher R, Daumé Ⅲ H. A neural network for factoid question answering over paragraphs. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2014, pp.633-644.

[7] Bordes A, Chopra S, Weston J. Question answering with subgraph embeddings. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2014, pp.615-620

[8] Blei D M, Ng A Y, Jordan M I. Latent Dirichlet allocation. The Journal of Machine Learning Research, 2003, 3:993-1022.

[9] Yih W T, Chang M W, Meek C, Pastusiak A. Question answering using enhanced lexical semantic models. In Proc. the 51st Annual Meeting of the Association for Computational Linguistics, August 2013, pp.1744-1753.

[10] Zhang J, Salwen J, Glass M, Gliozzo A. Word semantic representations using Bayesian probabilistic tensor factorization. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2014, pp.1522-1531.

[11] Lei T, Xin Y, Zhang Y, Barzilay R, Jaakkola T. Low-rank tensors for scoring dependency structures. In Proc. the 52nd Annual Meeting of the Association for Computational Linguistics, June 2014, pp.1381-1391.

[12] Pei W, Ge T, Chang B. Max margin tensor neural network for Chinese word segmentation. In Proc. the 52nd Annual Meeting of the Association for Computational Linguistics, June 2014, pp.293-303.

[13] Chang K W, Yih W T, Yang B, Meek C. Typed tensor decomposition of knowledge bases for relation extraction. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing, October 2014, pp.1568-1579.

[14] Yan Z, Zhou J. A new approach to answerer recommendation in community question answering services. In Proc. the 34th Advances in Information Retrieval, April 2012, pp.121-132.

[15] Qiu X, Tian L, Huang X. Latent semantic tensor indexing for community-based question answering. In Proc. the 51st Annual Meeting of the Association for Computational Linguistics, August 2013, pp.434-439.

[16] Zhou X, Hu B, Chen Q et al. Answer sequence learning with neural networks for answer selection in community question answering. arXiv:1506.06490, 2015. http://arxiv.org/abs/1506.06490,June 2016.

[17] Qiu X, Huang X. Convolutional neural tensor network architecture for community-based question answering. In Proc. the 24th International Joint Conference on Artificial Intelligence, July 2015, pp.1305-1311.

[18] Mansur M, Pei W, Chang B. Feature-based neural language model and Chinese word segmentation. In Proc. the 6th International Joint Conference on Natural Language Processing, October 2013, pp.1271-1277.

[19] Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv:1301.3781, 2013. http://arxiv.org/abs/1301.3781,June 2016.

[20] Vincent P, Larochelle H, Bengio Y, Manzagol P A. Extracting and composing robust features with denoising autoencoders. In Proc. the 25th International Conference on Machine Learning, June 2008, pp.1096-1103.

[21] Socher R, Pennington J, Huang E H, Ng A Y, Manning C D. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proc. the Conference on Empirical Methods in Natural Language Processing, July 2011, pp.151-161.

[22] Silberer C, Lapata M. Learning grounded meaning representations with autoencoders. InProc. the 52nd Annual Meeting of the Association for Computational Linguistics, June 2014, pp.721-732.

[23] Zhang H, Yu H, Xiong D, Liu Q. HHmm-based Chinese lexical analyzer ICTCLAS.In Proc. the 2nd SIGHAN Workshop on Chinese Language Processing, July 2003, pp.184-187.

[24] Hu B, Lu Z, Li H, Chen Q. Convolutional neural network architectures for matching natural language sentences. In Proc. Advances in Neural Information Processing Systems, December 2015, pp.2042-2050.
No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] Li Minghui;. CAD System of Microprogrammed Digital Systems[J]. , 1987, 2(3): 226 -235 .
[2] Zhou Qihai;. An Improved Graphic Representation for Structured Program Design[J]. , 1991, 6(2): 205 -208 .
[3] Wang Hui; Liu Dayou; Wang Yafei;. Sequential Back-Propagation[J]. , 1994, 9(3): 252 -260 .
[4] Zeng Jianchao; Hidehilio Sanada; Yoshikazu Tezuka;. A Form Evaluation System and Its Data Structure for Brush-Written Chinese Characters[J]. , 1995, 10(1): 35 -41 .
[5] Wang Shijun; Wang Shulin;. Research and Design of a Fuzzy Neural Expert System[J]. , 1995, 10(2): 112 -123 .
[6] Zhao Zhuxing; Min Yinghua; Li Zhongcheng;. Path Sensitization[J]. , 1997, 12(3): 271 -282 .
[7] Hao Ruibing; Wu Jianping;. A Formal Approach to Protocol Interoperability Testing[J]. , 1998, 13(1): 79 -90 .
[8] XIAO Limin; ZHU Mingfa;. Exploiting the Capabilities of the Interconnection Network on Dawning-1000[J]. , 1999, 14(1): 49 -55 .
[9] WEI Hua; LUO Yupin; YANG Shiyuan;. Fault Tolerance of Reconfigurable Bi-Directional Double-Loop LANs[J]. , 1999, 14(4): 379 -385 .
[10] ZHANG Shugong; LIU Ying; FENG Guochen;. The Multiplicity of Zeros of Algebraic System in Eigenvalue Method[J]. , 1999, 14(5): 510 -517 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

Home
Editorial Board
Author Guidelines
Subscription
Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved