基于知识图谱非对称映射的单词嵌入方法
Learning Better Word Embedding by Asymmetric Low-Rank Projection of Knowledge Graph
-
摘要: 单词嵌入是一种将自然单词映射到低维稠密向量空间的方法。单词嵌入已经在很多自然语言处理任务中显示出了很好的效果。但是, 现有的嵌入技术受限于自然文本中的信息错误和丢失问题。为此, 有很多工作致力于利用知识图谱作为额外信息源来提升单词嵌入向量的质量。尽管这些工作取得了一定成功, 它们仍然忽略了知识图谱的一些重要特征:1)知识图谱中的很多关系不是一对一映射, 而是多对一, 一对多甚至多对多的;2)知识图谱中绝大多数的头实体和尾实体来自于不同的语义空间。为了解决这些问题, 在本论文中, 我们提出了ProjectNet算法。ProjectNet首先通过不同的低秩映射矩阵来转换知识图谱中的头实体和尾实体, 之后再对关系进行建模。低秩矩阵可以建模头尾实体之间的非一对一关系, 而头尾实体不同的映射矩阵可以将它们放置于不同的语义空间中。实验结果显示与之前的工作相比, ProjectNet能够产生更准确的单词嵌入向量, 从而显著提升了各项自然语言处理任务的性能。Abstract: Word embedding, which refers to low-dimensional dense vector representations of natural words, has demonstrated its power in many natural language processing tasks. However, it may suffer from the inaccurate and incomplete information contained in the free text corpus as training data. To tackle this challenge, there have been quite a few studies that leverage knowledge graphs as an additional information source to improve the quality of word embedding. Although these studies have achieved certain success, they have neglected some important facts about knowledge graphs: 1) many relationships in knowledge graphs are many-to-one, one-to-many or even many-to-many, rather than simply one-to-one; 2) most head entities and tail entities in knowledge graphs come from very different semantic spaces. To address these issues, in this paper, we propose a new algorithm named ProjectNet. ProjectNet models the relationships between head and tail entities after transforming them with different low-rank projection matrices. The low-rank projection can allow non oneto-one relationships between entities, while different projection matrices for head and tail entities allow them to originate in different semantic spaces. The experimental results demonstrate that ProjectNet yields more accurate word embedding than previous studies, and thus leads to clear improvements in various natural language processing tasks.
-
Keywords:
- natural language processing /
- word embedding /
- neural network /
- knowledge graph
-
-
[1] Bengio Y, Ducharme R, Vincent P, Jauvin C. A neural probabilistic language model. The Journal of Machine Learning Research, 2003, 3: 1137-1155.
[2] Collobert R, Weston J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proc. the 25th International Conference on Machine Learning, July 2008, pp.160-167.
[3] Huang E, Socher R, Manning C, Ng A. Improving word representations via global context and multiple word prototypes. In Proc. the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers, July 2012, Volume 1, pp.873-882.
[4] Mikolov T, Chen K, Corrado G, Dean J. Efficient estimation of word representations in vector space. arXiv Preprint, arXiv:1301.3781, 2013. http://arxiv.org/pdf/1301.3781.pdf, Mar. 2016.
[5] Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of words and phrases and their compositionality. In Proc. the 27th NIPS, December 2013, pp.3111-3119.
[6] Pennington J, Socher R, Manning C. GloVe: Global vectors for word representation. In Proc. the Empirical Methods in Natural Language Processing (EMNLP), October 2014, pp.1532-1543.
[7] Miller G. WordNet: A lexical database for English. Communications of the ACM, 1995, 38(11): 39-41.
[8] Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J. Freebase: A collaboratively created graph database for structuring human knowledge. In Proc. the 2008 ACM SIGMOD International Conference on Management of Data, June 2008, pp.1247-1250.
[9] Bian J, Gao B, Liu T. Knowledge-powered deep learning for word embedding. In Proc. European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Sept. 2014, pp.132-148.
[10] Wang Z, Zhang J, Feng J, Chen Z. Knowledge graph and text jointly embedding. In Proc. the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2014, pp.1591-1601.
[11] Xu C, Bai Y, Bian J, Gao B, Wang G, Liu X, Liu T. RCNET: A general framework for incorporating knowledge into word representations. In Proc. the 23rd ACM International Conference on Information and Knowledge Management, November 2014, pp.1219-1228.
[12] Zheng Z, Si X, Li F, Chang E, Zhu X. Entity disambiguation with freebase. In Proc. the 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology, December 2012, Volume 01, pp.82-89.
[13] Socher R, Chen D, Manning C, Ng A. Reasoning with neural tensor networks for knowledge base completion. In Proc. the 27th Neural Information Processing Systems, December 2013, pp.926-934.
[14] Wang Z, Zhang J, Feng J, Chen Z. Knowledge graph embedding by translating on hyperplanes. In Proc. the 28th AAAI Conference on Artificial Intelligence, July 2014, pp.1112-1119.
[15] Bordes A,Weston J, Collobert R, Bengio Y. Learning structured embeddings of knowledge bases. In Proc. the 25th AAAI Conference on Artificial Intelligence, August 2011, pp.301-307.
[16] Turian J, Ratinov L, Bengio Y. Word representations: A simple and general method for semi-supervised learning. In Proc. the 48th Annual Meeting of the Association for Computational Linguistics, July 2010, pp.384-394.
[17] Botha J A, Blunsom P. Compositional morphology for word representations and language modelling. In Proc. the 31st International Conference on Machine Learning, June 2014, pp.1899-1907.
[18] Tian F, Dai H, Bian J, Gao B, Zhang R, Chen E, Liu T Y. A probabilistic model for learning multi-prototype word embeddings. In Proc. the 25th International Conference on Computational Linguistics, August 2014, pp.151-160.
[19] Luong M T, Socher R, Manning C. Better word representations with recursive neural networks for morphology. In Proc. the 17th Conference on Computational Natural Language Learning, August 2013, pp.104-113.
[20] Cui Q, Gao B, Bian J, Qiu S, Dai H, Liu T. KNET: A general framework for learning word embedding using morphological knowledge. ACM Transactions on Information Systems (TOIS), 2015, 34(1): Article No. 4.
[21] Yu M, Dredze M. Improving lexical embeddings with semantic knowledge. In Proc. the 52nd Annual Meeting of the Association for Computational Linguistics (Short Papers), June 2014, pp.545-550.
[22] Bordes A, Usunier N, Garcia-Durán A, Weston J, Yakhnenko O. Translating embeddings for modeling multirelational data. In Proc. the 27th Neural Information Processing Systems, December 2013, pp.2787-2795.
[23] Jenatton R, Le Roux N, Bordes A, Obozinski G R. A latent factor model for highly multi-relational data. In Proc. the 26th Neural Information Processing Systems, December 2012, pp.3167-3175.
[24] Horn R A, Johnson C R. Matrix Analysis. Cambridge University Press, 1990.
[25] Lin Y, Liu Z, Sun M, Liu Y, Zhu X. Learning entity and relation embeddings for knowledge graph completion. In Proc. the 29th AAAI Conference on Artificial Intelligence, January 2015, pp.2181-2187.
[26] Finkelstein L, Gabrilovich E, Matias Y, Rivlin E, Solan Z, Wolfman G, Ruppin E. Placing search in context: The concept revisited. In Proc. the 10th International Conference on World Wide Web, May 2001, pp.406-414.
计量
- 文章访问数: 44
- HTML全文浏览量: 0
- PDF下载量: 1054
下载:
