We use cookies to improve your experience with our site.

Indexed in:

SCIE, EI, Scopus, INSPEC, DBLP, CSCD, etc.

Submission System
(Author / Reviewer / Editor)
Advanced Search
Volume 36 Issue 6
November  2021
Turn off MathJax
Article Contents
Abstract
References
Related Articles
Supplements
Dan-Hao Zhu, Xin-Yu Dai, Jia-Jun Chen. Pre-Train and Learn: Preserving Global Information for Graph Neural Networks[J]. Journal of Computer Science and Technology, 2021, 36(6): 1420-1430. DOI: 10.1007/s11390-020-0142-x
Citation: Dan-Hao Zhu, Xin-Yu Dai, Jia-Jun Chen. Pre-Train and Learn: Preserving Global Information for Graph Neural Networks[J]. Journal of Computer Science and Technology, 2021, 36(6): 1420-1430. DOI: 10.1007/s11390-020-0142-x
shu

Pre-Train and Learn: Preserving Global Information for Graph Neural Networks

  • 1 Library, Jiangsu Police Institute, Nanjing 210031, China;
    2 Department of Computer Science and Technology, Nanjing University, Nanjing 210093, China

Funds: This work was partially supported by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant No. 18kJB510010, the Social Science Foundation of Jiangsu Province of China under Grant No. 19TQD002, and the National Nature Science Foundation of China under Grant No. 61976114.
More Information
  • Corresponding author:

    Xin-Yu Dai E-mail: daixinyu@nju.edu.cn

  • Received Date: October 29, 2019
  • Revised Date: October 08, 2020
  • Published Date: November 29, 2021
    Abstract
  • Graph neural networks (GNNs) have shown great power in learning on graphs. However, it is still a challenge for GNNs to model information faraway from the source node. The ability to preserve global information can enhance graph representation and hence improve classification precision. In the paper, we propose a new learning framework named G-GNN (Global information for GNN) to address the challenge. First, the global structure and global attribute features of each node are obtained via unsupervised pre-training, and those global features preserve the global information associated with the node. Then, using the pre-trained global features and the raw attributes of the graph, a set of parallel kernel GNNs is used to learn different aspects from these heterogeneous features. Any general GNN can be used as a kernal and easily obtain the ability of preserving global information, without having to alter their own algorithms. Extensive experiments have shown that state-of-the-art models, e.g., GCN, GAT, Graphsage and APPNP, can achieve improvement with G-GNN on three standard evaluation datasets. Specially, we establish new benchmark precision records on Cora (84.31%) and Pubmed (80.95%) when learning on attributed graphs.
    • FullText(HTML)
    • References (31)
  • [1]
    Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks. arXiv:1609.02907, 2016. https://arxiv.org/abs/1609.02907, September 2020.
    [2]
    Hamilton W, Ying Z, Leskovec J. Inductive representation learning on large graphs. In Proc. the 31st International Conference on Neural Information Processing Systems, December 2017, pp.1024-1034.
    [3]
    Velickovic P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y. Graph attention networks. arXiv:1710.10903, 2017. https://arxiv.org/abs/1710.10903, October 2020.
    [4]
    Li Q, Han Z, Wu X M. Deeper insights into graph convolutional networks for semi-supervised learning. In Proc. the 32nd AAAI Conference on Artificial Intelligence, February 2018, pp.3538-3545.
    [5]
    Abu-El-Haija S, Kapoor A, Perozzi B, Lee J. N-GCN: Multi-scale graph convolution for semi-supervised node classification. arXiv:1802.08888, 2018. https://arxiv.org/abs/1802.08888, August 2020.
    [6]
    Klicpera J, Bojchevski A, Günnemann S. Predict then propagate: Graph neural networks meet personalized PageRank. arXiv preprint arXiv:1810.05997, 2018. https://arxiv.org/abs/1810.05997, October 2020.
    [7]
    Perozzi B, Al-Rfou, R, Skiena S. DeepWalk: Online learning of social representations. In Proc. the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2014, pp.701-710. DOI: 10.1145/2623330.2623732.
    [8]
    Grover A, Leskovec J. node2vec: Scalable feature learning for networks. In Proc. the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 2016, pp.855-864. DOI: 10.1145/29396-72.2939754.
    [9]
    Albert R, Jeong H, Barabasi A L. Internet: Diameter of the world-wide web. Nature, 1999, 401(6749): 130-131. DOI: 10.1038/43601.
    [10]
    Broder A, Kumar R, Maghoul F, Raghavan P, Rajagopalan S, Stata R, Tomkins A, Wiener J. Graph structure in the web. Computer Networks, 2000, 33(1/2/3/4/5/6): 309-320. DOI: 10.1016/S1389-1286(00)00083-9.
    [11]
    Mikolov T, Sutskever I, Chen K, Corrado G, Dean J. Distributed representations of words and phrases and their compositionality. In Proc. the 26th International Conference on Neural Information Processing Systems, December 2013, pp.3111-3119.
    [12]
    Zhu D, Dai X Y, Yang K, Chen J, He Y. PCANE: Preserving context attributes for network embedding. In Proc. the 23rd Pacific-Asia Conference on Knowledge Discovery and Data Mining, April 2019, pp.156-168. DOI: 10.1007/978-3-030-16142-213.
    [13]
    Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, Kaiser L, Polosukhin I. Attention is all you need. In Proc. the 31st International Conference on Neural Information Processing Systems, December 2017, pp.5998-6008.
    [14]
    Bengio Y, Lamblin P, Popovici D, Larochelle H. Greedy layer-wise training of deep networks. In Proc. the 19th International Conference on Neural Information Processing Systems, December 2006, pp.153-160.
    [15]
    Lin T Y, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. In Proc. the 2017 IEEE International Conference on Computer Vision, October 2017, pp.2980-2988. DOI: 10.1109/ICCV.2017.324.
    [16]
    Sen P, Namata G, Bilgic M et al. Collective classification in network data. AI Magazine, 2008, 29(3): 93-106. DOI: 10.1609/aimag.v29i3.2157.
    [17]
    Yang Z, Cohen W W, Salakhutdinov R. Revisiting semisupervised learning with graph embeddings. arXiv:160-3.08861, 2016. https://arxiv.org/abs/1603.08861, March 2021.
    [18]
    Bojchevski A, Gunnemann S. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. arXiv:1707.03815, 2017. https://arxiv.org/abs/1707.03815, March 2021.
    [19]
    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
    [20]
    Kingma D P, Ba J. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014. https://arxiv.org/abs/14-12.6980, December 2020.
    [21]
    Gao Y, Yang H, Zhang P, Zhou C, Hu Y. Graphnas: Graph neural architecture search with reinforcement learning. arXiv:1904.09981, 2019. https://arxiv.org/abs/1904.09981, April 2021.
    [22]
    Abu-El-Haija S, Perozzi B, Kapoor A, Harutyunyan H, Alipourfard N, Lerman K, Steeg G, Galstyan A. MixHop: Higher-order graph convolution architectures via sparsified neighborhood mixing. arXiv:1905.00067, 2019. https://arxiv.org/abs/1905.00067, May 2021.
    [23]
    Tu C, Zhang W, Liu Z, Sun M. Max-margin DeepWalk: Discriminative learning of network representation. In Proc. the 25th International Joint Conference on Artificial Intelligence, July 2016, pp.3889-3895.
    [24]
    Chen W, Mao X, Li X, Zhang Y, Li X. PNE: Label embedding enhanced network embedding. In Proc. the 21st Pacific-Asia Conference on Knowledge Discovery and Data Mining, May 2017, pp.547-560. DOI: 10.1007/978-3-319-57454-743.
    [25]
    Zhu X, Ghahramani Z, Lafferty J D. Semi-supervised learning using Gaussian fields and harmonic functions. In Proc. the 20th International Conference on Machine Learning, August 2003, pp.912-919.
    [26]
    Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q. LINE: Large-scale information network embedding. In Proc. the 24th International Conference on World Wide Web, May 2015, pp.1067-1077. DOI: 10.1145/2736277.2741093.
    [27]
    Jacob Y, Denoyer L, Gallinari P. Learning latent representations of nodes for classifying in heterogeneous social networks. In Proc. the 7th ACM International Conference on Web Search and Data Mining, February 2014, pp.373-382. DOI: 10.1145/2556195.2556225.
    [28]
    Chen J, Ma T, Xiao C. FastGCN: Fast learning with graph convolutional networks via importance sampling. arXiv:1801.10247, 2018. https://arxiv.org/abs/1801.10247, January 2021.
    [29]
    You J, Ying R, Leskovec J. Position-aware graph neural networks. arXiv:1906.04817, 2019. https://arxiv.org/abs/1-906.04817, April 2021.
    [30]
    Xu K, Li C, Tian Y, Sonobe T, Kawarabayashi K I, Jegelka S. Representation learning on graphs with jumping knowledge networks. arXiv:1806.03536, 2018. https://arxiv.org/abs/1806.03536, June 2021.
    [31]
    Tran P V. Learning to make predictions on graphs with autoencoders. In Proc. the 5th IEEE International Conference on Data Science and Advanced Analytics, October 2018, pp.237-245. DOI: 10.1109/DSAA.2018.00034.
    • Relative Articles

  • Related Articles

    [1]Yong-Duo Sui, Xiang Wang, Tianlong Chen, Meng Wang, Xiang-Nan He, Tat-Seng Chua. Inductive Lottery Ticket Learning for Graph Neural Networks[J]. Journal of Computer Science and Technology, 2024, 39(6): 1223-1237. DOI: 10.1007/s11390-023-2583-5
    [2]Peng-Yi Hao, Si-Hao Liu, Cong Bai. Intent-Aware Graph-Level Embedding Learning Based Recommendation[J]. Journal of Computer Science and Technology, 2024, 39(5): 1138-1152. DOI: 10.1007/s11390-024-3522-9
    [3]Quentin Pajon, Swan Serre, Hugo Wissocq, Léo Rabaud, Siba Haidar, Antoun Yaacoub. Balancing Accuracy and Training Time in Federated Learning for Violence Detection in Surveillance Videos: A Study of Neural Network Architectures[J]. Journal of Computer Science and Technology, 2024, 39(5): 1029-1039. DOI: 10.1007/s11390-024-3702-7
    [4]Ke-Ting Cen, Hua-Wei Shen, Qi Cao, Bing-Bing Xu, Xue-Qi Cheng. Identity-Preserving Adversarial Training for Robust Network Embedding[J]. Journal of Computer Science and Technology, 2024, 39(1): 177-191. DOI: 10.1007/s11390-023-2256-4
    [5]Long-Yin Zhang, Xin Tan, Fang Kong, Pei-Feng Li, Guo-Dong Zhou. Top-down Text-Level Discourse Rhetorical Structure Parsing with Bidirectional Representation Learning[J]. Journal of Computer Science and Technology, 2023, 38(5): 985-1001. DOI: 10.1007/s11390-022-1167-0
    [6]Da-Wei Cheng, Yi Tu, Zhen-Wei Ma, Zhi-Bin Niu, Li-Qing Zhang. BHONEM: Binary High-Order Network Embedding Methods for Networked-Guarantee Loans[J]. Journal of Computer Science and Technology, 2019, 34(3): 657-669. DOI: 10.1007/s11390-019-1934-8
    [7]Xi-Jin Zhang, Yi-Fan Lu, Song-Hai Zhang. Multi-Task Learning for Food Identification and Analysis with Deep Convolutional Neural Networks[J]. Journal of Computer Science and Technology, 2016, 31(3): 489-500. DOI: 10.1007/s11390-016-1642-6
    [8]Zhou Yi, Wu ShiLin. NNF and NNPrF—Fuzzy Petri Nets Based on Neural Network for Knowledge Representation, Reasoning and Learning[J]. Journal of Computer Science and Technology, 1996, 11(2): 133-149.
    [9]Yao Shu, Zhang Bo. The Learning Convergence of CMAC in Cyclic Learning[J]. Journal of Computer Science and Technology, 1994, 9(4): 320-328.
    [10]Wu Xindong. Inductive Learning[J]. Journal of Computer Science and Technology, 1993, 8(2): 22-36.
    • Supplements (2)

  • Others

  • Cited by

    Periodical cited type(13)

    1. Tiesunlong Shen, Jin Wang, Xuejie Zhang. Knowledge distillation via adaptive meta-learning for graph neural network. Information Sciences, 2025, 689: 121505. DOI:10.1016/j.ins.2024.121505
    2. Usman Malik, Simon Bernard, Alexandre Pauchet, et al. Pseudo-labeling with Large Language Models for Multi-label Emotion Classification of French Tweets. IEEE Access, 2024. DOI:10.1109/ACCESS.2024.3354705
    3. Xin-Cheng Wen, Cuiyun Gao, Jiaxin Ye, et al. Meta-Path Based Attentional Graph Learning Model for Vulnerability Detection. IEEE Transactions on Software Engineering, 2024, 50(3): 360. DOI:10.1109/TSE.2023.3340267
    4. Michael Adjeisah, Xinzhong Zhu, Huiying Xu, et al. Graph Contrastive Multi-view Learning: A Pre-training Framework for Graph Classification. Knowledge-Based Systems, 2024, 299: 112112. DOI:10.1016/j.knosys.2024.112112
    5. Haitao He, Sheng Wang, Yanmin Wang, et al. VulTR: Software Vulnerability Detection Model Based on Multi-layer Key Feature Enhancement. Computers & Security, 2024. DOI:10.1016/j.cose.2024.104139
    6. Yexin Wang, Zhi Yang, Junqi Liu, et al. Scapin: Scalable Graph Structure Perturbation by Augmented Influence Maximization. Proceedings of the ACM on Management of Data, 2023, 1(2): 1. DOI:10.1145/3589291
    7. Michael Adjeisah, Xinzhong Zhu, Huiying Xu, et al. Towards data augmentation in graph neural network: An overview and evaluation. Computer Science Review, 2023, 47: 100527. DOI:10.1016/j.cosrev.2022.100527
    8. Kai Wang, Zhonghang Zhang, Chaozhi Cai, et al. Intelligent Whistling System of Rail Train Based on YOLOv4 and U-Net. Applied Sciences, 2023, 13(3): 1695. DOI:10.3390/app13031695
    9. José Marcio Duarte, Lilian Berton. A review of semi-supervised learning for text classification. Artificial Intelligence Review, 2023, 56(9): 9401. DOI:10.1007/s10462-023-10393-8
    10. Jiadong Ren, Jiao Zhang, Jiazheng Li, et al. Slice-level vulnerability detection model based on graph neural network. Proceedings of the 2024 3rd International Conference on Cryptography, Network Security and Communication Technology, DOI:10.1145/3673277.3673287
    11. Yu Shen, Xinyuyang Ren, Yupeng Lu, et al. Rover: An Online Spark SQL Tuning Service via Generalized Transfer Learning. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, DOI:10.1145/3580305.3599953
    12. Walid Guettala, László Gulyás. Intelligent Information and Database Systems. Lecture Notes in Computer Science, DOI:10.1007/978-981-97-4985-0_23
    13. Xin-Cheng Wen, Yupan Chen, Cuiyun Gao, et al. Vulnerability Detection with Graph Simplification and Enhanced Graph Representation Learning. 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), DOI:10.1109/ICSE48619.2023.00191

    Other cited types(0)

Catalog

    Get Citation
    PDF
    XML
    Read Online
    Article views (139) PDF downloads (2) Cited by(13)
    Related

    Copyright ©2025 JCST, All Rights Reserved     京ICP备05002829号-1

    Institute of Computing Technology, Chinese Academy of Sciences
    P.O. Box 2704, Beijing 100190, P.R. China

    Tel.: 86-10-62610746 E-mail: jcst@ict.ac.cn

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return