计算机科学技术学报 ›› 2022,Vol. 37 ›› Issue (1): 252-265.doi: 10.1007/s11390-021-0655-y

所属专题： Artificial Intelligence and Pattern Recognition； Theory and Algorithms

• • 上一篇    下一篇

互相关Hebbian算法的离散时间动力学特性分析

  

  • 收稿日期:2020-05-21 修回日期:2021-07-20 接受日期:2021-12-29 出版日期:2022-01-28 发布日期:2022-01-28

On the Discrete-Time Dynamics of Cross-Coupled Hebbian Algorithm

Xiao-Wei Feng (冯晓伟), Xiang-Yu Kong* (孔祥玉), Chuan He (何川), and Dong-Hui Xu (徐东辉)        

  1. Xi'an Research Institute of High Technology, Xi'an 710025, China
  • Received:2020-05-21 Revised:2021-07-20 Accepted:2021-12-29 Online:2022-01-28 Published:2022-01-28
  • Contact: Xiang-Yu Kong E-mail:xiangyukong01@163.com
  • About author:Xiang-Yu Kong received his Bachelor's degree in control science and engineering from Beijing Institute of Technology, Beijing, in 1990, his Master's degree in control science and engineering from Xi'an Research Institute of High Technology, Xi'an, in 2000, and his Ph.D. degree in control science and engineering from the Xi'an Jiaotong University, Xi'an, in 2005. He is now a professor at Xi'an Research Institute of High Technology, Xi'an. He has authored or coauthored more than 100 refereed articles on IEEE Transactions on Signal Processing, IEEE Transactions on Neural Networks and Learning Systems and others journals, and has published five monographs, including one entitled Principal Component Analysis Networks and Algorithms (Springer, 2017). His research interests cover stochastic system analysis, nonlinear system modeling and its application and complex system fault diagnosis.
  • Supported by:
    The work was supported by the National Natural Science Foundation of China under Grant Nos.61903375, 61673387 and 61773389, the Natural Science Foundation of Shaanxi Province of China under Grant Nos.2020JM-356 and 2020JQ-298, and the Postdoctoral Science Foundation of China under Grant No.2019M663635.

PDF (PC)

32

补充材料

1

迄今为止，确定性离散时间(DDT)方法已被广泛应用于主/次成分分析(PCA/MCA)和广义主/次成分分析(GPCA/GMCA)神经网络算法的收敛性分析。然而，DDT方法却无法分析针对两路随机信号的互相关矩阵进行奇异值分解(SVD)的神经网络算法的收敛特性。这是因为这类算法在一对耦合方程中同时包含两个交叉耦合的变量(即左奇异向量和右奇异向量)。因此，学者通常采用雅可比法或李亚普诺夫法来分析SVD算法的收敛性。然而，这些传统方法不能揭示神经网络算法的动态行为。本文以互相关Hebbian算法为例，将两个变量进行巧妙结合得到了一个只有一个变量的类PCA算法，进而通过求条件期望得到了一种类PCA算法的DDT系统，该系统在表达形式上与PCA的DDT系统一致，从而为基于DDT方法的算法动态行为分析提供了可能性。在此基础上，利用DDT方法详细研究了互相关Hebbian算法的离散时间动态行为和稳定性，然而因为此时的Rayleigh商不再是全为正数，因而系统的分析过程较常规DDT分析方法更为复杂。最终，通过该思路实现了对SVD神经网络算法的动力学特性的间接分析。据作者所知，已有文献中没有做过类似的研究工作。

关键词: 离散时间系统, 奇异值分解, 互相关矩阵, 离散时间动力学特性

Abstract: Principal/minor component analysis (PCA/MCA), generalized principal/minor component analysis (GPCA/GMCA), and singular value decomposition (SVD) algorithms are important techniques for feature extraction. In the convergence analysis of these algorithms, the deterministic discrete-time (DDT) method can reveal the dynamic behavior of PCA/MCA and GPCA/GMCA algorithms effectively. However, the dynamic behavior of SVD algorithms has not been studied quantitatively because of their special structure. In this paper, for the first time, we utilize the advantages of the DDT method in PCA algorithms analysis to study the dynamics of SVD algorithms. First, taking the cross-coupled Hebbian algorithm as an example, by concatenating the two cross-coupled variables into a single vector, we successfully get a PCA-like DDT system. Second, we analyze the discrete-time dynamic behavior and stability of the PCA-like DDT system in detail based on the DDT method, and obtain the boundedness of the weight vectors and learning rate. Moreover, further discussion shows the universality of the proposed method for analyzing other SVD algorithms. As a result, the proposed method provides a new way to study the dynamical convergence properties of SVD algorithms.

Key words: deterministic discrete-time (DDT) system, singular value decomposition (SVD), cross-correlation matrix, discrete-time dynamic behavior

[1] Ma H W, Lin Y Z, Nie Z H. Physical interpretation of principal component analysis for structural dynamics through string vibration. International Journal of Structural Stability & Dynamics, 2019, 19(9): Article No.1950109. DOI: 10.1142/S0219455419501098.
[2] Du B Y, Kong X Y, Feng X W. Generalized principal component analysis-based subspace decomposition of fault deviations and its application to fault reconstruction. IEEE Access, 2020, 8: 34177-34186. DOI: 10.1109/ACCESS.2020.2971507.
[3] Yuan G, Shen L, Zheng W S. A decomposition algorithm for the sparse generalized eigenvalue problem. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.6113-6122. DOI: 10.1109/CVPR.2019.00627.
[4] Ma J, Yuan Y. Dimension reduction of image deep feature using PCA. Journal of Visual Communication & Image Representation, 2019, 63: Article No.102578. DOI: 10.1016/j.jvcir.2019.102578.
[5] Omar H M, Morsli M, Yaichi S. Image compression using principal component analysis. In Proc. the 2nd International Conference on Mathematics and Information Technology, Feb. 2020, pp.226-231. DOI: 10.1109/ICMIT47780.2020.9047014.
[6] Harmouche J, Delpha C, Diallo D. Incipient fault detection and diagnosis based on Kullback-Leibler divergence using principal component analysis: Part II. Signal Processing, 2015, 109: 334-344. DOI: 10.1016/j.sigpro.2014.06.023.
[7] Yu D, Wang M, Cheng X. A method for the compound fault diagnosis of gearboxes based on morphological component analysis. Measurement, 2016, 91: 519-531. DOI: 10.1016/j.measurement.2016.05.087.
[8] Shahbazpanahi S, Gershman A, Luo Z Q, Wong K M. Robust adaptive beamforming for general-rank signal models. IEEE Transactions on Signal Processing, 2003, 51(9): 2257-2269. DOI: 10.1109/TSP.2003.815395.
[9] Morgan D R. Adaptive algorithms for solving generalized eigenvalue signal enhancement problems. Signal Processing, 2004, 84(6): 957-968. DOI: 10.1016/j.sigpro.2004.02.002.
[10] Jian Y, Xi C, Xi H. Fast adaptive extraction algorithm for multiple principal generalized eigenvectors. International Journal of Intelligent Systems, 2013, 28(3): 289-306. DOI: 10.1002/int.21570.
[11] Chu D, Liao L Z, Ng M K P, Wang X. Incremental linear discriminant analysis: A fast algorithm and comparisons. IEEE Transactions on Neural Networks & Learning Systems, 2015, 26(11): 2716-2735. DOI: 10.1109/TNNLS.2015.2391201.
[12] Chen Y, Tong S, Cong F, Xu J. Symmetrical singular value decomposition representation for pattern recognition. Neurocomputing, 2016, 214: 143-154. DOI: 10.1016/j.neucom.2016.05.075.
[13] Wang J, Shi D, Cheng D, Zhang Y, Gao J. LRSR: Low-rank-sparse representation for subspace clustering. Neurocomputing, 2016, 214: 1026-1037. DOI: 10.1016/j.neucom.2016.07.015.
[14] Peng X, Tang H, Zhang L. A unified framework for representation-based subspace clustering of out-of-sample and large-scale data. IEEE Transactions on Neural Networks & Learning Systems, 2015, 27(12): 2499-2512. DOI: 10.1109/TNNLS.2015.2490080.
[15] Abbass H H, Mousa Z R. A proposed method of face recognition based on edge detection and SVD. Journal of Engineering & Applied Sciences, 2019, 14(18): 6560-6566. DOI: 10.36478/jeasci.2019.6560.6566.
[16] Peng D, Yi Z, Xiang Y. A unified learning algorithm to extract principal and minor components. Digital Signal Processing, 2009, 19(4): 640-649. DOI: 10.1016/j.dsp.2009.03.004.
[17] Cirrincione G, Cirrincione M, Herault J, Van Huffel S. The MCA EXIN neuron for the minor component analysis. IEEE Transactions on Neural Networks, 2002, 13(1): 160-187. DOI: 10.1109/72.977295.
[18] Kong X Y, Hu C, Ma H G. A unified self-stabilizing neural network algorithm for principal and minor components extraction. IEEE Transactions on Neural Networks & Learning Systems, 2012, 23(2): 185-198. DOI: 10.1109/TNNLS.2011.2178564.
[19] Ben X, Meng W, Wang K, Yan R. An adaptive neural networks formulation for the two-dimensional principal component analysis. Neural Computing & Applications, 2016, 27(5): 1245-1261. DOI: 10.1007/s00521-015-1922-z.
[20] Nguyen T D, Yamada I. A unified convergence analysis of normalized PAST algorithms for estimating principal and minor components. Signal Processing, 2013, 93(1): 176-184. DOI: 10.1016/j.sigpro.2012.07.020.
[21] Nguyen T D, Yamada I. Adaptive normalized quasi-Newton algorithms for extraction of generalized eigen-pairs and their convergence analysis. IEEE Transactions on Signal Processing, 2013, 61(6): 1404-1418. DOI: 10.1109/TSP.2012.2234744.
[22] Feng X W, Kong X W, Ma H G, Si X S. A novel unified and self-stabilizing algorithm for generalized eigenpairs extraction. IEEE Transactions on Neural Networks & Learning Systems, 2017, 28(12): 3032-3044. DOI: 10.1109/TNNLS.2016.2614130.
[23] Ouyang S, Hua Y. Bi-iterative least square versus bi-iterative singular value decomposition for subspace tracking. In Proc. the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, May 2004, pp.353-356. DOI: 10.1109/ICASSP.2004.1326267.
[24] Feng D Z, Bao Z, Zhang X D. A cross-associative neural network for SVD of nonsquared data matrix in signal processing. IEEE Transactions on Neural Networks, 2001, 12(5): 1215-1221. DOI: 10.1109/72.950149.
[25] Hasan M A. Low-rank approximations with applications to principal singular component learning systems. In Proc. the 47th IEEE Conference on Decision & Control, Jan. 2009, pp.3293-3298. DOI: 10.1109/CDC.2008.4739112.
[26] Feng X W, Kong X Y, Xu D H, Qin J Q. A fast and effective principal singular subspace tracking algorithm. Neurocomputing, 2017, 267: 201-209. DOI: 10.1016/j.neucom.2017.06.006.
[27] Peng D Z, Yi Z. Convergence analysis of a deterministic discrete time system of Feng's MCA learning algorithm. IEEE Transactions on Signal Processing, 2006, 54(9): 3626-3632. DOI: 10.1109/TSP.2006.877662.
[28] Oja E. A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 1982, 15(3):267-273. DOI: 10.1007/BF00275687.
[29] Ljung L. Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control, 1977, 22(4): 551-575. DOI: 10.1109/TAC.1977.1101561.
[30] Zufiria P J. On the discrete-time dynamics of the basic Hebbian neural network node. IEEE Transactions on Neural Networks, 2002, 13(6): 1342-1352. DOI: 10.1109/TNN.2002.805752.
[31] Mao Y. Global convergence analysis of a self-stabilizing MCA learning algorithm. Neurocomputing, 2005, 67: 321-327. DOI: 10.1016/j.neucom.2005.01.002.
[32] Zhang Y, Mao Y, Jian C L, Kok K T. Convergence analysis of a deterministic discrete time system of Oja's PCA learning algorithm. IEEE Transactions on Neural Networks, 2005, 16(6): 1318-1328. DOI: 10.1109/TNN.2005.852236.
[33] Peng D Z, Yi Z. Global convergence of an adaptive minor component extraction algorithm. Chaos Solitons & Fractals, 2008, 35(3): 550-561. DOI: 10.1016/j.chaos.2006.05.051.
[34] Gao Y B, Kong X Y, Hu C H, Zhang H H, Hou L A. Convergence analysis of Möller algorithm for estimating minor component. Neural Processing Letters, 2015, 42(2): 355-368. DOI: 10.1007/s11063-014-9360-y.
[35] Kong X Y, Hu C, Duan Z S. Deterministic discrete-time system for the analysis of iterative algorithms. In Principal Component Analysis Networks and Algorithms, Kong X Y, Hu C, Duan Z S (eds.), Springer, 2017, pp.149-184. DOI: 10.1007/978-981-10-2915-8-6.
[36] Peng D Z, Yi Z. Dynamics of generalized PCA and MCA learning algorithms. IEEE Transactions on Neural Networks, 2007, 18(6): 1777-1784. DOI: 10.1109/TNN.2007.895821.
[37] Tuan D N, Noriyuki T, Isao Y. An adaptive extraction of generalized eigensubspace by using exact nested orthogonal complement structure. Multidimensional Systems & Signal Processing, 2013, 24(3): 457-483. DOI: 10.1007/s11045-012-0172-9.
[38] Karser A, Schenck W, Möller R. Coupled singular value decomposition of a cross-covariance matrix. International Journal of Neural Systems, 2010, 20(4): 293-318. DOI: 10.1142/S0129065710002437.
[39] Feng D Z, Zhang X D, Bao Z. A neural network learning for adaptively extracting cross-correlation features between two high-dimensional data streams. IEEE Transactions on Neural Networks, 2004, 15(6): 1541-1554. DOI: 10.1109/TNN.2004.838523.
[40] Kong X Y, Ma H G, An Q S, Zhang Q. An effective neural learning algorithm for extracting cross-correlation feature between two high-dimensional data streams. Neural Processing Letters, 2015, 42(2): 459-477. DOI: 10.1007/s11063-014-9367-4.
[41] Diamantaras K, Kung S Y. Cross-correlation neural network models. IEEE Transactions on Signal Processing, 1994, 42(11): 3218-3223. DOI: 10.1109/78.330379.
[42] Haykin S. Adaptive Filter Theory (5th edition). Pearson, 2002.
[43] Oja E, Karhunen J. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of Mathematical Analysis & Applications, 1985, 106(1): 69-84. DOI: 10.1016/0022-247X(85)90131-3.
[44] Stewart G W, Sun J G. Matrix Perturbation Theory (1st edition). Academic Press, 1990.
No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Qing-Bin Liu, Shi-Zhu He, Kang Liu, Sheng-Ping Liu, Jun Zhao. 一种用于对话状态跟踪的统一共享私有网络和去燥方法[J]. 计算机科学技术学报, 2021, 36(6): 1407 -1419 .
[2] . Online First Under Construction [J]. 计算机科学技术学报, 0, (): 1 .
[3] Dan-Hao Zhu, Xin-Yu Dai, Jia-Jun Chen. 预训练和学习：在图神经网络中保留全局信息[J]. 计算机科学技术学报, 2021, 36(6): 1420 -1430 .
[4] Bo Chen, Liang Liu, Hua-Dong Ma. CDM:以信息为中心的网络的内容扩散模型[J]. 计算机科学技术学报, 2021, 36(6): 1431 -1451 .
[5] Zhi-Neng Chen, Chong-Wah Ngo, Wei Zhang, Juan Cao, Yu-Gang Jiang. 网络视频人脸—姓名关联：大规模数据库，基准实验和开放性问题[J]. , 2014, 29(5): 785 -798 .
[6] Fei Xia, De-Jun Jiang, Jin Xiong, Ning-Hui Sun. PCM内存系统研究综述[J]. , 2015, 30(1): 121 -144 .
[7] Zhi-Xing Li, Yue Yu, Tao Wang, Gang Yin, Xin-Jun Mao, Huai-Min Wang. 基于文本和变更相似度的重复性合并请求检测技术[J]. 计算机科学技术学报, 2021, 36(1): 191 -206 .
[8] André Brinkmann, Kathryn Mohror, Weikuan Yu, Philip Carns, Toni Cortes, Scott A. Klasky, Alberto Miranda, Franz-Josef Pfreundt, Robert B. Ross, Marc-André Vef. 高性能计算专用文件系统[J]. 计算机科学技术学报, 2020, 35(1): 4 -26 .
[9] Yu-Tong Lu, Peng Cheng, Zhi-Guang Chen. Tianhe-2数据存储与管理系统设计与实现[J]. 计算机科学技术学报, 2020, 35(1): 27 -46 .
[10] Reza Jafari Ziarani, Reza Ravanmehr. 推荐系统中的意外效应：系统文献综述[J]. 计算机科学技术学报, 2021, 36(2): 375 -396 .
版权所有 © 《计算机科学技术学报》编辑部
本系统由北京玛格泰克科技发展有限公司设计开发 技术支持：support@magtech.com.cn
总访问量：