Journal of Computer Science and Technology ›› 2022, Vol. 37 ›› Issue (1): 252-265.doi: 10.1007/s11390-021-0655-y

Special Issue: Artificial Intelligence and Pattern Recognition; Theory and Algorithms

• Regular Paper • Previous Articles     Next Articles

On the Discrete-Time Dynamics of Cross-Coupled Hebbian Algorithm

Xiao-Wei Feng (冯晓伟), Xiang-Yu Kong* (孔祥玉), Chuan He (何川), and Dong-Hui Xu (徐东辉)        

  1. Xi'an Research Institute of High Technology, Xi'an 710025, China
  • Received:2020-05-21 Revised:2021-07-20 Accepted:2021-12-29 Online:2022-01-28 Published:2022-01-28
  • Contact: Xiang-Yu Kong E-mail:xiangyukong01@163.com
  • About author:Xiang-Yu Kong received his Bachelor's degree in control science and engineering from Beijing Institute of Technology, Beijing, in 1990, his Master's degree in control science and engineering from Xi'an Research Institute of High Technology, Xi'an, in 2000, and his Ph.D. degree in control science and engineering from the Xi'an Jiaotong University, Xi'an, in 2005. He is now a professor at Xi'an Research Institute of High Technology, Xi'an. He has authored or coauthored more than 100 refereed articles on IEEE Transactions on Signal Processing, IEEE Transactions on Neural Networks and Learning Systems and others journals, and has published five monographs, including one entitled Principal Component Analysis Networks and Algorithms (Springer, 2017). His research interests cover stochastic system analysis, nonlinear system modeling and its application and complex system fault diagnosis.
  • Supported by:
    The work was supported by the National Natural Science Foundation of China under Grant Nos.61903375, 61673387 and 61773389, the Natural Science Foundation of Shaanxi Province of China under Grant Nos.2020JM-356 and 2020JQ-298, and the Postdoctoral Science Foundation of China under Grant No.2019M663635.

PDF (PC)

32

Supplement

1

Principal/minor component analysis (PCA/MCA), generalized principal/minor component analysis (GPCA/GMCA), and singular value decomposition (SVD) algorithms are important techniques for feature extraction. In the convergence analysis of these algorithms, the deterministic discrete-time (DDT) method can reveal the dynamic behavior of PCA/MCA and GPCA/GMCA algorithms effectively. However, the dynamic behavior of SVD algorithms has not been studied quantitatively because of their special structure. In this paper, for the first time, we utilize the advantages of the DDT method in PCA algorithms analysis to study the dynamics of SVD algorithms. First, taking the cross-coupled Hebbian algorithm as an example, by concatenating the two cross-coupled variables into a single vector, we successfully get a PCA-like DDT system. Second, we analyze the discrete-time dynamic behavior and stability of the PCA-like DDT system in detail based on the DDT method, and obtain the boundedness of the weight vectors and learning rate. Moreover, further discussion shows the universality of the proposed method for analyzing other SVD algorithms. As a result, the proposed method provides a new way to study the dynamical convergence properties of SVD algorithms.

Key words: deterministic discrete-time (DDT) system; singular value decomposition (SVD); cross-correlation matrix; discrete-time dynamic behavior;

[1] Ma H W, Lin Y Z, Nie Z H. Physical interpretation of principal component analysis for structural dynamics through string vibration. International Journal of Structural Stability & Dynamics, 2019, 19(9): Article No.1950109. DOI: 10.1142/S0219455419501098.
[2] Du B Y, Kong X Y, Feng X W. Generalized principal component analysis-based subspace decomposition of fault deviations and its application to fault reconstruction. IEEE Access, 2020, 8: 34177-34186. DOI: 10.1109/ACCESS.2020.2971507.
[3] Yuan G, Shen L, Zheng W S. A decomposition algorithm for the sparse generalized eigenvalue problem. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.6113-6122. DOI: 10.1109/CVPR.2019.00627.
[4] Ma J, Yuan Y. Dimension reduction of image deep feature using PCA. Journal of Visual Communication & Image Representation, 2019, 63: Article No.102578. DOI: 10.1016/j.jvcir.2019.102578.
[5] Omar H M, Morsli M, Yaichi S. Image compression using principal component analysis. In Proc. the 2nd International Conference on Mathematics and Information Technology, Feb. 2020, pp.226-231. DOI: 10.1109/ICMIT47780.2020.9047014.
[6] Harmouche J, Delpha C, Diallo D. Incipient fault detection and diagnosis based on Kullback-Leibler divergence using principal component analysis: Part II. Signal Processing, 2015, 109: 334-344. DOI: 10.1016/j.sigpro.2014.06.023.
[7] Yu D, Wang M, Cheng X. A method for the compound fault diagnosis of gearboxes based on morphological component analysis. Measurement, 2016, 91: 519-531. DOI: 10.1016/j.measurement.2016.05.087.
[8] Shahbazpanahi S, Gershman A, Luo Z Q, Wong K M. Robust adaptive beamforming for general-rank signal models. IEEE Transactions on Signal Processing, 2003, 51(9): 2257-2269. DOI: 10.1109/TSP.2003.815395.
[9] Morgan D R. Adaptive algorithms for solving generalized eigenvalue signal enhancement problems. Signal Processing, 2004, 84(6): 957-968. DOI: 10.1016/j.sigpro.2004.02.002.
[10] Jian Y, Xi C, Xi H. Fast adaptive extraction algorithm for multiple principal generalized eigenvectors. International Journal of Intelligent Systems, 2013, 28(3): 289-306. DOI: 10.1002/int.21570.
[11] Chu D, Liao L Z, Ng M K P, Wang X. Incremental linear discriminant analysis: A fast algorithm and comparisons. IEEE Transactions on Neural Networks & Learning Systems, 2015, 26(11): 2716-2735. DOI: 10.1109/TNNLS.2015.2391201.
[12] Chen Y, Tong S, Cong F, Xu J. Symmetrical singular value decomposition representation for pattern recognition. Neurocomputing, 2016, 214: 143-154. DOI: 10.1016/j.neucom.2016.05.075.
[13] Wang J, Shi D, Cheng D, Zhang Y, Gao J. LRSR: Low-rank-sparse representation for subspace clustering. Neurocomputing, 2016, 214: 1026-1037. DOI: 10.1016/j.neucom.2016.07.015.
[14] Peng X, Tang H, Zhang L. A unified framework for representation-based subspace clustering of out-of-sample and large-scale data. IEEE Transactions on Neural Networks & Learning Systems, 2015, 27(12): 2499-2512. DOI: 10.1109/TNNLS.2015.2490080.
[15] Abbass H H, Mousa Z R. A proposed method of face recognition based on edge detection and SVD. Journal of Engineering & Applied Sciences, 2019, 14(18): 6560-6566. DOI: 10.36478/jeasci.2019.6560.6566.
[16] Peng D, Yi Z, Xiang Y. A unified learning algorithm to extract principal and minor components. Digital Signal Processing, 2009, 19(4): 640-649. DOI: 10.1016/j.dsp.2009.03.004.
[17] Cirrincione G, Cirrincione M, Herault J, Van Huffel S. The MCA EXIN neuron for the minor component analysis. IEEE Transactions on Neural Networks, 2002, 13(1): 160-187. DOI: 10.1109/72.977295.
[18] Kong X Y, Hu C, Ma H G. A unified self-stabilizing neural network algorithm for principal and minor components extraction. IEEE Transactions on Neural Networks & Learning Systems, 2012, 23(2): 185-198. DOI: 10.1109/TNNLS.2011.2178564.
[19] Ben X, Meng W, Wang K, Yan R. An adaptive neural networks formulation for the two-dimensional principal component analysis. Neural Computing & Applications, 2016, 27(5): 1245-1261. DOI: 10.1007/s00521-015-1922-z.
[20] Nguyen T D, Yamada I. A unified convergence analysis of normalized PAST algorithms for estimating principal and minor components. Signal Processing, 2013, 93(1): 176-184. DOI: 10.1016/j.sigpro.2012.07.020.
[21] Nguyen T D, Yamada I. Adaptive normalized quasi-Newton algorithms for extraction of generalized eigen-pairs and their convergence analysis. IEEE Transactions on Signal Processing, 2013, 61(6): 1404-1418. DOI: 10.1109/TSP.2012.2234744.
[22] Feng X W, Kong X W, Ma H G, Si X S. A novel unified and self-stabilizing algorithm for generalized eigenpairs extraction. IEEE Transactions on Neural Networks & Learning Systems, 2017, 28(12): 3032-3044. DOI: 10.1109/TNNLS.2016.2614130.
[23] Ouyang S, Hua Y. Bi-iterative least square versus bi-iterative singular value decomposition for subspace tracking. In Proc. the 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, May 2004, pp.353-356. DOI: 10.1109/ICASSP.2004.1326267.
[24] Feng D Z, Bao Z, Zhang X D. A cross-associative neural network for SVD of nonsquared data matrix in signal processing. IEEE Transactions on Neural Networks, 2001, 12(5): 1215-1221. DOI: 10.1109/72.950149.
[25] Hasan M A. Low-rank approximations with applications to principal singular component learning systems. In Proc. the 47th IEEE Conference on Decision & Control, Jan. 2009, pp.3293-3298. DOI: 10.1109/CDC.2008.4739112.
[26] Feng X W, Kong X Y, Xu D H, Qin J Q. A fast and effective principal singular subspace tracking algorithm. Neurocomputing, 2017, 267: 201-209. DOI: 10.1016/j.neucom.2017.06.006.
[27] Peng D Z, Yi Z. Convergence analysis of a deterministic discrete time system of Feng's MCA learning algorithm. IEEE Transactions on Signal Processing, 2006, 54(9): 3626-3632. DOI: 10.1109/TSP.2006.877662.
[28] Oja E. A simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 1982, 15(3):267-273. DOI: 10.1007/BF00275687.
[29] Ljung L. Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control, 1977, 22(4): 551-575. DOI: 10.1109/TAC.1977.1101561.
[30] Zufiria P J. On the discrete-time dynamics of the basic Hebbian neural network node. IEEE Transactions on Neural Networks, 2002, 13(6): 1342-1352. DOI: 10.1109/TNN.2002.805752.
[31] Mao Y. Global convergence analysis of a self-stabilizing MCA learning algorithm. Neurocomputing, 2005, 67: 321-327. DOI: 10.1016/j.neucom.2005.01.002.
[32] Zhang Y, Mao Y, Jian C L, Kok K T. Convergence analysis of a deterministic discrete time system of Oja's PCA learning algorithm. IEEE Transactions on Neural Networks, 2005, 16(6): 1318-1328. DOI: 10.1109/TNN.2005.852236.
[33] Peng D Z, Yi Z. Global convergence of an adaptive minor component extraction algorithm. Chaos Solitons & Fractals, 2008, 35(3): 550-561. DOI: 10.1016/j.chaos.2006.05.051.
[34] Gao Y B, Kong X Y, Hu C H, Zhang H H, Hou L A. Convergence analysis of Möller algorithm for estimating minor component. Neural Processing Letters, 2015, 42(2): 355-368. DOI: 10.1007/s11063-014-9360-y.
[35] Kong X Y, Hu C, Duan Z S. Deterministic discrete-time system for the analysis of iterative algorithms. In Principal Component Analysis Networks and Algorithms, Kong X Y, Hu C, Duan Z S (eds.), Springer, 2017, pp.149-184. DOI: 10.1007/978-981-10-2915-8-6.
[36] Peng D Z, Yi Z. Dynamics of generalized PCA and MCA learning algorithms. IEEE Transactions on Neural Networks, 2007, 18(6): 1777-1784. DOI: 10.1109/TNN.2007.895821.
[37] Tuan D N, Noriyuki T, Isao Y. An adaptive extraction of generalized eigensubspace by using exact nested orthogonal complement structure. Multidimensional Systems & Signal Processing, 2013, 24(3): 457-483. DOI: 10.1007/s11045-012-0172-9.
[38] Karser A, Schenck W, Möller R. Coupled singular value decomposition of a cross-covariance matrix. International Journal of Neural Systems, 2010, 20(4): 293-318. DOI: 10.1142/S0129065710002437.
[39] Feng D Z, Zhang X D, Bao Z. A neural network learning for adaptively extracting cross-correlation features between two high-dimensional data streams. IEEE Transactions on Neural Networks, 2004, 15(6): 1541-1554. DOI: 10.1109/TNN.2004.838523.
[40] Kong X Y, Ma H G, An Q S, Zhang Q. An effective neural learning algorithm for extracting cross-correlation feature between two high-dimensional data streams. Neural Processing Letters, 2015, 42(2): 459-477. DOI: 10.1007/s11063-014-9367-4.
[41] Diamantaras K, Kung S Y. Cross-correlation neural network models. IEEE Transactions on Signal Processing, 1994, 42(11): 3218-3223. DOI: 10.1109/78.330379.
[42] Haykin S. Adaptive Filter Theory (5th edition). Pearson, 2002.
[43] Oja E, Karhunen J. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of Mathematical Analysis & Applications, 1985, 106(1): 69-84. DOI: 10.1016/0022-247X(85)90131-3.
[44] Stewart G W, Sun J G. Matrix Perturbation Theory (1st edition). Academic Press, 1990.
No related articles found!
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] Qing-Bin Liu, Shi-Zhu He, Kang Liu, Sheng-Ping Liu, Jun Zhao. A Unified Shared-Private Network with Denoising for Dialogue State Tracking[J]. Journal of Computer Science and Technology, 2021, 36(6): 1407 -1419 .
[2] . Online First Under Construction [J]. Journal of Computer Science and Technology, 0, (): 1 .
[3] Dan-Hao Zhu, Xin-Yu Dai, Jia-Jun Chen. Pre-Train and Learn: Preserving Global Information for Graph Neural Networks[J]. Journal of Computer Science and Technology, 2021, 36(6): 1420 -1430 .
[4] Bo Chen, Liang Liu, Hua-Dong Ma. CDM: Content Diffusion Model for Information-Centric Networks[J]. Journal of Computer Science and Technology, 2021, 36(6): 1431 -1451 .
[5] Zhi-Neng Chen, Chong-Wah Ngo, Wei Zhang, Juan Cao, Yu-Gang Jiang. Name-Face Association in Web Videos: A Large-Scale Dataset, Baselines, and Open Issues[J]. , 2014, 29(5): 785 -798 .
[6] Fei Xia, De-Jun Jiang, Jin Xiong, Ning-Hui Sun. A Survey of Phase Change Memory Systems[J]. , 2015, 30(1): 121 -144 .
[7] Zhi-Xing Li, Yue Yu, Tao Wang, Gang Yin, Xin-Jun Mao, Huai-Min Wang. Detecting Duplicate Contributions in Pull-based Model Combining Textual and Change Similarities[J]. Journal of Computer Science and Technology, 2021, 36(1): 191 -206 .
[8] André Brinkmann, Kathryn Mohror, Weikuan Yu, Philip Carns, Toni Cortes, Scott A. Klasky, Alberto Miranda, Franz-Josef Pfreundt, Robert B. Ross, Marc-André Vef. Ad Hoc File Systems for High-Performance Computing[J]. Journal of Computer Science and Technology, 2020, 35(1): 4 -26 .
[9] Yu-Tong Lu, Peng Cheng, Zhi-Guang Chen. Design and Implementation of the Tianhe-2 Data Storage and Management System[J]. Journal of Computer Science and Technology, 2020, 35(1): 27 -46 .
[10] Reza Jafari Ziarani, Reza Ravanmehr. Serendipity in Recommender Systems: A Systematic Literature Review[J]. Journal of Computer Science and Technology, 2021, 36(2): 375 -396 .

ISSN 1000-9000(Print)

         1860-4749(Online)
CN 11-2296/TP

 Home
Editorial Board
Author Guidelines
Subscription		 Journal of Computer Science and Technology
Institute of Computing Technology, Chinese Academy of Sciences
P.O. Box 2704, Beijing 100190 P.R. China
Tel.:86-10-62610746
E-mail: jcst@ict.ac.cn
 
  Copyright ©2015 JCST, All Rights Reserved