|
计算机科学技术学报 ›› 2023,Vol. 38 ›› Issue (1): 3-24.doi: 10.1007/s11390-023-2879-5
所属专题： Data Management and Data Mining； Computer Networks and Distributed Computing
HXPY: 一个高效处理金融时间序列数据的软件包
HXPY: A High-Performance Data Processing Package for Financial Time-Series Data
1,2 (郭家栋), 1 (彭靖姝), 2,3 (苑 航), and 3,1,* (倪明选), Life Fellow, IEEE 
- 1 The Hong Kong University of Science and Technology, Hong Kong, China
2 International Digital Economy Academy, Shenzhen 518048, China
3 The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511455, China
1. 研究背景：
海量的金融时间序列数据在全球各地每天产生，这样的数据需要被快速分析以提供最大化的价值，许多学术研究和工业应用场景也需要高性能的金融时间序列计算框架。然而，传统的金融时间序列数据计算框架在性能及功能覆盖上表露出一些不足，并且在对多线程计算和CUDA的利用上具有缺陷。
2. 目的：
本篇论文的研究目标是提供一种新的金融时间序列计算框架，兼容Python Pandas的接口，优化单线程的性能表现，并支持多线程和CUDA进行计算加速，同时实现了更多金融时间序列的函数。
3. 方法:
本文提出了HXPY，一个新的金融时间序列计算框架，基于单指令流多数据流(SIMD)、流式算法、内存布局优化等技术，在现代C++中实现并优化了相应函数，并提供了接近原生体验的Python接口，也兼容和其他Python库的相互转换。
4. 结果:
HXPY显现出了显著的性能优势，在单线程的性能对比中，HXPY在文本文件读写上相比Python Pandas取得了5~10倍的性能提升，在时间序列函数上取得了2~3000倍的性能提升，在分组函数中提供了15~200倍的性能提升。同时，HXPY在多线程中相比基于Ray的Modin取得了2~200倍的性能提升，在CUDA测试中相比NVIDIA的cuDF取得了2~400倍的性能提升。
5. 结论:
HXPY实现了一种新的数据框结构，能够高效地处理和计算金融时间序列数据，我们在递增分析中观察到了从最初版本到所有优化手段被采用后带来的显著性能改进。HXPY具有较佳的应用价值，已经在一些研究机构和合作伙伴中进行内部测试和使用。在未来，我们将持续优化并添加更多的函数支持。
| <table class="reference-tab" style="background-color:#FFFFFF;width:914.104px;color:#333333;font-family:Calibri, Arial, 微软雅黑, "font-size:16px;"> <tbody> <tr class="document-box" id="b1"> <td valign="top" class="td1"> [1] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Farnoosh A, Azari B, Ostadabbas S. Deep switching auto-regressive factorization: Application to time series forecasting. <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>, 2021, 35(8): 7394–7403. DOI: <a href="http://dx.doi.org/10.1609/aaai.v35i8.16907">10.1609/aaai.v35i8.16907</a>. </div> </td> </tr> <tr class="document-box" id="b2"> <td valign="top" class="td1"> [2] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Rasul K, Seward C, Schuster I, Vollgraf R. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In <i>Proc</i>. <i>the 38th International Conference on Machine Learning</i>, Jul. 2021, pp.8857–8868. </div> </td> </tr> <tr class="document-box" id="b3"> <td valign="top" class="td1"> [3] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Pan Q Y, Hu W B, Chen N. Two birds with one stone: Series saliency for accurate and interpretable multivariate time series forecasting. In <i>Proc. the 30th International Joint Conference on Artificial Intelligence</i>, Aug. 2021, pp.2884–2891. DOI: <a href="http://dx.doi.org/10.24963/ijcai.2021/397">10.24963/ijcai.2021/397</a>. </div> </td> </tr> <tr class="document-box" id="b4"> <td valign="top" class="td1"> [4] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Lee D, Lee S, Yu H. Learnable dynamic temporal pooling for time-series classification. <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>, 2021, 35(9): 8288–8296. DOI: <a href="http://dx.doi.org/10.1609/aaai.v35i9.17008">10.1609/aaai.v35i9.17008</a>. </div> </td> </tr> <tr class="document-box" id="b5"> <td valign="top" class="td1"> [5] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Mbouopda M F. Uncertain time series classification. In <i>Proc</i>. <i>the 30th International Joint Conference on Artificial Intelligence</i>, Aug. 2021, pp.4903–4904. DOI: <a href="http://dx.doi.org/10.24963/ijcai.2021/683">10.24963/ijcai.2021/683</a>. </div> </td> </tr> <tr class="document-box" id="b6"> <td valign="top" class="td1"> [6] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Yue Z H, Wang Y J, Duan J Y, Yang T M, Huang C R, Tong Y H, Xu B X. TS2Vec: Towards universal representation of time series. <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>, 2022, 36(8): 8980–8987. DOI: <a href="http://dx.doi.org/10.1609/aaai.v36i8.20881">10.1609/aaai.v36i8.20881</a>. </div> </td> </tr> <tr class="document-box" id="b7"> <td valign="top" class="td1"> [7] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Eldele E, Ragab M, Chen Z H, Wu M, Kwoh C K, Li X L, Guan C T. Time-series representation learning via temporal and contextual contrasting. In <i>Proc</i>. <i>the 30th International Joint Conference on Artificial Intelligence</i>, Aug. 2021, pp.2352–2359. <a href="https://doi.org/10.24963/ijcai.2021/324">DOI: </a><a class="mainColor ref-doi" href="http://dx.doi.org/10.24963/ijcai.2021/3243C/a3E" target="_blank">10.24963/ijcai.2021/324</a>. </div> </td> </tr> <tr class="document-box" id="b8"> <td valign="top" class="td1"> [8] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Deng A L, Hooi B. Graph neural network-based anomaly detection in multivariate time series. <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>, 2021, 35(5): 4027–4035. DOI: <a href="http://dx.doi.org/10.1609/aaai.v35i5.16523">10.1609/aaai.v35i5.16523</a>. </div> </td> </tr> <tr class="document-box" id="b9"> <td valign="top" class="td1"> [9] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Kim S, Choi K, Choi H S, Lee B, Yoon S. Towards a rigorous evaluation of time-series anomaly detection. <i>Proceedings of the AAAI Conference on Artificial Intelligence</i>, 2022, 36(7): 7194–7201. DOI: <a href="http://dx.doi.org/10.1609/aaai.v36i7.20680">10.1609/aaai.v36i7.20680</a>. </div> </td> </tr> <tr class="document-box" id="b10"> <td valign="top" class="td1"> [10] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> McGowan M J. The rise of computerized high frequency trading: Use and controversy. <i>Duke L</i>. <i>& Tech</i>. <i>Rev</i>., 2010, 16. </div> </td> </tr> <tr class="document-box" id="b11"> <td valign="top" class="td1"> [11] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Yang X, Liu W Q, Zhou D, Bian J, Liu T Y. Qlib: An AI-oriented quantitative investment platform. arXiv: 2009.11189, 2021. <a href="https://arxiv.org/abs/2009.11189">https://arxiv.org/abs/2009.11189</a>, Dec. 2022. </div> </td> </tr> <tr class="document-box" id="b12"> <td valign="top" class="td1"> [12] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Ding Q G, Wu S F, Sun H, Guo J D, Guo J. Hierarchical multi-scale Gaussian transformer for stock movement prediction. In <i>Proc</i>. <i>the 29th International Joint Conference on Artificial Intelligence</i>, Jul. 2020, pp.4640–4646. DOI: <a href="http://dx.doi.org/10.24963/ijcai.2020/640">10.24963/ijcai.2020/640</a>. </div> </td> </tr> <tr class="document-box" id="b13"> <td valign="top" class="td1"> [13] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Wang J Y, Zhang Y, Tang K, Wu J J, Xiong Z. Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks. In <i>Proc</i>. <i>the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining</i>, Jul. 2019, pp.1900–1908. DOI: <a href="http://dx.doi.org/10.1145/3292500.3330647">10.1145/3292500.3330647</a>. </div> </td> </tr> <tr class="document-box" id="b14"> <td valign="top" class="td1"> [14] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> McKinney W. Pandas: A foundational Python library for data analysis and statistics. <i>Python for High Performance and Scientific Computing</i>, 2011, 14(9): 1–9. </div> </td> </tr> <tr class="document-box" id="b15"> <td valign="top" class="td1"> [15] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Petersohn D. Dataframe systems: Theory, architecture, and implementation. Technical Report No. UCB/EECS-2021-193, University of California, Berkeley, 2021. <a href="https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.html">https://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-193.html</a>, Dec. 2022. </div> </td> </tr> <tr class="document-box" id="b16"> <td valign="top" class="td1"> [16] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Petersohn D, Macke S, Xin D, Ma W, Lee D J L, Mo X X, Gonzalez J E, Hellerstein J M, Joseph A D, Ganesh A. Towards scalable dataframe systems. <i>Proceedings of the VLDB Endowment</i>, 2020, 13(12): 203–204. DOI: <a href="http://dx.doi.org/10.14778/3407790.3407807">10.14778/3407790.3407807</a>. </div> </td> </tr> <tr class="document-box" id="b17"> <td valign="top" class="td1"> [17] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Moritz P, Nishihara R, Wang S, Tumanov A, Liaw R, Liang E, Elibol M, Yang Z H, Paul W, Jordan M I, Stoica I. Ray: A distributed framework for emerging AI applications. In <i>Proc. the 13th USENIX Symposium on Operating Systems Design and Implementation</i>, Oct. 2018, pp.561–577. <a href="https://www.usenix.org/system/files/osdi18-moritz.pdf">https://www.usenix.org/system/files/osdi18-moritz.pdf</a>, Jan. 2023. </div> </td> </tr> <tr class="document-box" id="b18"> <td valign="top" class="td1"> [18] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Petersohn D, Tang D X, Durrani R, Melik-Adamyan A, Gonzalez J E, Joseph A D, Parameswaran A G. Flexible rule-based decomposition and metadata independence in modin: A parallel dataframe system. <i>Proceedings of the VLDB Endowment</i>, 2021, 15(3): 739–751. DOI: <a href="http://dx.doi.org/10.14778/3494124.3494152">10.14778/3494124.3494152</a>. </div> </td> </tr> <tr class="document-box" id="b19"> <td valign="top" class="td1"> [19] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Hord R M. The Illiac IV: The First Supercomputer. Springer Science & Business Media, 2013. </div> </td> </tr> <tr class="document-box" id="b20"> <td valign="top" class="td1"> [20] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Langdale G, Lemire D. Parsing gigabytes of JSON per second. <i>The VLDB Journal</i>, 2019, 28(6): 941–960. DOI: <a href="http://dx.doi.org/10.1007/s00778-019-00578-5">10.1007/s00778-019-00578-5</a>. </div> </td> </tr> <tr class="document-box" id="b21"> <td valign="top" class="td1"> [21] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Watanabe H, Nakagawa K M. SIMD vectorization for the Lennard-Jones potential with AVX2 and AVX-512 instructions. <i>Computer Physics Communications</i>, 2019, 237: 1–7. DOI: <a href="http://dx.doi.org/10.1016/j.cpc.2018.10.028">10.1016/j.cpc.2018.10.028</a>. </div> </td> </tr> <tr class="document-box" id="b22"> <td valign="top" class="td1"> [22] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Kahan W. IEEE standard 754 for binary floatingpoint arithmetic. <i>Lecture Notes on the Status of IEEE</i> <i>754</i>, 1996. </div> </td> </tr> <tr class="document-box" id="b23"> <td valign="top" class="td1"> [23] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Ihaka R, Gentleman R. R: A language for data analysis and graphics. <i>Journal of Computational and Graphical Statistics</i>, 1996, 5(3): 299–314. DOI: <a href="http://dx.doi.org/10.2307/1390807">10.2307/1390807</a>. </div> </td> </tr> <tr class="document-box" id="b24"> <td valign="top" class="td1"> [24] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Bezanson J, Edelman A, Karpinski S, Shah V B. Julia: A fresh approach to numerical computing. <i>SIAM Review</i>, 2017, 59(1): 65–98. DOI: <a href="http://dx.doi.org/10.1137/141000671">10.1137/141000671</a>. </div> </td> </tr> <tr class="document-box" id="b25"> <td valign="top" class="td1"> [25] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Duvinage M, Mazza P, Petitjean M. The intra-day performance of market timing strategies and trading systems based on Japanese candlesticks. <i>Quantitative Finance</i>, 2013, 13(7): 1059–1070. DOI: <a href="http://dx.doi.org/10.1080/14697688.2013.768774">10.1080/14697688.2013.768774</a>. </div> </td> </tr> <tr class="document-box" id="b26"> <td valign="top" class="td1"> [26] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Nelson D M Q, Pereira A C M, De Oliveira R A. Stock market’s price movement prediction with LSTM neural networks. In <i>Proc. International Joint Conference on Neural Networks</i> (<i>IJCNN</i>), May 2017, pp.1419–1426. DOI: <a href="http://dx.doi.org/10.1109/IJCNN.2017.7966019">10.1109/IJCNN.2017.7966019</a>. </div> </td> </tr> <tr class="document-box" id="b27"> <td valign="top" class="td1"> [27] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Tummon E, Raja M A, Ryan C. Trading cryptocurrency with deep deterministic policy gradients. In <i>Proc. the 21st International Conference on Intelligent Data Engineering and Automated Learning</i>, Nov. 2020, pp.245–256. DOI: <a href="http://dx.doi.org/10.1007/978-3-030-62362-3_22">10.1007/978-3-030-62362-3_22</a>. </div> </td> </tr> <tr class="document-box" id="b28"> <td valign="top" class="td1"> [28] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> De Guzman J, Nuffer D. The Spirit parser library: Inline parsing in C++. <i>CC Plus Plus Users Journal</i>, 2003, 21(9): 22–46. </div> </td> </tr> <tr class="document-box" id="b29"> <td valign="top" class="td1"> [29] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Alon N, Matias Y, Szegedy M. The space complexity of approximating the frequency moments. <i>Journal of Computer and System Sciences</i>, 1999, 58(1): 137–147. DOI: <a href="http://dx.doi.org/10.1006/jcss.1997.1545">10.1006/jcss.1997.1545</a>. </div> </td> </tr> <tr class="document-box" id="b30"> <td valign="top" class="td1"> [30] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, Tzoumas K. Apache Flink<span style="line-height:inherit;vertical-align:baseline;">TM</span>: Stream and batch processing in a single engine. <i>Bulletin of the IEEE Computer Society Technical Committee on Data Engineering</i>, 2015, 36(4): 28–38. </div> </td> </tr> <tr class="document-box" id="b31"> <td valign="top" class="td1"> [31] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Iqbal M H, Soomro T R. Big data analysis: Apache storm perspective. <i>International Journal of Computer Trends and Technology</i>, 2015, 19(1): 9–14. DOI: <a href="http://dx.doi.org/10.14445/22312803/IJCTT-V19P103">10.14445/22312803/IJCTT-V19P103</a>. </div> </td> </tr> <tr class="document-box" id="b32"> <td valign="top" class="td1"> [32] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Foley D, Danskin J. Ultra-performance Pascal GPU and NVLink interconnect. <i>IEEE Micro</i>, 2017, 37(2): 7–17. DOI: <a href="http://dx.doi.org/10.1109/MM.2017.37">10.1109/MM.2017.37</a>. </div> </td> </tr> <tr class="document-box" id="b33"> <td valign="top" class="td1"> [33] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Grant D A. The Latin square principle in the design and analysis of psychological experiments. <i>Psychological Bulletin</i>, 1948, 45(5): 427–442. DOI: <a href="http://dx.doi.org/10.1037/h0053912">10.1037/h0053912</a>. </div> </td> </tr> <tr class="document-box" id="b34"> <td valign="top" class="td1"> [34] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Agarap A F. Deep learning using rectified linear units (ReLU). arXiv: 1803.08375, 2018. <a href="https://arxiv.org/abs/1803.08375">https://arxiv.org/abs/1803.08375</a>, Dec. 2022. </div> </td> </tr> <tr class="document-box" id="b35"> <td valign="top" class="td1"> [35] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Harris C R, Millman K J, Van Der walt S J, Gommers R, Virtanen P, Cournapeau D, Wieser E, Taylor J, Berg S, Smith N J, Kern R, Picus M, Hoyer S, Van Kerkwijk M H, Brett M, Haldane A, Del Río J F, Wiebe M, Peterson P, Gérard-Marchant P, Sheppard K, Reddy T, Weckesser W, Abbasi H, Gohlke C, Oliphant T E. Array programming with NumPy. <i>Nature</i>, 2020, 585(7825): 357–362. DOI: <a href="http://dx.doi.org/10.1038/s41586-020-2649-2">10.1038/s41586-020-2649-2</a>. </div> </td> </tr> <tr class="document-box" id="b36"> <td valign="top" class="td1"> [36] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z M, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J J, Chintala S. PyTorch: An imperative style, high-performance deep learning library. In <i>Proc</i>. <i>the 33rd International Conference on Neural Information Processing Systems</i>, Dec. 2019, Article No. 712. </div> </td> </tr> <tr class="document-box" id="b37"> <td valign="top" class="td1"> [37] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Dagum L, Menon R. OpenMP: An industry standard API for shared-memory programming. <i>IEEE Computational Science and Engineering</i>, 1998, 5(1): 46–55. DOI: <a href="http://dx.doi.org/10.1109/99.660313">10.1109/99.660313</a>. </div> </td> </tr> <tr class="document-box" id="b38"> <td valign="top" class="td1"> [38] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Rodriguez S, Cardiff P. A general approach for running Python codes in OpenFOAM using an embedded Pybind11 Python interpreter. <i>OpenFOAM® Journal</i>, 2022, 2: 166–182. DOI: <a href="http://dx.doi.org/10.51560/ofj.v2.79">10.51560/ofj.v2.79</a>. </div> </td> </tr> <tr class="document-box" id="b39"> <td valign="top" class="td1"> [39] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Titus A J, Kishore S, Stavish T, Rogers S M, Ni K. PySEAL: A Python wrapper implementation of the SEAL homomorphic encryption library. arXiv: 1803.01891, 2018. <a href="https://arxiv.org/abs/1803.01891">https://arxiv.org/abs/1803.01891</a>, Dec. 2022. </div> </td> </tr> <tr class="document-box" id="b40"> <td valign="top" class="td1"> [40] </td> <td class="td2"> <div class="reference-en" style="margin:0px;padding:0px;"> Anderson C. Docker [software engineering]. <i>IEEE Software</i>, 2015, 32(3): 102-c3. DOI: <a href="http://dx.doi.org/10.1109/MS.2015.62">10.1109/MS.2015.62</a>. </div> </td> </tr> </tbody> </table> |
| No related articles found! |
|
| 版权所有 © 《计算机科学技术学报》编辑部 本系统由北京玛格泰克科技发展有限公司设计开发 技术支持：support@magtech.com.cn 总访问量： |