The 110th original article focuses on "personal growth and wealth freedom, the logic of world operation, and AI quantitative investment".
01 Some feelings
A grain of sand of the times, falling on everyone is like a mountain.
In the past three years, I believe many people can empathize with this sentence.
Looking at the vicissitudes of history, let's just say it lightly-the major changes that have not been seen in a century, just like the Song, Yuan, Ming and Qing Dynasties, just went all the way. For each individual, life is only a hundred years, whether it is long or not, but try to isolate yourself for 14 days?
There are many things we cannot control. Turning the history, the eyes are full of absurd things, just waiting for future generations to comment.
Change what can be changed, accept what cannot be changed, and prepare ourselves for uncertainty.
Open the diary and look back at the past year. Compared with the present, people and things are still relatively familiar, but looking back at today three years ago, many entangled things and entangled people have changed. We always overestimate the changes that will happen in the next year and underestimate the changes that will come in the next ten. Yes, ten years is too long, let's look forward three years, corresponding to my "continuous action" plan, today is the 110th original article, which is the 110th day of the action plan.
I believe that on the 1000th day, the epidemic will dissipate and everyone will be safe. The three years of the pandemic have given us new thinking about life and the future, how to face possible similar black swan events, your financial preparations, your physical health, what we can and should do is to do this well Prepare.
What is the "most important thing" - financial freedom . Just imagine, if your passive income is enough to cover living expenses , it means that you don't have to be tied to a certain city or a certain area. Then, even in the current epidemic, you can go to the countryside and live in low-risk places. Even overseas. This is the key. The more stable your finances, the more free your life and the stronger your ability to resist uncertain risks. At least increase the proportion of passive income in your income structure, the higher the better.
The ability to invest and manage money is a very important skill, which means that you have the ability to make money make money. The follow-up will focus on "quantification for profitability". Two directions, traditional quantification, based on ETF rotation, data processing is simple, and large-scale asset allocation can be done. Reinforcement learning (including machine learning), doing cryptocurrency, etc.
Import a reinforcement learning and finRL framework today.
02 Reinforcement Learning and finRL
finRL is the first open source framework for financial reinforcement learning. The purpose of the project is to unleash the great potential of reinforcement learning in financial transactions. From an ecological point of view, it has integrated hundreds of financial market data sources, cutting-edge reinforcement learning algorithms, financial applications (such as portfolio optimization, cryptocurrency, high-frequency trading, etc.), real offers, cloud deployment, etc.
11.3.1.1 finRL installation
The FinRL framework supports mac, linux and windows10 environments. This article uses windows10 as an example to install.
FinRL does not provide direct installation from pip, you can download the source package from github (download address: https://github.com/AI4Finance-Foundation/FinRL). After downloading and decompressing, execute pip install . in the current development environment (virtual environment) to install.
11.3.2 Source code structure
FinRL is lightweight, and the core code does not exceed 800 lines. The following is the source code structure, mainly three subfolders of applications, agents and meta.
The functions contained in the Meta folder are to build quantitative environment and financial data processing.
The Agents folder contains functions for state-of-the-art reinforcement learning models. Algorithms that support three reinforcement learning packages, elegantrl (developed by finRL), rllib and stablebaseline3.
The Applications folder implements specific applications, such as encryption currency, portfolio management, etc.
Reinforcement learning applied to financial transactions is more "complex" than machine learning. Reinforcement learning requires a gym-like trading environment, and then apply the reinforcement learning model-the algorithm model has a mature baseline in the industry. We understand its principles after learning, can use it proficiently, and can tune parameters. This aspect is different from that of Similar to traditional machine learning, we don't need to implement the gdbt algorithm from scratch, and ordinary people are unlikely to do better than professional teams.
Data preparation and data preprocessing, whether it is traditional rule strategy development, machine learning quantification, or reinforcement learning, are similar, and this part can be unified.
The format of data input is determined by the model, and there is some adaptation work here.
The backtesting and result interpretation after model input is a similar process and can also be reused.
When choosing a model, we focused on ease of use and efficiency. Needless to say, ease of use, our core focus is on strategy, and we should not spend too much energy on the model itself. The efficiency of deep learning is also very important. Otherwise, it will take half a day to run the model once, which will also greatly affect the strategy debugging.
In terms of algorithm selection for reinforcement learning, elegantRL, rllib and stablebaseline3 are available. At the beginning of FinRL, it encapsulated sb3, and later integrated rllib, and then independently developed eletangRL.
StableBaseline3 is a pytorch version of the reinforcement learning algorithm library. It is actively updated and maintained, with detailed documents and rich cases. However, it is a general version and is not customized for finance.
finRL officially believes that its elegantRL (Chinese name: Xiaoya) is much more stable and efficient than sb3, and the finRL ecosystem itself is born for transactions.
The author of ElegantRL's inventory of each framework:
Berkeley's RLlib ray: It is the DRL open source library with the most advantages. It implements a variety of mainstream DRL algorithms, supports distributed training, and supports three deep learning frameworks (TensorFlow1, PyTorch, and TensorFlow2). This library also has disadvantages: a large amount of code, difficult to get started, high coupling, difficult to change, and a large number of dependencies during installation.
OpenAI's baselines: very early open source DRL library, training is very slow and unstable
stable baselines developed by hill: The instability of baselines gave birth to it. It uses the old style of TensorFlow 1, which is less readable and the training is still not stable enough
stable baselines 3: The inconvenience of TensorFlow 1 gave birth to stable baselines 3 based on PyTorch. It's a pity that the code is literally translated according to TF1, and the old framework of baselines is used, which is not suitable for the DRL algorithm after 2018. Until March 2021, it does not support multi-GPU training.
The teaching code of "Mo Fan": A few years ago, the Chinese community only had his teaching code, which sacrificed performance for readability: in an entry-level environment where training only takes a few seconds, his code needs to be trained for more than ten minutes; Train in a slightly more difficult environment. It has contributed to popularizing reinforcement learning, but the community needs a good DRL library.
ElegantRL features:
Lightweight: The amount of code is less than 1000 lines.
Efficient: The performance is closer to Ray RLlib.
Stable: More stable than Stable Baseline 3.
In the future, we will focus on using the algorithm of the elegantRL library and apply it to high-frequency trading of cryptocurrencies.