The author of backtrader tested in this blog that using pypy can significantly improve the speed of backtrader backtesting, saving at least half of the time.
The author looked at the reddit forum and someone said that backtrader could not handle 1.6 million k lines.
The author uses backtrader to perform the following tests:
1 Data
The author first generates 100 stocks, each of which generates 20,000 bars , so there are a total of 2 million bars , which are placed in 100 csv files.
2 Test platform
i7 cpu 32G memory, win10
compares two python interpreters: traditional CPython 3.6.1 and fast pypy 3.6.0.0
3 Strategy The
test strategy is a simple double moving average strategy. All stocks are bought with golden forks and sold with dead forks.
4 Test results of non-executing transactions The
so-called non-executing transactions means that the next method of the strategy is empty, which is equivalent to just loading data. In addition, no indicators are calculated, just simple iterations.
4.1 In the default batch run mode (runonce mode) of backtrader
CPython: Peak memory is 348M, total time is 135 seconds, of which 76 seconds are used to preload data, and the rest is iteration time. An average of 14,713 k-lines are processed per second.
pypy: Peak memory is 269M, total time is 57 seconds, 34971 k lines are processed per second. Compared with CPython, the time is saved by more than half.
4.2 In the mode of exactbars=True
If you do not use the default mode, set exactbars=True and stdstats=False in the cerebro operating parameters, then all data will not be preloaded at once. Let's take a look at the test results.
CPython: 75M peak memory, 114 seconds total time. On average, 17494 k lines are processed per second.
pypy: Peak memory is 49M, total time is 66 seconds, and 30025 k lines are processed per second.
Compared with 4.1, the memory saving is obvious, but the speed saving is not obvious.
5 Test results of executing trades
If pypy executes dual moving average trading logic in the strategy next method, this is the same test as the actual strategy. Run in batch mode runonce mode. Only report the results under pypy.
Peak memory consumption: 1.3G
Execution time: 156 seconds (including index calculation + transaction), 12743 bars per second are processed.
6 Conclusion
Backtrader can handle millions of bars by default.
Try to use pypy (for example, if you don’t need plot output)
7 Test 8000 stocks?
Under pypy, if you backtest 8,000 stocks, each with 20,000 bars, which is equivalent to a total of 160 million bars. The time spent is roughly estimated to be 80 times the above, that is, 156x80=12480 seconds=3.5 hours. The memory may consume several gigabytes.