[FinRL] CH3. 플랫폼 구성-(3)Backtest

Financial Reinforcement Learning(FinRL) Platform

Contents

1. Installation
2. Framework Overview
3. Main Component
   • Data
   • Train
   • Backtest
4. Examples

 

CH3. Main Component - (3)Backtest

Introduction

• CH3-(2)에서 학습된 에이전트를 로드하고 테스트 진행
• API 로드(Part1): 각종 코드에서 사용될 API를 설치하고 Import
• 에이전트 추론(Part2): 강화학습으로 학습된 5가지 에이전트를 통해 각각 가격 추론 진행
• 비교군 MVO 제작(Part3): 비교를 위해 수학적 퀀트 방법인 Mean-Valiance Optimization(MVO) 적용
• 비교군 DJIA index 제작(Part4): 비교를 위해 시장 지수인 Donw Jones Index 적용
• 에이전트와 성능 비교(Part5): DJIA vs MVO vs 에이전트를 통해 최종 Backtest 진행

GitHub - AI4Finance-Foundation/FinRL-Tutorials: Tutorials. Please star.

---

Stock NeurIPS2018 Part 3. Backtest¶

This series is a reproduction of paper the process in the paper Practical Deep Reinforcement Learning Approach for Stock Trading.

This is the third and last part of the NeurIPS2018 series, introducing how to use use the agents we trained to do backtest, and compare with baselines such as Mean Variance Optimization and DJIA index.

Other demos can be found at the repo of FinRL-Tutorials).

Part 1. Install Packages¶

In [ ]:

## install required packages
!pip install swig
!pip install wrds
!pip install pyportfolioopt
## install finrl library
!pip install git+https://github.com/AI4Finance-Foundation/FinRL.git

In [ ]:

import sys
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from finrl.meta.preprocessor.yahoodownloader import YahooDownloader
from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv
from finrl.agents.stablebaselines3.models import DRLAgent
from stable_baselines3 import A2C, DDPG, PPO, SAC, TD3

%matplotlib inline
from finrl.config import INDICATORS, TRAINED_MODEL_DIR

In [ ]:

Part 2. Backtesting¶

To backtest the agents, upload trade_data.csv in the same directory of this notebook. For Colab users, just upload trade_data.csv to the default directory.

In [ ]:

train = pd.read_csv('train_data.csv')
trade = pd.read_csv('trade_data.csv')

# If you are not using the data generated from part 1 of this tutorial, make sure 
# it has the columns and index in the form that could be make into the environment. 
# Then you can comment and skip the following lines.
train = train.set_index(train.columns[0])
train.index.names = ['']
trade = trade.set_index(trade.columns[0])
trade.index.names = ['']

Then, upload the trained agent to the same directory, and set the corresponding variable to True.

In [ ]:

if_using_a2c = True
if_using_ddpg = True
if_using_ppo = True
if_using_td3 = True
if_using_sac = True

Load the agents

In [ ]:

trained_a2c = A2C.load(TRAINED_MODEL_DIR + "/agent_a2c") if if_using_a2c else None
trained_ddpg = DDPG.load(TRAINED_MODEL_DIR + "/agent_ddpg") if if_using_ddpg else None
trained_ppo = PPO.load(TRAINED_MODEL_DIR + "/agent_ppo") if if_using_ppo else None
trained_td3 = TD3.load(TRAINED_MODEL_DIR + "/agent_td3") if if_using_td3 else None
trained_sac = SAC.load(TRAINED_MODEL_DIR + "/agent_sac") if if_using_sac else None

Trading (Out-of-sample Performance)¶

We update periodically in order to take full advantage of the data, e.g., retrain quarterly, monthly or weekly. We also tune the parameters along the way, in this notebook we use the in-sample data from 2009-01 to 2020-07 to tune the parameters once, so there is some alpha decay here as the length of trade date extends.

Numerous hyperparameters – e.g. the learning rate, the total number of samples to train on – influence the learning process and are usually determined by testing some variations.

In [ ]:

stock_dimension = len(trade.tic.unique())
state_space = 1 + 2*stock_dimension + len(INDICATORS)*stock_dimension
print(f"Stock Dimension: {stock_dimension}, State Space: {state_space}")

Stock Dimension: 29, State Space: 291

In [ ]:

buy_cost_list = sell_cost_list = [0.001] * stock_dimension
num_stock_shares = [0] * stock_dimension

env_kwargs = {
    "hmax": 100,
    "initial_amount": 1000000,
    "num_stock_shares": num_stock_shares,
    "buy_cost_pct": buy_cost_list,
    "sell_cost_pct": sell_cost_list,
    "state_space": state_space,
    "stock_dim": stock_dimension,
    "tech_indicator_list": INDICATORS,
    "action_space": stock_dimension,
    "reward_scaling": 1e-4
}

In [ ]:

e_trade_gym = StockTradingEnv(df = trade, turbulence_threshold = 70,risk_indicator_col='vix', **env_kwargs)
env_trade, obs_trade = e_trade_gym.get_sb_env()

In [ ]:

df_account_value_a2c, df_actions_a2c = DRLAgent.DRL_prediction(
    model=trained_a2c, 
    environment = e_trade_gym) if if_using_a2c else (None, None)

hit end!

In [ ]:

df_account_value_ddpg, df_actions_ddpg = DRLAgent.DRL_prediction(
    model=trained_ddpg, 
    environment = e_trade_gym) if if_using_ddpg else (None, None)

hit end!

In [ ]:

df_account_value_ppo, df_actions_ppo = DRLAgent.DRL_prediction(
    model=trained_ppo, 
    environment = e_trade_gym) if if_using_ppo else (None, None)

hit end!

In [ ]:

df_account_value_td3, df_actions_td3 = DRLAgent.DRL_prediction(
    model=trained_td3, 
    environment = e_trade_gym) if if_using_td3 else (None, None)

hit end!

In [ ]:

df_account_value_sac, df_actions_sac = DRLAgent.DRL_prediction(
    model=trained_sac, 
    environment = e_trade_gym) if if_using_sac else (None, None)

hit end!

Part 3: Mean Variance Optimization¶

Mean Variance optimization is a very classic strategy in portfolio management. Here, we go through the whole process to do the mean variance optimization and add it as a baseline to compare.

First, process dataframe to the form for MVO weight calculation.

In [ ]:

def process_df_for_mvo(df):
  df = df.sort_values(['date','tic'],ignore_index=True)[['date','tic','close']]
  fst = df
  fst = fst.iloc[0:stock_dimension, :]
  tic = fst['tic'].tolist()

  mvo = pd.DataFrame()

  for k in range(len(tic)):
    mvo[tic[k]] = 0

  for i in range(df.shape[0]//stock_dimension):
    n = df
    n = n.iloc[i * stock_dimension:(i+1) * stock_dimension, :]
    date = n['date'][i*stock_dimension]
    mvo.loc[date] = n['close'].tolist()
  
  return mvo

Helper functions for mean returns and variance-covariance matrix¶

In [ ]:

# Codes in this section partially refer to Dr G A Vijayalakshmi Pai

# https://www.kaggle.com/code/vijipai/lesson-5-mean-variance-optimization-of-portfolios/notebook

def StockReturnsComputing(StockPrice, Rows, Columns): 
  import numpy as np 
  StockReturn = np.zeros([Rows-1, Columns]) 
  for j in range(Columns):        # j: Assets 
    for i in range(Rows-1):     # i: Daily Prices 
      StockReturn[i,j]=((StockPrice[i+1, j]-StockPrice[i,j])/StockPrice[i,j])* 100 
      
  return StockReturn

Calculate the weights for mean-variance¶

In [ ]:

StockData = process_df_for_mvo(train)
TradeData = process_df_for_mvo(trade)

TradeData.to_numpy()

Out[ ]:

array([[ 89.25041962, 230.55337524,  90.06215668, ...,  45.19828033,
         35.11460114, 113.78933716],
       [ 89.25041962, 233.37295532,  90.33027649, ...,  45.29749298,
         36.05947113, 113.33300781],
       [ 91.63790131, 231.57455444,  92.48486328, ...,  45.66953659,
         37.07305145, 113.02877808],
       ...,
       [146.9400177 , 194.42747498, 177.35267639, ...,  46.44018173,
         44.49522781, 145.47285461],
       [147.61225891, 195.41072083, 176.02963257, ...,  46.88179779,
         43.67258072, 144.20289612],
       [147.14762878, 193.44422913, 173.1890564 , ...,  46.48434067,
         42.59680557, 143.02020264]])

In [ ]:

#compute asset returns
arStockPrices = np.asarray(StockData)
[Rows, Cols]=arStockPrices.shape
arReturns = StockReturnsComputing(arStockPrices, Rows, Cols)

#compute mean returns and variance covariance matrix of returns
meanReturns = np.mean(arReturns, axis = 0)
covReturns = np.cov(arReturns, rowvar=False)
 
#set precision for printing results
np.set_printoptions(precision=3, suppress = True)

#display mean returns and variance-covariance matrix of returns
print('Mean returns of assets in k-portfolio 1\n', meanReturns)
print('Variance-Covariance matrix of returns\n', covReturns)

Use PyPortfolioOpt¶

In [ ]:

from pypfopt.efficient_frontier import EfficientFrontier

ef_mean = EfficientFrontier(meanReturns, covReturns, weight_bounds=(0, 0.5))
raw_weights_mean = ef_mean.max_sharpe()
cleaned_weights_mean = ef_mean.clean_weights()
mvo_weights = np.array([1000000 * cleaned_weights_mean[i] for i in range(29)])
mvo_weights

Out[ ]:

array([424250.,      0.,      0.,      0.,      0., 108650.,      0.,
            0.,      0.,      0., 181450.,      0.,      0.,      0.,
            0.,      0.,      0.,      0.,      0.,      0.,  16960.,
            0.,      0.,      0., 133540., 135150.,      0.,      0.,
            0.])

In [ ]:

LastPrice = np.array([1/p for p in StockData.tail(1).to_numpy()[0]])
Initial_Portfolio = np.multiply(mvo_weights, LastPrice)
Initial_Portfolio

Out[ ]:

array([4744.487,    0.   ,    0.   ,    0.   ,    0.   ,  579.993,
          0.   ,    0.   ,    0.   ,    0.   ,  782.279,    0.   ,
          0.   ,    0.   ,    0.   ,    0.   ,    0.   ,    0.   ,
          0.   ,    0.   ,   85.833,    0.   ,    0.   ,    0.   ,
        474.034,  715.534,    0.   ,    0.   ,    0.   ])

In [ ]:

Portfolio_Assets = TradeData @ Initial_Portfolio
MVO_result = pd.DataFrame(Portfolio_Assets, columns=["Mean Var"])
# MVO_result

Part 4: DJIA index¶

Add DJIA index as a baseline to compare with.

In [ ]:

TRAIN_START_DATE = '2009-01-01'
TRAIN_END_DATE = '2020-07-01'
TRADE_START_DATE = '2020-07-01'
TRADE_END_DATE = '2021-10-29'

In [ ]:

df_dji = YahooDownloader(start_date = TRADE_START_DATE,
                     end_date = TRADE_END_DATE,
                     ticker_list = ['dji']).fetch_data()
# df_dji

[*********************100%%**********************]  1 of 1 completed
Shape of DataFrame:  (319, 8)

In [ ]:

df_dji = df_dji[['date','close']]
fst_day = df_dji['close'][0]
dji = pd.merge(df_dji['date'], df_dji['close'].div(fst_day).mul(1000000), 
               how='outer', left_index=True, right_index=True).set_index('date')
# dji

Part 5: Backtesting Results¶

Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy.

In [ ]:

df_result_a2c = df_account_value_a2c.set_index(df_account_value_a2c.columns[0]) if if_using_a2c else None
df_result_ddpg = df_account_value_ddpg.set_index(df_account_value_ddpg.columns[0]) if if_using_ddpg else None
df_result_ppo = df_account_value_ppo.set_index(df_account_value_ppo.columns[0]) if if_using_ppo else None
df_result_td3 = df_account_value_td3.set_index(df_account_value_td3.columns[0]) if if_using_td3 else None
df_result_sac = df_account_value_sac.set_index(df_account_value_sac.columns[0]) if if_using_sac else None

result = pd.DataFrame()
if if_using_a2c: result = pd.merge(result, df_result_a2c, how='outer', left_index=True, right_index=True)
if if_using_ddpg: result = pd.merge(result, df_result_ddpg, how='outer', left_index=True, right_index=True, suffixes=('', '_drop'))
if if_using_ppo: result = pd.merge(result, df_result_ppo, how='outer', left_index=True, right_index=True, suffixes=('', '_drop'))
if if_using_td3: result = pd.merge(result, df_result_td3, how='outer', left_index=True, right_index=True, suffixes=('', '_drop'))
if if_using_sac: result = pd.merge(result, df_result_sac, how='outer', left_index=True, right_index=True, suffixes=('', '_drop'))
result = pd.merge(result, MVO_result, how='outer', left_index=True, right_index=True)
result = pd.merge(result, dji, how='outer', left_index=True, right_index=True).fillna(method='bfill')

/var/folders/pc/8l9dz1f949ddzztd4yfcytx80000gn/T/ipykernel_75154/3895181226.py:14: FutureWarning: DataFrame.fillna with 'method' is deprecated and will raise in a future version. Use obj.ffill() or obj.bfill() instead.
  result = pd.merge(result, dji, how='outer', left_index=True, right_index=True).fillna(method='bfill')

In [ ]:

col_name = []
col_name.append('A2C') if if_using_a2c else None
col_name.append('DDPG') if if_using_ddpg else None
col_name.append('PPO') if if_using_ppo else None
col_name.append('TD3') if if_using_td3 else None
col_name.append('SAC') if if_using_sac else None
col_name.append('Mean Var')
col_name.append('djia') 
result.columns = col_name

In [ ]:

result

Out[ ]:

	A2C	DDPG	PPO	TD3	SAC	Mean Var	djia
date
2020-07-01	1.000000e+06	1.000000e+06	1.000000e+06	1.000000e+06	1.000000e+06	1.001918e+06	1.000000e+06
2020-07-02	1.000796e+06	1.000881e+06	1.000164e+06	1.000382e+06	1.000484e+06	1.004235e+06	1.021449e+06
2020-07-06	1.004186e+06	1.007218e+06	1.000465e+06	1.008231e+06	1.004442e+06	1.023225e+06	1.021449e+06
2020-07-07	9.957335e+05	9.938203e+05	9.993283e+05	9.976369e+05	9.974599e+05	1.014021e+06	1.006031e+06
2020-07-08	9.983700e+05	9.988223e+05	9.995208e+05	1.001148e+06	9.983567e+05	1.029461e+06	1.012912e+06
...	...	...	...	...	...	...	...
2021-10-22	1.379239e+06	1.345651e+06	1.313267e+06	1.460297e+06	1.236512e+06	1.535668e+06	1.386322e+06
2021-10-25	1.382223e+06	1.350216e+06	1.320532e+06	1.462491e+06	1.236275e+06	1.542078e+06	1.388813e+06
2021-10-26	1.384209e+06	1.347201e+06	1.320763e+06	1.462833e+06	1.230254e+06	1.545514e+06	1.389427e+06
2021-10-27	1.367469e+06	1.334625e+06	1.318830e+06	1.452726e+06	1.216500e+06	1.534916e+06	1.379083e+06
2021-10-28	NaN	NaN	NaN	NaN	NaN	NaN	1.388401e+06

336 rows × 7 columns

Now, everything is ready, we can plot the backtest result.

In [ ]:

plt.rcParams["figure.figsize"] = (15,5)
plt.figure()
result.plot()

Out[ ]:

<Axes: xlabel='date'>

<Figure size 1500x500 with 0 Axes>