[FinRL] CH2. 플랫폼 Overview

Financial Reinforcement Learning(FinRL) Platform

Contents

1. Installation
2. Framework Overview
3. Main Component
   • Dataset
   • Train
   • Backtest
4. Examples

 

CH2. Framework Overview

Introduction

• FinRL은 AI 기반 퀀트 투자 연구 목적으로 Columbia Univ에서 제작한 툴로 두가지 특장점이 있음
• Framework UX: 다양한 Layer로 퀀트 투자에 필요한 모든 기능이 포함된 all-in-one 플랫폼임
• Framework UI: Configuration 정의와 python 실행 명령어로 코딩없이 누구든 쉽게 사용하게 만들었음

GitHub - AI4Finance-Foundation/FinRL: FinRL: Financial Reinforcement Learning. 🔥

FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance

---

1. Framework UX(User Experience)

• FinRL은 세 가지 레이어로 구성: 시장 환경(Market Environment), 강화학습 에이전트(DLR Agents), 응용 프로그램(Applications). Applications에는 트레이딩이 있고(맨 위), 중간에는 DLR Agents가 있어(가운데) Market Environment과 상호 작용하며 순차적으로 결정을 내립니다(가장 아래).

1. Application Layer: FinRL aims to provide hundreds of demonstrative trading tasks, serving as stepping stones.
2. DLR Agent Layer: FinRL supports fine-tuned DRL algorithms from DRL libraries in a plug-and-play manner.
3. Market Environment Layer: FinRL aims to wrap historical data and live trading APIs of hundreds of markets into training environments.

FinRL Framework Overview

 

2. Framework UI(User Interface)

• Configuration Setting: 소스코딩 없이 간단한 Configuration file의 옵션 변경으로 쉽게 다양한 실험 가능

# directory
from __future__ import annotations

DATA_SAVE_DIR = "datasets"
TRAINED_MODEL_DIR = "trained_models"
TENSORBOARD_LOG_DIR = "tensorboard_log"
RESULTS_DIR = "results"

# date format: '%Y-%m-%d'
TRAIN_START_DATE = "2022-12-01"  # bug fix: set Monday right, start date set 2014-01-01 ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 1658 and the array at index 1 has size 1657
TRAIN_END_DATE = "2022-12-31"  

TEST_START_DATE = "2023-01-01"
TEST_END_DATE = "2023-12-31"

TRADE_START_DATE = "2024-01-01"
TRADE_END_DATE = "2024-01-10"

# stockstats technical indicator column names
# check https://pypi.org/project/stockstats/ for different names
INDICATORS = [
    "macd",
    "boll_ub",
    "boll_lb",
    "rsi_30",
    "cci_30",
    "dx_30",
    "close_30_sma",
    "close_60_sma",
]


# Model Parameters
A2C_PARAMS = {"n_steps": 5, "ent_coef": 0.01, "learning_rate": 0.0007}
PPO_PARAMS = {
    "n_steps": 2048,
    "ent_coef": 0.01,
    "learning_rate": 0.00025,
    "batch_size": 64,
}
DDPG_PARAMS = {"batch_size": 128, "buffer_size": 50000, "learning_rate": 0.001}
TD3_PARAMS = {"batch_size": 100, "buffer_size": 1000000, "learning_rate": 0.001}
SAC_PARAMS = {
    "batch_size": 64,
    "buffer_size": 100000,
    "learning_rate": 0.0001,
    "learning_starts": 100,
    "ent_coef": "auto_0.1",
}
ERL_PARAMS = {
    "learning_rate": 3e-5,
    "batch_size": 2048,
    "gamma": 0.985,
    "seed": 312,
    "net_dimension": 512,
    "target_step": 5000,
    "eval_gap": 30,
    "eval_times": 64,  # bug fix:KeyError: 'eval_times' line 68, in get_model model.eval_times = model_kwargs["eval_times"]
}
RLlib_PARAMS = {"lr": 5e-5, "train_batch_size": 500, "gamma": 0.99}


# Possible time zones
TIME_ZONE_SHANGHAI = "Asia/Shanghai"  # Hang Seng HSI, SSE, CSI
TIME_ZONE_USEASTERN = "US/Eastern"  # Dow, Nasdaq, SP
TIME_ZONE_PARIS = "Europe/Paris"  # CAC,
TIME_ZONE_BERLIN = "Europe/Berlin"  # DAX, TECDAX, MDAX, SDAX
TIME_ZONE_JAKARTA = "Asia/Jakarta"  # LQ45
TIME_ZONE_SELFDEFINED = "xxx"  # If neither of the above is your time zone, you should define it, and set USE_TIME_ZONE_SELFDEFINED 1.
USE_TIME_ZONE_SELFDEFINED = 0  # 0 (default) or 1 (use the self defined)

# parameters for data sources
ALPACA_API_KEY = "xxx"  # your ALPACA_API_KEY
ALPACA_API_SECRET = "xxx"  # your ALPACA_API_SECRET
ALPACA_API_BASE_URL = "https://paper-api.alpaca.markets"  # alpaca url
BINANCE_BASE_URL = "https://data.binance.vision/"  # binance url

 

• Run the framework

python finrl/main.py