Predict future price movements using advanced statistical and ML techniques
Financial time series analysis is the backbone of quantitative trading. Unlike traditional data analysis, time series data has temporal dependencies where past values influence future ones. This lesson will teach you to harness these patterns for profitable trading strategies.
Let's prepare our environment with specialized libraries for time series analysis and forecasting.
Market Memory: Financial markets exhibit memory - today's price movement affects tomorrow's behavior. Unlike rolling dice where each outcome is independent, stock prices show serial correlation where past movements influence future ones. This "market memory" creates predictable patterns that quantitative traders exploit.
Professional Edge: Institutional traders use sophisticated time series models to forecast volatility for options pricing, predict mean reversion for pairs trading, and identify momentum for trend-following strategies. These aren't theoretical exercises - they're billion-dollar trading strategies used by hedge funds and investment banks.
Statistical Arbitrage: Time series analysis lets us identify when markets deviate from their statistical norms, creating arbitrage opportunities. When a stock's volatility spikes beyond historical patterns, options become mispriced. When correlations break down, pairs trades become profitable. The math isn't just academic - it directly translates to trading profits.
Specialized Library Ecosystem: Time series analysis requires specialized tools beyond basic pandas. Statsmodels provides econometric tests and ARIMA models used by quantitative researchers. The ARCH library implements GARCH volatility models that hedge funds use for options pricing. TensorFlow/Keras enables deep learning approaches that capture non-linear patterns traditional models miss.
Statistical Testing Framework: Professional time series work relies heavily on statistical tests - Augmented Dickey-Fuller for stationarity, Ljung-Box for residual autocorrelation, Jarque-Bera for normality. These aren't academic exercises; they validate the assumptions our trading models depend on. Failed statistical tests mean failed trading strategies.
Reproducibility Requirements: We set random seeds because time series model training involves random initialization. In production, reproducible results are crucial for model validation, regulatory compliance, and debugging. Professional teams use version-controlled random seeds to ensure consistent model behavior across environments.
Computational Considerations: Time series models can be computationally intensive, especially LSTM networks with long sequences. Professional implementations use GPU acceleration for neural networks and parallel processing for hyperparameter optimization. The infrastructure setup reflects these performance requirements.
# Install required packages
# pip install yfinance pandas numpy matplotlib seaborn plotly
# pip install statsmodels scikit-learn arch
# pip install pmdarima tensorflow keras
import yfinance as yf
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.graph_objects as go
from plotly.subplots import make_subplots
# Time series specific libraries
import statsmodels.api as sm
from statsmodels.tsa.stattools import adfuller, kpss
from statsmodels.tsa.seasonal import seasonal_decompose
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.statespace.sarimax import SARIMAX
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
# Machine learning
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error, mean_absolute_error
from sklearn.ensemble import RandomForestRegressor
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout
# Volatility modeling
from arch import arch_model
# Warnings
import warnings
warnings.filterwarnings('ignore')
# Set random seeds for reproducibility
np.random.seed(42)
tf.random.set_seed(42)
print("Time Series Analysis Environment Ready!")
print("All libraries loaded successfully")
# Configure plotting
plt.style.use('seaborn-v0_8')
plt.rcParams['figure.figsize'] = (12, 8)Statsmodels for Econometrics: We use statsmodels because it provides econometric tests and models specifically designed for financial data. Unlike scikit-learn's general-purpose ML algorithms, statsmodels implements the specialized statistical tests (ADF, KPSS, Ljung-Box) that financial professionals use to validate model assumptions and ensure regulatory compliance.
ARCH for Volatility Modeling: The ARCH library is purpose-built for financial volatility modeling. It implements GARCH models that capture volatility clustering - the tendency for high-volatility periods to be followed by high-volatility periods. This phenomenon is crucial for options pricing and risk management but isn't captured by standard ML libraries.
TensorFlow for Deep Learning: We use TensorFlow/Keras for LSTM networks because they handle the complex memory states and gradient flow required for long sequence modeling. Financial time series often have long-term dependencies (momentum can persist for months) that require sophisticated neural architectures unavailable in simpler ML frameworks.
Proper data preparation is crucial for time series modeling. Let's fetch and prepare our financial data.
Returns vs. Prices: We analyze returns instead of raw prices because returns are more stationary and comparable across different assets and time periods. A 5% move has the same statistical meaning whether the stock is $10 or $1000. Professional portfolio managers think in terms of returns because that's what matters for performance measurement.
Log Returns vs. Simple Returns: Log returns are continuously compounded and have better mathematical properties - they're additive over time and approximately normal for short periods. This makes them ideal for time series modeling and risk calculations. When you see "log-normal" distribution in Black-Scholes options pricing, this is why.
Volatility Calculation: We annualize volatility by multiplying by √252 (trading days per year) because volatility scales with the square root of time. This isn't arbitrary - it comes from the random walk model of stock prices and is fundamental to options pricing and risk management.
Returns vs. Prices Philosophy: We calculate both simple and log returns because they serve different purposes in professional finance. Simple returns are intuitive and additive across assets (portfolio returns), while log returns are additive across time and approximately normal for short periods. Options pricing models like Black-Scholes assume log-normal price distributions, making log returns the natural choice for volatility modeling.
Rolling Volatility Calculation: Our 20-day rolling volatility with √252 annualization reflects industry standards. The 20-day window captures monthly volatility patterns while being responsive to regime changes. The √252 scaling comes from Brownian motion theory - volatility scales with the square root of time in geometric Brownian motion, the foundation of modern financial modeling.
Technical Indicators as Features: We include RSI and moving averages not for their predictive power alone, but because they represent different market dynamics. Moving averages capture trend-following behavior of momentum traders, while RSI captures mean-reversion tendencies of contrarian traders. These represent the behavioral forces that create predictable patterns in time series data.
Data Quality Imperatives: Professional time series analysis is extremely sensitive to data quality. Missing values, splits, dividends, and survivorship bias can completely invalidate models. Our data preparation pipeline includes quality checks because garbage in, garbage out applies especially strongly to time series modeling where small errors compound over time.
# Enhanced data fetching for time series analysis
def get_financial_time_series(symbol, period="2y", interval="1d"):
"""
Fetch and prepare financial time series data
"""
print(f"Fetching {symbol} data for time series analysis...")
# Get stock data
stock = yf.Ticker(symbol)
data = stock.history(period=period, interval=interval)
if data.empty:
print(f"❌ No data available for {symbol}")
return None
# Calculate returns and log returns
data['Returns'] = data['Close'].pct_change()
data['Log_Returns'] = np.log(data['Close'] / data['Close'].shift(1))
# Calculate volatility (rolling standard deviation)
data['Volatility'] = data['Returns'].rolling(window=20).std() * np.sqrt(252)
# Price transformations
data['Log_Price'] = np.log(data['Close'])
data['Price_Diff'] = data['Close'].diff()
# Technical indicators for features
data['SMA_20'] = data['Close'].rolling(window=20).mean()
data['SMA_50'] = data['Close'].rolling(window=50).mean()
data['RSI'] = calculate_rsi_simple(data['Close'])
# Remove NaN values
data = data.dropna()
print(f"✅ Prepared {len(data)} observations")
print(f"Date range: {data.index[0].date()} to {data.index[-1].date()}")
return data
def calculate_rsi_simple(prices, period=14):
"""Calculate RSI for feature engineering"""
delta = prices.diff()
gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
rs = gain / loss
return 100 - (100 / (1 + rs))
# Fetch Apple stock data
symbol = "AAPL"
data = get_financial_time_series(symbol, period="2y")
# Display basic time series properties
print(f"\n=== {symbol} Time Series Properties ===")
print(f"Total observations: {len(data)}")
print(f"Average daily return: {data['Returns'].mean()*100:.3f}%")
print(f"Return volatility: {data['Returns'].std()*100:.3f}%")
print(f"Annualized volatility: {data['Returns'].std()*np.sqrt(252)*100:.1f}%")
print(f"Sharpe ratio: {data['Returns'].mean()/data['Returns'].std()*np.sqrt(252):.2f}")
# Show data structure
print(f"\n=== Data Structure ===")
print(data[['Close', 'Returns', 'Log_Returns', 'Volatility']].head(10))Sharpe Ratio Significance: The Sharpe ratio calculation (mean return / volatility * √252) immediately tells us whether this asset has sufficient risk-adjusted return to justify inclusion in a portfolio. Professional portfolio managers use Sharpe ratios > 1.0 as a minimum threshold for strategy consideration. This quick calculation filters opportunities before we invest time in complex modeling.
Volatility Annualization Logic: Multiplying daily volatility by √252 assumes returns follow a random walk where variance scales linearly with time. This assumption underlies most financial models, from Black-Scholes to VaR calculations. When this assumption breaks down (due to volatility clustering or mean reversion), our models need adjustment.
Log vs. Simple Returns Trade-offs: Log returns are preferred for time series modeling because they're symmetric (a 50% gain followed by a 33% loss nets to zero in log space) and more stationary. However, simple returns are better for portfolio aggregation and performance reporting. Professional systems maintain both and use each for its appropriate purpose.
Understanding the components of a time series helps us identify patterns and choose appropriate modeling techniques.
Trend Component: The long-term directional movement reflects fundamental factors like earnings growth, economic expansion, or industry cycles. Institutional investors with long time horizons drive trend components through systematic buying or selling programs.
Seasonal Component: Even in daily stock prices, we find weekly patterns (Monday effects, Friday profit-taking) and monthly patterns (options expiration, earnings seasons). These patterns exist because market participants have predictable behavior schedules - pension fund rebalancing, mutual fund flows, etc.
Residual Component: This captures the "noise" - unpredictable price movements driven by news, rumors, and random market microstructure events. A high residual variance suggests the stock is driven more by news than predictable patterns, which affects our trading strategy choice.
Multiplicative vs. Additive Models: We use multiplicative decomposition for financial data because market volatility is proportional to price levels. A 1% move matters more for a $1000 stock than a $10 stock in dollar terms. Multiplicative models capture this scaling relationship, while additive models assume constant absolute variations regardless of price level.
Period Selection Logic: We use a 5-day period for seasonal decomposition to capture weekly patterns - Monday effects, Friday profit-taking, mid-week momentum. These patterns exist because institutional trading has weekly rhythms (fund meetings on Mondays, month-end rebalancing). Professional systems test multiple periodicities to identify all seasonal patterns.
Trend Strength Interpretation: High trend strength (close to 1.0) suggests the stock is driven by fundamental factors with persistent directional movement. Low trend strength suggests the stock is more news-driven with random walk characteristics. This classification helps us choose between momentum strategies (high trend strength) and mean-reversion strategies (low trend strength).
Residual Analysis Importance: Large residuals indicate unpredictable price movements driven by news and market microstructure. High residual variance suggests the stock is difficult to model and predict, affecting our position sizing and holding period decisions. Professional managers reduce position sizes for high-residual assets to account for increased unpredictability.
# Time series decomposition
def analyze_time_series_components(data, symbol, column='Close'):
"""
Decompose time series into trend, seasonal, and residual components
"""
print(f"Analyzing time series components for {symbol}...")
# Perform seasonal decomposition
# Note: For daily financial data, we'll use a weekly cycle (5 days)
decomposition = seasonal_decompose(data[column], model='multiplicative', period=5)
# Create decomposition plot
fig, axes = plt.subplots(4, 1, figsize=(15, 12))
# Original series
axes[0].plot(data.index, data[column], color='blue', linewidth=1)
axes[0].set_title(f'{symbol} Original Price Series')
axes[0].set_ylabel('Price ($)')
axes[0].grid(True, alpha=0.3)
# Trend
axes[1].plot(decomposition.trend.index, decomposition.trend, color='red', linewidth=2)
axes[1].set_title('Trend Component')
axes[1].set_ylabel('Trend')
axes[1].grid(True, alpha=0.3)
# Seasonal
axes[2].plot(decomposition.seasonal.index, decomposition.seasonal, color='green', linewidth=1)
axes[2].set_title('Seasonal Component')
axes[2].set_ylabel('Seasonal')
axes[2].grid(True, alpha=0.3)
# Residual
axes[3].plot(decomposition.resid.index, decomposition.resid, color='orange', linewidth=1)
axes[3].set_title('Residual Component')
axes[3].set_ylabel('Residual')
axes[3].set_xlabel('Date')
axes[3].grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Analyze components
trend_strength = 1 - (decomposition.resid.var() / (decomposition.trend + decomposition.resid).var())
seasonal_strength = 1 - (decomposition.resid.var() / (decomposition.seasonal + decomposition.resid).var())
print(f"Trend strength: {trend_strength:.3f}")
print(f"Seasonal strength: {seasonal_strength:.3f}")
return decomposition
# Perform decomposition
decomposition = analyze_time_series_components(data, symbol)
# Analyze autocorrelation
def plot_autocorrelation_analysis(data, symbol, lags=40):
"""Plot ACF and PACF for autocorrelation analysis"""
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
# Price autocorrelation
plot_acf(data['Close'].dropna(), lags=lags, ax=axes[0, 0], title=f'{symbol} Price ACF')
plot_pacf(data['Close'].dropna(), lags=lags, ax=axes[0, 1], title=f'{symbol} Price PACF')
# Returns autocorrelation
plot_acf(data['Returns'].dropna(), lags=lags, ax=axes[1, 0], title=f'{symbol} Returns ACF')
plot_pacf(data['Returns'].dropna(), lags=lags, ax=axes[1, 1], title=f'{symbol} Returns PACF')
plt.tight_layout()
plt.show()
# Plot autocorrelation analysis
print("Analyzing autocorrelation patterns...")
plot_autocorrelation_analysis(data, symbol)ACF vs. PACF Interpretation: Autocorrelation Function (ACF) shows total correlation including indirect effects, while Partial Autocorrelation Function (PACF) shows direct correlation at each lag. For AR processes, PACF cuts off sharply while ACF decays gradually. For MA processes, the pattern reverses. These patterns help us identify the appropriate ARIMA model order.
Price vs. Returns Autocorrelation: Stock prices typically show strong positive autocorrelation (trending behavior) while returns show much weaker autocorrelation. This difference explains why we can't easily predict price levels but might be able to predict return direction over short periods. The autocorrelation patterns directly inform our trading strategy choice.
Lag Structure Significance: Significant autocorrelation at specific lags reveals market microstructure effects. Lag-1 autocorrelation might reflect bid-ask bounce, lag-5 could indicate weekly patterns, lag-20 might show monthly institutional rebalancing. Professional algorithms exploit these patterns through careful trade timing and execution strategies.
Statistical Significance Bounds: The blue confidence bands show statistical significance at 95% confidence. Autocorrelations outside these bands are statistically significant and potentially exploitable. However, statistical significance doesn't guarantee economic significance - transaction costs might eliminate profits from weak but statistically significant patterns.
Most time series models require stationary data. Let's test for stationarity and apply transformations if needed.
Statistical Foundation: Stationarity means the statistical properties (mean, variance, autocorrelation) don't change over time. This is crucial because our trading models assume that patterns we observe in historical data will persist in the future. Non-stationary data breaks this assumption and leads to unreliable forecasts.
Price vs. Returns: Stock prices are almost always non-stationary (they trend up or down over time), but returns are often stationary. This is why we model return processes rather than price levels. The efficient market hypothesis suggests that price changes (returns) should be unpredictable, which implies stationarity.
Trading Implications: If returns are stationary, we can use mean reversion strategies. If they're non-stationary with trends, momentum strategies work better. The stationarity tests help us choose the right trading approach and validate our model assumptions before risking capital.
Tests null hypothesis that the series has a unit root (non-stationary)
Decision: p-value < 0.05 → Stationary
Tests null hypothesis that the series is stationary
Decision: p-value > 0.05 → Stationary
Transform non-stationary series by taking differences
Method: Δy(t) = y(t) - y(t-1)
ADF vs. KPSS Test Logic: We use both tests because they have opposite null hypotheses. ADF tests if data is non-stationary (null: unit root exists), while KPSS tests if data is stationary (null: stationarity exists). When both tests agree, we have confidence in the conclusion. When they disagree, we need more investigation or different transformations.
Critical Value Interpretation: The 1%, 5%, and 10% critical values represent different confidence levels for rejecting the null hypothesis. Professional applications typically use 5% significance, but regulatory requirements might demand more conservative 1% thresholds. The choice affects our confidence in model assumptions and subsequent trading decisions.
Transformation Sequence Logic: We test prices first (usually non-stationary), then returns (often stationary), then higher-order differences if needed. This systematic approach follows the integrated (I) component of ARIMA modeling. Most financial returns are I(0) - stationary without differencing - which allows for meaningful statistical modeling.
Trading Strategy Implications: Stationary series mean-revert to their historical average, enabling mean-reversion strategies. Non-stationary series with trends enable momentum strategies. The stationarity tests directly inform our strategic approach - they're not just statistical exercises but fundamental strategy selection tools.
# Stationarity testing functions
def test_stationarity(series, title="Time Series"):
"""
Perform comprehensive stationarity tests
"""
print(f"\n=== Stationarity Tests for {title} ===")
# Augmented Dickey-Fuller test
adf_result = adfuller(series.dropna())
print(f"ADF Test:")
print(f" ADF Statistic: {adf_result[0]:.6f}")
print(f" p-value: {adf_result[1]:.6f}")
print(f" Critical Values:")
for key, value in adf_result[4].items():
print(f" {key}: {value:.3f}")
adf_stationary = adf_result[1] <= 0.05
print(f" ADF Result: {'Stationary' if adf_stationary else 'Non-Stationary'}")
# KPSS test
try:
kpss_result = kpss(series.dropna(), regression='c')
print(f"\nKPSS Test:")
print(f" KPSS Statistic: {kpss_result[0]:.6f}")
print(f" p-value: {kpss_result[1]:.6f}")
print(f" Critical Values:")
for key, value in kpss_result[3].items():
print(f" {key}: {value:.3f}")
kpss_stationary = kpss_result[1] >= 0.05
print(f" KPSS Result: {'Stationary' if kpss_stationary else 'Non-Stationary'}")
# Combined conclusion
if adf_stationary and kpss_stationary:
conclusion = "Stationary"
elif not adf_stationary and not kpss_stationary:
conclusion = "Non-Stationary"
else:
conclusion = "Inconclusive (mixed results)"
print(f"\nCombined Conclusion: {conclusion}")
except Exception as e:
print(f"KPSS test failed: {e}")
conclusion = "Stationary" if adf_stationary else "Non-Stationary"
print(f"Conclusion based on ADF only: {conclusion}")
return conclusion
# Test different series for stationarity
print("Testing stationarity for different transformations...")
# Test price levels
price_stationarity = test_stationarity(data['Close'], f"{symbol} Price")
# Test returns
returns_stationarity = test_stationarity(data['Returns'], f"{symbol} Returns")
# Test log returns
log_returns_stationarity = test_stationarity(data['Log_Returns'], f"{symbol} Log Returns")
# Test first difference of prices
price_diff_stationarity = test_stationarity(data['Price_Diff'], f"{symbol} Price Differences")
# Create difference series if needed
if price_stationarity == "Non-Stationary":
data['Price_Diff2'] = data['Close'].diff().diff()
price_diff2_stationarity = test_stationarity(data['Price_Diff2'], f"{symbol} Second Difference")
# Summary
print(f"\n=== Stationarity Summary ===")
print(f"Price levels: {price_stationarity}")
print(f"Returns: {returns_stationarity}")
print(f"Log returns: {log_returns_stationarity}")
print(f"Price differences: {price_diff_stationarity}")
# Visualize different transformations
def plot_stationarity_comparison(data, symbol):
"""Plot different transformations for visual comparison"""
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
# Original prices
axes[0, 0].plot(data.index, data['Close'], color='blue', linewidth=1)
axes[0, 0].set_title(f'{symbol} Original Prices')
axes[0, 0].set_ylabel('Price ($)')
axes[0, 0].grid(True, alpha=0.3)
# Returns
axes[0, 1].plot(data.index, data['Returns'], color='red', linewidth=1)
axes[0, 1].set_title(f'{symbol} Returns')
axes[0, 1].set_ylabel('Returns')
axes[0, 1].axhline(y=0, color='black', linestyle='--', alpha=0.5)
axes[0, 1].grid(True, alpha=0.3)
# Log returns
axes[1, 0].plot(data.index, data['Log_Returns'], color='green', linewidth=1)
axes[1, 0].set_title(f'{symbol} Log Returns')
axes[1, 0].set_ylabel('Log Returns')
axes[1, 0].axhline(y=0, color='black', linestyle='--', alpha=0.5)
axes[1, 0].grid(True, alpha=0.3)
# Price differences
axes[1, 1].plot(data.index, data['Price_Diff'], color='orange', linewidth=1)
axes[1, 1].set_title(f'{symbol} Price Differences')
axes[1, 1].set_ylabel('Price Difference')
axes[1, 1].axhline(y=0, color='black', linestyle='--', alpha=0.5)
axes[1, 1].grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
# Plot transformations
plot_stationarity_comparison(data, symbol)Visual Confirmation of Tests: The plots provide intuitive confirmation of statistical tests. Stationary series should fluctuate around a constant mean without obvious trends or level shifts. Non-stationary prices typically show trending behavior with changing variance over time. Professional analysts use visual inspection to validate statistical test results and identify structural breaks.
Returns vs. Price Patterns: Notice how price charts show clear trends and changing levels, while return charts fluctuate around zero with relatively constant variance. This visual difference immediately explains why we model returns rather than prices - returns have the stable statistical properties required for reliable forecasting models.
Transformation Effectiveness: Comparing original prices to their transformations shows how differencing removes trends and stabilizes variance. First differences (returns) usually achieve stationarity for financial data. If not, second differences might be needed, though this is rare and often indicates data quality issues or structural breaks requiring investigation.
ARIMA (AutoRegressive Integrated Moving Average) models are the foundation of time series forecasting in finance.
AutoRegressive (AR) Component: Captures momentum effects - when prices move up, they tend to keep moving up for a while. This reflects institutional order flow, where large trades are broken into smaller pieces over time, creating serial correlation in returns.
Moving Average (MA) Component: Captures mean reversion after shocks - when an unexpected news event moves prices, they tend to partially reverse as the market digests the information. This models the overreaction and correction cycle that creates trading opportunities.
Integrated (I) Component: Handles the non-stationarity in price levels by taking differences. This is why we can predict changes in prices (returns) even though we can't predict price levels. It's the mathematical foundation for why technical analysis works.
Practical Application: Hedge funds use ARIMA-type models to forecast short-term volatility for options trading, predict mean reversion timing for pairs trades, and optimize execution algorithms to minimize market impact.
AIC-Based Model Selection: We use Akaike Information Criterion (AIC) for model selection because it balances model fit against complexity. Lower AIC indicates better models, but AIC penalizes additional parameters to prevent overfitting. Professional model selection always includes this bias-variance tradeoff because overfit models fail catastrophically in live trading.
Grid Search Approach: We systematically test different (p,d,q) combinations because ARIMA model order significantly affects performance. Professional implementations often test hundreds of combinations with cross-validation. The automated search removes human bias in model selection and ensures we find the optimal specification.
Order Interpretation Logic: AR(p) captures momentum effects - how much today's return depends on past returns. MA(q) captures mean-reversion after shocks - how much returns respond to recent forecast errors. I(d) handles non-stationarity through differencing. Each component corresponds to different market microstructure phenomena.
Forecasting Horizon Considerations: ARIMA models work best for short-term forecasting (1-30 days) because financial time series have limited predictability. Beyond this horizon, forecasts converge to the long-term mean. Professional systems use ARIMA for tactical trading decisions, not long-term investment allocation.
# ARIMA model implementation
def find_optimal_arima_order(series, max_p=5, max_d=2, max_q=5):
"""
Find optimal ARIMA order using AIC criterion
"""
print("Searching for optimal ARIMA parameters...")
best_aic = np.inf
best_order = None
best_model = None
# Grid search for best parameters
for p in range(max_p + 1):
for d in range(max_d + 1):
for q in range(max_q + 1):
try:
model = ARIMA(series, order=(p, d, q))
fitted_model = model.fit()
if fitted_model.aic < best_aic:
best_aic = fitted_model.aic
best_order = (p, d, q)
best_model = fitted_model
except Exception as e:
continue
print(f"Best ARIMA order: {best_order}")
print(f"Best AIC: {best_aic:.2f}")
return best_order, best_model
# Find optimal ARIMA model for returns
print("Building ARIMA model for returns...")
best_order, arima_model = find_optimal_arima_order(data['Returns'].dropna(), max_p=3, max_d=1, max_q=3)
# Print model summary
print("\n=== ARIMA Model Summary ===")
print(arima_model.summary())
# Generate forecasts
def generate_arima_forecast(model, steps=30):
"""Generate ARIMA forecasts with confidence intervals"""
forecast = model.forecast(steps=steps)
conf_int = model.get_forecast(steps=steps).conf_int()
return forecast, conf_int
# Generate 30-day forecast
forecast, conf_int = generate_arima_forecast(arima_model, steps=30)
print(f"\n=== ARIMA Forecast (Next 30 days) ===")
print("Forecasted returns:")
for i, value in enumerate(forecast[:10]):
print(f"Day {i+1}: {value:.4f}")
# Plot ARIMA results
def plot_arima_results(data, model, forecast, conf_int, symbol, days_to_show=100):
"""Plot ARIMA model results and forecasts"""
# Get recent data for plotting
recent_data = data['Returns'].dropna().tail(days_to_show)
# Create forecast dates
last_date = data.index[-1]
forecast_dates = pd.date_range(start=last_date + pd.Timedelta(days=1), periods=len(forecast), freq='D')
# Plot
plt.figure(figsize=(15, 8))
# Historical data
plt.plot(recent_data.index, recent_data, label='Historical Returns', color='blue', linewidth=1)
# Forecast
plt.plot(forecast_dates, forecast, label='ARIMA Forecast', color='red', linewidth=2)
# Confidence intervals
plt.fill_between(forecast_dates, conf_int.iloc[:, 0], conf_int.iloc[:, 1],
color='red', alpha=0.2, label='Confidence Interval')
plt.title(f'{symbol} ARIMA{best_order} Returns Forecast')
plt.xlabel('Date')
plt.ylabel('Returns')
plt.legend()
plt.grid(True, alpha=0.3)
plt.axhline(y=0, color='black', linestyle='--', alpha=0.5)
plt.tight_layout()
plt.show()
# Plot ARIMA results
plot_arima_results(data, arima_model, forecast, conf_int, symbol)
# Model diagnostics
def plot_arima_diagnostics(model):
"""Plot ARIMA model diagnostics"""
residuals = model.resid
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
# Residuals plot
residuals.plot(ax=axes[0, 0], title='Residuals')
axes[0, 0].grid(True, alpha=0.3)
# Residuals histogram
residuals.hist(ax=axes[0, 1], bins=30, title='Residuals Histogram')
axes[0, 1].grid(True, alpha=0.3)
# Q-Q plot
sm.qqplot(residuals, line='s', ax=axes[1, 0])
axes[1, 0].set_title('Q-Q Plot')
axes[1, 0].grid(True, alpha=0.3)
# ACF of residuals
plot_acf(residuals, ax=axes[1, 1], title='ACF of Residuals')
plt.tight_layout()
plt.show()
# Ljung-Box test for residual autocorrelation
lb_stat, lb_pvalue = sm.stats.diagnostic.acorr_ljungbox(residuals, lags=10, return_df=False)[:2]
print(f"Ljung-Box test p-value: {lb_pvalue:.4f}")
print(f"Residuals {'appear to be' if lb_pvalue > 0.05 else 'may not be'} white noise")
# Run diagnostics
print("Running ARIMA model diagnostics...")
plot_arima_diagnostics(arima_model)Residual Analysis Importance: ARIMA model residuals should be white noise - no patterns, constant variance, normal distribution. Patterns in residuals indicate the model missed some predictable structure, suggesting model misspecification. Professional validation always includes residual analysis because systematic residual patterns indicate trading opportunities the model failed to capture.
Ljung-Box Test Significance: This test checks if residuals are truly random or still contain autocorrelation. P-values > 0.05 suggest residuals are white noise, validating our model. Failed Ljung-Box tests indicate model inadequacy and require either higher-order ARIMA models or different approaches entirely.
Q-Q Plot Interpretation: The quantile-quantile plot compares residual distribution to theoretical normal distribution. Points following the diagonal line indicate normality. Deviations suggest fat tails or skewness, common in financial data. While violations don't invalidate the model, they affect confidence interval accuracy and risk calculations.
Confidence Interval Reliability: ARIMA confidence intervals assume normally distributed residuals with constant variance. Real financial data often violates these assumptions, making confidence intervals unreliable. Professional implementations often use bootstrap methods or GARCH models to generate more realistic confidence intervals for risk management.
GARCH models capture volatility clustering - a key characteristic of financial time series where periods of high volatility are followed by high volatility.
Volatility Clustering Reality: GARCH models capture the most important feature of financial returns - volatility clustering. Periods of high volatility tend to be followed by high volatility, and calm periods by calm periods. This isn't just a statistical curiosity; it's fundamental to options pricing, risk management, and trading strategy design.
Options Trading Applications: GARCH volatility forecasts directly feed into options pricing models. When GARCH predicts rising volatility, options become more valuable. Professional options traders use GARCH forecasts to identify mispriced options and optimize their volatility trading strategies. The economic value of accurate volatility forecasting is enormous in options markets.
Risk Management Integration: Value-at-Risk (VaR) calculations rely on volatility forecasts to estimate potential losses. GARCH provides dynamic volatility estimates that adapt to changing market conditions, unlike static historical volatility measures. Professional risk management systems update VaR calculations using GARCH forecasts to maintain accurate risk measurements.
Regime Detection Capabilities: GARCH models automatically detect volatility regime changes - transitions between calm and turbulent market periods. These regime changes often precede major market moves, making GARCH useful for tactical asset allocation and market timing strategies beyond pure volatility forecasting.
# GARCH volatility modeling
def build_garch_model(returns, p=1, q=1):
"""
Build and fit GARCH model for volatility forecasting
"""
print(f"Building GARCH({p},{q}) model...")
# Remove any remaining NaN values
clean_returns = returns.dropna() * 100 # Convert to percentage for better numerical stability
# Define GARCH model
garch_model = arch_model(clean_returns, vol='GARCH', p=p, q=q, rescale=False)
# Fit the model
garch_fitted = garch_model.fit(disp='off')
print("GARCH model fitted successfully!")
return garch_fitted
# Fit GARCH model
garch_model = build_garch_model(data['Returns'])
# Print GARCH summary
print("\n=== GARCH Model Summary ===")
print(garch_model.summary())
# Generate volatility forecasts
def forecast_volatility(garch_model, horizon=30):
"""Generate GARCH volatility forecasts"""
volatility_forecast = garch_model.forecast(horizon=horizon)
return volatility_forecast
# Generate volatility forecast
vol_forecast = forecast_volatility(garch_model, horizon=30)
print(f"\n=== GARCH Volatility Forecast ===")
print("Forecasted volatility (next 10 days):")
for i in range(10):
vol_value = np.sqrt(vol_forecast.variance.iloc[-1, i])
print(f"Day {i+1}: {vol_value:.3f}%")
# Plot GARCH results
def plot_garch_results(data, garch_model, vol_forecast, symbol):
"""Plot GARCH volatility modeling results"""
# Extract conditional volatility
cond_vol = garch_model.conditional_volatility
fig, axes = plt.subplots(3, 1, figsize=(15, 12))
# Returns
axes[0].plot(data.index, data['Returns'] * 100, label='Returns', linewidth=1, alpha=0.7)
axes[0].set_title(f'{symbol} Returns')
axes[0].set_ylabel('Returns (%)')
axes[0].grid(True, alpha=0.3)
axes[0].legend()
# Conditional volatility
vol_dates = data.index[-len(cond_vol):]
axes[1].plot(vol_dates, cond_vol, label='GARCH Volatility', color='red', linewidth=2)
axes[1].set_title('GARCH Conditional Volatility')
axes[1].set_ylabel('Volatility (%)')
axes[1].grid(True, alpha=0.3)
axes[1].legend()
# Squared returns vs fitted volatility
squared_returns = (data['Returns'] * 100) ** 2
axes[2].plot(vol_dates, squared_returns.iloc[-len(cond_vol):],
label='Squared Returns', alpha=0.6, linewidth=1)
axes[2].plot(vol_dates, cond_vol ** 2,
label='GARCH Variance', color='red', linewidth=2)
axes[2].set_title('Squared Returns vs GARCH Variance')
axes[2].set_ylabel('Variance')
axes[2].set_xlabel('Date')
axes[2].grid(True, alpha=0.3)
axes[2].legend()
plt.tight_layout()
plt.show()
# Plot GARCH results
plot_garch_results(data, garch_model, vol_forecast, symbol)Conditional Volatility Insights: GARCH conditional volatility adapts in real-time to market conditions, spiking during stress periods and declining during calm periods. This dynamic behavior makes GARCH superior to simple rolling volatility for risk management. Professional trading systems use conditional volatility for position sizing - reducing positions when volatility spikes, increasing when it's low.
Percentage Scaling Logic: We multiply returns by 100 for numerical stability in GARCH estimation. Financial returns are small numbers (typically 0.01-0.05), which can cause numerical precision issues in optimization algorithms. Professional implementations always include such scaling considerations to ensure robust parameter estimation.
Forecast Persistence Analysis: GARCH volatility forecasts decay toward long-term average volatility over time. The rate of decay depends on model parameters - high persistence means volatility shocks have long-lasting effects. This persistence directly affects options pricing and risk management horizons.
Squared Returns vs. Variance: The comparison between squared returns (realized volatility) and GARCH variance (predicted volatility) shows model accuracy. Good GARCH models should anticipate volatility spikes before they occur, not just react to them. This predictive capability is what makes GARCH valuable for proactive risk management.
Modern ML techniques can capture complex non-linear patterns in financial time series that traditional models might miss.
Long-Term Memory Advantage: LSTM networks solve the vanishing gradient problem that prevents traditional neural networks from learning long-term dependencies. Financial markets have memory that spans weeks or months - momentum trends, seasonal patterns, regime persistence. LSTMs can capture these extended dependencies that simpler models miss.
Feature Engineering Strategy: We include price, volume, and technical indicators as features because LSTM networks benefit from diverse input signals. Unlike ARIMA models that focus on single series, LSTMs can integrate multiple data streams. Professional implementations often include fundamental data, news sentiment, and macro indicators for richer feature sets.
Sequence Length Selection: The 60-day lookback window captures approximately three months of market memory, spanning quarterly earnings cycles and monthly institutional rebalancing. This window length balances historical context against computational efficiency. Professional systems often test multiple lookback periods and ensemble the results.
Multi-Step Architecture Logic: Our three-layer LSTM with dropout reflects best practices for financial time series. Multiple layers capture hierarchical patterns, while dropout prevents overfitting to training data. The architecture mirrors successful implementations at quantitative hedge funds and high-frequency trading firms.
# LSTM implementation for time series forecasting
def prepare_lstm_data(data, target_col='Close', lookback=60, forecast_horizon=1):
"""
Prepare data for LSTM training
"""
print(f"Preparing LSTM data with {lookback} lookback period...")
# Select features
feature_cols = [target_col, 'Volume', 'SMA_20', 'SMA_50', 'RSI']
df = data[feature_cols].dropna().copy()
# Scale the data
scaler = MinMaxScaler()
scaled_data = scaler.fit_transform(df)
# Create sequences
X, y = [], []
for i in range(lookback, len(scaled_data) - forecast_horizon + 1):
X.append(scaled_data[i-lookback:i])
y.append(scaled_data[i + forecast_horizon - 1, 0]) # Predict target_col
X, y = np.array(X), np.array(y)
print(f"Created {len(X)} sequences")
print(f"Input shape: {X.shape}")
print(f"Output shape: {y.shape}")
return X, y, scaler
# Prepare LSTM data
X, y, scaler = prepare_lstm_data(data, target_col='Close', lookback=60)
# Split data
train_size = int(len(X) * 0.8)
X_train, X_test = X[:train_size], X[train_size:]
y_train, y_test = y[:train_size], y[train_size:]
print(f"Training samples: {len(X_train)}")
print(f"Testing samples: {len(X_test)}")
# Build LSTM model
def build_lstm_model(input_shape):
"""Build LSTM model architecture"""
model = Sequential([
LSTM(50, return_sequences=True, input_shape=input_shape),
Dropout(0.2),
LSTM(50, return_sequences=True),
Dropout(0.2),
LSTM(50),
Dropout(0.2),
Dense(25),
Dense(1)
])
model.compile(optimizer='adam', loss='mse', metrics=['mae'])
return model
# Build model
lstm_model = build_lstm_model((X_train.shape[1], X_train.shape[2]))
print("\nLSTM Model Architecture:")
lstm_model.summary()
# Train LSTM model
print("\nTraining LSTM model...")
history = lstm_model.fit(
X_train, y_train,
epochs=50,
batch_size=32,
validation_split=0.2,
verbose=1,
callbacks=[
tf.keras.callbacks.EarlyStopping(patience=10, restore_best_weights=True),
tf.keras.callbacks.ReduceLROnPlateau(factor=0.5, patience=5)
]
)
# Plot training history
def plot_training_history(history):
"""Plot LSTM training history"""
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))
# Loss
ax1.plot(history.history['loss'], label='Training Loss')
ax1.plot(history.history['val_loss'], label='Validation Loss')
ax1.set_title('Model Loss')
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss')
ax1.legend()
ax1.grid(True, alpha=0.3)
# MAE
ax2.plot(history.history['mae'], label='Training MAE')
ax2.plot(history.history['val_mae'], label='Validation MAE')
ax2.set_title('Model MAE')
ax2.set_xlabel('Epoch')
ax2.set_ylabel('MAE')
ax2.legend()
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
plot_training_history(history)
# Make predictions
print("Generating LSTM predictions...")
y_pred = lstm_model.predict(X_test)
# Inverse transform predictions
def inverse_transform_predictions(predictions, scaler):
"""Inverse transform scaled predictions"""
# Create dummy array with same shape as original features
dummy = np.zeros((len(predictions), scaler.n_features_in_))
dummy[:, 0] = predictions.flatten()
# Inverse transform
inversed = scaler.inverse_transform(dummy)
return inversed[:, 0]
y_pred_actual = inverse_transform_predictions(y_pred, scaler)
y_test_actual = inverse_transform_predictions(y_test, scaler)
# Calculate metrics
mse = mean_squared_error(y_test_actual, y_pred_actual)
mae = mean_absolute_error(y_test_actual, y_pred_actual)
rmse = np.sqrt(mse)
print(f"\n=== LSTM Model Performance ===")
print(f"MSE: {mse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"RMSE: {rmse:.4f}")
# Directional accuracy
actual_direction = np.sign(np.diff(y_test_actual))
pred_direction = np.sign(np.diff(y_pred_actual))
directional_accuracy = np.mean(actual_direction == pred_direction)
print(f"Directional Accuracy: {directional_accuracy:.4f}")
# Plot LSTM predictions
def plot_lstm_predictions(y_test, y_pred, symbol, days_to_show=100):
"""Plot LSTM predictions vs actual values"""
# Limit to recent data for clarity
if len(y_test) > days_to_show:
y_test_plot = y_test[-days_to_show:]
y_pred_plot = y_pred[-days_to_show:]
else:
y_test_plot = y_test
y_pred_plot = y_pred
plt.figure(figsize=(15, 8))
plt.plot(y_test_plot, label='Actual Prices', linewidth=2)
plt.plot(y_pred_plot, label='LSTM Predictions', linewidth=2, alpha=0.8)
plt.title(f'{symbol} LSTM Price Predictions')
plt.xlabel('Time Steps')
plt.ylabel('Price ($)')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
plot_lstm_predictions(y_test_actual, y_pred_actual, symbol)Scaling and Normalization Rationale: MinMaxScaler transforms features to [0,1] range, preventing features with larger magnitudes (like volume) from dominating the learning process. Neural networks are sensitive to input scales, and proper normalization significantly improves convergence and final performance. Professional implementations often use more sophisticated scaling methods like RobustScaler for outlier-heavy financial data.
Train-Test Split Methodology: We use temporal splitting (80% train, 20% test) rather than random splitting because time series have temporal structure. Using future data to predict the past creates unrealistic performance estimates. Professional validation uses walk-forward analysis where models are continuously retrained on expanding windows of historical data.
Early Stopping and Learning Rate Scheduling: These callbacks prevent overfitting and optimize training efficiency. Early stopping monitors validation loss and stops training when improvement ceases, preventing memorization of training data. Learning rate reduction helps models converge to better local minima when learning stagnates.
Directional Accuracy Significance: In trading, predicting direction matters more than exact price levels. Directional accuracy > 0.55 can be profitable after transaction costs, while high price prediction accuracy might not translate to trading profits. Professional systems optimize for directional accuracy and Sharpe ratio rather than traditional ML metrics like MSE.
Apply time series analysis to build your own forecasting models!
Compare different forecasting approaches:
# Your multi-model comparison
def compare_forecasting_models(data, symbol, test_size=50):
"""
Compare ARIMA, GARCH, and LSTM forecasting performance
"""
# Split data
train_data = data[:-test_size]
test_data = data[-test_size:]
results = {}
# ARIMA Model
print("Building ARIMA model...")
# Your ARIMA implementation here
# GARCH Model
print("Building GARCH model...")
# Your GARCH implementation here
# LSTM Model
print("Building LSTM model...")
# Your LSTM implementation here
# Ensemble Model
print("Creating ensemble predictions...")
# Your ensemble implementation here
# Compare performance
# Your comparison code here
return results
# Test your comparison
# comparison_results = compare_forecasting_models(data, symbol)Build a trading strategy using your time series forecasts:
Signal Generation Logic: Effective forecast-based strategies don't just buy when forecasts are positive - they consider forecast magnitude, confidence, and recent accuracy. Professional systems use forecast z-scores (how many standard deviations above/below average) to determine position sizing. Stronger signals get larger positions, weak signals get smaller positions.
Multi-Model Ensemble Approach: Rather than relying on single model forecasts, professional strategies combine ARIMA, GARCH, and LSTM predictions with different weights. ARIMA captures linear patterns, GARCH forecasts volatility for position sizing, and LSTM captures non-linear relationships. The ensemble approach reduces model risk and improves robustness.
Transaction Cost Integration: Forecast-based strategies can generate frequent trading signals, making transaction costs crucial. Professional implementations include bid-ask spreads, market impact, and slippage in their optimization. Strategies might ignore weak forecasts if the expected profit doesn't exceed transaction costs.
# Time series-based trading strategy
def create_forecast_strategy(data, model, lookback=30):
"""
Create trading signals based on time series forecasts
"""
signals = pd.DataFrame(index=data.index)
signals['Signal'] = 0
# Your strategy logic here:
# 1. Generate rolling forecasts
# 2. Compare forecast vs current price
# 3. Generate buy/sell signals
# 4. Add risk management rules
return signals
# Implement and test your strategy
# strategy_signals = create_forecast_strategy(data, lstm_model)
# Backtest the strategy
# Your backtesting code hereYou've mastered professional-grade time series analysis techniques used by quantitative hedge funds:
The Forecasting Paradox: The better we become at forecasting, the more we realize how difficult prediction really is. Professional quants spend more time understanding what they can't predict than what they can. This humility separates successful systematic traders from those who blow up accounts by overconfidence in their models.
Multi-Model Perspective: No single model captures all market dynamics. ARIMA reveals linear patterns, GARCH captures volatility clustering, LSTM learns non-linear relationships. Professional success comes from combining models intelligently, not from finding the "perfect" model. Diversification applies to modeling approaches as much as to portfolios.
Implementation Excellence: The gap between academic time series analysis and profitable trading lies in implementation details: data quality, computational efficiency, transaction costs, and behavioral factors. Great models fail with poor implementation, while modest models succeed with excellent execution. Focus on getting the basics right before pursuing exotic techniques.
Next, we'll put these forecasting skills to work by building our first complete momentum trading strategy with machine learning signals!
You've just mastered the mathematical and computational techniques that form the backbone of modern quantitative finance. From stationarity testing through GARCH volatility modeling to LSTM neural networks, you now possess the same analytical tools used by billion-dollar hedge funds and investment banks.
More importantly, you understand when and why to use each technique: ARIMA for short-term return forecasting, GARCH for volatility-dependent strategies like options trading, and LSTM for capturing complex patterns that classical models miss. This conceptual understanding is what separates skilled practitioners from those who just run algorithms.
The forecasting skills you've developed are immediately applicable to real trading strategies. Whether you're building momentum systems, volatility trading algorithms, or risk management frameworks, the time series analysis techniques in this lesson provide the mathematical foundation for systematic trading success.