{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "handed-robinson",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This document concerns how to use the GluonTS implementation of [Level Set Forecaster](https://papers.nips.cc/paper/2021/hash/32b127307a606effdcc8e51f60a45922-Abstract.html). Level Set Forecaster (LSF) is an algorithm for a tabular point prediction algorithm into a probabilistic one.\n",
    "\n",
    "A tabular point prediction algorithm is an algorithm that takes in tabular data and, at inference, outputs real numbers. Examples include: linear regression, random forests, XGBoost, etc. A tabular probabilistic prediction algorithm is one that at inference outputs an estimate of the conditional distribution.\n",
    "\n",
    "The high-level description of the LSF algorithm is that it bins the training data in such a manner that the predictions of the feature vectors in each bin are similar. At inference, it then uses the empirical distribution of true values for the bin associated with the new feature vector. We invite those who are interested to read the details and the theoretical guarantees in the [NeurIPS paper](https://papers.nips.cc/paper/2021/hash/32b127307a606effdcc8e51f60a45922-Abstract.html). \n",
    "\n",
    "The better your underlying model, the better performing its LSF wrapping will be."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numerous-marks",
   "metadata": {},
   "source": [
    "# Retrofitting an Existing Model\n",
    "\n",
    "Assume that you have trained a tabular point prediction algorithm and named it `underlying_model`. In order to wrap it with LSF, you have to feed into LSF the a dataset from the same distribution that `underlying_model` was trained on:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "destroyed-champion",
   "metadata": {},
   "source": [
    "```\n",
    "from gluonts.model.rotbaum._model import LSF\n",
    "model = LSF(model=underlying_model)\n",
    "model.fit(X_train, y_train, model_is_already_trained=True, min_bin_size=100)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aboriginal-arkansas",
   "metadata": {},
   "source": [
    "It is possible to provide LSF the exact same dataset that `underlying_model` was trained on, but it is not necessary. There is also no assumption that the dataset provided to LSF is the same size as the one provided to `underlying_model`.\n",
    "\n",
    "The hyperparameter `min_bin_size` is by default `100`. While the NeurIPS paper ensures consistency, under certain conditions, with an increasing `min_bin_size` (of order of magnitude `(ln(n))^2`), in practice we have found that keeping this hyperparameter at `100` works in a surprisingly wide variety of datasets.\n",
    "\n",
    "Note that some `underlying_model`s assume that `X_train` is a list of lists (or numpy array of numpy arrays), whereas some assume that it is a pandas dataframe. By default LSF works with the former. To apply LSF to the latter, simply set `x_train_is_dataframe=True` in `model.fit`.\n",
    "\n",
    "Note that while LSF is native to the tabular use case, it is possible to apply it with neural networks as well. To that end, LSF can come in at the embedding level of the architecture. To be a little more explicit, if f1 is the portion of the neural network that embeds the data into a fixed dimensional space, and the remainder of the network is f2, then after training you would cache pairs of embeddings and true target values, and later feed them into LSF together with f2 as the base algorithm. At inference, rather than feeding in the raw data, you would feed the data into f1 to obtain a fixed length feature vector, and only then feed it into the LSF."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "altered-hepatitis",
   "metadata": {},
   "source": [
    "# Letting LSF Train the Tabular Model As Well\n",
    "\n",
    "Letting `model_is_already_trained=False` (as is the default), LSF will train `underlying_model` first and only then wrap it with LSF.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dimensional-orange",
   "metadata": {},
   "source": [
    "```\n",
    "from gluonts.model.rotbaum._model import LSF\n",
    "from xgboost import XGBModel\n",
    "underlying_model = XGBModel(\n",
    "         base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
    "         colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n",
    "         importance_type='gain', interaction_constraints='',\n",
    "         learning_rate=0.300000012, max_delta_step=0, max_depth=3,\n",
    "         min_child_weight=1, monotone_constraints='()',\n",
    "         n_estimators=50, n_jobs=-1, num_parallel_tree=1,\n",
    "         objective='reg:squarederror', random_state=0, reg_alpha=0,\n",
    "         reg_lambda=1, scale_pos_weight=1, subsample=1, tree_method='exact',\n",
    "         validate_parameters=1, verbosity=1)\n",
    "model = LSF(model=underlying_model)\n",
    "model.fit(X_train, y_train)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "hearing-vermont",
   "metadata": {},
   "source": [
    "In case you want LSF to use less data to create the bins than the `underyling_model` uses to train, simply set `max_sample_size`. LSF will then sample `min(max_sample_size, len(X_train)` many data points from the training data without replacement for the purpose of creating the bins."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "funded-plymouth",
   "metadata": {},
   "source": [
    "# LSF Wrapping XGBoost\n",
    "\n",
    "By default, without specifying an `underlying_model`, LSF will wrap XGBoost with some default parameters:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "nominated-skirt",
   "metadata": {},
   "source": [
    "```\n",
    "from gluonts.model.rotbaum._model import LSF\n",
    "model.fit(X_train, y_train)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "capable-boating",
   "metadata": {},
   "source": [
    "# Inference\n",
    "\n",
    "If we want to estimate the conditional quantile for a specific quantile (between 0 and 1), simply query it thusly:\n",
    "```\n",
    "model.predict(X_test, quantile)\n",
    "```\n",
    "\n",
    "One can also retrieve the bin in its entirety, which in turn can be interpreted as an estimated sampling from the conditional distribution. To be precise `model.estimate_dist(X_test)` will output, in pseudocode, `[list of true values whose associated feature vectors are in the same bin as x for x in X_test]`.\n",
    "\n",
    "One can then plot histograms of the estimated conditional distributions:\n",
    "```\n",
    "from matplotlib import pyplot as plt\n",
    "plt.hist(model.estimate_dist(X_test)[0])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "whole-start",
   "metadata": {},
   "source": [
    "# Quick Synthetic Example to Get You Started"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "casual-fever",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/hashilaf/repos/gluonts-lightgbmfork2/gluon-ts/src/gluonts/json.py:101: UserWarning: Using `json`-module for json-handling. Consider installing one of `orjson`, `ujson` to speed up serialization and deserialization.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from gluonts.model.rotbaum._model import LSF\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "great-retailer",
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_quantile_loss(true, pred, quantile):\n",
    "    denom = sum(np.abs(true))\n",
    "    num = sum(\n",
    "        [\n",
    "            (1 - quantile) * abs(y_hat - y)\n",
    "            if y_hat > y\n",
    "            else quantile * abs(y_hat - y)\n",
    "            for y_hat, y in zip(pred, true)\n",
    "        ]\n",
    "    )\n",
    "    if denom != 0:\n",
    "        return 2 * num / denom\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "changing-creation",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = [\n",
    "    [np.random.normal(0, 1), np.random.normal(0, 1)] for i in range(10000)\n",
    "]\n",
    "y_train = [i + j + np.random.normal(0, abs(i + j)) for i, j in X_train]\n",
    "X_test = [\n",
    "    [np.random.normal(0, 1), np.random.normal(0, 1)] for i in range(10000)\n",
    "]\n",
    "y_test = [i + j + np.random.normal(0, abs(i + j)) for i, j in X_test]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "protective-freight",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "underlying_model = LinearRegression()\n",
    "underlying_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "quarterly-nowhere",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LSF(model=underlying_model, min_bin_size=100)\n",
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "catholic-milwaukee",
   "metadata": {},
   "outputs": [],
   "source": [
    "P10 = model.predict(X_test, 0.1)\n",
    "P50 = model.predict(X_test, 0.5)\n",
    "P90 = model.predict(X_test, 0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "amazing-florida",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.30368475167009445\n",
      "0.6872476043155556\n",
      "0.3032539971588588\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(array([ 4.,  4., 14., 19., 28., 14.,  9.,  6.,  1.,  1.]),\n",
       " array([-0.99932272, -0.48932945,  0.02066383,  0.53065711,  1.04065038,\n",
       "         1.55064366,  2.06063693,  2.57063021,  3.08062348,  3.59061676,\n",
       "         4.10061004]),\n",
       " <BarContainer object of 10 artists>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": " \n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(weighted_quantile_loss(y_test, P10, 0.1))\n",
    "print(weighted_quantile_loss(y_test, P50, 0.5))\n",
    "print(weighted_quantile_loss(y_test, P90, 0.9))\n",
    "plt.hist(model.estimate_dist([[0.5, 0.5]])[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "standing-trouble",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2.288148404849137,\n",
       " 1.574696116254158,\n",
       " 0.6568102103240447,\n",
       " 2.2993520742685227,\n",
       " 0.39109241176636667,\n",
       " 1.7025821798202085,\n",
       " 0.6855239431277019,\n",
       " 1.0452498711450835,\n",
       " 1.512439052988055,\n",
       " 0.8593260692605851,\n",
       " -0.6962483603681517,\n",
       " 0.139817808528079,\n",
       " 1.64273291259174,\n",
       " 1.1756085357251278,\n",
       " 1.2286820644502852,\n",
       " 2.327047163989284,\n",
       " 0.44771284640985787,\n",
       " 1.467409266835301,\n",
       " 0.9302439674991858,\n",
       " 1.03700532459507,\n",
       " 0.008129538932130087,\n",
       " 1.1368726098692654,\n",
       " 1.6079078731856582,\n",
       " -0.2317150940367927,\n",
       " 0.675471791654082,\n",
       " 2.0313939483686605,\n",
       " 0.8312038376464166,\n",
       " 1.645705787029204,\n",
       " 0.48299689048994443,\n",
       " 1.579827037650825,\n",
       " 1.165238373622516,\n",
       " 1.190270554221262,\n",
       " 1.4542366173260606,\n",
       " 1.2644800116574475,\n",
       " 1.4307864289204961,\n",
       " 0.7068177077124251,\n",
       " 0.13010745707289473,\n",
       " 1.2251120802386763,\n",
       " 0.3693318284416732,\n",
       " 1.1352689844230193,\n",
       " 1.9552669587197022,\n",
       " 2.4357911384654534,\n",
       " -0.0026836468661575186,\n",
       " 1.2065815093215613,\n",
       " 2.0651693948249856,\n",
       " 0.6721360437208823,\n",
       " 1.0491761310070071,\n",
       " 3.2172861439696208,\n",
       " 1.4453451621110442,\n",
       " 2.97958136611339,\n",
       " 2.1761274430887587,\n",
       " 0.3963338828779327,\n",
       " 2.267448034530609,\n",
       " 1.3228653106456072,\n",
       " 1.1575712211495195,\n",
       " 0.26233424616648937,\n",
       " 1.8000744038081407,\n",
       " 2.8989967920415873,\n",
       " 1.454721393152594,\n",
       " 1.031930958227963,\n",
       " 0.8561592385698967,\n",
       " 1.5360408249685844,\n",
       " 1.3960998629765138,\n",
       " 1.0472666019682473,\n",
       " 1.354787671331422,\n",
       " 2.71855795548974,\n",
       " 1.1237632317683988,\n",
       " 4.100610036246034,\n",
       " 0.15966831658768088,\n",
       " -0.6382599558124438,\n",
       " 0.36119013986411597,\n",
       " 1.5507273097237748,\n",
       " 0.7619370183841324,\n",
       " 1.5985185902942805,\n",
       " 0.1754941535388811,\n",
       " 0.4230780642143368,\n",
       " 0.6177496086652821,\n",
       " 1.6052831235499059,\n",
       " 2.65428516371382,\n",
       " 2.4405959721339556,\n",
       " 1.3484173092663088,\n",
       " 0.3135162527573676,\n",
       " 1.4690866663891833,\n",
       " 1.029260031206032,\n",
       " 0.46131889403271176,\n",
       " 2.3861704901341687,\n",
       " -0.9993227221284857,\n",
       " 1.659012249613435,\n",
       " 1.9533371600879148,\n",
       " 0.646100948352357,\n",
       " 0.876088514976303,\n",
       " 2.830688738531131,\n",
       " 0.8617456612312852,\n",
       " -0.7929279473938164,\n",
       " 0.5993728411844842,\n",
       " 1.3607497734619327,\n",
       " 1.3667941564055293,\n",
       " 0.6506879219033462,\n",
       " 2.6083040470905017,\n",
       " -0.29895103619716923]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.estimate_dist([[0.5, 0.5]])[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "typical-adrian",
   "metadata": {},
   "source": [
    "The bins in their entirety are stored in `model.id_to_bins`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "classified-fiction",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([  0.,   0.,   0.,   0.,   0., 100.,   0.,   0.,   0.,   0.]),\n",
       " array([ 99.5,  99.6,  99.7,  99.8,  99.9, 100. , 100.1, 100.2, 100.3,\n",
       "        100.4, 100.5]),\n",
       " <BarContainer object of 10 artists>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": " \n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist([len(model.id_to_bins[id]) for id in model.id_to_bins])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unique-belarus",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bestenv",
   "language": "python",
   "name": "bestenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
