2021DCIC智慧交通-基于车联网大数据的碰撞识别(TOP2方案).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 知识点标签\n",
    "\n",
    "有监督学习 》分类任务 》异常检测\n",
    "\n",
    "\n",
    "## 场景标签\n",
    "\n",
    "大数据 》交通预测 》碰撞识别"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 概述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-05T08:31:50.743802Z",
     "start_time": "2021-05-05T08:31:50.734802Z"
    }
   },
   "source": [
    "- **案例知识点**：\n",
    "    - [特征工程](https://discussion.datafountain.cn/articles/detail/477)\n",
    "    - [数据不均衡问题](https://discussion.datafountain.cn/articles/detail/76)\n",
    "    \n",
    "- **任务描述**: 本教程以赛题：[2021DCIC智慧交通-基于车联网大数据的碰撞识别](https://www.datafountain.cn/competitions/500/datasets)为例，此赛题使用的数据为采集车辆信号。车辆信息非常多，而且用户路况信息和使用偏好千人千面，很难找到一种准确识别碰撞的方法，希望参赛者通过车联网大数据识别车辆碰撞和碰撞时间。车辆标签信息如下：\n",
    "|车号|Label|CollectTime|\n",
    "|-|-|-|\n",
    "|1|1|2020/8/30 21:36:09|\n",
    "|2|0| |\n",
    "|3|1|2020/8/12 8:36:46|\n",
    "\n",
    "- **运行环境**：\n",
    "```\n",
    "joblib=1.0.1\n",
    "lightgbm=3.2.0\n",
    "numpy=1.20.2\n",
    "pandas=1.2.3\n",
    "scikit-learn=0.24.1\n",
    "tqdm=4.60.0\n",
    "```\n",
    "- **赛题解读**：\n",
    "\n",
    "1.原始数据时间缺失值多，存在一定的异常值\n",
    "\n",
    "2.原始数据特征间的相关性强，原始数据特征与碰撞标签的相关性弱\n",
    "\n",
    "3.数据严重不均衡\n",
    "\n",
    "4.精准碰撞时刻的预测存在困难\n",
    "\n",
    "\n",
    "- **方法概述**：首先针对数据特点进行数据预处理和特征工程；考虑到数据不均衡，进而对数据进行采样；最后，考虑到赛题对是否碰撞和碰撞时间预测的两个需求，将预测模型分为两层，第一层是预测样本碰撞状态，第二层模型在第一层的基础上进行精准的时间预测。\n",
    "![](./pictures/1.png)\n",
    "- **建议学习时长**：60分钟\n",
    "- **<span class=\"mark\">数据集版权说明：本实践使用互联网开源数据集，请勿用于商业用途。</span>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:55:15.816300Z",
     "start_time": "2021-05-08T02:55:15.810300Z"
    }
   },
   "outputs": [],
   "source": [
    "# !unzip -o 'datasets/data.zip' -d 'datasets/' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:14.390300Z",
     "start_time": "2021-05-08T02:57:13.420300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Package                           Version\n",
      "--------------------------------- -------------------\n",
      "anaconda-client                   1.7.2\n",
      "anaconda-navigator                1.10.0\n",
      "anyio                             2.2.0\n",
      "argon2-cffi                       20.1.0\n",
      "async-generator                   1.10\n",
      "attrs                             20.3.0\n",
      "Babel                             2.9.0\n",
      "backcall                          0.2.0\n",
      "backports.functools-lru-cache     1.6.3\n",
      "backports.tempfile                1.0\n",
      "backports.weakref                 1.0.post1\n",
      "beautifulsoup4                    4.9.3\n",
      "bleach                            3.3.0\n",
      "brotlipy                          0.7.0\n",
      "certifi                           2020.12.5\n",
      "cffi                              1.14.5\n",
      "chardet                           4.0.0\n",
      "click                             7.1.2\n",
      "clyent                            1.2.2\n",
      "colorama                          0.4.4\n",
      "conda                             4.10.0\n",
      "conda-build                       3.20.5\n",
      "conda-package-handling            1.7.2\n",
      "conda-verify                      3.4.2\n",
      "cryptography                      3.4.7\n",
      "cycler                            0.10.0\n",
      "decorator                         5.0.5\n",
      "defusedxml                        0.7.1\n",
      "entrypoints                       0.3\n",
      "filelock                          3.0.12\n",
      "future                            0.18.2\n",
      "glob2                             0.7\n",
      "idna                              2.10\n",
      "importlib-metadata                3.10.0\n",
      "ipykernel                         5.5.3\n",
      "ipython                           7.22.0\n",
      "ipython-genutils                  0.2.0\n",
      "ipywidgets                        7.6.3\n",
      "jedi                              0.18.0\n",
      "Jinja2                            2.11.3\n",
      "joblib                            1.0.1\n",
      "json5                             0.9.5\n",
      "jsonschema                        3.2.0\n",
      "jupyter-client                    6.1.12\n",
      "jupyter-contrib-core              0.3.3\n",
      "jupyter-contrib-nbextensions      0.5.1\n",
      "jupyter-core                      4.7.1\n",
      "jupyter-highlight-selected-word   0.2.0\n",
      "jupyter-latex-envs                1.4.6\n",
      "jupyter-nbextensions-configurator 0.4.1\n",
      "jupyter-packaging                 0.7.12\n",
      "jupyter-server                    1.5.1\n",
      "jupyterlab                        3.0.12\n",
      "jupyterlab-pygments               0.1.2\n",
      "jupyterlab-server                 2.4.0\n",
      "jupyterlab-widgets                1.0.0\n",
      "kiwisolver                        1.3.1\n",
      "libarchive-c                      2.9\n",
      "lightgbm                          3.2.0\n",
      "MarkupSafe                        1.1.1\n",
      "matlab                            0.1\n",
      "matplotlib                        3.4.1\n",
      "menuinst                          1.4.16\n",
      "mistune                           0.8.4\n",
      "navigator-updater                 0.2.1\n",
      "nbclassic                         0.2.6\n",
      "nbclient                          0.5.3\n",
      "nbconvert                         6.0.7\n",
      "nbformat                          5.1.3\n",
      "nest-asyncio                      1.5.1\n",
      "notebook                          6.3.0\n",
      "numpy                             1.20.2\n",
      "olefile                           0.46\n",
      "packaging                         20.9\n",
      "pandas                            1.2.3\n",
      "pandocfilters                     1.4.2\n",
      "parso                             0.8.2\n",
      "pickleshare                       0.7.5\n",
      "Pillow                            8.1.2\n",
      "pip                               21.0.1\n",
      "pkginfo                           1.7.0\n",
      "prometheus-client                 0.10.0\n",
      "prompt-toolkit                    3.0.18\n",
      "psutil                            5.8.0\n",
      "pycosat                           0.6.3\n",
      "pycparser                         2.20\n",
      "Pygments                          2.8.1\n",
      "pyOpenSSL                         20.0.1\n",
      "pyparsing                         2.4.7\n",
      "PyQt5                             5.12.3\n",
      "PyQt5-sip                         4.19.18\n",
      "PyQtChart                         5.12\n",
      "PyQtWebEngine                     5.12.1\n",
      "pyrsistent                        0.17.3\n",
      "PySocks                           1.7.1\n",
      "python-dateutil                   2.8.1\n",
      "pytz                              2021.1\n",
      "pywin32                           300\n",
      "pywinpty                          0.5.7\n",
      "PyYAML                            5.4.1\n",
      "pyzmq                             22.0.3\n",
      "QtPy                              1.9.0\n",
      "requests                          2.25.1\n",
      "ruamel-yaml-conda                 0.15.80\n",
      "scikit-learn                      0.24.1\n",
      "scipy                             1.6.2\n",
      "seaborn                           0.11.1\n",
      "Send2Trash                        1.5.0\n",
      "setuptools                        49.6.0.post20210108\n",
      "six                               1.15.0\n",
      "sniffio                           1.2.0\n",
      "soupsieve                         2.0.1\n",
      "terminado                         0.9.4\n",
      "testpath                          0.4.4\n",
      "threadpoolctl                     2.1.0\n",
      "tornado                           6.1\n",
      "tqdm                              4.60.0\n",
      "traitlets                         5.0.5\n",
      "urllib3                           1.26.4\n",
      "vboxapi                           1.0\n",
      "wcwidth                           0.2.5\n",
      "webencodings                      0.5.1\n",
      "wheel                             0.36.2\n",
      "widgetsnbextension                3.5.1\n",
      "win-inet-pton                     1.1.0\n",
      "wincertstore                      0.2\n",
      "xmltodict                         0.12.0\n",
      "zipp                              3.4.1\n"
     ]
    }
   ],
   "source": [
    "# 使用pip命令安装指定版本工具包,lightgbm对模型结果有影响\n",
    "!pip list\n",
    "# !pip install lightgbm==3.2.0\n",
    "# !pip install joblib==1.0.1 lightgbm==3.2.0 numpy==1.20.2 pandas==1.2.3 scikit-learn==0.24.1 tqdm==4.60.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:55:33.884300Z",
     "start_time": "2021-05-08T02:55:15.975300Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from joblib import Parallel,delayed\n",
    "from tqdm.notebook import tqdm\n",
    "import os\n",
    "\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "from sklearn.model_selection import KFold,StratifiedKFold,StratifiedShuffleSplit\n",
    "from sklearn.metrics import f1_score,precision_score,recall_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:55:33.919300Z",
     "start_time": "2021-05-08T02:55:33.890300Z"
    }
   },
   "outputs": [],
   "source": [
    "# csv文件合并并行设置,请根据CPU支持核数设置，默认4核\n",
    "jobs = 4\n",
    "\n",
    "label_path = './datasets/data/train/train_labels.csv'\n",
    "trn_path = './datasets/data/train'\n",
    "test_path = './datasets/data/test'\n",
    "\n",
    "result_path = './work/prediction_result'\n",
    "if not os.path.exists(result_path): # 如果不存在则创建目录\n",
    "    os.makedirs(result_path) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练集清洗"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **标签数据特点**：\n",
    "\n",
    "(1) 由于电动车电池具备防短路功能，车辆发生剧烈冲击时，安全气囊控制器会将信号发送给电池管理系统(BMS)，BMS通过电池继电器断开高压，防止电池起火。发生碰撞时，电池主负继电器状态由连接变为断开；\n",
    "\n",
    "(2) 绝大部分数据在碰撞瞬间，电池主负继电器状态处于断开状态；\n",
    "\n",
    "(3) 发生碰撞时，车辆速度会发生明显的变化；\n",
    "\n",
    "- **训练集异常标签清洗**：\n",
    "根据业务背景，可以发现训练集数据中部分标注碰撞的标签存在异常，不符合常识，对以下5台车标签进行修正。\n",
    "|车号|修改原因|\n",
    "|-|-|\n",
    "|5|车辆传感器故障，碰撞前后时刻车速一直为0|\n",
    "|19|没有碰撞时间相关特征数据|\n",
    "|36|碰撞前后时刻继电器一直处于连接状态，车速为20.953km/h未发生变化|\n",
    "|77|碰撞标签附近车速一直为0，而前6分钟车辆继电器由连接到断开的时刻，3s内速度由58km/h变为0|\n",
    "|94|没有碰撞时间相关特征数据|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:55:34.244300Z",
     "start_time": "2021-05-08T02:55:33.924300Z"
    }
   },
   "outputs": [],
   "source": [
    "df_label = pd.read_csv(label_path)\n",
    "# 车号5，19,36,94为异常数据，标签修正为0\n",
    "df_label.loc[4,:] = (5,0,np.NaN)\n",
    "df_label.loc[18,:] = (19,0,np.NaN)\n",
    "df_label.loc[35,:] = (36,0,np.NaN)\n",
    "df_label.loc[93,:] = (94,0,np.NaN)\n",
    "# 车号77碰撞时间有问题，修正为13:37:12\n",
    "df_label.loc[76,:] = (77,1,'2020/10/20 13:37:12')\n",
    "# df_label.to_csv('./原始数据/train_labels_revised.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 碰撞状态预测模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在碰撞状态预测模型中，先采用规则预测出发生强烈碰撞的车辆，再利用LightGBM模型对剩余车辆进行预测，该部分模型特征工程和采样有部分重合，因此下面主要介绍特征工程和采样的细节，规则预测模型细节见3.1.4，LightGBM模型细节见3.2.4：\n",
    "- **特征工程**\n",
    "\n",
    "特征工程主要从状态信息和运动信息两个方面考虑，对于状态信息而言，最重要的是启停状态两个特征，当继电器由连接变为断开时，if_off逐步由-5变为-1，其余时刻均为0，当继电器由断开变为连接时，if_on逐步由-1变为-5，其余时刻均为0。在车辆运动信息中，主要构造了车辆的瞬时加速度，局部加速度，加速度统计几个特征，并对几个主要特征进行了分桶操作。这些新特征与碰撞标签具备较强的相关性，使得后续样本采样以及的模型构造更加容易。\n",
    "![](./pictures/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-07T14:18:13.367300Z",
     "start_time": "2021-05-07T14:18:13.356300Z"
    }
   },
   "source": [
    "- **函数说明**\n",
    "\n",
    "```csv_finds()```函数批处理读csv文件位置，```csvs```存储了训练集文件下夹车号1-120的csv文件，```csvs_test```存储了测试集文件夹车号121-260的csv文件;\n",
    "\n",
    "```read_csv()```函数对每个csv函数进行数据清洗和特征工程；\n",
    "\n",
    "```applyParallel_concat()```函数合并所有清洗后的数据；\n",
    "\n",
    "```col_feature1()```函数对数据进行欠采样,供规则预测模型使用；\n",
    "\n",
    "```col_feature2()```函数对数据进行欠采样和特征工程,供LightGBM预测模型使用；"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 规则预测模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 文件读取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:55:34.265300Z",
     "start_time": "2021-05-08T02:55:34.249300Z"
    }
   },
   "outputs": [],
   "source": [
    "def csv_find(parent_path, file_flag):\n",
    "    '''批处理读csv文件位置'''\n",
    "    df_paths = []\n",
    "    for root, dirs, files in os.walk(parent_path):  # os.walk输出[目录路径，子文件夹，子文件]\n",
    "        for file in files:\n",
    "            if \"\".join(file).find(file_flag) != -1:  # 判断是否csv文件(files是list格式，需要转成str)\n",
    "                df_paths.append(root + '/' + file)  # 所有csv文件\n",
    "    return df_paths\n",
    "\n",
    "csvs = csv_find(trn_path,file_flag='csv')\n",
    "csvs_test = csv_find(test_path,file_flag='csv')\n",
    "csvs.remove(label_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征工程\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:10.549300Z",
     "start_time": "2021-05-08T02:55:34.272300Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ffde2f265784088a0323388a7ea3d0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/120 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df_trnall (3928449, 29)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "210fb2a41c784c76a8f48dd9040c3f5c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/140 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df_testall (4285948, 29)\n"
     ]
    }
   ],
   "source": [
    "def read_csv(path):\n",
    "    del_cols = ['车辆行驶里程','驾驶员需求扭矩值']\n",
    "    # '车速'单独考虑\n",
    "    num_cols = [ '低压蓄电池电压','整车当前总电流','整车当前总电压']\n",
    "    \n",
    "    '''数据处理'''\n",
    "    # 读取文件\n",
    "    df0 = pd.read_csv(path,index_col=False,low_memory=False)\n",
    "    # 删除缺失值\n",
    "    df = df0.dropna(axis=0,thresh=17)\n",
    "    df = df.rename(columns={'采集时间':'CollectTime'})\n",
    "    df['CollectTime'] = pd.to_datetime(df['CollectTime'],format='%Y-%m-%d %H:%M:%S')\n",
    "    # 去重\n",
    "    df.drop_duplicates(subset=['车号','CollectTime'],keep='first',inplace=True)\n",
    "    # 排序\n",
    "    df = df.sort_values('CollectTime').reset_index(drop=True)\n",
    "    # 删除无效特征\n",
    "    df = df.drop(del_cols,axis=1)\n",
    "    # 档位状态\n",
    "    df['整车当前档位状态'] = df['整车当前档位状态'].replace('驻车','空档')\n",
    "    # 电池包主负继电器状态\n",
    "    df['电池包主负继电器状态'] = df['电池包主负继电器状态'].replace('粘连','断开')\n",
    "    # 主驾驶座占用状态\n",
    "    df = df[df['主驾驶座占用状态'] != '传感器故障']\n",
    "    \n",
    "    '''特征构造'''\n",
    "    df['time_delta'] = df['CollectTime'].diff().dt.total_seconds()\n",
    "    df['time_delta_5'] = (df['CollectTime']-df['CollectTime'].shift(5)).dt.total_seconds()\n",
    "         \n",
    "    # 特征构造1: 判断数据是启动or停车——162 80(2020-12-02 13:04:13)\n",
    "    a = pd.DataFrame()\n",
    "    b = pd.DataFrame()\n",
    "    df['电池包主负继电器状态cate'] = df['电池包主负继电器状态'].astype('category').cat.codes\n",
    "    for i in np.arange(5):\n",
    "        a['电池状态'+str(i)] = df['电池包主负继电器状态cate']-df['电池包主负继电器状态cate'].shift(i+1)\n",
    "        b['电池状态'+str(i)] = df['电池包主负继电器状态cate']-df['电池包主负继电器状态cate'].shift(-i-1)\n",
    "    df['if_off'] = a.sum(axis=1)\n",
    "    df['if_on'] = b.sum(axis=1)\n",
    "    \n",
    "    # 特征构造2: 车速特征\n",
    "    df['v_diff1'] = df['车速'].diff()/df['time_delta']#用于规则分类\n",
    "    df['v_diff2'] = -df['车速'].shift(3).rolling(window=3).mean()/df['time_delta_5']#用于规则分类(前5条数据的3条取平均)\n",
    "    df['v_diff3'] = df['车速'].diff() # 用于规则&LGBM\n",
    "    df['v_diff4'] = df['v_diff1'].shift(-1) # 用于时间预测\n",
    "    df['a_min5'] = df['v_diff1'].rolling(window=3).min() # 用于LGBM模型\n",
    "    df['a_mean5'] = df['v_diff1'].rolling(window=3).mean() # 用于LGBM模型\n",
    "    df['a_max3'] = df['v_diff1'].rolling(window=3).min() # 用于LGBM模型\n",
    "    \n",
    "    df = df.iloc[5:,:]\n",
    "\n",
    "    # 删除时间戳太长的数据\n",
    "    df = df[df['time_delta_5']<90]\n",
    "    \n",
    "    return df\n",
    "\n",
    "def applyParallel_concat(paths, func, jobs=4):\n",
    "    ret = Parallel(n_jobs=jobs)(delayed(func)(csv)for csv in tqdm(paths))\n",
    "    return pd.concat(ret)\n",
    "\n",
    "\n",
    "df_trnall = applyParallel_concat(csvs,read_csv,jobs=jobs)\n",
    "print('df_trnall',df_trnall.shape)\n",
    "df_testall = applyParallel_concat(csvs_test,read_csv,jobs=jobs)\n",
    "print('df_testall',df_testall.shape)\n",
    "# df_trnall.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **数据欠采样**\n",
    "\n",
    "根据业务知识，汽车碰撞后电池包主负继电器处于断开状态；其次，训练集的所有标签均分布在继电器断开瞬间附近，即if_off处于-3~-5的区间，考虑到可能的停车时被追尾，我们增加了车速>0的条件，同时增加了条件3，剔除启动阶段出现一部分正常低车速数据。（条件3在```col_feature2```函数中）\n",
    "![](./pictures/3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:14.097300Z",
     "start_time": "2021-05-08T02:56:10.552300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn (9178, 29)\n",
      "test (10772, 29)\n"
     ]
    }
   ],
   "source": [
    "def col_feature1(df):\n",
    "    # 条件筛选数据\n",
    "    df = df [(df['电池包主负继电器状态cate']==0)] # 停车条件1\n",
    "    # 68的off_or_on==-3\n",
    "    df = df [(df['if_off']<-2) | (df['车速']!=0)] # 停车条件2\n",
    "    return df\n",
    "\n",
    "df_trnall2 = col_feature1(df_trnall)\n",
    "df_testall2 = col_feature1(df_testall)\n",
    "print('trn',df_trnall2.shape)\n",
    "print('test',df_testall2.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 集成标签(重采样)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-05T12:46:42.942802Z",
     "start_time": "2021-05-05T12:46:42.932802Z"
    }
   },
   "source": [
    "- **数据重采样**\n",
    "\n",
    "考虑到碰撞是一个连续过程，将碰撞时间前后5s时间均标记为碰撞，由此，训练集的碰撞标签由49变为了154个，提高了模型预测的泛化能力。\n",
    "![](./pictures/4.png)\n",
    "- 代码思路:依次读取训练集Label==1车号中的时间，增加其上下5s时间，并将其Label赋值为1，然后与训练数据进行合并。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:14.925300Z",
     "start_time": "2021-05-08T02:56:14.099300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "标签1数据量 (154, 30)\n",
      "标签1车辆数量 49\n"
     ]
    }
   ],
   "source": [
    "# 对训练集标签进行重采样\n",
    "df_label['CollectTime'] = pd.to_datetime(df_label['CollectTime'],format='%Y-%m-%d %H:%M:%S')\n",
    "# 增加label数量\n",
    "df_label_new = df_label\n",
    "df_label1 = df_label[df_label['Label']==1]\n",
    "for kind,kind_df in df_label1.groupby('车号'):\n",
    "    # 增加上下5s时间的Label\n",
    "    for t in np.arange(5):\n",
    "        new_row1 = pd.DataFrame({'车号': kind, 'Label':1,\n",
    "                                'CollectTime': kind_df['CollectTime'].iloc[0] + pd.Timedelta(seconds=t+1)},index=[1])\n",
    "        new_row2 = pd.DataFrame({'车号': kind, 'Label':1,\n",
    "                                'CollectTime': kind_df['CollectTime'].iloc[0] - pd.Timedelta(seconds=t+1)},index=[1])\n",
    "        df_label_new = df_label_new.append(new_row1, ignore_index=True)\n",
    "        df_label_new = df_label_new.append(new_row2, ignore_index=True)\n",
    "# 将筛选过的训练数据和重采样的Label和并\n",
    "df = pd.merge(df_trnall2,df_label_new, on=['车号', 'CollectTime'], how='left')\n",
    "df['Label'] = df['Label'].fillna(0)\n",
    "df1 = df[df['Label']==1]\n",
    "\n",
    "print('标签1数据量',df1.shape)\n",
    "print('标签1车辆数量',df1['车号'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 规则分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从构造的几个速度相关的特征来看，发生强烈碰撞的标签是比较好区分的，因此首先利用规则模型预测发生强烈碰撞的车辆里考虑到模型的泛化能力将分隔阈值设的很大，以避免过拟合。\n",
    "![](./pictures/5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:14.965300Z",
     "start_time": "2021-05-08T02:56:14.930300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn规则分类车辆数: 30\n",
      "trn规则分类车辆: [  1   3  12  15  17  35  38  41  44  47  50  56  58  61  62  64  65  68\n",
      "  69  74  76  77  85  98 105 110 112 113 119 120]\n",
      "----------------------------------------------------------------------------------------------------\n",
      "test规则分类车辆数: 45\n",
      "test规则分类车辆: [132 133 135 138 140 147 150 152 154 158 161 163 164 168 170 172 175 176\n",
      " 186 187 193 197 201 202 205 206 207 209 210 212 213 222 223 227 230 231\n",
      " 233 235 244 249 252 253 254 255 260]\n"
     ]
    }
   ],
   "source": [
    "# 按车速差分类\n",
    "feat = (df['v_diff1']<-9)|(df['v_diff2']<-6)|(df['v_diff3']<-40)\n",
    "feat_test = (df_testall2['v_diff1']<-9)|(df_testall2['v_diff2']<-6)|(df_testall2['v_diff3']<-40)\n",
    "print('trn规则分类车辆数:',df[feat]['车号'].nunique())\n",
    "print('trn规则分类车辆:',df[feat]['车号'].sort_values().unique())\n",
    "print('-'*100)\n",
    "print('test规则分类车辆数:',df_testall2[feat_test]['车号'].nunique())\n",
    "print('test规则分类车辆:',df_testall2[feat_test]['车号'].sort_values().unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 结果保存\n",
    "- 代码思路：提取出训练集中和测试集中label==1的数据(```df_trn_label```和```df_testall2```)，按车号进行```groupby()```操作，取时间最早的数据作为结果，存储到```sub_trn```和```sub_rule```中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:15.350300Z",
     "start_time": "2021-05-08T02:56:14.969300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn预测最大时间差: 0 days 00:00:04\n",
      "trn需预测数据: (90, 3)\n"
     ]
    }
   ],
   "source": [
    "## 提取结果\n",
    "df_trn_label = df[feat] # 预测的车号\n",
    "sub_trn = pd.DataFrame(columns=['车号','Label_pred','CollectTime_pred'])\n",
    "sub_trn = pd.merge(df_label,sub_trn,on=['车号'],how='left')\n",
    "for kind,kind_df in df_trn_label.groupby('车号'):\n",
    "    sub_trn.loc[sub_trn[sub_trn['车号']==kind].index,'Label_pred'] = 1\n",
    "    sub_trn.loc[sub_trn[sub_trn['车号']==kind].index,'CollectTime_pred'] = kind_df['CollectTime'].iloc[0]\n",
    "sub_trn['CollectTime_pred'] = pd.to_datetime(sub_trn['CollectTime_pred'],format='%Y-%m-%d %H:%M:%S')\n",
    "print('trn预测最大时间差:',np.abs(sub_trn['CollectTime_pred']-sub_trn['CollectTime']).max())\n",
    "#提取出还未预测的车辆\n",
    "cols = ['车号','Label','CollectTime']\n",
    "trn_for_pred = sub_trn[sub_trn['Label_pred']!=1][cols]\n",
    "print('trn需预测数据:',trn_for_pred.shape)\n",
    "\n",
    "# 提交结果\n",
    "sub_rule = pd.DataFrame(columns=['车号','Label','CollectTime'])\n",
    "sub_rule['车号'] = np.arange(121,261)\n",
    "for kind,kind_df in df_testall2[feat_test].groupby('车号'):\n",
    "    sub_rule.loc[sub_rule[sub_rule['车号']==kind].index,'Label'] = 1\n",
    "    sub_rule.loc[sub_rule[sub_rule['车号']==kind].index,'CollectTime'] = kind_df['CollectTime'].iloc[0]\n",
    "test_for_pred = sub_rule[sub_rule['Label']!=1][cols]\n",
    "# trn_for_pred.to_csv('./train_label_for_pred.csv',index=False)\n",
    "# test_for_pred.to_csv('./test_label_for_pred.csv',index=False)\n",
    "# sub_rule.to_csv('./submit_rule.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LightGBM预测模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 文件读取\n",
    "由于规则预测模型已经预测出了部分车号，因此LightGBM只需要预测```trn_for_pred```和```test_for_pred```中存储的剩余的车号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:15.368300Z",
     "start_time": "2021-05-08T02:56:15.354300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集csv数: 90\n",
      "测试集csv数: 95\n"
     ]
    }
   ],
   "source": [
    "csvs_LGB = []\n",
    "csvs_test_LGB = []\n",
    "for i in trn_for_pred['车号']:\n",
    "    path = trn_path + '/' + str(i) + '.csv'\n",
    "    csvs_LGB.append(path)\n",
    "for i in test_for_pred['车号']:\n",
    "    path = test_path + '/' + str(i)+'.csv'\n",
    "    csvs_test_LGB.append(path)\n",
    "print('训练集csv数:',len(csvs_LGB))\n",
    "print('测试集csv数:',len(csvs_test_LGB))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:39.559300Z",
     "start_time": "2021-05-08T02:56:15.371300Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e4a79fe9e6149f09fd0e68a792ea4ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/90 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df_trnall (2923559, 29)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "efd333cdb48b4e4da1bf3f4c857011e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/95 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df_testall (2954171, 29)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Index(['车号', 'CollectTime', '加速踏板位置', '电池包主负继电器状态', '电池包主正继电器状态', '制动踏板状态',\n",
       "       '驾驶员离开提示', '主驾驶座占用状态', '驾驶员安全带状态', '手刹状态', '整车钥匙状态', '低压蓄电池电压',\n",
       "       '整车当前档位状态', '整车当前总电流', '整车当前总电压', '车速', '方向盘转角', 'time_delta',\n",
       "       'time_delta_5', '电池包主负继电器状态cate', 'if_off', 'if_on', 'v_diff1',\n",
       "       'v_diff2', 'v_diff3', 'v_diff4', 'a_min5', 'a_mean5', 'a_max3'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_trnall_LGB = applyParallel_concat(csvs_LGB,read_csv,jobs=jobs)\n",
    "print('df_trnall',df_trnall_LGB.shape)\n",
    "df_testall_LGB = applyParallel_concat(csvs_test_LGB,read_csv,jobs=jobs)\n",
    "print('df_testall',df_testall_LGB.shape)\n",
    "df_trnall_LGB.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 特征工程\n",
    "LightGBM模型中的特征和规则预测模型中略有不同，这里用```col_feature2（）```函数进行特征工程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:50.002300Z",
     "start_time": "2021-05-08T02:56:39.561300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn (4528, 9)\n",
      "test (5100, 9)\n",
      "Index(['车号', 'CollectTime', 'if_off', 'v_diff1', 'a_min5', 'a_mean5', 'a_max3',\n",
      "       '整车钥匙状态catestd', 'v_diff3_bin'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "def col_feature2(df):\n",
    "    cate_cols = ['制动踏板状态', '驾驶员离开提示','主驾驶座占用状态', '驾驶员安全带状态',\n",
    "                 '手刹状态', '整车钥匙状态', '整车当前档位状态']\n",
    "    cate_cols2 = []\n",
    "    \n",
    "    # 类别特征编码\n",
    "    for col in cate_cols:\n",
    "        df[col+'cate'] = df[col].astype('category').cat.codes\n",
    "        cate_cols2.append(col+'cate')\n",
    "    df_code_dict = {col:{code:cate for code,cate in enumerate(df[col].astype('category').cat.categories)} \n",
    "                    for col in cate_cols}\n",
    "    df['整车钥匙状态catestd'] = df['整车钥匙状态cate'].rolling(window=5,center=True).std()\n",
    "    \n",
    "    # 条件筛选数据\n",
    "    df = df [(df['电池包主负继电器状态cate']==0)] # 停车条件1\n",
    "    df = df [(df['if_on']==0) | (df['车速']>20)] # 停车条件2 判断是否启动阶段，或者车速过大的异常清空\n",
    "    df = df [(df['if_off']<-3) | (df['车速']!=0)] # 停车条件3  判断是否停车阶段，或者车速异常大于0的情况\n",
    "    \n",
    "    # 车速分桶\n",
    "    bin1 = [-0.1,0.01,0.5,1.5,3,100]\n",
    "    df['v_bin'] = pd.cut(df['车速'], bin1, labels=False)\n",
    "    \n",
    "    # 加速踏板特征\n",
    "    df['a0'] = df.apply(lambda x:  1 if (x['加速踏板位置']>0) else 0,axis=1)\n",
    "    df['a_min5'] = df['a_min5'] - df['v_bin']*df['v_bin']/2 - df['a0']*1.5\n",
    "    df['a_mean5'] = df['a_mean5'] - df['v_bin']*df['v_bin']/2 - df['a0']*1.5\n",
    "#     df['a_min5'] = df['a_min5'] - df['v_bin']*df['v_bin'] - df['a0']*3\n",
    "#     df['a_mean5'] = df['a_mean5'] - df['v_bin']*df['v_bin'] - df['a0']*3\n",
    "    df['v_diff1'] = df['v_diff1'] - df['v_bin']*df['v_bin'] - df['a0']*3\n",
    "    df['v_diff3'] = df['v_diff3'] - df['v_bin']*df['v_bin'] - df['a0']*3\n",
    "    \n",
    "    # 加速度分桶\n",
    "    bin1 = [-100,-15,-10,-6,-3,-0.01,0.1,100]\n",
    "    df['v_diff3_bin'] = pd.cut(df['v_diff3'], bin1, labels=False)\n",
    "    \n",
    "    ori_cols = ['加速踏板位置', '电池包主负继电器状态', '电池包主正继电器状态', '制动踏板状态',\n",
    "       '驾驶员离开提示', '主驾驶座占用状态', '驾驶员安全带状态', '手刹状态', '整车钥匙状态', '低压蓄电池电压',\n",
    "       '整车当前档位状态', '整车当前总电流', '整车当前总电压','车速', '方向盘转角']\n",
    "    cate_cols = ['电池包主负继电器状态cate','制动踏板状态cate', '驾驶员离开提示cate','主驾驶座占用状态cate',\n",
    "                 '驾驶员安全带状态cate', '手刹状态cate', '整车钥匙状态cate','整车当前档位状态cate',]\n",
    "    choose_cols = ['time_delta', 'time_delta_5','if_on','a0','v_bin','v_diff3','v_diff2','v_diff4']\n",
    "    df = df.drop(ori_cols+cate_cols+choose_cols,axis=1)\n",
    "\n",
    "    return df,df_code_dict\n",
    "\n",
    "\n",
    "df_trnall2_LGB , df_code_dict1= col_feature2(df_trnall_LGB)\n",
    "print('trn',df_trnall2_LGB.shape)\n",
    "df_testall2_LGB, df_code_dict2 = col_feature2(df_testall_LGB)\n",
    "print('test',df_testall2_LGB.shape)\n",
    "print(df_trnall2_LGB.columns)\n",
    "# df_testall2_LGB.to_csv('./test_data.csv',encoding='GBK',index=False)\n",
    "# df_code_dict1 == df_code_dict2\n",
    "# df_code_dict1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 集成标签(重采样)\n",
    "- 代码思路：由于规则预测模型中已经对生成了重采样标签```df_label_new```，这里直接用```pd.merge()```函数进行合并即可"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:50.037300Z",
     "start_time": "2021-05-08T02:56:50.006300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练数据label==1数量: (47, 10)\n",
      "训练数据label==1车数量: 19\n"
     ]
    }
   ],
   "source": [
    "df_LGB = pd.merge(df_trnall2_LGB,df_label_new, on=['车号', 'CollectTime'], how='left')\n",
    "df_LGB['Label'] = df_LGB['Label'].fillna(0)\n",
    "df1_LGB = df_LGB[df_LGB['Label']==1]\n",
    "print('训练数据label==1数量:',df1_LGB.shape)\n",
    "print('训练数据label==1车数量:',df1_LGB['车号'].nunique())\n",
    "# df.to_csv('./train_data.csv',encoding='GBK',index=False)\n",
    "# df1.to_csv('./train_label_data.csv',encoding='GBK',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LGBM模型分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-05T09:52:52.239802Z",
     "start_time": "2021-05-05T09:52:52.228802Z"
    }
   },
   "source": [
    "为了增强树模型的预测能力，我们对原始的特征进行了交叉修正，修正后，这几个主要特征与label的相关性更为明显。通过分层抽样、调参获得了状态预测结果，由于重采样预测结果中同一辆车可能有多个邻近的预测时间，取最早的一个时间作为预测结果。\n",
    "![](./pictures/6.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:50.120300Z",
     "start_time": "2021-05-08T02:56:50.039300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data (4528, 10)\n",
      "testdata (5100, 7)\n",
      "Index(['Num', 'CollectTime', 'if_off', 'v_diff1', 'a_min5', 'a_mean5',\n",
      "       'a_max3', 'key_std', 'v_diff3_bin', 'Label'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "## 重命名\n",
    "cols = {'车号':'Num','整车钥匙状态catestd':'key_std',}\n",
    "\n",
    "df_LGB = df_LGB.rename(columns=cols)\n",
    "df_test = df_testall2_LGB.rename(columns=cols)\n",
    "\n",
    "# 训练数据的特征\n",
    "f_names = [x for x in df_LGB.columns if x not in['Num','CollectTime','Label']]\n",
    "test_data = df_test[f_names]\n",
    "print('data',df_LGB.shape)\n",
    "print('testdata',test_data.shape)\n",
    "print(df_LGB.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 代码思路：抽样方式按照车号的标签采用了分层抽样，即`split.split()`函数中的抽样数据为`trn_for_pred`，`trn_data`和`val_data`是将根据`trn_for_pred`中抽样的车号数据获得，这样保证了按车数量抽样而不是标签数量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:56:53.072300Z",
     "start_time": "2021-05-08T02:56:50.125300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn Label 0.0    3080\n",
      "1.0      34\n",
      "Name: Label, dtype: int64\n",
      "val Label 0.0    1401\n",
      "1.0      13\n",
      "Name: Label, dtype: int64\n",
      "trn车数量 63\n",
      "val车数量 27\n"
     ]
    }
   ],
   "source": [
    "split = StratifiedShuffleSplit(n_splits=1,test_size=0.3,random_state=2021)\n",
    "# 按照label里的车号划分，再进入df中选择对应车号的行,保证trn和val的车号不相同\n",
    "# 这个地方一定要加.values，并且注意别写错地方了\n",
    "for trn_idx,val_idx in split.split(trn_for_pred['车号'],trn_for_pred['Label']):\n",
    "    trn_data = df_LGB[df_LGB['Num'].apply(lambda x: x in trn_for_pred.iloc[trn_idx]['车号'].values)].reset_index(drop=True)\n",
    "    val_data = df_LGB[df_LGB['Num'].apply(lambda x: x in trn_for_pred.iloc[val_idx]['车号'].values)].reset_index(drop=True)\n",
    "\n",
    "trn_x, trn_y =  trn_data[f_names], trn_data.loc[:,'Label']\n",
    "val_x, val_y =  val_data[f_names], val_data.loc[:,'Label']\n",
    "\n",
    "print('trn Label',trn_data['Label'].value_counts())\n",
    "print('val Label',val_data['Label'].value_counts())\n",
    "print('trn车数量',trn_data['Num'].nunique())\n",
    "print('val车数量',val_data['Num'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-07T14:36:14.157300Z",
     "start_time": "2021-05-07T14:36:14.148300Z"
    }
   },
   "source": [
    "- 代码思路：调参用了贝叶斯优化，由于上分点主要在于数据清洗和特征工程，以及不均衡数据的处理，调参也不是重点，限于篇幅原因不展开了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:04.298300Z",
     "start_time": "2021-05-08T02:56:53.074300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 2000 rounds\n",
      "[3000]\tvalid_0's binary_logloss: 0.00013201\n",
      "[6000]\tvalid_0's binary_logloss: 0.0001186\n",
      "[9000]\tvalid_0's binary_logloss: 0.000114999\n",
      "[12000]\tvalid_0's binary_logloss: 0.000114034\n",
      "[15000]\tvalid_0's binary_logloss: 0.000113444\n",
      "[18000]\tvalid_0's binary_logloss: 0.000112927\n",
      "[21000]\tvalid_0's binary_logloss: 0.000112591\n",
      "[24000]\tvalid_0's binary_logloss: 0.000112362\n",
      "[27000]\tvalid_0's binary_logloss: 0.000112196\n",
      "[30000]\tvalid_0's binary_logloss: 0.000112071\n",
      "Did not meet early stopping. Best iteration is:\n",
      "[29999]\tvalid_0's binary_logloss: 0.000112069\n",
      "trn F1 1.0\n",
      "val F1 1.0\n"
     ]
    }
   ],
   "source": [
    "# 调参val由0.83到0.93，线上0.85到0.925，对于不均衡样本colsample要尽可能小\n",
    "model =  LGBMClassifier(n_jobs=8,\n",
    "                        n_estimators=30000,\n",
    "                        objective='binary',\n",
    "                        boosting_type='gbdt',\n",
    "                        learning_rate=0.104962713,\n",
    "                        subsample=0.9,\n",
    "                        colsample_bytree=1,\n",
    "                        is_unbalance=True,\n",
    "#                         scale_pos_weight=30,\n",
    "                        reg_lambda=0,\n",
    "                        num_leaves=100,\n",
    "                        random_state=2021)\n",
    "model.fit(trn_x,trn_y,\n",
    "          eval_set=[(val_x,val_y)],\n",
    "          eval_metric='logloss',\n",
    "          early_stopping_rounds=2000,\n",
    "          verbose=3000)\n",
    "# 预测\n",
    "pred_val = model.predict(val_x)\n",
    "pred_trn = model.predict(trn_x)\n",
    "pred_test = model.predict(test_data)\n",
    "\n",
    "#得分\n",
    "f1_trn = f1_score(pred_trn,trn_y)\n",
    "f1_val = f1_score(pred_val,val_y)\n",
    "print('trn F1',f1_trn)\n",
    "print('val F1',f1_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 结果保存\n",
    "- 代码思路：和规则预测模型的思路一致，取时间最早的数据作为结果，测试集结果存储到和`sub`中，然后将`sub_rule`和`sub`合并，得到总预测结果`sub_all`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:04.353300Z",
     "start_time": "2021-05-08T02:57:04.303300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "submit_pred: (19, 3)\n"
     ]
    }
   ],
   "source": [
    "df_test['label_pred'] = pred_test\n",
    "df_test_sub = df_test[df_test['label_pred']==1]\n",
    "\n",
    "sub = pd.DataFrame(columns=['车号','Label','CollectTime'])\n",
    "sub['车号'] = np.arange(121,261)\n",
    "for kind,kind_df in df_test_sub.groupby('Num'):\n",
    "    sub.loc[sub[sub['车号']==kind].index,'Label'] = 1\n",
    "    sub.loc[sub[sub['车号']==kind].index,'CollectTime'] = kind_df['CollectTime'].iloc[0]\n",
    "print('submit_pred:',sub[sub['Label']==1].shape)\n",
    "# sub.to_csv('./submit_pred.csv',index=False)#,encoding='GBK'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:04.419300Z",
     "start_time": "2021-05-08T02:57:04.356300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "总提交正样本数: 64\n"
     ]
    }
   ],
   "source": [
    "### 提交\n",
    "# sub_all = pd.read_csv('./submit_rule.csv')\n",
    "sub_all = sub_rule.copy()\n",
    "a = sub[sub['Label']==1].index.values\n",
    "sub_all.iloc[a] = sub[sub['Label']==1]\n",
    "sub_all['Label'] = sub_all['Label'].fillna(0) #补0\n",
    "sub_all['Label'] = sub_all['Label'].astype(int)#转int\n",
    "# sub_all.to_csv('./submit_all.csv',index=False)\n",
    "print('总提交正样本数:',sub_all[sub_all['Label']==1].shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 碰撞时间预测模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "碰撞状态预测模型中，由于重采样以及构建的一些局部特征，时间预测并不准确，需要对状态预测模型结果中的时间进行修正。\n",
    "\n",
    "可以看到绝大部分标签处于if_off=-5的时间点，因此首先将训练集标签中if_off=-5的时间标注为1，其他车辆标注为0，将时间预测转化为一个二分类模型。接下来构建时间预测的瞬时特征，采用可解释程度高并且简单的K近邻算法。获得结果后，将Time_label=0标签中的碰撞时间修正为当前时刻的下个时刻，得到了最终的结果。\n",
    "![](./pictures/7.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 重采样\n",
    "由于需要对训练集和测试集的预测结果进行修正，因此这里对预测结果进行了重采样(注意前面的重采样是对真实标签重采样，这里是对预测标签，二者不一样)\n",
    "\n",
    "重采样后和特征数据合并，找到预测结果中附近的if_off==-5数据，对训练集标签进行修正\n",
    "\n",
    "- 代码思路：对预测结果进行重采样，得到训练集重采样标签时间`time1`和测试集重采样时间`time1_test`,然后和特征数据合并，取出`if_off==-5`的标签，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:04.517300Z",
     "start_time": "2021-05-08T02:57:04.421300Z"
    }
   },
   "outputs": [],
   "source": [
    "df_test_label  = sub_all[sub_all['Label']==1]\n",
    "df_test_label['CollectTime'] = pd.to_datetime(df_test_label['CollectTime'],format='%Y-%m-%d %H:%M:%S')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:06.292300Z",
     "start_time": "2021-05-08T02:57:04.521300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn (539, 2)\n",
      "trn车数量: (49, 3)\n",
      "test (704, 2)\n",
      "test车数量: (64, 3)\n"
     ]
    }
   ],
   "source": [
    "def label_resample(df):\n",
    "    df_new = df[['车号','CollectTime']]\n",
    "    for kind,kind_df in df.groupby('车号'):\n",
    "        for t in np.arange(5):\n",
    "            new_row1 = pd.DataFrame({'车号': kind,\n",
    "                                    'CollectTime': kind_df['CollectTime'].iloc[0] + pd.Timedelta(seconds=t+1)},index=[1])\n",
    "            new_row2 = pd.DataFrame({'车号': kind,\n",
    "                                    'CollectTime': kind_df['CollectTime'].iloc[0] - pd.Timedelta(seconds=t+1)},index=[1])\n",
    "            df_new = df_new.append(new_row1, ignore_index=True)\n",
    "            df_new = df_new.append(new_row2, ignore_index=True)\n",
    "    return df_new\n",
    "\n",
    "time1 = label_resample(df_label[df_label['Label']==1])\n",
    "print('trn',time1.shape)\n",
    "print('trn车数量:',df_label[df_label['Label']==1].shape)\n",
    "time1_test = label_resample(df_test_label)\n",
    "print('test',time1_test.shape)\n",
    "print('test车数量:',df_test_label.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:06.358300Z",
     "start_time": "2021-05-08T02:57:06.297300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trn Label==1总数量: (49, 30)\n",
      "Trn车if_off==-5 的数量: (38, 30)\n",
      "Test Label==1总数量: (64, 29)\n"
     ]
    }
   ],
   "source": [
    "# 找到预测结果中附近的if_off==-5数据\n",
    "df_trnall2_time = df_trnall2[df_trnall2['if_off']==-5]\n",
    "df_trn_time = pd.merge(df_trnall2_time,time1, on=['车号', 'CollectTime'], how='inner')\n",
    "# 找到Trn中的TimeLabel: 'if_off'==-5时 Time_Label=1\n",
    "df_trn_time = pd.merge(df_trn_time,df_label[df_label['Label']==1], on=['车号', 'CollectTime'], how='left')\n",
    "df_trn_time['Label'] = df_trn_time['Label'].fillna(0)\n",
    "print('Trn Label==1总数量:',df_trn_time.shape)\n",
    "print('Trn车if_off==-5 的数量:',df_trn_time[df_trn_time['Label']==1].shape)\n",
    "\n",
    "# Test一样\n",
    "df_testall2_time = df_testall2[df_testall2['if_off']==-5]\n",
    "df_test_time = pd.merge(df_testall2_time,time1_test, on=['车号', 'CollectTime'], how='inner')\n",
    "print('Test Label==1总数量:',df_test_time.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 特征工程\n",
    "由于碰撞时间受限于数据采集精度、传感器延迟、人为标注的影响，本身存在一定的随机误差，因此很难精确到1秒以内，\n",
    "\n",
    "这里因此仅构造了当前瞬时加速度，下一时刻瞬时加速度，追尾判断，主驾驶座占用状态4个特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:06.427300Z",
     "start_time": "2021-05-08T02:57:06.362300Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['车号', 'CollectTime', 'v_diff1', 'v_diff4', 'Label', '主驾驶座占用状态cate',\n",
      "       'a1'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "def col_feature3(df):\n",
    "    cate_cols = ['主驾驶座占用状态']\n",
    "    \n",
    "    # 类别特征编码\n",
    "    for col in cate_cols:\n",
    "        df[col+'cate'] = df[col].astype('category').cat.codes\n",
    "    \n",
    "    df['a1'] = df['v_diff1'].apply(lambda x: 1 if x>0 else 0)\n",
    "#     df['a2'] = df['v_diff4'].apply(lambda x: 1 if x>0 else 0)\n",
    "    \n",
    "    ori_cols = ['加速踏板位置', '电池包主负继电器状态', '电池包主正继电器状态', '制动踏板状态',\n",
    "       '驾驶员离开提示', '主驾驶座占用状态', '驾驶员安全带状态', '手刹状态', '整车钥匙状态', '低压蓄电池电压',\n",
    "       '整车当前档位状态', '整车当前总电流', '整车当前总电压','车速', '方向盘转角']\n",
    "    cate_cols = ['电池包主负继电器状态cate']\n",
    "    choose_cols = ['time_delta', 'time_delta_5','if_on','if_off','v_diff3','v_diff2', 'a_min5', 'a_mean5', 'a_max3']\n",
    "    df = df.drop(ori_cols+cate_cols+choose_cols,axis=1)\n",
    "\n",
    "    return df\n",
    "\n",
    "df_trn_time2 = col_feature3(df_trn_time)\n",
    "df_test_time2 = col_feature3(df_test_time)\n",
    "print(df_trn_time2.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 标准化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:06.556300Z",
     "start_time": "2021-05-08T02:57:06.432300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['v_diff1', 'v_diff4', '主驾驶座占用状态cate', 'a1'], dtype='object')\n",
      "(49, 4) (64, 4)\n"
     ]
    }
   ],
   "source": [
    "# 训练数据的特征\n",
    "f_names2 = [x for x in df_trn_time2.columns if x not in['车号','CollectTime','Label']]\n",
    "x_time_data = df_trn_time2[f_names2]\n",
    "y_time_data = df_trn_time2['Label']\n",
    "test_time_data = df_test_time2[f_names2]\n",
    "print(test_time_data.columns)\n",
    "std = StandardScaler()\n",
    "x_time_data = std.fit_transform(x_time_data)\n",
    "test_time_data = std.fit_transform(test_time_data)\n",
    "print(x_time_data.shape,test_time_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 五折交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:06.897300Z",
     "start_time": "2021-05-08T02:57:06.561300Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trn F1 [0.8857142857142858, 0.8823529411764706, 0.8695652173913044, 0.8695652173913044, 0.8732394366197184]\n",
      "val F1 [0.75, 0.888888888888889, 0.888888888888889, 0.888888888888889, 0.8750000000000001]\n",
      "trn_F1mean 0.8760874196586167\n",
      "val_F1mean 0.8583333333333334\n"
     ]
    }
   ],
   "source": [
    "pred_test_Kfold = pd.DataFrame()\n",
    "f1_trn,f1_val = [],[]\n",
    "fold = StratifiedKFold(n_splits=5,shuffle=True,random_state=0)\n",
    "\n",
    "for iteration ,(trn_idx,val_idx) in enumerate(fold.split(x_time_data,y_time_data)):\n",
    "#     print('------------{} fold -------------'.format(iteration))\n",
    "    x_trn_time,y_trn_time = x_time_data[trn_idx],y_time_data[trn_idx]\n",
    "    x_val_time,y_val_time = x_time_data[val_idx],y_time_data[val_idx]\n",
    "    \n",
    "    knn = KNeighborsClassifier(n_neighbors=5,p=1,metric='minkowski')\n",
    "    knn.fit(x_trn_time,y_trn_time)\n",
    "\n",
    "    # 预测\n",
    "    pred_trn = knn.predict(x_trn_time)\n",
    "    pred_val = knn.predict(x_val_time)\n",
    "    pred_test_Kfold['label'+str(iteration)] = knn.predict(test_time_data)\n",
    "\n",
    "    #得分\n",
    "    f1_trn.append(f1_score(pred_trn,y_trn_time))\n",
    "    f1_val.append(f1_score(pred_val,y_val_time))\n",
    "\n",
    "print('trn F1',f1_trn)\n",
    "print('val F1',f1_val)\n",
    "print('trn_F1mean',np.mean(f1_trn))\n",
    "print('val_F1mean',np.mean(f1_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:07.018300Z",
     "start_time": "2021-05-08T02:57:06.900300Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>车号</th>\n",
       "      <th>CollectTime</th>\n",
       "      <th>v_diff1</th>\n",
       "      <th>v_diff4</th>\n",
       "      <th>主驾驶座占用状态cate</th>\n",
       "      <th>a1</th>\n",
       "      <th>Time_Pred</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>132</td>\n",
       "      <td>2021-01-30 07:47:55</td>\n",
       "      <td>-15.2190</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>138</td>\n",
       "      <td>2020-11-26 19:07:39</td>\n",
       "      <td>-18.5310</td>\n",
       "      <td>-1.117500</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>168</td>\n",
       "      <td>2020-11-10 14:40:15</td>\n",
       "      <td>-14.1565</td>\n",
       "      <td>-10.703000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>207</td>\n",
       "      <td>2020-12-23 16:39:46</td>\n",
       "      <td>-14.6250</td>\n",
       "      <td>-7.078125</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     车号         CollectTime  v_diff1    v_diff4  主驾驶座占用状态cate  a1  Time_Pred\n",
       "1   132 2021-01-30 07:47:55 -15.2190   0.000000             1   0          0\n",
       "4   138 2020-11-26 19:07:39 -18.5310  -1.117500             1   0          0\n",
       "19  168 2020-11-10 14:40:15 -14.1565 -10.703000             1   0          0\n",
       "39  207 2020-12-23 16:39:46 -14.6250  -7.078125             0   0          0"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_test_Kfold['Time_Pred'] = pred_test_Kfold.mean(axis=1)\n",
    "pred_test_time = pred_test_Kfold['Time_Pred'].apply(lambda x: 1 if x>0.6 else 0)\n",
    "df_test_time2['Time_Pred'] = pred_test_time\n",
    "df_test_time2[df_test_time2['Time_Pred']==0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 结果提交\n",
    "- 代码思路：将预测标签中`Time_Pred=0`的数据找到，修正为`if_off==-4`的标签，作为最终结果提交"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:07.061300Z",
     "start_time": "2021-05-08T02:57:07.023300Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(64, 30)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 找到if_off==-4对应time \n",
    "df_testall2_time_if_off4 = df_testall2[df_testall2['if_off']==-4]\n",
    "df_test_time_if_off4 = pd.merge(df_testall2_time_if_off4,time1_test, on=['车号', 'CollectTime'], how='inner')\n",
    "df_test_time_if_off4['Time_Pred'] = pred_test_time\n",
    "df_test_time_if_off4.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-08T02:57:07.179300Z",
     "start_time": "2021-05-08T02:57:07.064300Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label1数量: (64, 3)\n",
      "-------------------end-------------------\n"
     ]
    }
   ],
   "source": [
    "Time1_test_sub = df_test_time2.loc[df_test_time2['Time_Pred']==1,['车号','CollectTime']]\n",
    "Time2_test_sub = df_test_time_if_off4.loc[df_test_time_if_off4['Time_Pred']==0,['车号','CollectTime']]\n",
    "Time_test_sub = pd.concat([Time1_test_sub,Time2_test_sub])\n",
    "\n",
    "# 提交最终结果\n",
    "sub_all_time = pd.merge(sub_all[['车号','Label']],Time_test_sub,on=['车号'], how='left')\n",
    "sub_all_time.to_csv(result_path+'/result.csv',index=False)\n",
    "print('Label1数量:',sub_all_time[sub_all_time['Label']==1].shape)\n",
    "print('-------------------end-------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](./pictures/8.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "307.188px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}