my_transformer_try/text summeray with ALBERT GOOGLE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1qfQAtRsMVl7"
   },
   "source": [
    "# Text summarization with ALBERT\n",
    "\n",
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sBBHbGvQN5vX"
   },
   "source": [
    "## 1.0 Setup\n",
    "\n",
    "Let's check out what kind of GPU our friends at Google gave us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "frTeTcy4WdbY"
   },
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D5RImM3oWbrZ"
   },
   "source": [
    "First, we clone the Hugging Face transformer library from Github.\n",
    "\n",
    "\n",
    "Note it's checking out a specific commit only because I've tested this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QOAoUwBFMQCg"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/google-research/albert.git\n",
    "!pip install -r albert/requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TRZned-8WJrj"
   },
   "outputs": [],
   "source": [
    "!pip install tensorboardX\n",
    "!pip install tensorflow-datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "UHCuzhPptH0M"
   },
   "source": [
    "## 2.0 Train Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OaQGsAiWXcnd"
   },
   "source": [
    "### 2.1 Get Training and Evaluation Data\n",
    "get CNN Daily Mail Datatest from the tensorflow dataset library\n",
    "https://www.tensorflow.org/datasets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dI6e-PfOXSnO"
   },
   "outputs": [],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "cnn_dailymail = tfds.load(name=\"cnn_dailymail\", data_dir=\"dataset\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "r8irmHEPQlgL"
   },
   "outputs": [],
   "source": [
    "cnn_dailymail['train'].size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tN9_VrsiDDPM"
   },
   "outputs": [],
   "source": [
    "# https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/evaluation/metrics.py\n",
    "\n",
    "def rouge(targets, predictions, score_keys=None):\n",
    "  \"\"\"Computes rouge score.\n",
    "  Args:\n",
    "    targets: list of strings\n",
    "    predictions: list of strings\n",
    "    score_keys: list of strings with the keys to compute.\n",
    "  Returns:\n",
    "    dict with score_key: rouge score across all targets and predictions\n",
    "  \"\"\"\n",
    "\n",
    "  if score_keys is None:\n",
    "    score_keys = [\"rouge1\", \"rouge2\", \"rougeLsum\"]\n",
    "  scorer = rouge_scorer.RougeScorer(score_keys)\n",
    "  aggregator = scoring.BootstrapAggregator()\n",
    "\n",
    "  def _prepare_summary(summary):\n",
    "    # Make sure the summary is not bytes-type\n",
    "    # Add newlines between sentences so that rougeLsum is computed correctly.\n",
    "    summary = summary.replace(\" . \", \" .\\n\")\n",
    "    return summary\n",
    "\n",
    "  for prediction, target in zip(predictions, targets):\n",
    "    target = _prepare_summary(target)\n",
    "    prediction = _prepare_summary(prediction)\n",
    "    aggregator.add_scores(scorer.score(target=target, prediction=prediction))\n",
    "  result = aggregator.aggregate()\n",
    "  for key in score_keys:\n",
    "    logging.info(\n",
    "        \"%s = %.2f, 95%% confidence [%.2f, %.2f]\",\n",
    "        key,\n",
    "        result[key].mid.fmeasure*100,\n",
    "        result[key].low.fmeasure*100,\n",
    "        result[key].high.fmeasure*100,\n",
    "    )\n",
    "  return {key: result[key].mid.fmeasure*100 for key in score_keys}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tpdfZdgVEPbN"
   },
   "outputs": [],
   "source": [
    "ALBERT_MODEL = 'base' #@param {type:\"string\"}\n",
    "OUTPUT_DIR = 'out_albert' #@param {type:\"string\"}\n",
    "ALBERT_MODEL_HUB = 'https://tfhub.dev/google/albert_' + ALBERT_MODEL + '/3'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dZ87q93GDeeL"
   },
   "source": [
    "### 2.2 Run training \n",
    "\n",
    "We can now train the model with the training set. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-Eg53t3QXZAb"
   },
   "outputs": [],
   "source": [
    "!python -m albert.run_classifier \\\n",
    "  --data_dir=\"dataset/cnn_dailymail\" \\\n",
    "  --output_dir=$OUTPUT_DIR \\\n",
    "  --albert_hub_module_handle=$ALBERT_MODEL_HUB \\\n",
    "  --spm_model_file=\"from_tf_hub\" \\\n",
    "  --do_train \\\n",
    "  --do_eval \\\n",
    "  --do_predict \\\n",
    "  --do_lower_case \\\n",
    "  --max_seq_length=512 \\\n",
    "  --optimizer=adamw \\\n",
    "  --task_name=MNLI \\\n",
    "  --warmup_step=1000 \\\n",
    "  --learning_rate=3e-5 \\\n",
    "  --train_step=10000 \\\n",
    "  --save_checkpoints_steps=100 \\\n",
    "  --train_batch_size=32"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "text summeray with ALBERT.ipynb",
   "private_outputs": true,
   "provenance": [
    {
     "file_id": "19nO__3UsYvOsfNu53V4Ajropr_GbjYre",
     "timestamp": 1585595980349
    },
    {
     "file_id": "https://github.com/spark-ming/albert-qa-demo/blob/master/Question_Answering_with_ALBERT.ipynb",
     "timestamp": 1585002674672
    }
   ],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}