00 - Downloading&Metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8b062ed1-7de2-4488-b3e6-2df10a1aee82",
   "metadata": {},
   "source": [
    "This part of the pipeline just derives some metadata files from exports and manually assembled metadata from NCBI RefSeq. We assume you've downloaded your genomes from NCBI as well. Otherwise, just make sure you follow the same file structure as our metadata file.\n",
    "\n",
    "There are multiple ways to get your genomics data from NCBI. For the genomics data, we recommend to use the NCBI Datasets CLI with a taxon ID for example. Important is to put all genome fastas (file extension `.fna`) in the folder `00-refseq/data`.\n",
    "\n",
    "For the metadata, it's the easiest to go via the NCBI website portal, submit a query to Assembly and copy-paste/download the metadata.\n",
    "\n",
    "Crucial is to make a spreadsheet based on the metadata from RefSeq with a header with at least the following columns in this order:\n",
    "\n",
    "`Species`, `Genome_accession`, `Taxonomic_cluster`, `Assembly_level`, `Size (bp)`, `No. genes`, `Completeness (%)`, `Contamination (%)`, `Status`, `Failed_checkM`, `Failed_Mash`, `Failed_QC`\n",
    "\n",
    "Afterwards, export the spreadsheet to a tab-separated text file named `genomes_metadata`, and leave this file in at the root of the pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "401bec19-a9bb-4f9f-a5be-0889cf14014d",
   "metadata": {},
   "source": [
    "### Paths and parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76a2613a-b70d-448a-81d6-18a85c6a3d7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_root=\"./00-refseq\"\n",
    "data=\"$task_root/data\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9df76335-521a-43ac-aee6-fdfbc6a87643",
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir -p $task_root $data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1399413f-3442-4a22-8273-6b01eb2a0451",
   "metadata": {},
   "source": [
    "### Setting up other metadata derivative files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86a03214-8854-47f1-8466-001752a1195f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cut -f -2 genomes_metadata > names_by_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff24310e-2586-46e7-97e6-489d0ed9980a",
   "metadata": {},
   "outputs": [],
   "source": [
    "cat genomes_metadata | awk -F '\\t' '{if ($10 == \"X\") {print $2}}' > $task_root/failed_checkm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4547a1a-d5c2-425e-b922-538f0c880a9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "wc -l genomes_metadata $task_root/failed_checkm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d7521bf-2279-4af4-8ef2-8ddccbee1b81",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}