\n",
"\n"
],
"text/plain": [
"\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"adata = ep.io.read_csv(\"medcat_data/pt_notes.csv\", columns_obs_only=[\"text\"])\n",
"adata_encoded = ep.pp.encode(adata, autodetect=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Prepare CAT object as per MedCAT workflow\n",
"\n",
"To leverage MedCAT for free text processing in ehrapy, we start by creating a `CAT` object. This is described in more details in for example the [MedCAT tutorials](https://medcat.readthedocs.io/en/latest/main.html#tutorials).\n",
"\n",
"Here, we simply load a pretrained model from the maintainers of MedCAT which can be downloaded & readily used.\n",
"For specific usecases, using more sophisticated refinements might be required. For this tutorial and a quick start, we will use this out-of-the-box model.\n",
"\n",
"We also set some TUI filters.\n",
"A full list of TUI’s can be found at: https://lhncbc.nlm.nih.gov/ii/tools/MetaMap/Docs/SemanticTypes_2018AB.txt"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"# create the main ehrapy medcat object used for medical entitiy analysis using ehrapy and medcat\n",
"cat = CAT.load_model_pack(\"./medcat_data/medmen_wstatus_2021_oct.zip\")\n",
"\n",
"# only use diseases and behavioural disorders diagnoses in this example by filtering by TUI\n",
"tuis=[\"T047\", \"T048\"]\n",
"cui_filters = set()\n",
"for type_id in tuis:\n",
" cui_filters.update(cat.cdb.addl_info[\"type_id2cuis\"][type_id])\n",
"cat.cdb.config.linking[\"filters\"][\"cuis\"] = cui_filters\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using this model pack we can already extract entities from an example note."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(kidney failure,)"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"text = \"He was diagnosed with kidney failure\"\n",
"doc = cat(text)\n",
"doc.ents"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'entities': {2: {'pretty_name': 'Kidney Failure',\n",
" 'cui': 'C0035078',\n",
" 'type_ids': ['T047'],\n",
" 'types': ['Disease or Syndrome'],\n",
" 'source_value': 'kidney failure',\n",
" 'detected_name': 'kidney~failure',\n",
" 'acc': 1.0,\n",
" 'context_similarity': 1.0,\n",
" 'start': 22,\n",
" 'end': 36,\n",
" 'icd10': [],\n",
" 'ontologies': [],\n",
" 'snomed': [],\n",
" 'id': 2,\n",
" 'meta_anns': {'Status': {'value': 'Affirmed',\n",
" 'confidence': 0.9999961853027344,\n",
" 'name': 'Status'}}}},\n",
" 'tokens': []}"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Example output of an extracted medcat entity; note that ehrapy will deal with this automatically and the here displayed manual extraction is not required.\n",
"# CUI: Concept Unique Identifier, which is just an unique identifier for each concept extracted\n",
"cat.get_entities(\"He was diagnosed with kidney failure\", only_cui=False)"
]
},
{
"cell_type": "markdown",
"metadata": {
"tags": []
},
"source": [
"## Extracting and visualizing all disease entities"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To extract all disease entities from our example dataset we require a complete annotation of the dataset. This step is computationally expensive and may take some time."
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [],
"source": [
"ep.tl.annotate_text(adata_encoded, cat, text_column=\"text\", n_proc=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The annotated results as extracted by MedCAT are transformed and stored into a Pandas DataFrame in `adata.uns`."
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | row_nr | \n", "pretty_name | \n", "cui | \n", "type_ids | \n", "types | \n", "meta_anns | \n", "
|---|---|---|---|---|---|---|
| 0 | \n", "0 | \n", "Diabetes | \n", "C0011847 | \n", "[T047] | \n", "[Disease or Syndrome] | \n", "Affirmed | \n", "
| 1 | \n", "0 | \n", "Sepsis | \n", "C0243026 | \n", "[T047] | \n", "[Disease or Syndrome] | \n", "Other | \n", "
| 2 | \n", "0 | \n", "Respiratory Distress Syndrome, Adult | \n", "C0035222 | \n", "[T047] | \n", "[Disease or Syndrome] | \n", "Affirmed | \n", "
| 3 | \n", "0 | \n", "Pulmonary Embolism | \n", "C0034065 | \n", "[T047] | \n", "[Disease or Syndrome] | \n", "Other | \n", "
| 4 | \n", "0 | \n", "Respiratory Failure | \n", "C1145670 | \n", "[T047] | \n", "[Disease or Syndrome] | \n", "Affirmed | \n", "
| \n", " | pretty_name | \n", "type_ids | \n", "types | \n", "n_patient_visit | \n", "n_patient_visit_percent | \n", "
|---|---|---|---|---|---|
| cui | \n", "\n", " | \n", " | \n", " | \n", " | \n", " |
| C0020538 | \n", "Hypertensive disease | \n", "T047 | \n", "Disease or Syndrome | \n", "432 | \n", "42.394504 | \n", "
| C0028754 | \n", "Obesity | \n", "T047 | \n", "Disease or Syndrome | \n", "245 | \n", "24.043180 | \n", "
| C0011847 | \n", "Diabetes | \n", "T047 | \n", "Disease or Syndrome | \n", "149 | \n", "14.622179 | \n", "
| C0010054 | \n", "Coronary Arteriosclerosis | \n", "T047 | \n", "Disease or Syndrome | \n", "127 | \n", "12.463199 | \n", "
| C0011849 | \n", "Diabetes Mellitus | \n", "T047 | \n", "Disease or Syndrome | \n", "118 | \n", "11.579980 | \n", "
| C0012634 | \n", "Disease | \n", "T047 | \n", "Disease or Syndrome | \n", "110 | \n", "10.794897 | \n", "
| C0038454 | \n", "Cerebrovascular accident | \n", "T047 | \n", "Disease or Syndrome | \n", "110 | \n", "10.794897 | \n", "
| C0004096 | \n", "Asthma | \n", "T047 | \n", "Disease or Syndrome | \n", "97 | \n", "9.519136 | \n", "
| C0024117 | \n", "Chronic Obstructive Airway Disease | \n", "T047 | \n", "Disease or Syndrome | \n", "92 | \n", "9.028459 | \n", "
| C0011570 | \n", "Mental Depression | \n", "T048 | \n", "Mental or Behavioral Dysfunction | \n", "91 | \n", "8.930324 | \n", "
| \n", " | text | \n", "chartdate | \n", "dob | \n", "category | \n", "gender | \n", "Diabetes | \n", "Congestive heart failure | \n", "
|---|---|---|---|---|---|---|---|
| 0 | \n", "HISTORY OF PRESENT ILLNESS:, The patient is a ... | \n", "2079-01-01 | \n", "2018-01-01 | \n", "General Medicine | \n", "F | \n", "True | \n", "False | \n", "
| 1 | \n", "HISTORY OF PRESENT ILLNESS: , A 71-year-old fe... | \n", "2079-01-01 | \n", "2018-01-01 | \n", "Rheumatology | \n", "F | \n", "False | \n", "False | \n", "
| 2 | \n", "HISTORY OF PRESENT ILLNESS:, The patient is a ... | \n", "2079-01-01 | \n", "2018-01-01 | \n", "Consult - History and Phy. | \n", "F | \n", "True | \n", "False | \n", "
| 3 | \n", "CHIEF COMPLAINT:,1. Infection.,2. Pelvic pai... | \n", "2037-01-01 | \n", "2018-01-01 | \n", "Consult - History and Phy. | \n", "F | \n", "False | \n", "False | \n", "
| 4 | \n", "SUBJECTIVE:, This is a 29-year-old Vietnamese... | \n", "2037-01-01 | \n", "2018-01-01 | \n", "Dermatology | \n", "F | \n", "False | \n", "False | \n", "