cwpk-32-iterating-full-extractions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CWPK \\#32: Iterating Over a Full Extraction\n",
    "=======================================\n",
    "\n",
    "It is Time to Explore Python Dictionaries and Packaging\n",
    "--------------------------\n",
    "\n",
    "<div style=\"float: left; width: 305px; margin-right: 10px;\">\n",
    "\n",
    "<img src=\"http://kbpedia.org/cwpk-files/cooking-with-kbpedia-305.png\" title=\"Cooking with KBpedia\" width=\"305\" />\n",
    "\n",
    "</div>\n",
    "\n",
    "In our last coding installments in this [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) series, prior to our single-installment detour to learn about files, we developed extraction routines for both structure (<code>rdfs:subClassOf</code>) and annotations (of various properties) using the fantastic package [owlready2](https://owlready2.readthedocs.io/en/latest/intro.html). In practice, these generic routines will loop over populations of certain object types in [KBpedia](https://kbpedia.org/), such as typologies or property types. We want a way to feed these variations to the generic routines in an efficient and understandable way.\n",
    "\n",
    "[Python](https://en.wikipedia.org/wiki/Python_(programming_language)) lists are one way to do so, and we have begun to gain a bit of experience in our prior work with lists and sets. But there is another structure in Python called a 'dictionary' that sets up [key-value pairs](https://en.wikipedia.org/wiki/Attribute%E2%80%93value_pair) of 2-[tuples](https://en.wikipedia.org/wiki/Tuple) that promises more flexibility and power. The 2-tuple sets up a relationship between an attribute name (a variable name) with a value, quite similar to the [associative arrays](https://en.wikipedia.org/wiki/Associative_array) in [JSON](https://en.wikipedia.org/wiki/JSON). The values in a dictionary can be any object in Python, including functions or other dictionaries, the latter which allows 'record'-like data structures. However, there may not be duplicate names for keys within a given dictionary (but names may be used again in other dictionaries without global reference).\n",
    "\n",
    "Dictionaries (<code>'dicts'</code>) are like Python lists except list elements are accessed by their position in the list using a numeric index, while we access <code>dict</code> elements via keys. This makes tracing the code easier. We have also indicated that dictionary structures may be forthcoming in other uses of KBpedia, such as [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) or [master data](https://en.wikipedia.org/wiki/Master_data). So, I decided to start gaining experience with <code>'dicts'</code> in this installment.\n",
    "\n",
    "(Other apparent advantages of dictionaries not directly related to our immediate needs include:\n",
    "\n",
    "- Dictionaries can be expanded without altering what is already there\n",
    "- From Python 3.7 onward, the order entered into a dict is preserved in loops\n",
    "- Dictionaries can handle extremely large data sets\n",
    "- <code>Dicts</code> are fast because they are implemented as a [hash table](https://en.wikipedia.org/wiki/Hash_table), and \n",
    "- They can be directly related to a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) should we go that route.)\n",
    "\n",
    "We can inspect this method with our standard statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dir(dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Basic Iteration Approach\n",
    "\n",
    "In installments [**CWPK #28**](https://www.mkbergman.com/2363/cwpk-28-extracting-structure-for-typologies/), [**CWPK #29**](https://www.mkbergman.com/2364/cwpk-29-extracting-object-and-data-properties/), and [**CWPK #30**](https://www.mkbergman.com/2365/cwpk-30-extracting-annotations/), we created generic prototype routines for extracting structure from typologies and properties and then annotations from classes (including typologies as a subset) and properties as well. We thus have generic extraction routines for:\n",
    "\n",
    "|  Structure                     | Annotations            |\n",
    "|:---------                      |:-----------            |\n",
    "| classes                        | classes                |\n",
    "| typologies &nbsp;&nbsp;&nbsp;  | typologies (possible)  |\n",
    "| properties                     | properties             |\n",
    "\n",
    "Our basic iteration approach, then, is to define dictionaries for the root objects in these categories and loop over them invoking these generic routines. In the process we want to write out results for each iteration, provide some progress messages, and then complete the looping elements for each root object. Labels and internal lookups to the namespace objects come from the dictionary. In generic terms, then, here is how we want these methods to be structured:\n",
    "\n",
    "- Initialize method\n",
    "- Message: starting method\n",
    "- Get <code>dict</code> iterator:\n",
    "  - Message: iterating current element\n",
    "  - Get owlready2 set iterator for element:\n",
    "    - Populate row\n",
    "    - Print to file\n",
    "- Return to prompt without error message.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Starting and Load\n",
    "To demonstrate this progression, we begin with our standard opening routine:\n",
    "\n",
    "<div style=\"background-color:#eee; border:1px dotted #aaa; vertical-align:middle; margin:15px 60px; padding:8px;\"><strong>Which environment?</strong> The specific load routine you should choose below depends on whether you are using the online MyBinder service (the 'raw' version) or local files. The example below is based on using local files (though replace with your own local directory specification). If loading from MyBinder, replace with the lines that are commented (<code>#</code>) out.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kbpedia = 'C:/1-PythonProjects/kbpedia/sandbox/kbpedia_reference_concepts.owl'\n",
    "# kbpedia = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kbpedia_reference_concepts.owl'\n",
    "skos_file = 'http://www.w3.org/2004/02/skos/core' \n",
    "kko_file = 'C:/1-PythonProjects/kbpedia/sandbox/kko.owl'\n",
    "# kko_file = 'https://raw.githubusercontent.com/Cognonto/CWPK/master/sandbox/builds/ontologies/kko.owl'\n",
    "\n",
    "from owlready2 import *\n",
    "world = World()\n",
    "kb = world.get_ontology(kbpedia).load()\n",
    "rc = kb.get_namespace('http://kbpedia.org/kko/rc/')               \n",
    "\n",
    "skos = world.get_ontology(skos_file).load()\n",
    "kb.imported_ontologies.append(skos)\n",
    "core = world.get_namespace('http://www.w3.org/2004/02/skos/core#')\n",
    "\n",
    "kko = world.get_ontology(kko_file).load()\n",
    "kb.imported_ontologies.append(kko)\n",
    "kko = kb.get_namespace('http://kbpedia.org/ontologies/kko#')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like always, we execute each cell as we progress down this notebook page by pressing <code>shift+enter</code> for the highlighted cell or by choosing Run from the notebook menu.\n",
    "\n",
    "### Creating the Dictionaries\n",
    "We will now create dictionaries for typologies and properties. We will construct them using our standard internal name as the 'key' for each element, with the value being the internal reference including the namespace prefix (easier than always concatenating using strings). I'll first begin with the smaller properties dictionary and explain the sytax afterwards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prop_dict = {\n",
    "        'objectProperties'    : 'kko.predicateProperties',\n",
    "        'dataProperties'      : 'kko.predicateDataProperties',\n",
    "        'annotationProperties': 'kko.representations',\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dictionary is declared either with the curly brackets (<code>{ }</code>) with the colon separator for key:value, or by using the <code>d = dict([(&lt;key&gt;, &lt;value&gt;)])</code> form. The 'key' field is normally quoted, except where the variable is globally defined. The 'value' field in this instance is the internal owlready2 notation of &lt;namespace&gt; + &lt;class&gt;. There is no need to align the colons except to enhance readability.\n",
    "\n",
    "Our longer listing is the typology one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "typol_dict = {\n",
    "        'ActionTypes'           : 'kko.ActionTypes',\n",
    "        'AdjunctualAttributes'  : 'kko.AdjunctualAttributes',\n",
    "        'Agents'                : 'kko.Agents',\n",
    "        'Animals'               : 'kko.Animals',\n",
    "        'AreaRegion'            : 'kko.AreaRegion',\n",
    "        'Artifacts'             : 'kko.Artifacts',\n",
    "        'Associatives'          : 'kko.Associatives',\n",
    "        'AtomsElements'         : 'kko.AtomsElements',\n",
    "        'AttributeTypes'        : 'kko.AttributeTypes',\n",
    "        'AudioInfo'             : 'kko.AudioInfo',\n",
    "        'AVInfo'                : 'kko.AVInfo',\n",
    "        'BiologicalProcesses'   : 'kko.BiologicalProcesses',\n",
    "        'Chemistry'             : 'kko.Chemistry',\n",
    "        'Concepts'              : 'kko.Concepts',\n",
    "        'ConceptualSystems'     : 'kko.ConceptualSystems',\n",
    "        'Constituents'          : 'kko.Constituents',\n",
    "        'ContextualAttributes'  : 'kko.ContextualAttributes',\n",
    "        'CopulativeRelations'   : 'kko.CopulativeRelations',\n",
    "        'Denotatives'           : 'kko.Denotatives',\n",
    "        'DirectRelations'       : 'kko.DirectRelations',\n",
    "        'Diseases'              : 'kko.Diseases',\n",
    "        'Drugs'                 : 'kko.Drugs',\n",
    "        'EconomicSystems'       : 'kko.EconomicSystems',\n",
    "        'EmergentKnowledge'     : 'kko.EmergentKnowledge',\n",
    "        'Eukaryotes'            : 'kko.Eukaryotes',\n",
    "        'EventTypes'            : 'kko.EventTypes',\n",
    "        'Facilities'            : 'kko.Facilities',\n",
    "        'FoodDrink'             : 'kko.FoodDrink',\n",
    "        'Forms'                 : 'kko.Forms',\n",
    "        'Generals'              : 'kko.Generals',\n",
    "        'Geopolitical'          : 'kko.Geopolitical',\n",
    "        'Indexes'               : 'kko.Indexes',\n",
    "        'Information'           : 'kko.Information',\n",
    "        'InquiryMethods'        : 'kko.InquiryMethods',\n",
    "        'IntrinsicAttributes'   : 'kko.IntrinsicAttributes',\n",
    "        'KnowledgeDomains'      : 'kko.KnowledgeDomains',\n",
    "        'LearningProcesses'     : 'kko.LearningProcesses',\n",
    "        'LivingThings'          : 'kko.LivingThings',\n",
    "        'LocationPlace'         : 'kko.LocationPlace',\n",
    "        'Manifestations'        : 'kko.Manifestations',\n",
    "        'MediativeRelations'    : 'kko.MediativeRelations',\n",
    "        'Methodeutic'           : 'kko.Methodeutic',\n",
    "        'NaturalMatter'         : 'kko.NaturalMatter',\n",
    "        'NaturalPhenomena'      : 'kko.NaturalPhenomena',\n",
    "        'NaturalSubstances'     : 'kko.NaturalSubstances',\n",
    "        'OrganicChemistry'      : 'kko.OrganicChemistry',\n",
    "        'OrganicMatter'         : 'kko.OrganicMatter',\n",
    "        'Organizations'         : 'kko.Organizations',\n",
    "        'Persons'               : 'kko.Persons',\n",
    "        'Places'                : 'kko.Places',\n",
    "        'Plants'                : 'kko.Plants',\n",
    "        'Predications'          : 'kko.Predications',\n",
    "        'PrimarySectorProduct'  : 'kko.PrimarySectorProduct',\n",
    "        'Products'              : 'kko.Products',\n",
    "        'Prokaryotes'           : 'kko.Prokaryotes',\n",
    "        'ProtistsFungus'        : 'kko.ProtistsFungus',\n",
    "        'RelationTypes'         : 'kko.RelationTypes',\n",
    "        'RepresentationTypes'   : 'kko.RepresentationTypes',\n",
    "        'SecondarySectorProduct': 'kko.SecondarySectorProduct',\n",
    "        'Shapes'                : 'kko.Shapes',\n",
    "        'SituationTypes'        : 'kko.SituationTypes',\n",
    "        'SocialSystems'         : 'kko.SocialSystems',\n",
    "        'Society'               : 'kko.Society',\n",
    "        'SpaceTypes'            : 'kko.SpaceTypes',\n",
    "        'StructuredInfo'        : 'kko.StructuredInfo',\n",
    "        'Symbolic'              : 'kko.Symbolic',\n",
    "        'Systems'               : 'kko.Systems',\n",
    "        'TertiarySectorService' : 'kko.TertiarySectorService',\n",
    "        'Times'                 : 'kko.Times',\n",
    "        'TimeTypes'             : 'kko.TimeTypes',\n",
    "        'TopicsCategories'      : 'kko.TopicsCategories',\n",
    "        'VisualInfo'            : 'kko.VisualInfo',\n",
    "        'WrittenInfo'           : 'kko.WrittenInfo'\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a listing of entries in a dictionary, simply reference its name and run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prop_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a variety of methods for nesting or merging dictionaries. We do not have need at present for that, but one example shows how we can create a new dictionary, relate it to an existing one, and then update (or merge) another dictionary with it, using the two dictionaries from above as examples: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "total_dict = dict(typol_dict)\n",
    "total_dict.update(prop_dict)\n",
    "print(total_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This now gives us a merged dictionary. However, whether keys match or vary in number means specific cases need to be evaluated individually. The <code>.update</code> may not always be an appropriate approach.\n",
    "\n",
    "In these <code>dicts</code>, we now have the population of items (sets) from which we want to obtain all of their members and get the individual extractions. We also have them organized into dictionaries that we can iterate over to complete a full extraction from KBpedia.\n",
    "\n",
    "### Marrying Iterators and Routines\n",
    "We can now return to our generic extraction prototypes and enhance them a bit to loop over these iterators. Let's take the structure extraction of <code>rdfs:subPropertyOf</code> from **CWPK #29** to extract out structural aspects of our properties. I will keep the form from the earlier installment and comment all lines of code added to accommodate the iterations loops and message feedback. First we will add the iterator: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for value in prop_dict.values():      # iterates over dictionary 'values' with each occurence a 'value'\n",
    "  root = eval(value)                  # need to convert value 'string' to internal variable\n",
    "  p_set=root.descendants()\n",
    "\n",
    "#  o_frag = set()                     # left over from prior development; commented out\n",
    "#  s_frag = set()                     # left over from prior development; commented out\n",
    "  p_item = 'rdfs:subPropertyOf'\n",
    "  for s_item in p_set:\n",
    "    o_set = s_item.is_a\n",
    "    for o_item in o_set:\n",
    "       print(s_item,',',p_item,',',o_item,'.','\\n', sep='', end='')\n",
    "#       o_frag.add(o_item)            # left over from prior development; commented out\n",
    "#    s_frag.add(s_item)               # left over from prior development; commented out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could do a <code>len()</code> to test output lines or make other tests to ensure you are iterating over the property groupings.\n",
    "\n",
    "The <code>eval()</code> function submits the string represented by <code>value</code> to the resident Python code base and in this case returns the owlready2 property object, which then allows proper processing of the <code>.descendants()</code> code. My understanding is that in open settings <code>eval()</code> can pose some security holes. I think it is OK in our case since we are doing local or internal processing, and not exposing this as a public method.\n",
    "\n",
    "We'll continue with this code block, but now print to file and remove the commented lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_file = 'C:/1-PythonProjects/kbpedia/sandbox/prop_struct_out.csv'                 # variable to physical file\n",
    "with open(out_file, mode='w', encoding='utf8') as out_put:                        # std file declaration (CWPK #31)\n",
    "  for value in prop_dict.values():      \n",
    "    root = eval(value)                  \n",
    "    p_set=root.descendants()\n",
    "    p_item = 'rdfs:subPropertyOf'\n",
    "    for s_item in p_set:\n",
    "      o_set = s_item.is_a\n",
    "      for o_item in o_set:\n",
    "        print(s_item,',',p_item,',',o_item,'.','\\n', sep='', end='', file=out_put) # add output file here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, then, we'll add some messages to the screen to see output as it whizzes by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Beginning property structure extraction . . .\n",
      "   . . . processing kko.predicateProperties\n",
      "   . . . processing kko.predicateDataProperties\n",
      "   . . . processing kko.representations\n"
     ]
    }
   ],
   "source": [
    "print('Beginning property structure extraction . . .')                            # print message\n",
    "out_file = 'C:/1-PythonProjects/kbpedia/sandbox/prop_struct_out.csv'\n",
    "with open(out_file, mode='w', encoding='utf8') as out_put:\n",
    "  for value in prop_dict.values():\n",
    "    print('   . . . processing', value)                                           # loop print message\n",
    "    root = eval(value)                  \n",
    "    p_set=root.descendants()\n",
    "    p_item = 'rdfs:subPropertyOf'\n",
    "    for s_item in p_set:\n",
    "      o_set = s_item.is_a\n",
    "      for o_item in o_set:\n",
    "        print(s_item,',',p_item,',',o_item,'.','\\n', sep='', end='', file=out_put)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, so this looks to be a complete routine as we desire. However, we are starting to accumulate a fair number of lines in our routines, and we need additional routines very similar to what is above for extracting classes, typologies and annotations. \n",
    "\n",
    "It is time to bring a bit more formality to our code writing and management, which I address in the next installment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Documentation\n",
    "\n",
    "Here is additional documentation related to today's **CWPK** installment:\n",
    "\n",
    "- [dict](https://python-reference.readthedocs.io/en/latest/docs/dict/) reference\n",
    "- RealPython's entry on [dictionaries](https://realpython.com/python-dicts/)\n",
    "- [Using Python dictionary as a database](https://developer.rhino3d.com/guides/rhinopython/python-dictionary-database/)\n",
    "- RealPython's [how to interate dictionaries](https://realpython.com/iterate-through-dictionary-python/).\n",
    "\n",
    "\n",
    " <div style=\"background-color:#efefff; border:1px dotted #ceceff; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "  <span style=\"font-weight: bold;\">NOTE:</span> This article is part of the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\" style=\"font-style: italic;\">Cooking with Python and KBpedia</a> series. See the <a href=\"https://www.mkbergman.com/cooking-with-python-and-kbpedia/\"><strong>CWPK</strong> listing</a> for other articles in the series. <a href=\"http://kbpedia.org/\">KBpedia</a> has its own Web site.\n",
    "  </div>\n",
    "\n",
    "<div style=\"background-color:#ebf8e2; border:1px dotted #71c837; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "\n",
    "<span style=\"font-weight: bold;\">NOTE:</span> This <strong>CWPK \n",
    "installment</strong> is available both as an online interactive\n",
    "file <a href=\"https://mybinder.org/v2/gh/Cognonto/CWPK/master\" ><img src=\"https://mybinder.org/badge_logo.svg\" style=\"display:inline-block; vertical-align: middle;\" /></a> or as a <a href=\"https://github.com/Cognonto/CWPK\" title=\"CWPK notebook\" alt=\"CWPK notebook\">direct download</a> to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the <code>*.ipynb</code> file. It may take a bit of time for the interactive option to load.</div>\n",
    "\n",
    "<div style=\"background-color:#feeedc; border:1px dotted #f7941d; vertical-align:middle; margin:15px 60px; padding:8px;\"> \n",
    "<div style=\"float: left; margin-right: 5px;\"><img src=\"http://kbpedia.org/cwpk-files/warning.png\" title=\"Caution!\" width=\"32\" /></div>I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment -- which is part of the fun of Python -- and to <a href=\"mailto:mike@mkbergman.com\">notify me</a> should you make improvements.    \n",
    "\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}