#!/usr/bin/env python
# coding: utf-8
# # Introduction
# Udapi is an API and framework for processing [Universal Dependencies](http://universaldependencies.org/). In this tutorial, we will focus on the Python version of Udapi. Perl and Java versions are [available](http://udapi.github.io/) as well, but they are missing some of the features.
#
# Udapi can be used from the shell (e.g. Bash), using the wrapper script `udapy`. It can be also used as a library, from Python, IPython or Jupyter notebooks. We will show both of these ways bellow.
#
# This tutorial uses Details sections for extra info (if you want to know more or if you run into problems). You need to click on it to show its content.
# Details
# It is a substitute for footnotes. The content may be long and showing it in the main text may be distracting.
# Details
#
# - The command below installs Udapi from GitHub (from the master branch). With
pip3 install --user --upgrade udapi, you can install the last version released on PyPI (possibly older).
# - The exclamation mark (!) in Jupyter or IPython means that the following command will be executed by the system shell (e.g. Bash).
#
# Details: If the previous command fails with "udapy: command not found"
# This means that Udapi is not properly installed. When installing Udapi with pip3 --user, it is installed into ~/.local/lib/python3.6/site-packages/udapi/ (or similar depending on your Python version) and the wrapper into ~/.local/bin. Thus you need to
#
# export PATH="$HOME/.local/bin/:$PATH"
#
# Details: executing from Bash, IPython, Jupyter
#
# - If you see "No such file or directory" error, make sure you executed the previous cell. Note that the
cd command is not prefixed by an exclamation mark because that would run in a sub-shell, which "forgets" the changed directory when finished. It is prefixed by a percent sign, which marks it as IPython magic.
# cat is another IPython magic command, this time an alias for the shell command of the same name (so you can prefix cat with an exclamation mark, if you prefer), which prints a given file. With automagic on, you can use it without the percent sign.
# - In this tutorial, we use
| head to show just the first 10 lines of the output (preventing thus big ipynb file size). You can ignore the "cat: write error: Broken pipe" warning.
# - When using Jupyter, you can omit the
| head because long outputs are automatically wrapped in a text box with a scrollbar.
# - When running this from IPython or Bash, you can use a pager:
less UD_Ancient_Greek/sample.conllu
#
# Details:
#
# - You may be used to see dependency trees where the root node is on the top and words are ordered horizontally (left to right). Here, the root is on left and words are ordered vertically (top to bottom).
#
- The colors are implemented using the colorama package and ANSI escape codes. When running this from IPython or Bash and using
less, you need to instruct it to display the colors with -R:
#
# cat UD_Ancient_Greek/sample.conllu | udapy -T | less -R
#
# - You can also use
udapy -T -N to disable the colors.
# udapy -q suppresses all Udapi messages (warnings, info, debug) printed on the standard error output, so only fatal errors are printed. By default only debug messages are suppresses, but these can be printed with udapy -v.
# - But you already know this because you have read
udapy -h, am I right?
#
# Details:
#
# doc = udapi.Document(filename) is a shortcut for
#
# import udapi.core.document
# doc = udapi.core.document.Document(filename)
#
# - We can print the whole document using
doc.draw().
# doc.draw(**kwargs) is a shortcut for creating a write.TextModeTrees block and applying it on the document:
#
# import udapi.block.write.textmodetrees
# block = udapi.block.write.textmodetrees.TextModeTrees(**kwargs)
# block.run(doc)
#
#
# Hint 1:
# In some dependency grammars, adpositions govern noun (i.e. adposition is the *parent* of a given noun node). In other dependency grammars, adpositions depend on nouns (i.e. noun is the *parent* of a given adposition). Find out which style is being used by UD. Check the UD documentation or inspect some of the tree visualizations and guess.
# Hint 2:
# See the Node documentation and find out how to obtain dependency parent and dependency children. Note that these are properties of a given node, rather than methods, so you should not write parentheses () after the property name.
# Hint 3:
# doc.nodes iterates over all nodes in the document sorted by the word order, but this would be cumbersome to exploit. Find a method of Node to detect the relative word order of two nodes (within the same tree/sentence).
# Hint 4:
# Use node.parent and node.precedes(another_node).
# The latter is a shortcut for node.ord < another_node.ord.
# Solution:
#
# for node in doc.nodes:
# if node.upos == "ADP":
# if node.precedes(node.parent):
# prepositions += 1
# else:
# postpositions += 1
#
# Solution 1:
#
# for node in doc.nodes:
# if node.upos == "ADP" and node.parent.precedes(node):
# node.parent.draw()
# count[node.lemma + " " + node.deprel] += 1
#
# Hint 1:
# We can see there are many particles of phrase verbs, e.g. "busted up".
# These seem to be correctly annotated as ADP according to the UD guidelines.
# Let's filter out those cases and focus on the rest and let's switch to the big train data.
# Solution 2:
#
# count = collections.Counter()
# for node in big_doc.nodes:
# if node.upos == "ADP" and node.parent.precedes(node) and node.parent.upos != "VERB":
# count[node.lemma + " " + node.deprel] += 1
# count.most_common()
#
# Alternatively to node.parent.upos != "VERB",
# you could also filter out node.deprel != "compound:prt",
# or directly focus on node.deprel == "case"
# Partial answer:
# Most of the occurrences are actually annotated correctly,
# although they are not typically considered as postpositions.
# For example, node.deprel == "fixed" is being used for multi-word adpositions,
# such as "because of", where "of" depends on "because" from technical (and consistency) reasons,
# but the whole multi-word adpositions precedes its governing nound, so it is actually a multi-word preposition.
#
# What about the remaining occurrences, after filtering out node.deprel not in {"compound:prt", "fixed"}?
#