Notebook

This document is organized in 3 sections:

Recap of Eli Bendersky's XML parsing from his online article
Parsing and helper functions for Kanjidic2
Parsing and helper functions for JMDict

Eli Bendersky's article tests¶

Basic parsing¶

<doc> <branch name="testing" hash="1cdf045c"> text,source </branch> <branch name="release01" hash="f200013e"> <sub-branch name="subrelease01"> xml,sgml </sub-branch> </branch> <branch name="invalid"> </branch> </doc>

In [2]:

import xml.etree.cElementTree as ET
tree = ET.ElementTree(file='doc1.xml')

In [80]:

tree.getroot()

Out[80]:

<Element 'doc' at 0x05349EC0>

In [83]:

root = tree.getroot()
root.tag, root.attrib

Out[83]:

('doc', {})

In [84]:

for child_of_root in root:
    print child_of_root.tag, child_of_root.attrib

branch {'hash': '1cdf045c', 'name': 'testing'}
branch {'hash': 'f200013e', 'name': 'release01'}
branch {'name': 'invalid'}

In [85]:

root[0].tag, root[0].text

Out[85]:

('branch', '\n        text,source\n    ')

Find interesting stuff¶

In [86]:

for elem in tree.iter():
    print elem.tag, elem.attrib

doc {}
branch {'hash': '1cdf045c', 'name': 'testing'}
branch {'hash': 'f200013e', 'name': 'release01'}
sub-branch {'name': 'subrelease01'}
branch {'name': 'invalid'}

In [87]:

for elem in tree.iter(tag='branch'):
    print elem.tag, elem.attrib

branch {'hash': '1cdf045c', 'name': 'testing'}
branch {'hash': 'f200013e', 'name': 'release01'}
branch {'name': 'invalid'}

Using XPath¶

In [97]:

for elem in tree.iterfind('branch/sub-branch'):
    print elem.tag, elem.attrib

sub-branch {'name': 'subrelease01'}

In [99]:

for elem in tree.iterfind('branch'):
    print elem.tag, elem.attrib

branch {'hash': '1cdf045c', 'name': 'testing'}
branch {'hash': 'f200013e', 'name': 'release01'}
branch {'name': 'invalid'}

In [90]:

for elem in tree.iterfind('branch[@name="release01"]'):
    print elem.tag, elem.attrib

branch {'hash': 'f200013e', 'name': 'release01'}

Kanjidic2¶

Example file¶

<character> <literal>本</literal> <codepoint> <cp_value cp_type="ucs">672c</cp_value> <cp_value cp_type="jis208">43-60</cp_value> </codepoint> <radical> <rad_value rad_type="classical">75</rad_value> <rad_value rad_type="nelson_c">2</rad_value> </radical> <misc> <grade>1</grade> <stroke_count>5</stroke_count> <variant var_type="jis208">52-81</variant> <freq>10</freq> <jlpt>4</jlpt> </misc> <dic_number> <dic_ref dr_type="nelson_c">96</dic_ref> <dic_ref dr_type="nelson_n">2536</dic_ref> <dic_ref dr_type="halpern_njecd">3502</dic_ref> <dic_ref dr_type="halpern_kkld">2183</dic_ref> <dic_ref dr_type="heisig">211</dic_ref> <dic_ref dr_type="gakken">15</dic_ref> <dic_ref dr_type="oneill_names">212</dic_ref> <dic_ref dr_type="oneill_kk">20</dic_ref> <dic_ref dr_type="moro" m_vol="6" m_page="0026">14421</dic_ref> <dic_ref dr_type="henshall">70</dic_ref> <dic_ref dr_type="sh_kk">25</dic_ref> <dic_ref dr_type="sakade">45</dic_ref> <dic_ref dr_type="jf_cards">61</dic_ref> <dic_ref dr_type="henshall3">76</dic_ref> <dic_ref dr_type="tutt_cards">47</dic_ref> <dic_ref dr_type="crowley">6</dic_ref> <dic_ref dr_type="kanji_in_context">37</dic_ref> <dic_ref dr_type="busy_people">2.1</dic_ref> <dic_ref dr_type="kodansha_compact">1046</dic_ref> <dic_ref dr_type="maniette">215</dic_ref> </dic_number> <query_code> <q_code qc_type="skip">4-5-3</q_code> <q_code qc_type="sh_desc">0a5.25</q_code> <q_code qc_type="four_corner">5023.0</q_code> <q_code qc_type="deroo">1855</q_code> </query_code> <reading_meaning> <rmgroup> <reading r_type="pinyin">ben3</reading> <reading r_type="korean_r">bon</reading> <reading r_type="korean_h">본</reading> <reading r_type="ja_on">ホン</reading> <reading r_type="ja_kun">もと</reading> <meaning>book</meaning> <meaning>present</meaning> <meaning>main</meaning> <meaning>true</meaning> <meaning>real</meaning> <meaning>counter for long cylindrical things</meaning> <meaning m_lang="fr">livre</meaning> <meaning m_lang="fr">présent</meaning> <meaning m_lang="fr">essentiel</meaning> <meaning m_lang="fr">origine</meaning> <meaning m_lang="fr">principal</meaning> <meaning m_lang="fr">réalité</meaning> <meaning m_lang="fr">vérité</meaning> <meaning m_lang="fr">compteur d'objets allongés</meaning> <meaning m_lang="es">libro</meaning> <meaning m_lang="es">origen</meaning> <meaning m_lang="es">base</meaning> <meaning m_lang="es">contador de cosas alargadas</meaning> <meaning m_lang="pt">livro</meaning> <meaning m_lang="pt">presente</meaning> <meaning m_lang="pt">real</meaning> <meaning m_lang="pt">verdadeiro</meaning> <meaning m_lang="pt">principal</meaning> <meaning m_lang="pt">sufixo p/ contagem De coisas longas</meaning> </rmgroup> <nanori>まと</nanori> </reading_meaning> </character>

In [105]:

import xml.etree.cElementTree as ET
tree = ET.ElementTree(file='kanjidic2_example.xml')

First of all, what does the tree look like in this example file?

In [25]:

elems = [elem for elem in tree.iter()][:10]
elems

Out[25]:

[<Element 'character' at 0x050E6848>,
 <Element 'literal' at 0x050E68A8>,
 <Element 'codepoint' at 0x050E68D8>,
 <Element 'cp_value' at 0x050E6908>,
 <Element 'cp_value' at 0x050E6920>,
 <Element 'radical' at 0x050E6938>,
 <Element 'rad_value' at 0x050E6968>,
 <Element 'rad_value' at 0x050E6980>,
 <Element 'misc' at 0x050E6998>,
 <Element 'grade' at 0x050E69C8>]

Getting the root: the 'character'.

In [15]:

root = tree.getroot()
root

Out[15]:

<Element 'character' at 0x050E6848>

Getting the literal.

In [16]:

literal = root[0]
literal

Out[16]:

<Element 'literal' at 0x050E68A8>

In [17]:

kanji = literal.text
kanji

Out[17]:

u'\u672c'

In [18]:

print kanji

本

Getting the meanings.

In [44]:

meanings = [elem for elem in tree.iter('meaning')]
[meaning.text for meaning in meanings]

Out[44]:

['book',
 'present',
 'main',
 'true',
 'real',
 'counter for long cylindrical things',
 'livre',
 u'pr\xe9sent',
 'essentiel',
 'origine',
 'principal',
 u'r\xe9alit\xe9',
 u'v\xe9rit\xe9',
 u"compteur d'objets allong\xe9s",
 'libro',
 'origen',
 'base',
 'contador de cosas alargadas',
 'livro',
 'presente',
 'real',
 'verdadeiro',
 'principal',
 'sufixo p/ contagem De coisas longas']

But here we only want english meanings.

In [46]:

meanings[10].attrib

Out[46]:

{'m_lang': 'fr'}

In [51]:

english_meanings = filter(lambda elem: elem.attrib == {}, meanings)
[meaning.text for meaning in english_meanings]

Out[51]:

['book',
 'present',
 'main',
 'true',
 'real',
 'counter for long cylindrical things']

Finally, we can get the Kanas.

In [60]:

readings = [elem for elem in tree.iter('reading')]
print [reading.text for reading in readings]

['ben3', 'bon', u'\ubcf8', u'\u30db\u30f3', u'\u3082\u3068']

Filtering for kanas.

In [64]:

readings[0].attrib['r_type']

Out[64]:

'pinyin'

In [67]:

'r_type' in readings[0].attrib

Out[67]:

True

In [73]:

kanas = filter(lambda reading: reading.attrib['r_type'] in ['ja_on', 'ja_kun'], readings)
kanas

Out[73]:

[<Element 'reading' at 0x050E6EF0>, <Element 'reading' at 0x050E6F08>]

In [75]:

for kana in kanas:
    print kana.text

ホン
もと

The whole Kanjidic2 file¶

In [1]:

import xml.etree.cElementTree as ET
tree = ET.ElementTree(file='kanjidic2.xml')
tree

Out[1]:

<ElementTree at 0x49cce50>

In [2]:

root = tree.getroot()
root

Out[2]:

<Element 'kanjidic2' at 0x049D8AB8>

In [3]:

root.findall('character/literal')[:10]

Out[3]:

[<Element 'literal' at 0x049D8440>,
 <Element 'literal' at 0x04A2D488>,
 <Element 'literal' at 0x04A2D7E8>,
 <Element 'literal' at 0x04A2DB48>,
 <Element 'literal' at 0x049FC260>,
 <Element 'literal' at 0x049FCD28>,
 <Element 'literal' at 0x04A276E0>,
 <Element 'literal' at 0x04A27AE8>,
 <Element 'literal' at 0x04A27E48>,
 <Element 'literal' at 0x04A31428>]

I understand now: you have to specify the exact branching in the findall command while iter works because it filters the depth first search.

Searching for the entry of a specific kanji.

In [4]:

search_kanji = u'本'
literals = root.findall('character/literal')
literals[:10]

Out[4]:

[<Element 'literal' at 0x049D8440>,
 <Element 'literal' at 0x04A2D488>,
 <Element 'literal' at 0x04A2D7E8>,
 <Element 'literal' at 0x04A2DB48>,
 <Element 'literal' at 0x049FC260>,
 <Element 'literal' at 0x049FCD28>,
 <Element 'literal' at 0x04A276E0>,
 <Element 'literal' at 0x04A27AE8>,
 <Element 'literal' at 0x04A27E48>,
 <Element 'literal' at 0x04A31428>]

In [5]:

len(literals)

Out[5]:

In [147]:

tree.find('character/literal')

Out[147]:

<Element 'literal' at 0x0A8ABDB8>

In [6]:

[literal.text for literal in literals].index(u'話')

Out[6]:

In [7]:

print literals[2948].text

話

Getting the parent node.

In [151]:

characters = root.findall('character')
characters[:10]

Out[151]:

[<Element 'character' at 0x0A8ABA10>,
 <Element 'character' at 0x0A8AB3E0>,
 <Element 'character' at 0x0A8A6F98>,
 <Element 'character' at 0x0A8A6C50>,
 <Element 'character' at 0x0A8A6350>,
 <Element 'character' at 0x0A8A18F0>,
 <Element 'character' at 0x0A89DE90>,
 <Element 'character' at 0x0A89DA70>,
 <Element 'character' at 0x0A89D758>,
 <Element 'character' at 0x0A89D170>]

In [153]:

print characters[2948][0].text

話

Defining helper functions¶

Find a specific kanji in the dictionary¶

In [155]:

def find_element_by_kanji(tree, kanji):
    root = tree.getroot()
    literals = root.findall('character/literal')
    index = [literal.text for literal in literals].index(kanji)
    return root.findall('character')[index]

In [158]:

kuruma = find_element_by_kanji(tree, u'車')
kuruma

Out[158]:

<Element 'character' at 0x0738E500>

In [160]:

print kuruma[0].text

車

Extract meaningful information from a 'character'¶

In [173]:

def extract_data(element):
    """returns the kanji, the kana and the meanings from an element"""
    kanji = element.find('literal').text
    kana = [elem.text for elem in filter(lambda reading: reading.attrib['r_type'] in ['ja_on', 'ja_kun'], element.findall('reading_meaning/rmgroup/reading'))]
    meanings = [elem.text for elem in filter(lambda elem: elem.attrib == {}, element.findall('reading_meaning/rmgroup/meaning'))]
    return (kanji, kana, meanings)

In [176]:

def disp_data(data):
    print data[0]
    for item in data[1]:
        print item
    for item in data[2]:
        print item    

data = extract_data(kuruma)
disp_data(data)

車
シャ
くるま
car

In [178]:

disp_data(extract_data(find_element_by_kanji(tree, u'話')))

話
ワ
はな.す
はなし
tale
talk

In [179]:

disp_data(extract_data(find_element_by_kanji(tree, u'尖')))

尖
セン
とが.る
さき
するど.い
be pointed
sharp
taper
displeased
angry
edgy

JMdict¶

Working with the example file¶

<entry> <ent_seq>1171270</ent_seq> <k_ele> <keb>右翼</keb> <ke_pri>ichi1</ke_pri> <ke_pri>news1</ke_pri> <ke_pri>nf04</ke_pri> </k_ele> <r_ele> <reb>うよく</reb> <re_pri>ichi1</re_pri> <re_pri>news1</re_pri> <re_pri>nf04</re_pri> </r_ele> <sense> <pos>adj-no;</pos> <gloss>right-wing</gloss> <gloss xml:lang="fr">aile droite (oiseau, armée, parti politique, base-ball)</gloss> <gloss xml:lang="ru">пра́вое крыло́</gloss> <gloss xml:lang="ru">пра́вый фланг</gloss> <gloss xml:lang="de">die Rechte</gloss> <gloss xml:lang="de">rechter Flügel</gloss> </sense> <sense> <pos>n;</pos> <gloss>right field (e.g. in sport)</gloss> <gloss>right flank</gloss> <gloss>right wing</gloss> <gloss xml:lang="de">{Sport}</gloss> <gloss xml:lang="de">rechte Flanke</gloss> <gloss xml:lang="de">rechter Flügel</gloss> </sense> </entry>

In [195]:

tree = ET.ElementTree(file='JMdict_example.xml')
tree

Out[195]:

<ElementTree at 0x36692dd0>

In [196]:

root = tree.getroot()
root

Out[196]:

<Element 'entry' at 0x366944E8>

Looking at the first few lines.

In [197]:

elems = [elem for elem in tree.iter()][:10]
elems

Out[197]:

[<Element 'entry' at 0x366944E8>,
 <Element 'ent_seq' at 0x36694728>,
 <Element 'k_ele' at 0x36694500>,
 <Element 'keb' at 0x366946E0>,
 <Element 'ke_pri' at 0x36694650>,
 <Element 'ke_pri' at 0x36694548>,
 <Element 'ke_pri' at 0x36694608>,
 <Element 'r_ele' at 0x36694518>,
 <Element 'reb' at 0x36694578>,
 <Element 're_pri' at 0x366947A0>]

In [198]:

expression = root.find('k_ele/keb').text
print expression

右翼

In [200]:

reading = root.find('r_ele/reb').text
print reading

うよく

In [203]:

senses = root.findall('sense/gloss')
senses

Out[203]:

[<Element 'gloss' at 0x366AF2D8>,
 <Element 'gloss' at 0x366AF320>,
 <Element 'gloss' at 0x366AF098>,
 <Element 'gloss' at 0x366AF368>,
 <Element 'gloss' at 0x366AF380>,
 <Element 'gloss' at 0x366AF3B0>,
 <Element 'gloss' at 0x366AF410>,
 <Element 'gloss' at 0x366AF428>,
 <Element 'gloss' at 0x366AF440>,
 <Element 'gloss' at 0x366AF470>,
 <Element 'gloss' at 0x366AF4A0>,
 <Element 'gloss' at 0x366AF4D0>]

In [208]:

senses = filter(lambda sense: sense.attrib == {}, senses)
senses

Out[208]:

[<Element 'gloss' at 0x366AF2D8>,
 <Element 'gloss' at 0x366AF410>,
 <Element 'gloss' at 0x366AF428>,
 <Element 'gloss' at 0x366AF440>]

In [209]:

for sense in senses:
    print sense.text

right-wing
right field (e.g. in sport)
right flank
right wing

Working with the whole file¶

In [3]:

tree = ET.ElementTree(file='JMdict.xml')
tree

Out[3]:

<ElementTree at 0x4a159b0>

In [4]:

root = tree.getroot()
root

Out[4]:

<Element 'JMdict' at 0x04995D28>

In [5]:

word_entries = tree.getroot().findall('entry/k_ele/keb')
words = [entry.text for entry in word_entries]

In [6]:

len(words)

Out[6]:

In [221]:

for word in words[:50]:
    print word

〃
仝
々
漢数字ゼロ
○
〇
ＡＢＣ順
ＣＤプレーヤー
ＣＤプレイヤー
Ｎ響
Ｏバック
ＲＳ２３２ケーブル
Ｔシャツ
Ｔバック
あうんの呼吸
阿吽の呼吸
明白
明白
偸閑
白地
明かん
悪どい
論う
馬酔木
彼処
彼所
あっと言う間に
あっという間に
あっとゆう間に
彼の
あの人
彼の人
あの方
彼の方
溢れる
阿呆陀羅
甘子
天魚
雨子
𩺊
彼
いい加減にしなさい
いい年をして
否々
否否
如何わしい
いかなる場合でも
如何にも
幾つも
行けない

In [237]:

words[49][0] in words[34]

Out[237]:

False

Ask for a specific kanji in an expression:

In [240]:

filtered_words = filter(lambda expression: u'寺' in expression, words)
for word in filtered_words:
    print word

駆け込み寺
駆込み寺
古社寺
山寺
寺
寺院
禅寺
僧寺
大寺院
中禅寺湖
尼寺
仏寺
末寺
古寺
寺社
社寺
国分寺
寺参り
寺子屋
寺小屋
回教寺院
縁切り寺
氏寺
檀那寺
勅願寺
寺男
寺銭
菩提寺
寺格
八百八寺
寺内
入寺
敵は本能寺にあり
敵は本能寺に在り
寺号
寺域
官寺
大覚寺統
脇寺
寺中
寺社奉行
寺預け
寺入り
南都七大寺
七大寺
本願寺派
仏光寺派
誠照寺派
少林寺拳法
寺
お寺
御寺
お寺様
お寺さま
御寺様
紅妙蓮寺
寺請
寺請け
寺請制度
寺檀制度
三井寺歩行虫
三井寺芥虫
寺子
少林寺流
寺領
本能寺の変
寺務
監寺
都寺
副寺
寺務所
私寺
お寺さん
御寺さん
道明寺粉
廃寺
当寺
寺内町
宮寺
神宮寺
諸寺

Outline of what could be done from this¶

build a sort of exploratory app that starts with a kanji or a word, then lists all compounds from the dictionary that contain the given kanjis and makes it able to reselect any one of them at a later stage while offering the possibility to visualize the data associated to each kanji
probably the most easy thing to do is classify words with respect to frequency
add support for reading Anki decks or better: integrate with Anki desktop, as it is written in Python

In [ ]: