You might want to consider the start of this tutorial.

Short introductions to other TF datasets:

In [1]:
%load_ext autoreload
%autoreload 2
In [2]:
from tf.app import use
In [3]:
VERSION = '2017'
In [4]:
A = use('bhsa', hoist=globals(), version=VERSION)
# A = use('bhsa:clone', checkout="clone", hoist=globals(), version=VERSION)
Using TF-app in /Users/dirk/text-fabric-data/annotation/app-bhsa/code:
	rv2.0.4=#7b3b9ffba7ee6dbc76a52b8d76475d17babf0daf offline under ~/text-fabric-data (local release)
Using data in /Users/dirk/text-fabric-data/etcbc/bhsa/tf/2017:
	rv1.6 offline under ~/text-fabric-data (local release)
Using data in /Users/dirk/text-fabric-data/etcbc/phono/tf/2017:
	r1.2=#1ac68e976ee4a7f23eb6bb4c6f401a033d0ec169 offline under ~/text-fabric-data (local release)
Using data in /Users/dirk/text-fabric-data/etcbc/parallels/tf/2017:
	r1.2=#395dfe2cb69c261862fab9f0289e594a52121d5c offline under ~/text-fabric-data (local release)
   |     0.00s Dataset without structure sections in otext:no structure functions in the T-API

Gaps and spans

Searches often do not deliver the results you expect. Besides typos, lack of familiarity with the template formalism and bugs in the system, there is another cause: difficult semantics of the data.

Most users reason about phrases, clauses and sentences as if they are consecutive blocks of words. But in the BHSA this is not the case: each of these objects may have gaps.

Most of the time, verse boundaries coincide with the boundaries of sentences, clauses, and phrases. But not always, there are verse spanning sentences.

Note These phenomena may wreak havoc with your intuitive reasoning about what search templates should deliver. Query templates do not require the objects to be consecutive and still they make sense. But that might not be your sense, unless you Mind the gap!

We are going to show these issues in depth.

Gaps

TF-search has no primitives to deal with gaps directly. Nodes correspond to textual objects such as words, phrases, clauses, verses, books. Usually these are consecutive sequences of one or more words, but in theory they can be arbitrary sets of slots.

And, as far as the BHSA corpus is concerned, in practice too. If we look at phrases, then the overwhelming majority is consecutive, without gaps, But there is also a substantial amount of phrases with gaps.

People that are familiar with MQL (see fromMQL) may remember that in MQL you can search for a gap. The MQL query

SELECT ALL OBJECTS WHERE

[phrase FOCUS
    [word lex='L']
    [gap]
]

looks for a phrase with a gap in it (i.e. one or more consecutive words between the start and the end of the phrase that do not belong to the phrase). The query then asks additionally for those gap-containing phrases that have a certain word in front of the gap.

We want this too!

Find the gap

We start with a query that aims to get the same results as the MQL query above.

In our template, we require that there is a word wPreGap in the phrase that is just before the gap, a word wGap that comes right after, so it is in the gap, and hence does not belong to the phrase. But this all must happen before the last word wLast of the phrase.

In [5]:
query = '''
verse
    p:phrase
      wPreGap:word lex=L
      wLast:word
      :=

wGap:word
wPreGap <: wGap
wGap < wLast
p || wGap
'''
In [6]:
results = A.search(query)
  0.99s 13 results

Nice and quick. Let's see the results.

In [7]:
A.table(results, skipCols="1")
npphrasewordwordword
1Genesis 17:7לְךָ֙ וּֽלְזַרְעֲךָ֖ אַחֲרֶֽיךָ׃ לְךָ֙ אַחֲרֶֽיךָ׃ לֵֽ
2Genesis 28:4לְךָ֙ לְךָ֖ וּלְזַרְעֲךָ֣ אִתָּ֑ךְ לְךָ֙ אִתָּ֑ךְ אֶת־
3Genesis 31:16לָ֥נוּ וּלְבָנֵ֑ינוּ לָ֥נוּ בָנֵ֑ינוּ ה֖וּא
4Exodus 30:21לָהֶ֧ם לֹ֥ו וּלְזַרְעֹ֖ו לָהֶ֧ם זַרְעֹ֖ו חָק־
5Leviticus 25:6לָכֶם֙ לְךָ֖ וּלְעַבְדְּךָ֣ וְלַאֲמָתֶ֑ךָ וְלִשְׂכִֽירְךָ֙ וּלְתֹושָׁ֣בְךָ֔ לָכֶם֙ תֹושָׁ֣בְךָ֔ לְ
6Numbers 20:15לָ֛נוּ וְלַאֲבֹתֵֽינוּ׃ לָ֛נוּ אֲבֹתֵֽינוּ׃ מִצְרַ֖יִם
7Numbers 32:33לָהֶ֣ם׀ לִבְנֵי־גָד֩ וְלִבְנֵ֨י רְאוּבֵ֜ן וְלַחֲצִ֣י׀ שֵׁ֣בֶט׀ מְנַשֶּׁ֣ה בֶן־יֹוסֵ֗ף לָהֶ֣ם׀ יֹוסֵ֗ף מֹשֶׁ֡ה
8Deuteronomy 1:36לֹֽו־וּלְבָנָ֑יו לֹֽו־בָנָ֑יו אֶתֵּ֧ן
9Deuteronomy 26:11לְךָ֛ וּלְבֵיתֶ֑ךָ לְךָ֛ בֵיתֶ֑ךָ יְהוָ֥ה
101_Samuel 25:31לְךָ֡ לַאדֹנִ֗י לְךָ֡ אדֹנִ֗י לְ
112_Kings 25:24לָהֶ֤ם וּלְאַנְשֵׁיהֶ֔ם לָהֶ֤ם אַנְשֵׁיהֶ֔ם גְּדַלְיָ֨הוּ֙
12Jeremiah 40:9לָהֶ֜ם וּלְאַנְשֵׁיהֶ֣ם לָהֶ֜ם אַנְשֵׁיהֶ֣ם גְּדַלְיָ֨הוּ
13Daniel 9:8לָ֚נוּ לִמְלָכֵ֥ינוּ לְשָׂרֵ֖ינוּ וְלַאֲבֹתֵ֑ינוּ לָ֚נוּ אֲבֹתֵ֑ינוּ בֹּ֣שֶׁת

Let's color the word in the gap differently.

In [8]:
A.displaySetup(skipCols="1", colorMap={1: 'aqua', 2: 'yellow', 4: 'magenta'}, condenseType="clause")
In [9]:
A.table(results, condensed=False)
npphrasewordwordword
1Genesis 17:7לְךָ֙ וּֽלְזַרְעֲךָ֖ אַחֲרֶֽיךָ׃ לְךָ֙ אַחֲרֶֽיךָ׃ לֵֽ
2Genesis 28:4לְךָ֙ לְךָ֖ וּלְזַרְעֲךָ֣ אִתָּ֑ךְ לְךָ֙ אִתָּ֑ךְ אֶת־
3Genesis 31:16לָ֥נוּ וּלְבָנֵ֑ינוּ לָ֥נוּ בָנֵ֑ינוּ ה֖וּא
4Exodus 30:21לָהֶ֧ם לֹ֥ו וּלְזַרְעֹ֖ו לָהֶ֧ם זַרְעֹ֖ו חָק־
5Leviticus 25:6לָכֶם֙ לְךָ֖ וּלְעַבְדְּךָ֣ וְלַאֲמָתֶ֑ךָ וְלִשְׂכִֽירְךָ֙ וּלְתֹושָׁ֣בְךָ֔ לָכֶם֙ תֹושָׁ֣בְךָ֔ לְ
6Numbers 20:15לָ֛נוּ וְלַאֲבֹתֵֽינוּ׃ לָ֛נוּ אֲבֹתֵֽינוּ׃ מִצְרַ֖יִם
7Numbers 32:33לָהֶ֣ם׀ לִבְנֵי־גָד֩ וְלִבְנֵ֨י רְאוּבֵ֜ן וְלַחֲצִ֣י׀ שֵׁ֣בֶט׀ מְנַשֶּׁ֣ה בֶן־יֹוסֵ֗ף לָהֶ֣ם׀ יֹוסֵ֗ף מֹשֶׁ֡ה
8Deuteronomy 1:36לֹֽו־וּלְבָנָ֑יו לֹֽו־בָנָ֑יו אֶתֵּ֧ן
9Deuteronomy 26:11לְךָ֛ וּלְבֵיתֶ֑ךָ לְךָ֛ בֵיתֶ֑ךָ יְהוָ֥ה
101_Samuel 25:31לְךָ֡ לַאדֹנִ֗י לְךָ֡ אדֹנִ֗י לְ
112_Kings 25:24לָהֶ֤ם וּלְאַנְשֵׁיהֶ֔ם לָהֶ֤ם אַנְשֵׁיהֶ֔ם גְּדַלְיָ֨הוּ֙
12Jeremiah 40:9לָהֶ֜ם וּלְאַנְשֵׁיהֶ֣ם לָהֶ֜ם אַנְשֵׁיהֶ֣ם גְּדַלְיָ֨הוּ
13Daniel 9:8לָ֚נוּ לִמְלָכֵ֥ינוּ לְשָׂרֵ֖ינוּ וְלַאֲבֹתֵ֑ינוּ לָ֚נוּ אֲבֹתֵ֑ינוּ בֹּ֣שֶׁת
In [10]:
A.show(results, end=3, condensed=False)

result 1

Genesis 17:7 
clause
phrase
lex=L
phrase
phrase
lex=L
phrase
lex=W
phrase

result 2

Genesis 28:4 
clause
phrase
lex=W
phrase
phrase
phrase
phrase

result 3

Genesis 31:16 
clause
phrase
phrase
phrase
lex=W
phrase
In [11]:
A.displayReset()

All gapped phrases

These were particular gaps. Now we want to get all gapped phrases.

We can just lift the special requirement that the preGapWord has to satisfy a special lexical condition.

In [12]:
query = '''
p:phrase
  wPreGap:word
  wLast:word
  :=

wGap:word
wPreGap <: wGap
wGap < wLast

p || wGap
'''
In [13]:
results = A.search(query)
  2.29s 715 results

Not too bad! We could wait for it. Here are some results.

In [14]:
A.table(results, start=5, end=10)
npphrasewordwordword
5Genesis 2:25שְׁנֵיהֶם֙ הָֽאָדָ֖ם וְאִשְׁתֹּ֑ו שְׁנֵיהֶם֙ אִשְׁתֹּ֑ו עֲרוּמִּ֔ים
6Genesis 4:4הֶ֨בֶל גַם־ה֛וּא הֶ֨בֶל ה֛וּא הֵבִ֥יא
7Genesis 7:8מִן־הַבְּהֵמָה֙ הַטְּהֹורָ֔ה וּמִן־הַ֨בְּהֵמָ֔ה וּמִ֨ן־הָעֹ֔וף וְכֹ֥ל בְּהֵמָ֔ה כֹ֥ל אֲשֶׁ֥ר
8Genesis 7:14הֵ֜מָּה וְכָל־הַֽחַיָּ֣ה לְמִינָ֗הּ וְכָל־הַבְּהֵמָה֙ לְמִינָ֔הּ וְכָל־הָרֶ֛מֶשׂ לְמִינֵ֑הוּ וְכָל־הָעֹ֣וף לְמִינֵ֔הוּ כֹּ֖ל צִפֹּ֥ור כָּל־כָּנָֽף׃ רֶ֛מֶשׂ כָּנָֽף׃ הָ
9Genesis 7:21כָּל־בָּשָׂ֣ר׀ בָּעֹ֤וף וּבַבְּהֵמָה֙ וּבַ֣חַיָּ֔ה וּבְכָל־הַשֶּׁ֖רֶץ וְכֹ֖ל הָאָדָֽם׃ בָּשָׂ֣ר׀ אָדָֽם׃ הָ
10Genesis 7:21כָּל־בָּשָׂ֣ר׀ בָּעֹ֤וף וּבַבְּהֵמָה֙ וּבַ֣חַיָּ֔ה וּבְכָל־הַשֶּׁ֖רֶץ וְכֹ֖ל הָאָדָֽם׃ שֶּׁ֖רֶץ אָדָֽם׃ הַ

If a phrase has multiple gaps, we encounter it multiple times in our results.

We show the two results in Genesis 7:21.

In [15]:
A.show(
    results, condensed=False, condenseType="clause",
    start=9, end=10,
    colorMap={1: 'lightgreen', 2: 'orange', 4: 'magenta'}
)

If we want just the phrases, and only once, we can run the query in shallow mode, see advanced:

In [16]:
gapQueryResults = A.search(query, shallow=True)
  2.88s 671 results

A different query

We can make an equivalent query to get the gaps.

In [17]:
query = '''
p:phrase
    =: wFirst:word
    wLast:word
    :=

wGap:word
wFirst < wGap
wLast > wGap

p || wGap
'''

Experience has shown that this is a slow query, so we handle it with care.

In [18]:
S.study(query)
S.showPlan(details=True)
  0.00s Checking search template ...
  0.00s Setting up search space for 4 objects ...
  0.25s Constraining search space with 7 relations ...
  1.14s 	2 edges thinned
  1.14s Setting up retrieval plan with strategy small_choice_multi ...
  1.17s Ready to deliver results from 1186145 nodes
Iterate over S.fetch() to get the results
See S.showPlan() to interpret the results
Search with 4 objects and 6 relations
Results are instantiations of the following objects:
node  0-phrase                                        253187   choices
node  1-word                                          253187   choices
node  2-word                                          253187   choices
node  3-word                                          426584   choices
Performance parameters:
	yarnRatio            =    1.25
	tryLimitFrom         =      40
	tryLimitTo           =      40
Instantiations are computed along the following relations:
node                                  0-phrase        253187   choices
edge        0-phrase           [[     2-word               1.0 choices
edge        2-word             :=     0-phrase             0   choices
edge        0-phrase           [[     1-word               1.0 choices
edge        1-word             =:     0-phrase             0   choices
edge      2,1-word            >,<     3-word           21329.2 choices
edge        3-word             ||     0-phrase             0   choices
  1.18s The results are connected to the original search template as follows:
 0     
 1 R0  p:phrase
 2 R1      =: wFirst:word
 3 R2      wLast:word
 4         :=
 5     
 6 R3  wGap:word
 7     wFirst < wGap
 8     wLast > wGap
 9     
10     p || wGap
11     
In [19]:
S.count(progress=1, limit=8)
  0.00s Counting results per 1 up to 8 ...
   |       10s 1
   |       10s 2
   |       10s 3
   |       10s 4
   |       10s 5
   |       10s 6
   |       18s 7
   |       18s 8
    18s Done: 8 results

This is a good example of a query that is slow to deliver even its first result. And that is bad, because it is such a straightforward query.

Why is this one so slow, while the previous one went so smoothly?

The crucial thing is the wGap word. In the latter template, wGap is not embedded in anything. It is constrained by wFirst < wGap and wGap < wLast. However, the way the search strategy works is by examining all possibilities for wFirst < wGap and only then checking whether wGap < wLast. The algorithm cannot check both conditions at the same time.

With embedding relations, things are better. Text-Fabric is heavily optimized to deal with embedding relationships.

In the former template, we see that the wGap is required to be adjacent to wPreGap, and this one is embedded in the phrase. Hence there are few cases to consider for wPreGap, and per instance there is only one wGap.

Lesson Try to prevent the use of free floating nodes in your template that become constrained by other spatial relationships than embedding.

To the rescue

The former template had it right. Can we rescue the latter template?

We can assume that the phrase and the gap each contain a word in one and the same verse. Note that phrase and gap may belong to different clauses and sentences. We assume that a phrase cannot belong to more than two verses, so either the first or the last word of the phrase is in the same verse as a word in the gap.

In [20]:
query = '''
p:phrase
    =: wFirst:word
    wLast:word
    :=

wGap:word
wFirst < wGap
wLast > wGap

p || wGap

v:verse

v [[ wFirst
v [[ wGap
'''
In [21]:
S.study(query)
S.showPlan(details=True)
S.count(progress=100, limit=3000)
  0.00s Checking search template ...
  0.00s Setting up search space for 5 objects ...
  0.22s Constraining search space with 9 relations ...
  1.07s 	2 edges thinned
  1.07s Setting up retrieval plan with strategy small_choice_multi ...
  1.12s Ready to deliver results from 1209358 nodes
Iterate over S.fetch() to get the results
See S.showPlan() to interpret the results
Search with 5 objects and 8 relations
Results are instantiations of the following objects:
node  0-phrase                                        253187   choices
node  1-word                                          253187   choices
node  2-word                                          253187   choices
node  3-word                                          426584   choices
node  4-verse                                          23213   choices
Performance parameters:
	yarnRatio            =    1.25
	tryLimitFrom         =      40
	tryLimitTo           =      40
Instantiations are computed along the following relations:
node                                  4-verse          23213   choices
edge        4-verse            [[     1-word              10.1 choices
edge        1-word             =:     0-phrase             1.0 choices (thinned)
edge        1-word             ]]     0-phrase             0   choices
edge        0-phrase           [[     2-word               1.0 choices
edge        2-word             :=     0-phrase             0   choices
edge        4-verse            [[     3-word              17.2 choices
edge      2,1-word            >,<     3-word               0   choices
edge        3-word             ||     0-phrase             0   choices
  1.13s The results are connected to the original search template as follows:
 0     
 1 R0  p:phrase
 2 R1      =: wFirst:word
 3 R2      wLast:word
 4         :=
 5     
 6 R3  wGap:word
 7     wFirst < wGap
 8     wLast > wGap
 9     
10     p || wGap
11     
12 R4  v:verse
13     
14     v [[ wFirst
15     v [[ wGap
16     
  0.00s Counting results per 100 up to 3000 ...
   |     0.29s 100
   |     0.55s 200
   |     0.77s 300
   |     0.89s 400
   |     0.98s 500
   |     1.21s 600
   |     1.26s 700
   |     1.53s 800
   |     1.62s 900
   |     1.75s 1000
   |     1.92s 1100
   |     2.04s 1200
   |     2.23s 1300
   |     2.76s 1400
   |     3.22s 1500
   |     3.48s 1600
   |     3.76s 1700
   |     3.93s 1800
   |     4.21s 1900
   |     4.49s 2000
   |     4.70s 2100
   |     4.87s 2200
   |     5.31s 2300
   |     6.33s 2400
   |     6.64s 2500
   |     6.90s 2600
   |     7.22s 2700
  7.24s Done: 2707 results
In [22]:
# ignore this
# S.tweakPerformance(yarnRatio=1)

We are going to run this query in shallow mode.

In [23]:
results = A.search(query, shallow=True)
    11s 671 results

Shallow mode tends to be quicker, but that does not always materialize. The number of results agrees with the first query. Yet we have been lucky, because we required the word in the gap to be in the same verse as the first word in the phrase. What if we require if it is the last word in the phrase?

In [24]:
query = '''
p:phrase
    =: wFirst:word
    wLast:word
    :=

wGap:word
wFirst < wGap
wLast > wGap

p || wGap

v:verse

v [[ wLast
v [[ wGap
'''
In [25]:
results = A.search(query, shallow=True)
    11s 660 results

Then we would not have found all results.

So, this road, although doable, is much less comfortable, performance-wise and logic-wise.

Check the gaps

In this misty landscape of gaps we need some corroboration that we found the right results.

  1. is every node in gapQueryResults a phrase?
  2. does every phrase in the gapQueryResults have a gap?
  3. is every gapped phrase contained in gapQueryResults?

We check all this by hand coding.

Here is a function that checks whether a phrase has a gap. If the distance between its end points is greater than the number of words it contains, it must have a gap.

In [26]:
def hasGap(p):
    words = L.d(p, otype='word')
    return words[-1] - words[0] + 1 > len(words)

Now we can perform the checks.

In [27]:
otypesGood = True
haveGaps = True

for p in gapQueryResults:
    otype = F.otype.v(p)
    if otype != 'phrase':
        print(f'Non phrase detected: {p}) is a {otype}')
        otypesGood = False
        break

    if not hasGap(p):
        print(f'Phrase without a gap: {p}')
        A.pretty(p)
        haveGaps = False
        break

print(f'{len(gapQueryResults)} nodes in query result')
if otypesGood:
    print('1. all nodes are phrases')
if haveGaps:
    print('2. all nodes have gaps')

inResults = True
for p in F.otype.s('phrase'):
    if hasGap(p):
        if p not in gapQueryResults:
            print(f'Gapped phrase outside query results: {p}')
            A.pretty(p)
            inResults = False
            break
            
if inResults:
    print('3. all gapped phrases are contained in the results')
671 nodes in query result
1. all nodes are phrases
2. all nodes have gaps
3. all gapped phrases are contained in the results

Note that by hand coding we can get the gapped phrases much more quickly and securely!

Custom sets for (non-)gapped phrases

We have obtained a set with all gapped phrases, and we have paid a price:

  • either an expensive query,
  • or an inconvenient bit of hand coding.

It would be nice if we could kick-start our queries using this set as a given. And that is exactly what we are going to do now.

We make two custom sets and give them a name, gapphrase for gapped phrases and conphrase for non-gapped phrases (consecutive phrases).

In [28]:
customSets = dict(
    gapphrase=gapQueryResults,
    conphrase=set(F.otype.s('phrase')) - gapQueryResults,
)

Suppose we want all verbs that occur in a gapped phrase.

In [29]:
query = '''
gapphrase
  word sp=verb
'''

Note that we have used the foreign name gapphrase in our search template, instead of phrase.

But we can still run search(), provided we tell it what we mean by gapphrase. We do that by passing the sets parameter to search(), which should be a dictionary of sets. Search will look up gapphrase in this dictionary, and will use its value, which should be a node set. That way, it understands that the expression gapphrase stands for the nodes in the given node set.

Here we go:

In [30]:
results = A.search(query, sets=customSets)
  0.43s 93 results
In [31]:
A.show(results, start=1, end=3, condenseType="clause")

result 1

Genesis 30:20 
clause
phrase
phrase
phrase
phrase

result 2

Genesis 30:35 
clause
phrase
sp=conj
phrase
phrase
sp=prep
sp=art
sp=art
phrase
phrase
sp=conj
phrase
sp=prep
sp=subs
sp=art
sp=art
sp=conj
sp=art
phrase
clause
phrase
sp=conj
phrase
sp=subs
phrase
sp=prep
sp=art

result 3

Genesis 30:35 
clause
phrase
sp=conj
phrase
phrase
sp=prep
sp=art
sp=art
phrase
phrase
sp=conj
phrase
sp=prep
sp=subs
sp=art
sp=art
sp=conj
sp=art
phrase
clause
phrase
sp=conj
phrase
sp=subs
phrase
sp=prep
sp=art

That looks good.

We can also apply feature conditions to gapphrase:

In [32]:
query = '''
gapphrase function=Subj
'''
results = A.search(query, sets=customSets)
A.table(results, start=1, end=3)
  0.00s 176 results
npphrase
1Genesis 2:25שְׁנֵיהֶם֙ הָֽאָדָ֖ם וְאִשְׁתֹּ֑ו
2Genesis 4:4הֶ֨בֶל גַם־ה֛וּא
3Genesis 7:14הֵ֜מָּה וְכָל־הַֽחַיָּ֣ה לְמִינָ֗הּ וְכָל־הַבְּהֵמָה֙ לְמִינָ֔הּ וְכָל־הָרֶ֛מֶשׂ לְמִינֵ֑הוּ וְכָל־הָעֹ֣וף לְמִינֵ֔הוּ כֹּ֖ל צִפֹּ֥ור כָּל־כָּנָֽף׃
In [33]:
A.show(results, start=1, end=3, condenseType="clause")

result 1

Genesis 2:25 
clause
phrase
function=Conj
phrase
function=Pred
phrase
function=Subj
phrase
function=PreC

result 2

Genesis 4:4 
clause
phrase
function=Conj
phrase
function=Subj
phrase
function=Pred
phrase
function=Subj

result 3

Genesis 7:14 
clause
phrase
function=Subj
phrase
function=Subj
phrase
function=Subj
phrase
function=Subj
phrase
function=Subj
phrase
function=Subj
clause
phrase
function=Subj
phrase
function=Subj
phrase
function=Subj
phrase
function=Subj

We reduce the details by setting the baseType to phrase. The highlighted phrases will now get a yellow background.

In [34]:
A.show(results, start=3, end=3, baseTypes="phrase")

result 3

Genesis 7:14 
verse
sentence
clause
phrase הֵ֜מָּה וְכָל־הַֽחַיָּ֣ה
function=Subj
phrase לְמִינָ֗הּ
function=Subj
phrase וְ
function=Subj
phrase כָל־הַבְּהֵמָה֙
function=Subj
phrase לְמִינָ֔הּ
function=Subj
phrase וְ
function=Subj
phrase כָל־הָרֶ֛מֶשׂ
function=Subj
clause
phrase הָ
function=Rela
phrase רֹמֵ֥שׂ
function=PreC
phrase עַל־הָאָ֖רֶץ
function=Cmpl
clause
phrase לְמִינֵ֑הוּ
function=Subj
phrase וְ
function=Subj
phrase כָל־הָעֹ֣וף
function=Subj
phrase לְמִינֵ֔הוּ
function=Subj
phrase כֹּ֖ל צִפֹּ֥ור כָּל־כָּנָֽף׃
function=Subj

We reduce the details by setting the baseType to phrase_atom. The highlighted phrases will not get a yellow background now.

In [35]:
A.show(results, start=3, end=3, baseTypes={"phrase_atom"})

result 3

Genesis 7:14 
verse
sentence
clause
phrase
function=Subj
הֵ֜מָּה וְכָל־הַֽחַיָּ֣ה
phrase
function=Subj
לְמִינָ֗הּ
phrase
function=Subj
וְ
phrase
function=Subj
כָל־הַבְּהֵמָה֙
phrase
function=Subj
לְמִינָ֔הּ
phrase
function=Subj
וְ
phrase
function=Subj
כָל־הָרֶ֛מֶשׂ
clause
phrase
function=Rela
הָ
phrase
function=PreC
רֹמֵ֥שׂ
phrase
function=Cmpl
עַל־הָאָ֖רֶץ
clause
phrase
function=Subj
לְמִינֵ֑הוּ
phrase
function=Subj
וְ
phrase
function=Subj
כָל־הָעֹ֣וף
phrase
function=Subj
לְמִינֵ֔הוּ
phrase
function=Subj
כֹּ֖ל צִפֹּ֥ור כָּל־כָּנָֽף׃

Two-phrase clauses

We can find the gaps, but do our minds always reckon with gaps? Gaps cause unexpected semantics. Here is a little puzzle.

Suppose we want to count the clauses consisting of exactly two phrases.

Here follows a little journey. We use a query to find the clauses, check the result with hand-coding, scratch our heads, refine the query, the hand-coding and our question until we are satisfied.

Attempt 1

By query

The following template should do it: a clause, starting with a phrase, followed by an adjacent phrase, which terminates the clause.

In [36]:
query = '''
clause
    =: phrase
    <: phrase
    :=
'''
In [37]:
# ignore this
# S.tweakPerformance(yarnRatio=1.2)
In [38]:
S.study(query)
  0.00s Checking search template ...
  0.00s Setting up search space for 3 objects ...
  0.09s Constraining search space with 5 relations ...
  0.61s 	2 edges thinned
  0.61s Setting up retrieval plan with strategy small_choice_multi ...
  0.63s Ready to deliver results from 264303 nodes
Iterate over S.fetch() to get the results
See S.showPlan() to interpret the results
In [39]:
S.showPlan(details=True)
Search with 3 objects and 5 relations
Results are instantiations of the following objects:
node  0-clause                                         88101   choices
node  1-phrase                                         88101   choices
node  2-phrase                                         88101   choices
Performance parameters:
	yarnRatio            =    1.25
	tryLimitFrom         =      40
	tryLimitTo           =      40
Instantiations are computed along the following relations:
node                                  0-clause         88101   choices
edge        0-clause           :=     2-phrase             1.0 choices (thinned)
edge        0-clause           [[     2-phrase             0   choices
edge        2-phrase           :>     1-phrase             0.2 choices
edge        1-phrase           ]]     0-clause             0   choices
edge        1-phrase           =:     0-clause             0   choices
  3.29s The results are connected to the original search template as follows:
 0     
 1 R0  clause
 2 R1      =: phrase
 3 R2      <: phrase
 4         :=
 5     
In [40]:
results = A.search(query)
A.table(results, end=7)
  1.00s 23483 results
npclausephrasephrase
1Genesis 1:3יְהִ֣י אֹ֑ור יְהִ֣י אֹ֑ור
2Genesis 1:4כִּי־טֹ֑וב כִּי־טֹ֑וב
3Genesis 1:7אֲשֶׁר֙ מִתַּ֣חַת לָרָקִ֔יעַ אֲשֶׁר֙ מִתַּ֣חַת לָרָקִ֔יעַ
4Genesis 1:7אֲשֶׁ֖ר מֵעַ֣ל לָרָקִ֑יעַ אֲשֶׁ֖ר מֵעַ֣ל לָרָקִ֑יעַ
5Genesis 1:10כִּי־טֹֽוב׃ כִּי־טֹֽוב׃
6Genesis 1:11מַזְרִ֣יעַ זֶ֔רַע מַזְרִ֣יעַ זֶ֔רַע
7Genesis 1:12כִּי־טֹֽוב׃ כִּי־טֹֽוב׃

If we want to have the clauses only, we run it in shallow mode:

In [41]:
clausesByQuery = sorted(A.search(query, shallow=True))
  1.03s 23483 results

Note result 3 above: it seems we have 3 phrases. Yet there are only 2. We take a closer look:

In [42]:
focus = results[2][0]
A.pretty(focus)

One phrase is chunked into two phrase atoms, let's make that more clear:

In [43]:
A.pretty(focus, showChunks=True)
Genesis 1:7 
clause
clause_atom
phrase
phrase_atom
phrase
phrase_atom
phrase

By hand

Let us check this with a piece of hand-written code. We want clauses that consist of exactly two phrases.

In [44]:
indent(reset=True)
info('counting ...')

clausesByHand = []
for clause in F.otype.s('clause'):
    phrases = L.d(clause, otype='phrase')
    if len(phrases) == 2:
        clausesByHand.append(clause)
clausesByHand = sorted(clausesByHand)
info(f'Done: found {len(clausesByHand)}')
  0.00s counting ...
  0.61s Done: found 23862

The difference

Strange, we end up with more cases. What is happening? Let us compare the results. We look at the first result where both methods diverge.

We put the difference finding in a little function.

In [45]:
def showDiff(queryResults, handResults):
    diff = [x for x in zip(queryResults, handResults) if x[0] != x[1]]
    if not diff:
        print(f'''
{len(queryResults):>6} queryResults
         are identical with
{len(handResults):>6} handResults
''')
        return
    (rQuery, rHand) = diff[0]
    if rQuery < rHand:
        print(f'clause {rQuery} is a query result but not found by hand')
        toShow = rQuery
    else:
        print(f'clause {rHand} is not a query result but has been found by hand')
        toShow = rHand
    colors = ['aqua', 'aquamarine', 'khaki', 'lavender', 'yellow']
    highlights = {}
    for (i, phrase) in enumerate(L.d(toShow, otype='phrase')):
        highlights[phrase] = colors[i % len(colors)]
        # for atom in L.d(phrase, otype='phrase_atom'):
        #     highlights[atom] = colors[i % len(colors)]
    A.pretty(toShow, withNodes=True, suppress={'lex', 'sp', 'vt', 'vs'}, highlights=highlights, baseTypes="phrase_atom")
In [46]:
showDiff(clausesByQuery, clausesByHand)
clause 427931 is not a query result but has been found by hand
Genesis 4:14 
clause:427931
phrase:652631
כָל־
clause:427931
phrase:652633
יַֽהַרְגֵֽנִי׃

Lo and behold:

  • the hand-written code is right in a sense: this is a clause that consists exactly of two phrases.
  • the query is also right in a sense: the two phrases are not adjacent: there is a gap in the clause between them!

Attempt 2

By hand

We modify the hand-written code such that only clauses qualify if the two phrases are adjacent.

In [47]:
indent(reset=True)
info('counting ...')

clausesByHand2 = []
for clause in F.otype.s('clause'):
    phrases = L.d(clause, otype='phrase')
    if len(phrases) == 2:
        if L.d(phrases[0], otype='word')[-1] + 1 == L.d(phrases[1], otype='word')[0]:
            clausesByHand2.append(clause)
clausesByHand2 = sorted(clausesByHand2)
info(f'Done: found {len(clausesByHand2)}')
  0.00s counting ...
  0.71s Done: found 23399

The difference

Now we have less cases. What is going on?

In [48]:
showDiff(clausesByQuery, clausesByHand2)
clause 428692 is a query result but not found by hand
Genesis 14:16 
clause:428692
phrase:655060
וְ
phrase:655061
גַם֩ אֶת־לֹ֨וט
phrase:655061
אָחִ֤יו
phrase:655061
וּ
phrase:655061
רְכֻשֹׁו֙
phrase:655062
הֵשִׁ֔יב
phrase:655061
וְ
phrase:655061
גַ֥ם אֶת־הַנָּשִׁ֖ים וְאֶת־הָעָֽם׃

Observe:

This clause has three phrases, but the third one lies inside the second one.

  • the hand-written code is right in a sense: this clause has three phrases.
  • the query is right in a sense: it contains two adjacent phrases that together span the whole clause.

Attempt 3

By query

Can we adjust the pattern to exclude cases like this? Yes, with custom sets, see advanced.

Instead of looking through all phrases, we can just consider non gapped phrases only.

Earlier in this notebook we have constructed the set of non-gapped phrases and put it under the name conphrase in the custom sets.

In [49]:
query = '''
clause
    =: conphrase
    <: conphrase
    :=
'''

clausesByQuery2 = sorted(A.search(query, sets=customSets, shallow=True))
  1.10s 23327 results

The difference

There is still a difference.

In [50]:
showDiff(clausesByQuery2, clausesByHand2)
clause 428374 is not a query result but has been found by hand
Genesis 10:14 
clause:428374
phrase:654063
וְֽ
phrase:654064
אֶת־פַּתְרֻסִ֞ים וְאֶת־כַּסְלֻחִ֗ים
clause:428374
phrase:654064
וְ
phrase:654064
אֶת־כַּפְתֹּרִֽים׃ ס

Observe:

This clause has two phrases, the second one has a gap, which coincides with a gap in the clause.

  • the hand-written code is right in a sense: this clause has two phrases, adjacent, and they span the whole clause, nothing left out.
  • the query is right in a sense: the second phrase is not consecutive.

Attempt 4

By hand

We modify the hand-written code, so that only consecutive clauses qualify.

In [51]:
indent(reset=True)
info('counting ...')

clausesByHand3 = []
for clause in F.otype.s('clause'):
    if hasGap(clause):
        continue
    phrases = L.d(clause, otype='phrase')
    if len(phrases) == 2:
        if L.d(phrases[0], otype='word')[-1] + 1 == L.d(phrases[1], otype='word')[0]:
            clausesByHand3.append(clause)
clausesByHand3 = sorted(clausesByHand3)
info(f'Done: found {len(clausesByHand3)}')
  0.00s counting ...
  1.00s Done: found 23327

The difference

Now the number of results agree. But are they really the same?

In [52]:
showDiff(clausesByQuery2, clausesByHand3)
 23327 queryResults
         are identical with
 23327 handResults

Conclusion

It took four attempts to arrive at the final concept of things that we were looking for.

Sometimes the search template had to be modified, sometimes the hand-written code.

The interplay and systematic comparison between the attempts helped to spot all relevant configurations of phrases within clauses.

Spans

Here is another cause of wrong query results: there are sentences that span multiple verses. Such sentences are not contained in any verse. That makes that they are easily missed out in queries.

We describe a scenario where that happens.

Mother clauses

A clause and its mother do not have to be in the same verse. We are going to fetch are the cases where they are in different verses.

All mother clauses

But first we fetch all pairs of clauses connected by a mother edge.

In [53]:
query = '''
clause
-mother> clause
'''
allMotherPairs = A.search(query)
A.table(results, end=7)
  0.15s 13907 results
npclausephrasephrase
1Genesis 1:3יְהִ֣י אֹ֑ור יְהִ֣י אֹ֑ור
2Genesis 1:4כִּי־טֹ֑וב כִּי־טֹ֑וב
3Genesis 1:7אֲשֶׁר֙ מִתַּ֣חַת לָרָקִ֔יעַ אֲשֶׁר֙ מִתַּ֣חַת לָרָקִ֔יעַ
4Genesis 1:7אֲשֶׁ֖ר מֵעַ֣ל לָרָקִ֑יעַ אֲשֶׁ֖ר מֵעַ֣ל לָרָקִ֑יעַ
5Genesis 1:10כִּי־טֹֽוב׃ כִּי־טֹֽוב׃
6Genesis 1:11מַזְרִ֣יעַ זֶ֔רַע מַזְרִ֣יעַ זֶ֔רַע
7Genesis 1:12כִּי־טֹֽוב׃ כִּי־טֹֽוב׃

Mother in another verse

Now we modify the query to the effect that mother and daughter must sit in distinct verses.

In [54]:
query = '''
cm:clause
-mother> cd:clause

v1:verse
v2:verse
v1 # v2

cm ]] v1
cd ]] v2
'''
diffMotherPairs = A.search(query)
A.table(diffMotherPairs, end=7, skipCols="3 4", withPassage="1 2")
  0.27s 721 results
nclauseclause
1Genesis 1:18  וְלִמְשֹׁל֙ בַּיֹּ֣ום וּבַלַּ֔יְלָה Genesis 1:17  לְהָאִ֖יר עַל־הָאָֽרֶץ׃
2Genesis 2:7  וַיִּיצֶר֩ יְהוָ֨ה אֱלֹהִ֜ים אֶת־הָֽאָדָ֗ם עָפָר֙ מִן־הָ֣אֲדָמָ֔ה Genesis 2:4  בְּיֹ֗ום
3Genesis 7:3  לְחַיֹּ֥ות זֶ֖רַע עַל־פְּנֵ֥י כָל־הָאָֽרֶץ׃ Genesis 7:2  מִכֹּ֣ל׀ הַבְּהֵמָ֣ה הַטְּהֹורָ֗ה תִּֽקַּח־לְךָ֛ שִׁבְעָ֥ה שִׁבְעָ֖ה אִ֣ישׁ וְאִשְׁתֹּ֑ו
4Genesis 22:17  כִּֽי־בָרֵ֣ךְ אֲבָרֶכְךָ֗ Genesis 22:16  כִּ֗י
5Genesis 24:44  הִ֣וא הָֽאִשָּׁ֔ה Genesis 24:43  הָֽעַלְמָה֙
6Genesis 27:45  עַד־שׁ֨וּב אַף־אָחִ֜יךָ מִמְּךָ֗ Genesis 27:44  עַ֥ד אֲשֶׁר־תָּשׁ֖וּב חֲמַ֥ת אָחִֽיךָ׃
7Genesis 36:16  אַלּֽוּף־קֹ֛רַח אַלּ֥וּף גַּעְתָּ֖ם אַלּ֣וּף עֲמָלֵ֑ק Genesis 36:15  בְּנֵ֤י אֱלִיפַז֙ בְּכֹ֣ור עֵשָׂ֔ו אַלּ֤וּף תֵּימָן֙ אַלּ֣וּף אֹומָ֔ר אַלּ֥וּף צְפֹ֖ו אַלּ֥וּף קְנַֽז׃

Mother in same verse

As a check, we modify the latter query and require v1 and v2 to be the same verse, to get the mother pairs of which both members are in the same verse.

In [55]:
query = '''
cm:clause
-mother> cd:clause

v1:verse
v2:verse
v1 = v2

cm ]] v1
cd ]] v2
'''
sameMotherPairs = A.search(query)
A.table(sameMotherPairs, end=7, skipCols="3 4", withPassage="1 2")
  0.29s 13160 results
nclauseclause
1Genesis 1:4  כִּי־טֹ֑וב Genesis 1:4  וַיַּ֧רְא אֱלֹהִ֛ים אֶת־הָאֹ֖ור
2Genesis 1:10  כִּי־טֹֽוב׃ Genesis 1:10  וַיַּ֥רְא אֱלֹהִ֖ים
3Genesis 1:12  כִּי־טֹֽוב׃ Genesis 1:12  וַיַּ֥רְא אֱלֹהִ֖ים
4Genesis 1:14  לְהַבְדִּ֕יל בֵּ֥ין הַיֹּ֖ום וּבֵ֣ין הַלָּ֑יְלָה Genesis 1:14  יְהִ֤י מְאֹרֹת֙ בִּרְקִ֣יעַ הַשָּׁמַ֔יִם
5Genesis 1:15  לְהָאִ֖יר עַל־הָאָ֑רֶץ Genesis 1:15  וְהָי֤וּ לִמְאֹורֹת֙ בִּרְקִ֣יעַ הַשָּׁמַ֔יִם
6Genesis 1:17  לְהָאִ֖יר עַל־הָאָֽרֶץ׃ Genesis 1:17  וַיִּתֵּ֥ן אֹתָ֛ם אֱלֹהִ֖ים בִּרְקִ֣יעַ הַשָּׁמָ֑יִם
7Genesis 1:18  וּֽלֲהַבְדִּ֔יל בֵּ֥ין הָאֹ֖ור וּבֵ֣ין הַחֹ֑שֶׁךְ Genesis 1:18  וְלִמְשֹׁל֙ בַּיֹּ֣ום וּבַלַּ֔יְלָה

The difference

Let's check if the numbers add up:

  • the first query asked for all pairs
  • the second query asked for pairs with members in different verses
  • the third query asked for pairs with members in the same verse

Then the results of the second and third query combined should equal the results of the first query.

That makes sense.

Still, let's check:

In [56]:
discrepancy = len(allMotherPairs) - len(diffMotherPairs) - len(sameMotherPairs)
print(discrepancy)
26

The numbers do not add up. We are missing cases. Why?

Clauses may cross verse boundaries. In that case they are not part of a verse, and hence our latter two queries do not detect them. Let's count how many verse boundary crossing clauses there are.

In [57]:
query = '''
clause
/with/
v1:verse
&& ..
v2:verse
&& ..
v1 < v2
/-/
'''
results = A.search(query)
  1.29s 50 results

You might think we can speed up the query by requiring v1 <: v2 (both verses are adjacent). There are less possibilities to consider, to maybe we gain something.

In [58]:
query = '''
clause
/with/
v1:verse
&& ..
v2:verse
&& ..
v1 <: v2
/-/
'''
results = A.search(query)
  1.43s 49 results

Indeed, slightly faster, but one result less! How can that be?

There must be a clause that spans at least two verses and in doing so, skips at least one verse.

Let's find that one:

In [59]:
query = '''
clause
/with/
v1:verse
&& ..
v2:verse
|| ..
v3:verse
&& ..
v1 < v2
v2 < v3
v1 < v3
/-/
'''
resultsX = A.search(query)
  2.47s 1 result
In [60]:
A.table(resultsX)
A.show(resultsX, baseTypes="clause_atom")
npclause
11_Kings 8:41וְגַם֙ אֶל־הַנָּכְרִ֔י אַתָּ֞ה תִּשְׁמַ֤ע הַשָּׁמַ֨יִם֙ מְכֹ֣ון שִׁבְתֶּ֔ךָ

result 1

1_Kings 8:41 
verse
sentence
clause
וְגַם֙ אֶל־הַנָּכְרִ֔י
clause
אֲשֶׁ֛ר לֹא־מֵעַמְּךָ֥ יִשְׂרָאֵ֖ל ה֑וּא
clause
וּבָ֛א מֵאֶ֥רֶץ רְחֹוקָ֖ה לְמַ֥עַן שְׁמֶֽךָ׃
1_Kings 8:43 
verse
sentence
clause
אַתָּ֞ה תִּשְׁמַ֤ע הַשָּׁמַ֨יִם֙ מְכֹ֣ון שִׁבְתֶּ֔ךָ
sentence
clause
וְעָשִׂ֕יתָ כְּכֹ֛ל
clause
אֲשֶׁר־יִקְרָ֥א אֵלֶ֖יךָ הַנָּכְרִ֑י
sentence
clause
לְמַ֣עַן יֵדְעוּן֩ כָּל־עַמֵּ֨י הָאָ֜רֶץ אֶת־שְׁמֶ֗ךָ
clause
לְיִרְאָ֤ה אֹֽתְךָ֙ כְּעַמְּךָ֣ יִשְׂרָאֵ֔ל
clause
וְלָדַ֕עַת
clause
כִּי־שִׁמְךָ֣ נִקְרָ֔א עַל־הַבַּ֥יִת הַזֶּ֖ה
clause
אֲשֶׁ֥ר בָּנִֽיתִי׃

A more roundabout way to find the same clauses:

In [61]:
query = '''
clause
    =: first:word
    last:word
    :=
v1:verse
    w1:word
v2:verse
    w2:word
    
first = w1
last = w2
v1 # v2
'''
results = A.search(query)
  2.58s 50 results

Some of these verse spanning clauses do not have mothers or are not mothers. Let's count the cases where two clauses are in a mother relation and at least one of them spans a verse.

We need two queries for that. These queries are almost similar. One retrieves the clause pairs where the mother crosses verse boundaries, and the other where the daughter does so.

But we are programmers. We do not have to repeat ourselves:

In [62]:
queryCommon = '''
c1:clause
-mother> c2:clause

c3:clause
/with/
v1:verse
&& ..
v2:verse
&& ..
v1 < v2
/-/
'''

query1 = f'''
{queryCommon}
c1 = c3
'''
query2 = f'''
{queryCommon}
c2 = c3
'''

results1 = A.search(query1, silent=True)
results2 = A.search(query2, silent=True)
spannersByQuery = {(r[0], r[1]) for r in results1 + results2}
print(f'{len(spannersByQuery):>3} spanners are missing')
print(f'{discrepancy:>3} missing cases were detected before')
print(f'{discrepancy - len(spannersByQuery):>3} is the resulting disagreement')
 26 spanners are missing
 26 missing cases were detected before
  0 is the resulting disagreement

We may find the mother clause pairs in which it least one member is verse spanning by hand-coding in an easier way:

Starting with the set of all mother pairs, we filter out any pair that has a verse spanner.

In [63]:
spannersByHand = set()

for (c1, c2) in allMotherPairs:
    if not (
        L.u(c1, otype='verse')
        and
        L.u(c2, otype='verse')
    ):
        spannersByHand.add((c1, c2))
        
len(spannersByHand)
Out[63]:
26

And, to be completely sure:

In [64]:
spannersByHand == spannersByQuery
Out[64]:
True

By custom sets

If we are content with the clauses that do not span verses, we can put them in a set, and modify the queries by replacing clause by conclause and bind the right set to it.

Here we go. In one cell we run the queries to get all pairs, the mother-daughter-in-separate-verses pairs, and the mother-daughter-in-same-verses pair and we do the math of checking.

In [65]:
conClauses = {c for c in F.otype.s('clause') if L.u(c, otype='verse')}
customSets = dict(conclause=conClauses)

print('All pairs')
allPairs = A.search('''
conclause
-mother> conclause
''', 
    sets=customSets,
)

print('Different verse pairs')
diffPairs = A.search('''
cm:conclause
-mother> cd:conclause

v1:verse
v2:verse
v1 # v2

cm ]] v1
cd ]] v2
''',
    sets=customSets,
)

print('Same verse pairs')
samePairs = A.search('''
cm:conclause
-mother> cd:conclause

v1:verse
v2:verse
v1 = v2

cm ]] v1
cd ]] v2
''',
    sets=customSets,
)

allPairSet = set(allPairs)
diffPairSet = {(r[0], r[1]) for r in diffPairs}
samePairSet = {(r[0], r[1]) for r in samePairs}

print(f'Intersection same-verse/different-verse pairs: {samePairSet & diffPairSet}')
print(f'All pairs is union of same-verse/different-verse pairs: {allPairSet == (samePairSet | diffPairSet)}')
All pairs
  0.17s 13881 results
Different verse pairs
  0.28s 721 results
Same verse pairs
  0.28s 13160 results
Intersection same-verse/different-verse pairs: set()
All pairs is union of same-verse/different-verse pairs: True

Lessons

  • mix programming with composing queries;
  • a good way to do so is custom sets;
  • use programming for processing results;
  • find the balance between queries and hand-coding.

All steps

  • start your first step in mastering the bible computationally
  • display become an expert in creating pretty displays of your text structures
  • search turbo charge your hand-coding with search templates

advanced sets relations quantifiers fromMQL rough gaps

You have now finished the search tutorial.

Share the work!


  • exportExcel make tailor-made spreadsheets out of your results
  • share draw in other people's data and let them use yours
  • export export your dataset as an Emdros database

CC-BY Dirk Roorda