Exercise 1: Pseudo-feedback with Two-component Mixture Model

First, let's import the Python bindings for MeTA:

In [1]:
import metapy

If you don't have metapy installed, you can install it with a

pip install metapy

on the command line on Linux, macOS, or Windows for either Python 2.7 or Python 3.x. (I will be using Python 3.6 in this tutorial.)

Double-check that you are running the latest version. Right now, that should be 0.2.10.

In [2]:
metapy.__version__
Out[2]:
'0.2.10'

Now, let's set MeTA to log to standard error so we can see progress output for long-running commands. (Only do this once, or you'll get double the output.)

In [3]:
metapy.log_to_stderr()

Now, let's download all of the files we need for the tutorial.

In [4]:
import urllib.request
import os
import tarfile

if not os.path.exists('sigir18-tutorial.tar.gz'):
    urllib.request.urlretrieve('https://meta-toolkit.org/data/2018-06-25/sigir18-tutorial.tar.gz',
                               'sigir18-tutorial.tar.gz')
    
if not os.path.exists('data'):
    with tarfile.open('sigir18-tutorial.tar.gz', 'r:gz') as files:
        files.extractall()

Let's index our data using the InvertedIndex format. In a search engine, we want to quickly determine what documents mention a specific query term, so the InvertedIndex stores a mapping from term to a list of documents that contain that term (along with how many times they do).

In [5]:
inv_idx = metapy.index.make_inverted_index('cranfield.toml')
 > Counting lines in file: [=================================] 100% ETA 00:00:00 
1529953996: [info]     Creating index: cranfield-idx/inv (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:119)
 > Tokenizing Docs: [>                                       ]   0% ETA 00:00:00 
1529953996: [warning]  Empty document (id = 470) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:228)
 
1529953996: [warning]  Empty document (id = 994) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:228)
 > Tokenizing Docs: [========================================] 100% ETA 00:00:00 
 > Merging: [================================================] 100% ETA 00:00:00 
1529953996: [info]     Created uncompressed postings file cranfield-idx/inv/postings.index (197.770000 KB) (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:148)
 > Compressing postings: [===================================] 100% ETA 00:00:00 
1529953996: [info]     Created compressed postings file (168.060000 KB) (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:279)
1529953996: [info]     Done creating index: cranfield-idx/inv (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/inverted_index.cpp:166)

This may take a minute at first, since the index needs to be built. Subsequent calls to make_inverted_index with this config file will simply load the index, which will not take any time.

Here's how we can interact with the index object:

In [6]:
inv_idx.num_docs()
Out[6]:
1400
In [7]:
inv_idx.unique_terms()
Out[7]:
4137
In [8]:
inv_idx.avg_doc_length()
Out[8]:
87.17857360839844
In [9]:
inv_idx.total_corpus_terms()
Out[9]:
122050

Let's search our index. We'll start by creating a ranker:

In [10]:
ranker = metapy.index.DirichletPrior()

Now we need a query. Let's create an example query.

In [11]:
query = metapy.index.Document()
query.content("flow equilibrium")

Now we can use this to search our index like so:

In [12]:
top_docs = ranker.score(inv_idx, query, num_results=5)
top_docs
Out[12]:
[(235, 1.2931444644927979),
 (1251, 1.256299614906311),
 (316, 1.1081531047821045),
 (655, 1.0878994464874268),
 (574, 1.076568841934204)]

We are returned a ranked list of (doc_id, score) pairs. The scores are from the ranker, which in this case was Okapi BM25. Since the tutorial.toml file we created for the cranfield dataset has store-full-text = true, we can verify the content of our top documents by inspecting the document metadata field "content".

In [13]:
for num, (d_id, _) in enumerate(top_docs):
    content = inv_idx.metadata(d_id).get('content')
    print("{}. {}...\n".format(num + 1, content[0:250]))
1. criteria for thermodynamic equilibrium in gas flow . when gases flow at high velocity, the rates of internal processes may not be fast enough to maintain thermodynamic equilibrium .  by defining quasi-equilibrium in flow as the condition in which the...

2. on the approach to chemical and vibrational equilibrium behind a strong normal shock wave . the concurrent approach to chemical and vibrational equilibrium of a pure diatomic gas passing through a strong normal shock wave is investigated .  it is dem...

3. non-equilibrium flow of an ideal dissociating gas . the theory of an'ideal dissociating'gas developed by lighthill/1957/for conditions of thermodynamic equilibrium is extended to non-equilibrium conditions by postulating a simple rate equation for th...

4. departure from dissociation equilibrium in a hypersonic nozzle . the equations of motion for the flow of an ideal dissociating gas through a nearly conical nozzle have been solved numerically, assuming a simple equation for the rate of dissociation, ...

5. atomic recombination in a hypersonic wind tunnel nozzle . the flow of an ideal dissociating gas through a nearly conical nozzle is considered . the equations of one-dimensional motion are solved numerically assuming a simple rate equation together wi...

Since we have the queries file and relevance judgements, we can do an IR evaluation.

In [14]:
ev = metapy.index.IREval('cranfield.toml')

We will loop over the queries file and add each result to the IREval object ev.

In [15]:
def evaluate_ranker(ranker, ev, num_results):
    ev.reset_stats()
    with open('data/cranfield/cranfield-queries.txt') as query_file:
        for query_num, line in enumerate(query_file):
            query.content(line.strip())
            results = ranker.score(inv_idx, query, num_results)                            
            avg_p = ev.avg_p(results, query_num + 1, num_results)
            print("Query {} average precision: {}".format(query_num + 1, avg_p))
            
evaluate_ranker(ranker, ev, 10)
Query 1 average precision: 0.19
Query 2 average precision: 0.5433333333333332
Query 3 average precision: 0.6541666666666666
Query 4 average precision: 0.5
Query 5 average precision: 0.35
Query 6 average precision: 0.0625
Query 7 average precision: 0.10666666666666666
Query 8 average precision: 0.0
Query 9 average precision: 0.6984126984126983
Query 10 average precision: 0.0625
Query 11 average precision: 0.028571428571428574
Query 12 average precision: 0.18
Query 13 average precision: 0.0
Query 14 average precision: 0.5
Query 15 average precision: 0.7
Query 16 average precision: 0.08333333333333333
Query 17 average precision: 0.07142857142857142
Query 18 average precision: 0.3333333333333333
Query 19 average precision: 0.0
Query 20 average precision: 0.2685185185185185
Query 21 average precision: 0.0
Query 22 average precision: 0.0
Query 23 average precision: 0.04722222222222222
Query 24 average precision: 0.3333333333333333
Query 25 average precision: 0.6507936507936507
Query 26 average precision: 0.13888888888888887
Query 27 average precision: 0.0
Query 28 average precision: 0.0
Query 29 average precision: 0.14166666666666666
Query 30 average precision: 0.11904761904761904
Query 31 average precision: 0.0
Query 32 average precision: 0.023809523809523808
Query 33 average precision: 0.5317460317460317
Query 34 average precision: 0.20555555555555557
Query 35 average precision: 0.0
Query 36 average precision: 0.25
Query 37 average precision: 0.05555555555555555
Query 38 average precision: 0.01111111111111111
Query 39 average precision: 0.12857142857142856
Query 40 average precision: 0.07222222222222222
Query 41 average precision: 0.7666666666666666
Query 42 average precision: 0.1638095238095238
Query 43 average precision: 0.48333333333333334
Query 44 average precision: 0.0
Query 45 average precision: 0.155
Query 46 average precision: 0.5380952380952382
Query 47 average precision: 0.15083333333333332
Query 48 average precision: 0.1433333333333333
Query 49 average precision: 0.0
Query 50 average precision: 0.0
Query 51 average precision: 0.16666666666666666
Query 52 average precision: 0.027777777777777776
Query 53 average precision: 0.15857142857142856
Query 54 average precision: 0.05555555555555555
Query 55 average precision: 0.11499999999999999
Query 56 average precision: 0.05
Query 57 average precision: 0.01
Query 58 average precision: 0.15873015873015872
Query 59 average precision: 0.03571428571428571
Query 60 average precision: 0.4666666666666667
Query 61 average precision: 0.45999999999999996
Query 62 average precision: 0.0
Query 63 average precision: 0.0
Query 64 average precision: 0.5
Query 65 average precision: 0.0
Query 66 average precision: 0.0
Query 67 average precision: 0.0365079365079365
Query 68 average precision: 0.1
Query 69 average precision: 0.06666666666666667
Query 70 average precision: 0.1
Query 71 average precision: 0.03125
Query 72 average precision: 0.0
Query 73 average precision: 0.2861111111111111
Query 74 average precision: 0.020833333333333332
Query 75 average precision: 0.03333333333333333
Query 76 average precision: 0.028571428571428574
Query 77 average precision: 0.28888888888888886
Query 78 average precision: 0.6666666666666666
Query 79 average precision: 0.0
Query 80 average precision: 0.0
Query 81 average precision: 0.25
Query 82 average precision: 0.09
Query 83 average precision: 0.0625
Query 84 average precision: 0.3
Query 85 average precision: 0.05
Query 86 average precision: 0.41666666666666663
Query 87 average precision: 0.0
Query 88 average precision: 0.594047619047619
Query 89 average precision: 0.075
Query 90 average precision: 0.19
Query 91 average precision: 0.0873015873015873
Query 92 average precision: 0.4726984126984126
Query 93 average precision: 0.5
Query 94 average precision: 0.4
Query 95 average precision: 0.5
Query 96 average precision: 0.4463095238095239
Query 97 average precision: 0.15416666666666665
Query 98 average precision: 0.0
Query 99 average precision: 0.19642857142857142
Query 100 average precision: 0.2185185185185185
Query 101 average precision: 0.6180555555555555
Query 102 average precision: 0.08333333333333333
Query 103 average precision: 0.05555555555555555
Query 104 average precision: 0.1
Query 105 average precision: 0.35333333333333333
Query 106 average precision: 0.24666666666666667
Query 107 average precision: 0.10476190476190476
Query 108 average precision: 0.6261904761904761
Query 109 average precision: 0.0
Query 110 average precision: 0.0
Query 111 average precision: 0.07619047619047618
Query 112 average precision: 0.3611111111111111
Query 113 average precision: 0.10416666666666666
Query 114 average precision: 0.0
Query 115 average precision: 0.0
Query 116 average precision: 0.03333333333333333
Query 117 average precision: 0.0
Query 118 average precision: 0.041666666666666664
Query 119 average precision: 0.5
Query 120 average precision: 0.17724867724867724
Query 121 average precision: 0.4768707482993197
Query 122 average precision: 0.05925925925925926
Query 123 average precision: 0.0
Query 124 average precision: 0.0
Query 125 average precision: 0.02
Query 126 average precision: 0.20833333333333331
Query 127 average precision: 0.025
Query 128 average precision: 0.0
Query 129 average precision: 0.4773809523809524
Query 130 average precision: 0.3933333333333333
Query 131 average precision: 0.09375
Query 132 average precision: 0.6592063492063491
Query 133 average precision: 0.12976190476190477
Query 134 average precision: 0.5
Query 135 average precision: 0.38690476190476186
Query 136 average precision: 0.03333333333333333
Query 137 average precision: 0.1875
Query 138 average precision: 0.25
Query 139 average precision: 0.0
Query 140 average precision: 0.10833333333333334
Query 141 average precision: 0.05555555555555555
Query 142 average precision: 0.0
Query 143 average precision: 0.6111111111111112
Query 144 average precision: 0.14722222222222223
Query 145 average precision: 0.07142857142857142
Query 146 average precision: 0.41666666666666663
Query 147 average precision: 0.14666666666666667
Query 148 average precision: 0.125
Query 149 average precision: 0.18285714285714286
Query 150 average precision: 1.0
Query 151 average precision: 0.0
Query 152 average precision: 0.0
Query 153 average precision: 0.19999999999999998
Query 154 average precision: 1.0
Query 155 average precision: 0.06481481481481481
Query 156 average precision: 0.6365476190476189
Query 157 average precision: 0.43809523809523804
Query 158 average precision: 0.175
Query 159 average precision: 0.015625
Query 160 average precision: 0.2
Query 161 average precision: 0.43333333333333335
Query 162 average precision: 0.020833333333333332
Query 163 average precision: 0.21666666666666667
Query 164 average precision: 0.40208333333333335
Query 165 average precision: 0.16666666666666666
Query 166 average precision: 0.0
Query 167 average precision: 0.325
Query 168 average precision: 0.08333333333333333
Query 169 average precision: 0.125
Query 170 average precision: 0.4611111111111111
Query 171 average precision: 0.4888888888888889
Query 172 average precision: 0.5416666666666666
Query 173 average precision: 0.8333333333333333
Query 174 average precision: 0.03333333333333333
Query 175 average precision: 0.02222222222222222
Query 176 average precision: 0.0
Query 177 average precision: 0.475
Query 178 average precision: 0.49166666666666664
Query 179 average precision: 0.125
Query 180 average precision: 0.34523809523809523
Query 181 average precision: 0.2
Query 182 average precision: 0.7
Query 183 average precision: 0.5308730158730158
Query 184 average precision: 0.06938775510204082
Query 185 average precision: 0.6388888888888888
Query 186 average precision: 0.016666666666666666
Query 187 average precision: 0.16666666666666666
Query 188 average precision: 0.09
Query 189 average precision: 0.015873015873015872
Query 190 average precision: 0.1
Query 191 average precision: 0.014285714285714285
Query 192 average precision: 0.5
Query 193 average precision: 0.6481481481481481
Query 194 average precision: 0.08888888888888889
Query 195 average precision: 0.1111111111111111
Query 196 average precision: 0.03333333333333333
Query 197 average precision: 0.6666666666666666
Query 198 average precision: 0.3125
Query 199 average precision: 0.05208333333333333
Query 200 average precision: 0.24074074074074073
Query 201 average precision: 0.3
Query 202 average precision: 0.26
Query 203 average precision: 0.07222222222222222
Query 204 average precision: 0.0
Query 205 average precision: 0.75
Query 206 average precision: 0.3333333333333333
Query 207 average precision: 0.15
Query 208 average precision: 0.6829365079365081
Query 209 average precision: 0.02
Query 210 average precision: 0.27777777777777773
Query 211 average precision: 0.0125
Query 212 average precision: 0.36666666666666664
Query 213 average precision: 0.5583333333333333
Query 214 average precision: 0.08333333333333333
Query 215 average precision: 0.0
Query 216 average precision: 0.0
Query 217 average precision: 0.05833333333333333
Query 218 average precision: 0.0
Query 219 average precision: 0.014285714285714285
Query 220 average precision: 0.05333333333333333
Query 221 average precision: 0.24333333333333332
Query 222 average precision: 0.5148148148148147
Query 223 average precision: 0.44375
Query 224 average precision: 0.0
Query 225 average precision: 0.13333333333333333

Afterwards, we can get the mean average precision of all the queries.

In [16]:
dp_map = ev.map()
print("MAP: {}".format(dp_map))
MAP: 0.21512203955656342

Now, let's use the two-component mixture model we discussed as an implementation of pseudo-feedback for retrieval and see if it helps improve performance. The actual ranking function used here is KL-divergence, where the query model is adjusted to include pseudo-feedback from the retrieved documents.

In order to work, the ranker needs to be able to quickly determine what words were used in the feedback document set. The InvertedIndex does not provide fast access to this (since it is a mapping from term to documents, rather than from documents to terms), so we will want to first create a ForwardIndex to get the document -> terms mapping.

In [17]:
fwd_idx = metapy.index.make_forward_index('cranfield.toml')
 > Counting lines in file: [=================================] 100% ETA 00:00:00 
1529954010: [info]     Creating forward index: cranfield-idx/fwd (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:239)
 > Tokenizing Docs: [>                                       ]   0% ETA 00:00:00 
1529954010: [warning]  Empty document (id = 470) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:335)
 
1529954010: [warning]  Empty document (id = 994) generated! (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:335)
 > Tokenizing Docs: [========================================] 100% ETA 00:00:00 
 > Merging: [================================================] 100% ETA 00:00:00 
1529954010: [info]     Done creating index: cranfield-idx/fwd (/tmp/pip-req-build-m473bt6z/deps/meta/src/index/forward_index.cpp:278)

Now we can construct the KL-divergence pseudo-feedback ranker. The main components are:

  1. The forward index
  2. A base language-model ranker (here we'll use DirichletPrior)
  3. $\alpha$, the query interpolation parameter (how strongly do we prefer terms from the feedback model? default 0.5)
  4. $\lambda$, the language-model interpolation parameter (how strong is the background model in the two-component mixture? default 0.5)
  5. $k$, the number of documents to retrieve for the feedback set (default 10)
  6. max_terms, the number of terms from the feedback model to incorporate into the new query model (default 50)
In [18]:
feedback = metapy.index.KLDivergencePRF(fwd_idx, metapy.index.DirichletPrior())
In [19]:
evaluate_ranker(feedback, ev, 10)
Query 1 average precision: 0.13999999999999999
Query 2 average precision: 0.524047619047619
Query 3 average precision: 0.6642857142857143
Query 4 average precision: 0.5
Query 5 average precision: 0.6875
Query 6 average precision: 0.0625
Query 7 average precision: 0.11666666666666665
Query 8 average precision: 0.0
Query 9 average precision: 0.49999999999999994
Query 10 average precision: 0.0625
Query 11 average precision: 0.023809523809523808
Query 12 average precision: 0.15714285714285714
Query 13 average precision: 0.0
Query 14 average precision: 0.5
Query 15 average precision: 0.6428571428571428
Query 16 average precision: 0.05555555555555555
Query 17 average precision: 0.05
Query 18 average precision: 0.16666666666666666
Query 19 average precision: 0.0
Query 20 average precision: 0.33201058201058203
Query 21 average precision: 0.0
Query 22 average precision: 0.0
Query 23 average precision: 0.075
Query 24 average precision: 0.38888888888888884
Query 25 average precision: 0.7530864197530864
Query 26 average precision: 0.061111111111111116
Query 27 average precision: 0.0
Query 28 average precision: 0.0
Query 29 average precision: 0.091005291005291
Query 30 average precision: 0.19999999999999998
Query 31 average precision: 0.0
Query 32 average precision: 0.023809523809523808
Query 33 average precision: 0.6666666666666666
Query 34 average precision: 0.21031746031746032
Query 35 average precision: 0.03333333333333333
Query 36 average precision: 0.6
Query 37 average precision: 0.1111111111111111
Query 38 average precision: 0.02
Query 39 average precision: 0.12857142857142856
Query 40 average precision: 0.07222222222222222
Query 41 average precision: 0.7916666666666666
Query 42 average precision: 0.14523809523809522
Query 43 average precision: 0.3333333333333333
Query 44 average precision: 0.0
Query 45 average precision: 0.13999999999999999
Query 46 average precision: 0.49071428571428566
Query 47 average precision: 0.14027777777777778
Query 48 average precision: 0.1433333333333333
Query 49 average precision: 0.0
Query 50 average precision: 0.0
Query 51 average precision: 0.22999999999999998
Query 52 average precision: 0.08125
Query 53 average precision: 0.15
Query 54 average precision: 0.037037037037037035
Query 55 average precision: 0.11833333333333333
Query 56 average precision: 0.05
Query 57 average precision: 0.0125
Query 58 average precision: 0.16666666666666666
Query 59 average precision: 0.03571428571428571
Query 60 average precision: 0.45999999999999996
Query 61 average precision: 0.4083333333333333
Query 62 average precision: 0.0
Query 63 average precision: 0.0
Query 64 average precision: 0.5
Query 65 average precision: 0.0
Query 66 average precision: 0.0
Query 67 average precision: 0.03111111111111111
Query 68 average precision: 0.06666666666666667
Query 69 average precision: 0.2
Query 70 average precision: 0.1
Query 71 average precision: 0.041666666666666664
Query 72 average precision: 0.0
Query 73 average precision: 0.38916666666666666
Query 74 average precision: 0.05555555555555555
Query 75 average precision: 0.02222222222222222
Query 76 average precision: 0.02040816326530612
Query 77 average precision: 0.34027777777777773
Query 78 average precision: 0.7916666666666666
Query 79 average precision: 0.0
Query 80 average precision: 0.0
Query 81 average precision: 0.25
Query 82 average precision: 0.07333333333333333
Query 83 average precision: 0.05
Query 84 average precision: 0.2375
Query 85 average precision: 0.05
Query 86 average precision: 0.30952380952380953
Query 87 average precision: 0.0
Query 88 average precision: 0.594047619047619
Query 89 average precision: 0.10833333333333334
Query 90 average precision: 0.1875
Query 91 average precision: 0.1111111111111111
Query 92 average precision: 0.5747619047619048
Query 93 average precision: 0.5
Query 94 average precision: 0.4
Query 95 average precision: 0.5
Query 96 average precision: 0.6042063492063492
Query 97 average precision: 0.24166666666666664
Query 98 average precision: 0.0
Query 99 average precision: 0.25
Query 100 average precision: 0.25555555555555554
Query 101 average precision: 0.5444444444444444
Query 102 average precision: 0.08333333333333333
Query 103 average precision: 0.1
Query 104 average precision: 0.1
Query 105 average precision: 0.4333333333333333
Query 106 average precision: 0.3746031746031746
Query 107 average precision: 0.1619047619047619
Query 108 average precision: 0.5488095238095239
Query 109 average precision: 0.0
Query 110 average precision: 0.0
Query 111 average precision: 0.014285714285714287
Query 112 average precision: 0.3611111111111111
Query 113 average precision: 0.10555555555555556
Query 114 average precision: 0.0
Query 115 average precision: 0.0
Query 116 average precision: 0.02857142857142857
Query 117 average precision: 0.0
Query 118 average precision: 0.037037037037037035
Query 119 average precision: 0.3333333333333333
Query 120 average precision: 0.1527777777777778
Query 121 average precision: 0.42857142857142855
Query 122 average precision: 0.07777777777777778
Query 123 average precision: 0.0
Query 124 average precision: 0.0
Query 125 average precision: 0.03333333333333333
Query 126 average precision: 0.20833333333333331
Query 127 average precision: 0.02222222222222222
Query 128 average precision: 0.0
Query 129 average precision: 0.6738095238095239
Query 130 average precision: 0.55
Query 131 average precision: 0.18333333333333335
Query 132 average precision: 0.7254365079365078
Query 133 average precision: 0.18010204081632653
Query 134 average precision: 0.5
Query 135 average precision: 0.37351190476190477
Query 136 average precision: 0.047619047619047616
Query 137 average precision: 0.1125
Query 138 average precision: 0.5
Query 139 average precision: 0.0
Query 140 average precision: 0.10833333333333334
Query 141 average precision: 0.041666666666666664
Query 142 average precision: 0.0
Query 143 average precision: 0.39285714285714285
Query 144 average precision: 0.20555555555555557
Query 145 average precision: 0.07142857142857142
Query 146 average precision: 0.30952380952380953
Query 147 average precision: 0.14999999999999997
Query 148 average precision: 0.075
Query 149 average precision: 0.17666666666666667
Query 150 average precision: 1.0
Query 151 average precision: 0.0
Query 152 average precision: 0.0
Query 153 average precision: 0.2571428571428571
Query 154 average precision: 1.0
Query 155 average precision: 0.06666666666666667
Query 156 average precision: 0.7532142857142856
Query 157 average precision: 0.40555555555555556
Query 158 average precision: 0.19375
Query 159 average precision: 0.020833333333333332
Query 160 average precision: 0.2
Query 161 average precision: 0.3611111111111111
Query 162 average precision: 0.013888888888888888
Query 163 average precision: 0.19444444444444442
Query 164 average precision: 0.40208333333333335
Query 165 average precision: 0.125
Query 166 average precision: 0.0
Query 167 average precision: 0.5
Query 168 average precision: 0.08333333333333333
Query 169 average precision: 0.25
Query 170 average precision: 0.46990740740740744
Query 171 average precision: 0.5555555555555555
Query 172 average precision: 0.5583333333333333
Query 173 average precision: 0.7
Query 174 average precision: 0.03333333333333333
Query 175 average precision: 0.02857142857142857
Query 176 average precision: 0.0
Query 177 average precision: 0.475
Query 178 average precision: 0.75
Query 179 average precision: 0.25
Query 180 average precision: 0.3880952380952381
Query 181 average precision: 0.1
Query 182 average precision: 0.7
Query 183 average precision: 0.6842063492063492
Query 184 average precision: 0.05215419501133787
Query 185 average precision: 0.6296296296296297
Query 186 average precision: 0.04722222222222222
Query 187 average precision: 0.125
Query 188 average precision: 0.19
Query 189 average precision: 0.022222222222222223
Query 190 average precision: 0.05
Query 191 average precision: 0.05357142857142857
Query 192 average precision: 0.575
Query 193 average precision: 0.6565255731922398
Query 194 average precision: 0.19999999999999998
Query 195 average precision: 0.3333333333333333
Query 196 average precision: 0.05
Query 197 average precision: 0.6666666666666666
Query 198 average precision: 0.1875
Query 199 average precision: 0.048611111111111105
Query 200 average precision: 0.40740740740740744
Query 201 average precision: 0.33999999999999997
Query 202 average precision: 0.22666666666666666
Query 203 average precision: 0.07333333333333333
Query 204 average precision: 0.0
Query 205 average precision: 0.7
Query 206 average precision: 0.38888888888888884
Query 207 average precision: 0.3
Query 208 average precision: 0.7345238095238096
Query 209 average precision: 0.03333333333333333
Query 210 average precision: 0.27777777777777773
Query 211 average precision: 0.025
Query 212 average precision: 0.36666666666666664
Query 213 average precision: 0.49309523809523814
Query 214 average precision: 0.08333333333333333
Query 215 average precision: 0.0
Query 216 average precision: 0.0
Query 217 average precision: 0.07857142857142857
Query 218 average precision: 0.0
Query 219 average precision: 0.014285714285714285
Query 220 average precision: 0.07333333333333333
Query 221 average precision: 0.24333333333333332
Query 222 average precision: 0.5238095238095238
Query 223 average precision: 0.4583333333333333
Query 224 average precision: 0.0
Query 225 average precision: 0.15
In [20]:
fb_map = ev.map()
print("Feedback MAP: {}".format(fb_map))
print("DP MAP: {}".format(dp_map))
Feedback MAP: 0.22816526133086987
DP MAP: 0.21512203955656342