Joblib is a data serialization format used in Big Data use cases.
pip3 install joblib
It looks attractive from the specifactions, but how will it perform?
As a simple test, we take the feature data for g_word_utf8
.
It is a map from the numbers 1 to 426584 to Hebrew word occurrences (Unicode strings).
In Text-Fabric we have a representation in plain text and a compressed, pickled representation.
Text-Fabric is much faster in loading this kind of data.
The size of the joblib uncompressed serialization is much bigger than the TF text representation.
The size of the gzipped joblib serialization is still 30% more than gzipped, pickled TF serialization.
name | kind | size | load time |
---|---|---|---|
g_word_utf8.tf | tf: plain unicode text | 5.4 MB | 1.6 s |
g_word_utf8.tfx | tf: gzipped binary | 3.2 MB | 0.2 s |
g_word_utf8.joblib | joblib: uncompressed | 11.2 MB | 2.0 s |
g_word_utf8.joblib.gz | joblib: gzipped | 3.2 MB | 2.7 s |
We do not see reasons to replace the TF feature data serialization by joblib.
import os
import gzip
import joblib
from tf.fabric import Fabric
GZIP_LEVEL = 2 # same as used in Text-Fabric
VERSION = 'c'
BHSA = f'BHSA/tf/{VERSION}'
PARA = f'parallels/tf/{VERSION}'
TF = Fabric(locations='~/github/etcbc', modules=[BHSA, PARA])
api = TF.load('')
api.makeAvailableIn(globals())
This is Text-Fabric 5.5.22 Api reference : https://annotation.github.io/text-fabric/Api/Fabric/ Tutorial : https://github.com/annotation/text-fabric/blob/master/docs/tutorial.ipynb Example data : https://github.com/annotation/text-fabric-data 117 features found and 0 ignored 0.00s loading features ... | 1.60s T g_word_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c 6.02s All features loaded/computed - for details use loadLog()
The load time is ~ 1.6 seconds.
But during this time, the textual data has been compiled and written to a binary form. Let's load again.
TF = Fabric(locations='~/github/etcbc', modules=[BHSA, PARA])
api = TF.load('')
api.makeAvailableIn(globals())
This is Text-Fabric 5.5.22 Api reference : https://annotation.github.io/text-fabric/Api/Fabric/ Tutorial : https://github.com/annotation/text-fabric/blob/master/docs/tutorial.ipynb Example data : https://github.com/annotation/text-fabric-data 117 features found and 0 ignored 0.00s loading features ... 5.27s All features loaded/computed - for details use loadLog()
loadLog()
| 0.03s B otype from /Users/dirk/github/etcbc/BHSA/tf/c | 0.53s B oslots from /Users/dirk/github/etcbc/BHSA/tf/c | 0.01s B book from /Users/dirk/github/etcbc/BHSA/tf/c | 0.01s B chapter from /Users/dirk/github/etcbc/BHSA/tf/c | 0.01s B verse from /Users/dirk/github/etcbc/BHSA/tf/c | 0.13s B g_cons from /Users/dirk/github/etcbc/BHSA/tf/c | 0.18s B g_cons_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.14s B g_lex from /Users/dirk/github/etcbc/BHSA/tf/c | 0.19s B g_lex_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.16s B g_word from /Users/dirk/github/etcbc/BHSA/tf/c | 0.33s B g_word_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.16s B lex0 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.25s B lex_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B qere from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B qere_trailer from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B qere_trailer_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.01s B qere_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.08s B trailer from /Users/dirk/github/etcbc/BHSA/tf/c | 0.09s B trailer_utf8 from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B __levels__ from otype, oslots, otext | 0.03s B __order__ from otype, oslots, __levels__ | 0.04s B __rank__ from otype, __order__ | 1.65s B __levUp__ from otype, oslots, __rank__ | 0.88s B __levDown__ from otype, __levUp__, __rank__ | 0.30s B __boundary__ from otype, oslots, __rank__ | 0.01s B __sections__ from otype, oslots, otext, __levUp__, __levels__, book, chapter, verse | 0.00s B book@am from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@ar from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@bn from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@da from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@de from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@el from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@en from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@es from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@fa from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@fr from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@he from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@hi from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@id from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@ja from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@ko from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@la from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@nl from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@pa from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@pt from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@ru from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@sw from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@syc from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@tr from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@ur from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@yo from /Users/dirk/github/etcbc/BHSA/tf/c | 0.00s B book@zh from /Users/dirk/github/etcbc/BHSA/tf/c
The load time of the feature g_word_utf8
is ~ 0.2 seconds.
tempDir = os.path.expanduser('~/github/annotation/text-fabric/_temp/joblib')
os.makedirs(tempDir, exist_ok=True)
feature = 'g_word_utf8'
data = TF.features[feature].data
print(len(data))
print(data[2])
426584 רֵאשִׁ֖ית
We write the feature data to an Avro data file.
dataFile = f'{tempDir}/{feature}.joblib'
indent(reset=True)
info('start writing')
joblib.dump(data, dataFile)
info('done')
0.00s start writing 3.20s done
We make also a gzipped data file.
indent(reset=True)
info('start writing')
dataFileZ = f'{dataFile}.gz'
joblib.dump(data, dataFileZ, compress=('gzip', GZIP_LEVEL))
info('done')
0.00s start writing 3.70s done
indent(reset=True)
info('start reading')
rData = joblib.load(dataFile)
info('done')
print(rData[2])
0.00s start reading 2.06s done רֵאשִׁ֖ית
Load time ~ 2.0 seconds.
indent(reset=True)
info('start reading')
rData = joblib.load(dataFileZ)
info('done')
print(rData[2])
0.00s start reading 2.73s done רֵאשִׁ֖ית
Load time ~ 2.7 seconds.