#!/usr/bin/env python
# coding: utf-8

# # Data Sets Tutorial
# This tutorial demonstrates how to create and use `DataSet` objects.  PyGSTi uses `DataSet` objects to hold experimental or simulated data in the form of outcome counts.  When a `DataSet` is used to hold time-independent data (the typical case, and all we'll look at in this tutorial), it essentially looks like a nested dictionary which associates operation sequences with dictionaries of (outcome-label,count) pairs so that `dataset[circuit][outcomeLabel]` can be used to read & write the number of `outcomeLabel` outcomes of the experiment given by the circuit `circuit`.
# 
# There are a few important differences between a `DataSet` and a dictionary-of-dictionaries:
# - `DataSet` objects can be in one of two modes: *static* or *non-static*.  When in *non-static* mode, data can be freely modified within the set, making this mode to use during the data-entry.  In the *static* mode, data cannot be modified and the `DataSet` is essentially read-only.  The `done_adding_data` method of a `DataSet` switches from non-static to static mode, and should be called, as the name implies, once all desired data has been added (or modified).  Once a `DataSet` is static, it is read-only for the rest of its life; to modify its data the best one can do is make a non-static *copy* via the `copy_nonstatic` member and modify the copy.
# 
# - Because `DataSet`s may contain time-dependent data, the dictionary-access syntax for a single outcome label (i.e. `dataset[circuit][outcomeLabel]`) *cannot* be used to write counts for new `circuit` keys; One should instead  use the `add_`*xxx* methods of the `DataSet` object.
# 
# Once a `DataSet` is constructed, filled with data, and made *static*, it is typically passed as a parameter to one of pyGSTi's algorithm or driver routines to find a `Model` estimate based on the data.  This tutorial focuses on how to construct a `DataSet` and modify its data.  Later tutorials will demonstrate the different GST algorithms.

# In[1]:


from __future__ import print_function
import pygsti


# ## Creating a `DataSet`
# There three basic ways to create `DataSet` objects in `pygsti`:
# * By creating an empty `DataSet` object and manually adding counts corresponding to operation sequences.  Remember that the `add_`*xxx* methods must be used to add data for operation sequences not yet in the `DataSet`.  Once the data is added, be sure to call `done_adding_data`, as this restructures the internal storage of the `DataSet` to optimize the access operations used by algorithms.
# * By loading from a text-format dataset file via `pygsti.io.load_dataset`.  The result is a ready-to-use-in-algorithms *static* `DataSet`, so there's no need to call `done_adding_data` this time.
# * By using a `Model` to generate "fake" data via `generate_fake_data`. This can be useful for doing simulations of GST, and comparing to your experimental results.
# 
# We do each of these in turn in the cells below.

# In[2]:


#1) Creating a data set from scratch
#    Note that tuples may be used in lieu of OpString objects
ds1 = pygsti.objects.DataSet(outcomeLabels=['0','1'])
ds1.add_count_dict( ('Gx',), {'0': 10, '1': 90} )
ds1.add_count_dict( ('Gx','Gy'), {'0': 40, '1': 60} )
ds1[('Gy',)] = {'0': 10, '1': 90} # dictionary assignment

#Modify existing data using dictionary-like access
ds1[('Gx',)]['0'] = 15
ds1[('Gx',)]['1'] = 85

#OpString objects can be used.
mdl = pygsti.objects.Circuit( ('Gx','Gy'))
ds1[mdl]['0'] = 45
ds1[mdl]['1'] = 55

ds1.done_adding_data()


# In[3]:


#2) By creating and loading a text-format dataset file.  The first
#    row is a directive which specifies what the columns (after the
#    first one) holds.  Note that "0" and "1" in are the 
#    SPAM labels and must match those of any Model used in 
#    conjuction with this DataSet.
dataset_txt = \
"""## Columns = 0 count, 1 count
{} 0 100
Gx 10 90
GxGy 40 60
Gx^4 20 90
"""
with open("../tutorial_files/Example_TinyDataset.txt","w") as tinydataset:
    tinydataset.write(dataset_txt)
ds2 = pygsti.io.load_dataset("../tutorial_files/Example_TinyDataset.txt")


# In[4]:


#3) By generating fake data (using the std1Q_XYI standard model module)
from pygsti.construction import std1Q_XYI

#Depolarize the perfect X,Y,I model
depol_gateset = std1Q_XYI.target_model().depolarize(op_noise=0.1)

#Compute the sequences needed to perform Long Sequence GST on 
# this Model with sequences up to lenth 512
circuit_list = pygsti.construction.make_lsgst_experiment_list(
    std1Q_XYI.target_model(), std1Q_XYI.prepStrs, std1Q_XYI.effectStrs,
    std1Q_XYI.germs, [1,2,4,8,16,32,64,128,256,512])

#Generate fake data (Tutorial 00)
ds3 = pygsti.construction.generate_fake_data(depol_gateset, circuit_list, nSamples=1000,
                                             sampleError='binomial', seed=100)
ds3b = pygsti.construction.generate_fake_data(depol_gateset, circuit_list, nSamples=50,
                                              sampleError='binomial', seed=100)

#Write the ds3 and ds3b datasets to a file for later tutorials
pygsti.io.write_dataset("../tutorial_files/Example_Dataset.txt", ds3, outcomeLabelOrder=['0','1']) 
pygsti.io.write_dataset("../tutorial_files/Example_Dataset_LowCnts.txt", ds3b) 


# ## Viewing `DataSets`

# In[5]:


#It's easy to just print them:
print("Dataset1:\n",ds1)
print("Dataset2:\n",ds2)
print("Dataset3 is too big to print, so here it is truncated to Dataset2's strings\n", ds3.truncate(ds2.keys()))


# **Note that the outcome labels `'0'` and `'1'` appear as `('0',)` and `('1',)`**.  This is because outcome labels in pyGSTi are tuples of time-ordered instrument element (see the [intermediate measurements tutorial](advanced/Instruments.ipynb)) and POVM effect labels.  In the special but common case when there are no intermediate measurements, the outcome label is a 1-tuple of just the final POVM effect label.  In this case, one may use the effect label itself (e.g. `'0'` or `'1'`) in place of the 1-tuple in almost all contexts, as it is automatically converted to the 1-tuple (e.g. `('0',)` or `('1',)`) internally.  When printing, however, the 1-tuple is still displayed to remind the user of the more general structure contained in the `DataSet`.

# ## Iteration over data sets

# In[6]:


# A DataSet's keys() method returns a list of OpString objects
ds1.keys()


# In[7]:


# There are many ways to iterate over a DataSet.  Here's one:
for circuit in ds1.keys():
    dsRow = ds1[circuit]
    for spamlabel in dsRow.counts.keys():
        print("Circuit = %s, SPAM label = %s, count = %d" % \
            (repr(circuit).ljust(13), str(spamlabel).ljust(7), dsRow[spamlabel]))


# ## Advanced features of data sets
# 
# ### `collisionAction` argument
# When creating a `DataSet` one may specify the `collisionAction` argument as either `"aggregate"` (the default) or `"keepseparate"`.  The former instructs the `DataSet` to simply add the counts of like outcomes when counts are added for an already existing gate sequence.  `"keepseparate"`, on the other hand, causes the `DataSet` to tag added count data by appending a fictitious `"#<n>"` operation label to a gate sequence that already exists, where `<n>` is an integer.  When retreiving the keys of a `keepseparate` data set, the `stripOccuranceTags` argument to `keys()` determines whether the `"#<n>"` labels are included in the output (if they're not - the default - duplicate keys may be returned).  Access to different occurances of the same data are provided via the `occurrance` argument of the `get_row` and `set_row` functions, which should be used instead of the usual bracket indexing.

# In[8]:


ds_agg = pygsti.objects.DataSet(outcomeLabels=['0','1'], collisionAction="aggregate") #the default
ds_agg.add_count_dict( ('Gx','Gy'), {'0': 10, '1': 90} )
ds_agg.add_count_dict( ('Gx','Gy'), {'0': 40, '1': 60} )
print("Aggregate-mode Dataset:\n",ds_agg)

ds_sep = pygsti.objects.DataSet(outcomeLabels=['0','1'], collisionAction="keepseparate")
ds_sep.add_count_dict( ('Gx','Gy'), {'0': 10, '1': 90} )
ds_sep.add_count_dict( ('Gx','Gy'), {'0': 40, '1': 60} )
print("Keepseparate-mode Dataset:\n",ds_sep)


# ## Related topics
# This concludes our overview of `DataSet` objects.  Here are a couple of related topics we didn't touch on that might be of interest:
# - the ability of a `DataSet` to store **time-dependent data** (in addition to the count data described above) is covered in the [timestamped data tutorial](advanced/TimestampedDataSets.ipynb).
# - the `MultiDataSet` object, which is similar to a dictionary of `DataSets` is described in the [MultiDataSet tutorial](advanced/MultiDataSet.ipynb).  `MultiDataSet` objects are useful when you have several data sets containing outcomes for the *same* set of circuits, like when you do multiple passes of the same experimental procedure.

# In[ ]: