#!/usr/bin/env python
# coding: utf-8

# # Fine-tuning BERT (base or large) on a question-answering task by using the libraries transformers (HF) and DeepSpeed (Microsoft)

# - **Credit**: Hugging Face and Microsoft
# - **Author**: [Pierre GUILLOU](https://www.linkedin.com/in/pierreguillou/)
# - **Date**: 18/06/2021
# - **Blog post**: [NLP | Como treinar um modelo de Question Answering em qualquer linguagem baseado no BERT large, melhorando o desempenho do modelo utilizando o BERT base? (estudo de caso em português)](https://medium.com/@pierre_guillou/nlp-como-treinar-um-modelo-de-question-answering-em-qualquer-linguagem-baseado-no-bert-large-1c899262dd96)
# - **Link to the model in the Model Hub of Hugging Face**: https://huggingface.co/pierreguillou/bert-large-cased-squad-v1.1-portuguese

# ## Context

# This notebook is an adaptation of the notebook [question_answering.ipynb](https://github.com/huggingface/notebooks/blob/master/examples/question_answering.ipynb) and of the script [run_qa.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa.py) of Hugging Face (HF) for fine-tuning a (transformer) Masked Language Model (MLM) like BERT on the QA task with the [Portuguese Squad 1.1 dataset](https://forum.ailab.unb.br/t/datasets-em-portugues/251/4).
# 
# In order to speed up the fine-tuning of the model on only one GPU, the library [DeepSpeed](https://www.deepspeed.ai/) is used by applying the configuration provided by HF in the notebook [transformers + deepspeed CLI](https://github.com/stas00/porting/blob/master/transformers/deepspeed/DeepSpeed_on_colab_CLI.ipynb).
# 
# *Note: the paragraph about Causal language modeling (CLM) is not included in this notebook, and all the non necessary code about Masked Model Language (MLM) has been deleted from the original notebook.*

# ## Installation

# If you're opening this Notebook on colab, you will probably need to install 🤗 Transformers and 🤗 Datasets, and DeepSpeed. Uncomment the following cells and run it.

# Pytorch

# In[1]:


# !pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html


# DeepSpeed

# In[2]:


# !pip install git+https://github.com/microsoft/deepspeed


# Datasets, Tokenizers, Transformers

# In[3]:


# git clone https://github.com/huggingface/transformers
# cd transformers
# # examples change a lot so let's pick a sha that we know this notebook will work with
# # comment out/remove the next line if you want the master
# # git checkout  d2753dcbec712350
# pip install -e .
# pip install -r examples/pytorch/translation/requirements.txt


# In this notebook's folder, you need to create symbolic links to 3 files in the transformers folder you just installed.

# In[5]:


#!ln -s ~/transformers/examples/pytorch/question-answering/run_qa.py
#!ln -s ~/transformers/examples/pytorch/question-answering/trainer_qa.py
#!ln -s ~/transformers/examples/pytorch/question-answering/utils_qa.py


# Let's check our installation.

# In[6]:


import sys; print('python:',sys.version)
import pathlib
from pathlib import Path

import torch; print('Pytorch:',torch.__version__)

import transformers; print('transformers:',transformers.__version__)
import tokenizers; print('tokenizers:',tokenizers.__version__)
import datasets; print('datasets:',datasets.__version__)

import deepspeed; print('deepspeed:',deepspeed.__version__)

# Versions installed:
# python: 3.8.10 (default, Jun  4 2021, 15:09:15) 
# [GCC 7.5.0]
# Pytorch: 1.8.1+cu111
# transformers: 4.7.0.dev0
# tokenizers: 0.10.3
# datasets: 1.8.0
# deepspeed: 0.4.1+fa7921e


# If you're opening this notebook locally, make sure your environment has an install from the last version of those libraries.
# 
# You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs [here](https://github.com/huggingface/transformers/tree/master/examples/question-answering).

# In this notebook, we will see how to fine-tune one of the [🤗 Transformers](https://github.com/huggingface/transformers) model to a question answering task, which is the task of extracting the answer to a question from a given context. We will see how to easily load a dataset for these kinds of tasks and use the `Trainer` API to fine-tune a model on it.
# 
# ![Widget inference representing the QA task](images/question_answering.png)
# 
# **Note:** This notebook finetunes models that answer question by taking a substring of a context, not by generating new text.

# This notebook is built to run on any question answering task with the same format as SQUAD (version 1 or 2), with any model checkpoint from the [Model Hub](https://huggingface.co/models) as long as that model has a version with a token classification head and a fast tokenizer (check on [this table](https://huggingface.co/transformers/index.html#bigtable) if this is the case). It might just need some small adjustments if you decide to use a different dataset than the one used here. Depending on you model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set those three parameters, then the rest of the notebook should run smoothly:

# In[7]:


# This flag is the difference between SQUAD v1 or 2 (if you're using another dataset, it indicates if impossible
# answers are allowed or not).
squad_v2 = False
batch_size = 16


# ## BERT model

# In[8]:


model_name_or_path = "neuralmind/bert-large-portuguese-cased"


# ## Dataset

# ### Loading the dataset

# For our example here, we'll use the [Portuguese Squad 1.1 dataset](https://forum.ailab.unb.br/t/datasets-em-portugues/251/4) which is a translation of the [English SQUAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook should work with any question answering dataset provided by the 🤗 Datasets library. If you're using your own dataset defined from a JSON or csv file (see the [Datasets documentation](https://huggingface.co/docs/datasets/loading_datasets.html#from-local-files) on how to load them), it might need some adjustments in the names of the columns used.

# In[9]:


dataset_name = "squad11pt"


# In[10]:


# %%time
# if dataset_name == "squad11pt":
    
#     # create dataset folder 
#     root = Path.cwd()
#     path_to_dataset = root.parent/'data'/dataset_name
#     path_to_dataset.mkdir(parents=True, exist_ok=True) 

#     # Get dataset SQUAD in Portuguese
#     %cd {path_to_dataset}
#     !wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1Q0IaIlv2h2BC468MwUFmUST0EyN7gNkn" -O squad-pt.tar.gz && rm -rf /tmp/cookies.txt

#     # unzip 
#     !tar -xvf squad-pt.tar.gz

#     # Get the train and validation json file in the HF script format 
#     # inspiration: file squad.py at https://github.com/huggingface/datasets/tree/master/datasets/squad

#     import json 
#     files = ['squad-train-v1.1.json','squad-dev-v1.1.json']

#     for file in files:

#         # Opening JSON file & returns JSON object as a dictionary 
#         f = open(file, encoding="utf-8") 
#         data = json.load(f) 

#         # Iterating through the json list 
#         entry_list = list()
#         id_list = list()

#         for row in data['data']: 
#             title = row['title']

#             for paragraph in row['paragraphs']:
#                 context = paragraph['context']

#                 for qa in paragraph['qas']:
#                     entry = {}

#                     qa_id = qa['id']
#                     question = qa['question']
#                     answers = qa['answers']

#                     entry['id'] = qa_id
#                     entry['title'] = title.strip()
#                     entry['context'] = context.strip()
#                     entry['question'] = question.strip()

#                     answer_starts = [answer["answer_start"] for answer in answers]
#                     answer_texts = [answer["text"].strip() for answer in answers]
#                     entry['answers'] = {}
#                     entry['answers']['answer_start'] = answer_starts
#                     entry['answers']['text'] = answer_texts

#                     entry_list.append(entry)

#         reverse_entry_list = entry_list[::-1]

#         # for entries with same id, keep only last one (corrected texts by the group Deep Learning Brasil)
#         unique_ids_list = list()
#         unique_entry_list = list()
#         for entry in reverse_entry_list:
#             qa_id = entry['id']
#             if qa_id not in unique_ids_list:
#                 unique_ids_list.append(qa_id)
#                 unique_entry_list.append(entry)

#         # Closing file 
#         f.close() 

#         new_dict = {}
#         new_dict['data'] = unique_entry_list

#         file_name = 'pt_' + str(file)
#         with open(file_name, 'w') as json_file:
#             json.dump(new_dict, json_file)
            
# %cd {root}


# ### Check the dataset

# We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download the data and get the metric we need to use for evaluation (to compare our model to the benchmark). This can be easily done with the functions `load_dataset` and `load_metric`.  

# In[ ]:


from datasets import load_dataset, load_metric

if dataset_name == "squad11pt":
    
    # dataset folder 
    root = Path.cwd()
    path_to_dataset = root.parent/'data'/dataset_name
    
    # paths to files
    train_file = str(path_to_dataset/'pt_squad-train-v1.1.json')
    validation_file = str(path_to_dataset/'pt_squad-dev-v1.1.json')
    
    datasets = load_dataset('json', 
                            data_files={'train': train_file, \
                                        'validation': validation_file, \
                                       }, 
                            field='data')


# The `datasets` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set.

# In[12]:


datasets


# To access an actual element, you need to select a split first, then give an index:

# In[13]:


datasets["train"][0]


# We can see the answers are indicated by their start position in the text (here at character 2) and their full text, which is a substring of the context as we mentioned above.

# ## DeepSpeed

# Let's setup the `DeepSpeed` configuration.

# To use a LR Linear Decay after warmup as scheduler (equivalent of the one by default in the HF Trainer), we changed `WarmupLR` to `WarmupDecayLR` in the DeepSpeed configuration file, and kept a copy of the scheduler initial code here:

# ````
# # the lr stays constant after the warmup (this is not equivalent to teh default scheduler of HF which is Linear)
#     "scheduler": {
#         "type": "WarmupLR",
#         "params": {
#             "warmup_min_lr": "auto",
#             "warmup_max_lr": "auto",
#             "warmup_num_steps": "auto"
#         }
#     },
# ````

# ### ZeRO-2

# source: https://huggingface.co/transformers/master/main_classes/deepspeed.html#zero-2-example

# In[14]:


get_ipython().run_cell_magic('bash', '', '\ncat <<\'EOT\' > ds_config_zero2.json\n{\n    "fp16": {\n        "enabled": "auto",\n        "loss_scale": 0,\n        "loss_scale_window": 1000,\n        "initial_scale_power": 16,\n        "hysteresis": 2,\n        "min_loss_scale": 1\n    },\n\n    "optimizer": {\n        "type": "AdamW",\n        "params": {\n            "lr": "auto",\n            "betas": "auto",\n            "eps": "auto",\n            "weight_decay": "auto"\n        }\n    },\n\n    "scheduler": {\n         "type": "WarmupDecayLR",\n         "params": {\n             "last_batch_iteration": -1,\n             "total_num_steps": "auto",\n             "warmup_min_lr": "auto",\n             "warmup_max_lr": "auto",\n             "warmup_num_steps": "auto"\n         }\n     },\n\n    "zero_optimization": {\n        "stage": 2,\n        "offload_optimizer": {\n            "device": "cpu",\n            "pin_memory": true\n        },\n        "allgather_partitions": true,\n        "allgather_bucket_size": 5e8,\n        "overlap_comm": true,\n        "reduce_scatter": true,\n        "reduce_bucket_size": 5e8,\n        "contiguous_gradients": true\n    },\n\n    "gradient_accumulation_steps": "auto",\n    "gradient_clipping": "auto",\n    "steps_per_print": 2000,\n    "train_batch_size": "auto",\n    "train_micro_batch_size_per_gpu": "auto",\n    "wall_clock_breakdown": false\n}\nEOT\n')


# ### ZeRO-3

# source: https://huggingface.co/transformers/master/main_classes/deepspeed.html#zero-3-example

# Compared to ZeRO-2, the `ZeRO-3` configuration allows to train larger models but also for a longer training time. For this reason, we will not be using the `ZeRO-3` configuration.

# In[15]:


get_ipython().run_cell_magic('bash', '', '\ncat <<\'EOT\' > ds_config_zero3.json\n{\n    "fp16": {\n        "enabled": "auto",\n        "loss_scale": 0,\n        "loss_scale_window": 1000,\n        "initial_scale_power": 16,\n        "hysteresis": 2,\n        "min_loss_scale": 1\n    },\n\n    "optimizer": {\n        "type": "AdamW",\n        "params": {\n            "lr": "auto",\n            "betas": "auto",\n            "eps": "auto",\n            "weight_decay": "auto"\n        }\n    },\n\n    "scheduler": {\n         "type": "WarmupDecayLR",\n         "params": {\n             "last_batch_iteration": -1,\n             "total_num_steps": "auto",\n             "warmup_min_lr": "auto",\n             "warmup_max_lr": "auto",\n             "warmup_num_steps": "auto"\n         }\n     },\n     \n    "zero_optimization": {\n        "stage": 3,\n        "offload_optimizer": {\n            "device": "cpu",\n            "pin_memory": true\n        },\n        "offload_param": {\n            "device": "cpu",\n            "pin_memory": true\n        },\n        "overlap_comm": true,\n        "contiguous_gradients": true,\n        "sub_group_size": 1e9,\n        "reduce_bucket_size": "auto",\n        "stage3_prefetch_bucket_size": "auto",\n        "stage3_param_persistence_threshold": "auto",\n        "stage3_max_live_parameters": 1e9,\n        "stage3_max_reuse_distance": 1e9,\n        "stage3_gather_fp16_weights_on_model_save": true\n    },\n\n    "gradient_accumulation_steps": "auto",\n    "gradient_clipping": "auto",\n    "steps_per_print": 2000,\n    "train_batch_size": "auto",\n    "train_micro_batch_size_per_gpu": "auto",\n    "wall_clock_breakdown": false\n}\nEOT\n')


# ## Training arguments

# Let's setup all the training arguments needed by the script `run_qa.py`.

# ### GPU

# In[16]:


num_gpus = 1 # run the script on only one gpu
gpu = 0 # select the gpu


# ### model, dataset, sequence

# In[17]:


# setup the training argument
do_train = True # False
do_eval = True 

if dataset_name == "squad11pt":
    
    # dataset folder 
    root = Path.cwd()
    path_to_dataset = root.parent/'data'/dataset_name
    
    # paths to files
    train_file = str(path_to_dataset/'pt_squad-train-v1.1.json')
    validation_file = str(path_to_dataset/'pt_squad-dev-v1.1.json')
                          
# if you want to test the trainer, set up the following variables
max_train_samples = 200 # None
max_eval_samples = 50 # None

# The maximum total input sequence length after tokenization. 
# Sequences longer than this will be truncated, sequences shorter will be padded.
max_seq_length = 384
                          
# Whether to pad all samples to `max_seq_length`.
# If False, will pad the samples dynamically when batching to the maximum length in the batch
# (which can be faster on GPU but will be slower on TPU).
pad_to_max_length = True
        
# If true, some of the examples do not have an answer.
version_2_with_negative = False

# When splitting up a long document into chunks, how much stride to take between chunks.
doc_stride = 128

# The total number of n-best predictions to generate when looking for an answer.
n_best_size = 20
                          
# The maximum length of an answer that can be generated. This is needed because the start
# and end predictions are not conditioned on one another.                          
max_answer_length = 30


# ### training_args()

# source: https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments

# If you keep the value 1e-8 for `adam_epsilon` in `fp16` mode, it is zero. The first non-zero value is 1e-7 in this mode. After some testing, we found that `adam_epsilon = 1e-4` gives the best results.

# We use the `ZeRO-2` mode for `DeepSpeed`.

# In[18]:


# epochs, bs, GA
evaluation_strategy = "epoch" # no
BS = batch_size
gradient_accumulation_steps = 1

# optimizer (AdamW)
learning_rate = 5e-5
weight_decay = 0.01 # 0.0
adam_beta1 = 0.9
adam_beta2 = 0.999
adam_epsilon = 1e-4 # 1e-08

# epochs
num_train_epochs = 3.

# scheduler
lr_scheduler_type = 'linear'
warmup_ratio = 0.0
warmup_steps = 0

# logs
logging_strategy = "steps"
logging_first_step = True # False
logging_steps = 500     # if strategy = "steps"
eval_steps = logging_steps # logging_steps

# checkpoints
save_strategy = "epoch" # steps
save_steps = 500 # if save_strategy = "steps"
save_total_limit = 1 # None

# no cuda, seed
no_cuda = False
seed = 42

# fp16
fp16 = True # False
fp16_opt_level = 'O1'
fp16_backend = "auto"
fp16_full_eval = False

# bar
disable_tqdm = False # True
remove_unused_columns = True
#label_names (List[str], optional) 

# best model
load_best_model_at_end = True # False
metric_for_best_model = "eval_f1"
greater_is_better = True

# deepspeed
zero = 2

if zero == 2:
    deepspeed_config = "ds_config_zero2.json"
elif zero == 3:
    deepspeed_config = "ds_config_zero3.json"


# In[19]:


# folder for training outputs
outputs = model_name_or_path.replace('/','-') + '-' + dataset_name \
+ '_wd' + str(weight_decay) + '_eps' + str(adam_epsilon) \
+ '_epochs' + str(num_train_epochs) \
+ '-lr' + str(learning_rate)
path_to_outputs = root/'models_outputs'/outputs

# subfolder for model outputs
output_dir = path_to_outputs/'output_dir' 
overwrite_output_dir = True # False

# logs
logging_dir = path_to_outputs/'logging_dir'


# ## Training + Evaluation

# #### Update the system path with the virtual environment path

# This is needed to launch the `deepspeed` command in our server configuration.

# In[ ]:


import os
PATH = os.getenv('PATH')
get_ipython().run_line_magic('env', 'PATH=/mnt/home/xxxx/anaconda3/envs/aaaa/bin:$PATH')
    
# xxxx is the folder name where anaconda was installed
# aaaa is the virtual ambiente name within this notebook is run


# #### Setup environment variables 

# The magic command `%env` corresponds to `export` in linux. It allows to setup the values of all arguments of the script `run_qa.py`.
# 
# *Note: as we noticed that the script runs without environment variables in this notebook but with local ones, we do not use this magic command.*

# #### Delete the output_dir (if exists)

# In[ ]:


get_ipython().system('rm -r {output_dir}')


# Now, we can launch the training :-) 

# In[24]:


# copy/paste/uncomment the 2 following lines in the following cell if you want to limit the number of data (useful for testing)
# --max_train_samples $max_train_samples \
# --max_eval_samples $max_eval_samples \


# In[ ]:


get_ipython().run_cell_magic('time', '', '# !deepspeed --num_gpus=$num_gpus run_qa.py \\\n!deepspeed --include localhost:$gpu run_qa.py \\\n--model_name_or_path $model_name_or_path \\\n--train_file $train_file \\\n--do_train $do_train \\\n--do_eval $do_eval \\\n--validation_file $validation_file \\\n--max_seq_length $max_seq_length \\\n--pad_to_max_length $pad_to_max_length \\\n--version_2_with_negative $version_2_with_negative \\\n--doc_stride $doc_stride \\\n--n_best_size $n_best_size \\\n--max_answer_length $max_answer_length \\\n--output_dir $output_dir \\\n--overwrite_output_dir $overwrite_output_dir \\\n--evaluation_strategy $evaluation_strategy \\\n--per_device_train_batch_size $batch_size \\\n--per_device_eval_batch_size $batch_size \\\n--gradient_accumulation_steps $gradient_accumulation_steps \\\n--learning_rate $learning_rate \\\n--weight_decay $weight_decay \\\n--adam_beta1 $adam_beta1 \\\n--adam_beta2 $adam_beta2 \\\n--adam_epsilon $adam_epsilon \\\n--num_train_epochs $num_train_epochs \\\n--warmup_ratio $warmup_ratio \\\n--warmup_steps $warmup_steps \\\n--logging_dir $logging_dir \\\n--logging_strategy $logging_strategy \\\n--logging_first_step $logging_first_step \\\n--logging_steps $logging_steps \\\n--eval_steps $eval_steps \\\n--save_strategy $save_strategy \\\n--save_steps $save_steps \\\n--save_total_limit $save_total_limit \\\n--no_cuda $no_cuda \\\n--seed $seed \\\n--fp16 $fp16 \\\n--fp16_opt_level $fp16_opt_level \\\n--fp16_backend $fp16_backend \\\n--fp16_full_eval $fp16_full_eval \\\n--disable_tqdm $disable_tqdm \\\n--remove_unused_columns $remove_unused_columns \\\n--load_best_model_at_end $load_best_model_at_end \\\n--metric_for_best_model $metric_for_best_model \\\n--greater_is_better $greater_is_better \\\n--deepspeed $deepspeed_config\n')


# **Results**

# ````
# [INFO|trainer_pt_utils.py:908] 2021-06-18 05:32:34,020 >> ***** eval metrics *****
# [INFO|trainer_pt_utils.py:913] 2021-06-18 05:32:34,020 >>   epoch            =     3.0
# [INFO|trainer_pt_utils.py:913] 2021-06-18 05:32:34,020 >>   eval_exact_match = 72.6774
# [INFO|trainer_pt_utils.py:913] 2021-06-18 05:32:34,020 >>   eval_f1          = 84.4315
# [INFO|trainer_pt_utils.py:913] 2021-06-18 05:32:34,020 >>   eval_samples     =   10917
# CPU times: user 5min 5s, sys: 51.3 s, total: 5min 56s
# Wall time: 3h 20min 36s
# ````

# ## TensorBoard

# In[4]:


#!pip install tensorboard


# In[26]:


get_ipython().run_line_magic('load_ext', 'tensorboard')
# %reload_ext tensorboard
get_ipython().run_line_magic('tensorboard', '--logdir {logging_dir} --bind_all')


# ## Getting The Model Weights Out

# To get back weights in fp32, read this: https://huggingface.co/transformers/master/main_classes/deepspeed.html#getting-the-model-weights-out

# In[21]:


get_ipython().system('wget https://raw.githubusercontent.com/microsoft/DeepSpeed/master/deepspeed/utils/zero_to_fp32.py')


# In[ ]:


# in this training, checkpoint-16734 contains the best model
get_ipython().run_line_magic('cd', "{output_dir}/'checkpoint-16734'")


# In[32]:


get_ipython().system('ls -al')


# We can observe that pytorch_model.bin has a size of 666 MB because the weights model have been saved with a fp16 format. Let's use the script `zero_to_fp32.py` from DeepSpeed in order to convert them to a fp32 format.

# In[34]:


path_to_zero_to_fp32 = root/'zero_to_fp32.py'
get_ipython().system('python $path_to_zero_to_fp32 -h')


# In[35]:


get_ipython().system('python $path_to_zero_to_fp32 global_step16734 pytorch_model.bin')


# In[38]:


get_ipython().system('ls -al')


# That's it! The size of our model (1.3 GB) means that the weights format is now fp32.

# In[ ]:


get_ipython().run_line_magic('cd', '{root}')


# ## Save the model to HF format

# In[40]:


import pathlib
from pathlib import Path


# In[41]:


# model source
source = root/output_dir/'checkpoint-16734/'

# model destination
dest = root/'HFmodels'
fname_HF = 'bert-large-cased-squad-v1.1-portuguese'
path_to_awesome_name_you_picked = dest/fname_HF
path_to_awesome_name_you_picked.mkdir(exist_ok=True, parents=True)


# In[42]:


# copy model to destination
get_ipython().system("cp {source}/'config.json' {dest/fname_HF}")
get_ipython().system("cp {source}/'pytorch_model.bin' {dest/fname_HF}")

# copy tokenizer to destination
get_ipython().system("cp {source}/'tokenizer_config.json' {dest/fname_HF}")
get_ipython().system("cp {source}/'special_tokens_map.json' {dest/fname_HF}")
get_ipython().system("cp {source}/'vocab.txt' {dest/fname_HF}")


# **Make your model work on all frameworks** ([source](https://huggingface.co/transformers/model_sharing.html#make-your-model-work-on-all-frameworks))

# In[46]:


from transformers import BertForQuestionAnswering
pt_model = BertForQuestionAnswering.from_pretrained(str(path_to_awesome_name_you_picked))
pt_model.save_pretrained(str(path_to_awesome_name_you_picked))


# In[48]:


import tensorflow
from transformers import TFBertForQuestionAnswering

tf_model = TFBertForQuestionAnswering.from_pretrained(str(path_to_awesome_name_you_picked), from_pt=True)
tf_model.save_pretrained(str(path_to_awesome_name_you_picked))


# In[49]:


get_ipython().system('ls -al {path_to_awesome_name_you_picked}')


# ## Model sharing and uploading to the HF models hub

# Don't forget to [upload your model](https://huggingface.co/transformers/model_sharing.html) on the [🤗 Model Hub](https://huggingface.co/models). You can then use it only to generate results like the one shown in the first picture of this notebook!|

# ## Use our QA model

# ### Gradio

# source: [Using & Mixing Hugging Face Models with Gradio 2.0](https://huggingface.co/blog/gradio)

# In[ ]:


import gradio as gr

iface = gr.Interface.load("huggingface/pierreguillou/bert-large-cased-squad-v1.1-portuguese",server_name='xxxx')
iface.launch()

# xxxx is your server name (alias)


# ### Code

# In[ ]:


### import transformers
import pathlib
from pathlib import Path


# In[51]:


from transformers import AutoModelForQuestionAnswering, AutoTokenizer

model_qa = AutoModelForQuestionAnswering.from_pretrained(path_to_awesome_name_you_picked)
tokenizer_qa = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)


# In[55]:


from transformers import pipeline
nlp = pipeline("question-answering", model=model_qa, tokenizer=tokenizer_qa)


# In[93]:


# source: https://pt.wikipedia.org/wiki/Pandemia_de_COVID-19
context = r"""A pandemia de COVID-19, também conhecida como pandemia de coronavírus, é uma pandemia em curso de COVID-19, 
uma doença respiratória causada pelo coronavírus da síndrome respiratória aguda grave 2 (SARS-CoV-2). 
O vírus tem origem zoonótica e o primeiro caso conhecido da doença remonta a dezembro de 2019 em Wuhan, na China. 
Em 20 de janeiro de 2020, a Organização Mundial da Saúde (OMS) classificou o surto 
como Emergência de Saúde Pública de Âmbito Internacional e, em 11 de março de 2020, como pandemia. 
Em 18 de junho de 2021, 177 349 274 casos foram confirmados em 192 países e territórios, 
com 3 840 181 mortes atribuídas à doença, tornando-se uma das pandemias mais mortais da história.
Os sintomas de COVID-19 são altamente variáveis, variando de nenhum a doenças com risco de morte. 
O vírus se espalha principalmente pelo ar quando as pessoas estão perto umas das outras. 
Ele deixa uma pessoa infectada quando ela respira, tosse, espirra ou fala e entra em outra pessoa pela boca, nariz ou olhos.
Ele também pode se espalhar através de superfícies contaminadas. 
As pessoas permanecem contagiosas por até duas semanas e podem espalhar o vírus mesmo se forem assintomáticas.
"""


# In[95]:


get_ipython().run_cell_magic('time', '', 'question = "Quando começou a pandemia de Covid-19 no mundo?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[96]:


get_ipython().run_cell_magic('time', '', 'question = "Qual é a data de início da pandemia Covid-19 em todo o mundo?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[98]:


get_ipython().run_cell_magic('time', '', 'question = "A Covid-19 tem algo a ver com animais?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[99]:


get_ipython().run_cell_magic('time', '', 'question = "Onde foi descoberta a Covid-19?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[100]:


get_ipython().run_cell_magic('time', '', 'question = "Quantos casos houve?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[101]:


get_ipython().run_cell_magic('time', '', 'question = "Quantos mortes?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[102]:


get_ipython().run_cell_magic('time', '', 'question = "Quantos paises tiveram casos?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[103]:


get_ipython().run_cell_magic('time', '', 'question = "Quais são sintomas de COVID-19"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# In[104]:


get_ipython().run_cell_magic('time', '', 'question = "Como se espalha o vírus?"\n\nresult = nlp(question=question, context=context)\n\nprint(f"Answer: \'{result[\'answer\']}\', score: {round(result[\'score\'], 4)}, start: {result[\'start\']}, end: {result[\'end\']}")\n')


# # END