Marian is an efficient, free Neural Machine Translation framework mainly being developed by the Microsoft Translator team (646+ pretrained models & pipelines in 192+ languages)
You need to specify the language your data is in as start_language
and the language you want to translate to as target_language
.
The language references must be ISO language codes
nlu.load('<start_language>.translate.<target_language>')
Translate Turkish to English:
nlu.load('tr.translate_to.en')
Translate English to French:
nlu.load('en.translate_to.fr')
Translate French to Hebrew:
nlu.load('en.translate_to.he')
import os
! apt-get update -qq > /dev/null
# Install java
! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]
! pip install nlu pyspark==2.4.7 > /dev/null
import nlu
Collecting nlu Downloading https://files.pythonhosted.org/packages/7e/6c/e5d13cdf2e06d7b29aa8ecf50ad182d25a0ef9376f47e729960b09cd53d2/nlu-1.1.1-py3-none-any.whl (158kB) |████████████████████████████████| 163kB 4.2MB/s Collecting pyspark==2.4.7 Downloading https://files.pythonhosted.org/packages/e2/06/29f80e5a464033432eedf89924e7aa6ebbc47ce4dcd956853a73627f2c07/pyspark-2.4.7.tar.gz (217.9MB) |████████████████████████████████| 217.9MB 67kB/s Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from nlu) (1.19.5) Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from nlu) (0.8) Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from nlu) (1.1.5) Collecting pyarrow>=0.16.0 Downloading https://files.pythonhosted.org/packages/33/67/2f4fcce1b41bcc7e88a6bfdb42046597ae72e5bc95c2789b7c5ac893c433/pyarrow-3.0.0-cp36-cp36m-manylinux2014_x86_64.whl (20.7MB) |████████████████████████████████| 20.7MB 1.4MB/s Collecting spark-nlp<2.8,>=2.7.1 Downloading https://files.pythonhosted.org/packages/32/9e/2f43d668eefea486e7417c1e83554c72a41e0786976e9429846b753f5014/spark_nlp-2.7.3-py2.py3-none-any.whl (138kB) |████████████████████████████████| 143kB 41.3MB/s Collecting py4j==0.10.7 Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB) |████████████████████████████████| 204kB 55.5MB/s Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->nlu) (2018.9) Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas->nlu) (2.8.1) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.7.3->pandas->nlu) (1.15.0) Building wheels for collected packages: pyspark Building wheel for pyspark (setup.py) ... done Created wheel for pyspark: filename=pyspark-2.4.7-py2.py3-none-any.whl size=218279465 sha256=71fc8754f32cbda4aac57e21f13941ded7cb7fbe4d0dda9bec994490d08df05f Stored in directory: /root/.cache/pip/wheels/34/1f/2e/1e7460f80acf26b08dbb8c53d7ff9e07146f2a68dd5c732be5 Successfully built pyspark Installing collected packages: pyarrow, spark-nlp, nlu, py4j, pyspark Found existing installation: pyarrow 0.14.1 Uninstalling pyarrow-0.14.1: Successfully uninstalled pyarrow-0.14.1 Successfully installed nlu-1.1.1 py4j-0.10.7 pyarrow-3.0.0 pyspark-2.4.7 spark-nlp-2.7.3
# Download bitcoin newsarticles
import pandas as pd
!wget http://ckl-it.de/wp-content/uploads/2020/12/small_btc.csv
df = pd.read_csv('/content/small_btc.csv').iloc[0:20].title
--2021-02-11 18:48:10-- http://ckl-it.de/wp-content/uploads/2020/12/small_btc.csv Resolving ckl-it.de (ckl-it.de)... 217.160.0.108, 2001:8d8:100f:f000::209 Connecting to ckl-it.de (ckl-it.de)|217.160.0.108|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 22244914 (21M) [text/csv] Saving to: ‘small_btc.csv’ small_btc.csv 100%[===================>] 21.21M 6.54MB/s in 3.5s 2021-02-11 18:48:14 (6.00 MB/s) - ‘small_btc.csv’ saved [22244914/22244914]
import nlu
translate_pipe = nlu.load('en.translate_to.de')
de_df = translate_pipe.predict(df)
de_df
translate_en_de download started this may take some time. Approx size to download 370.2 MB [OK!]
text | translation | sentence | title | |
---|---|---|---|---|
origin_index | ||||
0 | Bitcoin Price Update: Will China Lead us Down? | Bitcoin Price Update: Wird China uns nach unte... | Bitcoin Price Update: Will China Lead us Down? | Bitcoin Price Update: Will China Lead us Down? |
1 | Key Bitcoin Price Levels for Week 51 (15 – 22 ... | Preisniveaus für Bitcoin für Woche 51 (15.-22.... | Key Bitcoin Price Levels for Week 51 (15 – 22 ... | Key Bitcoin Price Levels for Week 51 (15 – 22 ... |
2 | National Australia Bank, Citing Highly Flawed ... | Die National Australia Bank, die hochflawed Da... | National Australia Bank, Citing Highly Flawed ... | National Australia Bank, Citing Highly Flawed ... |
3 | Chinese Bitcoin Ban Driven by Chinese Banking... | Chinesische Bitcoin Ban durch chinesische Bank... | Chinese Bitcoin Ban Driven by Chinese Banking... | Chinese Bitcoin Ban Driven by Chinese Banking... |
4 | Bitcoin Trade Update: Opened Position | Bitcoin Trade Update: Geöffnete Position | Bitcoin Trade Update: Opened Position | Bitcoin Trade Update: Opened Position |
5 | Key Bitcoin Price Levels for Week 52 (22 – 28 ... | Key Bitcoin Price Levels für Woche 52 (22. - 2... | Key Bitcoin Price Levels for Week 52 (22 – 28 ... | Key Bitcoin Price Levels for Week 52 (22 – 28 ... |
6 | Bitcoin Survival | Bitcoin Überleben | Bitcoin Survival | Bitcoin Survival |
7 | Massive Bitcoin Sell Going On | Massive Bitcoin verkaufen weiter | Massive Bitcoin Sell Going On | Massive Bitcoin Sell Going On |
8 | Why Bitcoin will rise on Monday 23rd by more t... | Warum Bitcoin am Montag um mehr als 10% steige... | Why Bitcoin will rise on Monday 23rd by more t... | Why Bitcoin will rise on Monday 23rd by more t... |
9 | Why Bitcoin is falling, and will rise again | Warum Bitcoin fällt und wieder aufsteigt | Why Bitcoin is falling, and will rise again | Why Bitcoin is falling, and will rise again |
10 | Bitcoin Price in 2014 | Bitcoin Preis 2014 | Bitcoin Price in 2014 | Bitcoin Price in 2014 |
11 | This is probably the best time to invest in Bi... | Dies ist wahrscheinlich die beste Zeit, um in ... | This is probably the best time to invest in Bi... | This is probably the best time to invest in Bi... |
12 | Comparing Bitcoins and Oranges | Vergleich von Bitcoins und Orangen | Comparing Bitcoins and Oranges | Comparing Bitcoins and Oranges |
13 | Key Bitcoin Price Levels for the Week Ahead | Die wichtigsten Preisniveaus für Bitcoin für d... | Key Bitcoin Price Levels for the Week Ahead | Key Bitcoin Price Levels for the Week Ahead |
14 | The 2014 Bitcoin War Has Started— And it is Bo... | Der Bitcoin-Krieg 2014 hat begonnen - Und es i... | The 2014 Bitcoin War Has Started— And it is Bo... | The 2014 Bitcoin War Has Started— And it is Bo... |
15 | Two new Bitcoin ASIC Miners : Scam or Real? | Zwei neue Bitcoin ASIC Miners: Scam oder Real? | Two new Bitcoin ASIC Miners : Scam or Real? | Two new Bitcoin ASIC Miners : Scam or Real? |
16 | KnCMiner Announces 2TH/s Bitcoin ASIC Miner | KnCMiner kündigt 2. Bitcoin ASIC Miner an | KnCMiner Announces 2TH/s Bitcoin ASIC Miner | KnCMiner Announces 2TH/s Bitcoin ASIC Miner |
17 | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax | PSA: Fallen Sie nicht für den Apple Bitcoin Mi... | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax |
18 | Breaking News: HashCows Mining Pool Hacked | Nachrichten: | Breaking News: | Breaking News: HashCows Mining Pool Hacked |
18 | Breaking News: HashCows Mining Pool Hacked | HashCows Bergbau Pool gehackt | HashCows Mining Pool Hacked | Breaking News: HashCows Mining Pool Hacked |
19 | Primecoin Cloud Mining using Digital Ocean – T... | Primecoin Cloud Mining mit Digital Ocean - Der... | Primecoin Cloud Mining using Digital Ocean – T... | Primecoin Cloud Mining using Digital Ocean – T... |
translate_pipe = nlu.load('en.translate_to.fr')
tr_df = translate_pipe.predict(df)
tr_df
translate_en_fr download started this may take some time. Approx size to download 376 MB [OK!]
text | translation | sentence | title | |
---|---|---|---|---|
origin_index | ||||
0 | Bitcoin Price Update: Will China Lead us Down? | Bitcoin Mise à jour de prix: La Chine nous mèn... | Bitcoin Price Update: Will China Lead us Down? | Bitcoin Price Update: Will China Lead us Down? |
1 | Key Bitcoin Price Levels for Week 51 (15 – 22 ... | Niveaux clés des prix Bitcoin pour la semaine ... | Key Bitcoin Price Levels for Week 51 (15 – 22 ... | Key Bitcoin Price Levels for Week 51 (15 – 22 ... |
2 | National Australia Bank, Citing Highly Flawed ... | Banque nationale d'Australie, Citant des donné... | National Australia Bank, Citing Highly Flawed ... | National Australia Bank, Citing Highly Flawed ... |
3 | Chinese Bitcoin Ban Driven by Chinese Banking... | L'interdiction de Bitcoin chinois sous l'influ... | Chinese Bitcoin Ban Driven by Chinese Banking... | Chinese Bitcoin Ban Driven by Chinese Banking... |
4 | Bitcoin Trade Update: Opened Position | Bitcoin Trade Update: position ouverte | Bitcoin Trade Update: Opened Position | Bitcoin Trade Update: Opened Position |
5 | Key Bitcoin Price Levels for Week 52 (22 – 28 ... | Niveaux clés des prix Bitcoin pour la semaine ... | Key Bitcoin Price Levels for Week 52 (22 – 28 ... | Key Bitcoin Price Levels for Week 52 (22 – 28 ... |
6 | Bitcoin Survival | Survie Bitcoin | Bitcoin Survival | Bitcoin Survival |
7 | Massive Bitcoin Sell Going On | Vente de Bitcoin massif allant sur | Massive Bitcoin Sell Going On | Massive Bitcoin Sell Going On |
8 | Why Bitcoin will rise on Monday 23rd by more t... | Pourquoi Bitcoin montera le lundi 23ème de plu... | Why Bitcoin will rise on Monday 23rd by more t... | Why Bitcoin will rise on Monday 23rd by more t... |
9 | Why Bitcoin is falling, and will rise again | Pourquoi Bitcoin est en baisse, et va se lever... | Why Bitcoin is falling, and will rise again | Why Bitcoin is falling, and will rise again |
10 | Bitcoin Price in 2014 | Prix Bitcoin en 2014 | Bitcoin Price in 2014 | Bitcoin Price in 2014 |
11 | This is probably the best time to invest in Bi... | C'est probablement le meilleur moment pour inv... | This is probably the best time to invest in Bi... | This is probably the best time to invest in Bi... |
12 | Comparing Bitcoins and Oranges | Comparaison des Bitcoins et des Oranges | Comparing Bitcoins and Oranges | Comparing Bitcoins and Oranges |
13 | Key Bitcoin Price Levels for the Week Ahead | Principaux niveaux de prix Bitcoin pour la sem... | Key Bitcoin Price Levels for the Week Ahead | Key Bitcoin Price Levels for the Week Ahead |
14 | The 2014 Bitcoin War Has Started— And it is Bo... | La guerre de Bitcoin 2014 a commencé - Et c'es... | The 2014 Bitcoin War Has Started— And it is Bo... | The 2014 Bitcoin War Has Started— And it is Bo... |
15 | Two new Bitcoin ASIC Miners : Scam or Real? | Deux nouveaux Bitcoin ASIC Miners: Scam ou Real? | Two new Bitcoin ASIC Miners : Scam or Real? | Two new Bitcoin ASIC Miners : Scam or Real? |
16 | KnCMiner Announces 2TH/s Bitcoin ASIC Miner | KNCMiner annonce 2TH/s Bitcoin ASIC Miner | KnCMiner Announces 2TH/s Bitcoin ASIC Miner | KnCMiner Announces 2TH/s Bitcoin ASIC Miner |
17 | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax | PSA: Ne tombez pas pour le Bitcoin Apple Miner... | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax |
18 | Breaking News: HashCows Mining Pool Hacked | Nouvelles de la nouvelle : | Breaking News: | Breaking News: HashCows Mining Pool Hacked |
18 | Breaking News: HashCows Mining Pool Hacked | HashCows mine Pool piraté | HashCows Mining Pool Hacked | Breaking News: HashCows Mining Pool Hacked |
19 | Primecoin Cloud Mining using Digital Ocean – T... | Primecoin Cloud Mining utilisant l'océan numér... | Primecoin Cloud Mining using Digital Ocean – T... | Primecoin Cloud Mining using Digital Ocean – T... |
translate_pipe = nlu.load('en.translate_to.he')
tr_df = translate_pipe.predict(df)
tr_df
translate_en_he download started this may take some time. Approx size to download 399.3 MB [OK!]
text | translation | sentence | title | |
---|---|---|---|---|
origin_index | ||||
0 | Bitcoin Price Update: Will China Lead us Down? | עדכון מחיר הביטקוין: האם סין תוביל אותנו למטה? | Bitcoin Price Update: Will China Lead us Down? | Bitcoin Price Update: Will China Lead us Down? |
1 | Key Bitcoin Price Levels for Week 51 (15 – 22 ... | רמת מחירי Bitcoin מפתח לשבוע 51 (15 - 22 Dec) | Key Bitcoin Price Levels for Week 51 (15 – 22 ... | Key Bitcoin Price Levels for Week 51 (15 – 22 ... |
2 | National Australia Bank, Citing Highly Flawed ... | בנק אוסטרליה הלאומי, ציוץ מידע שגוי ביותר, קור... | National Australia Bank, Citing Highly Flawed ... | National Australia Bank, Citing Highly Flawed ... |
3 | Chinese Bitcoin Ban Driven by Chinese Banking... | סיני Bitcoin באן מונע על ידי משבר הבנקאות הסיני? | Chinese Bitcoin Ban Driven by Chinese Banking... | Chinese Bitcoin Ban Driven by Chinese Banking... |
4 | Bitcoin Trade Update: Opened Position | עדכון ביטקוין: מיקום פתוח | Bitcoin Trade Update: Opened Position | Bitcoin Trade Update: Opened Position |
5 | Key Bitcoin Price Levels for Week 52 (22 – 28 ... | רמת מחירי Bitcoin מפתח לשבוע 52 (22 - 28 Dec) ... | Key Bitcoin Price Levels for Week 52 (22 – 28 ... | Key Bitcoin Price Levels for Week 52 (22 – 28 ... |
6 | Bitcoin Survival | הישרדות הביטקוין | Bitcoin Survival | Bitcoin Survival |
7 | Massive Bitcoin Sell Going On | מכירת Bitcoin מסיבית ממשיכה | Massive Bitcoin Sell Going On | Massive Bitcoin Sell Going On |
8 | Why Bitcoin will rise on Monday 23rd by more t... | למה Bitcoin יעלה ביום שני 23 על ידי יותר מ -10% | Why Bitcoin will rise on Monday 23rd by more t... | Why Bitcoin will rise on Monday 23rd by more t... |
9 | Why Bitcoin is falling, and will rise again | למה ביטקוין נופל, ויתרומם שוב | Why Bitcoin is falling, and will rise again | Why Bitcoin is falling, and will rise again |
10 | Bitcoin Price in 2014 | Bitcoin Price בשנת 2014 | Bitcoin Price in 2014 | Bitcoin Price in 2014 |
11 | This is probably the best time to invest in Bi... | זה כנראה הזמן הטוב ביותר להשקיע בביטקוין. | This is probably the best time to invest in Bi... | This is probably the best time to invest in Bi... |
12 | Comparing Bitcoins and Oranges | השוואה בין ביטקוין ותפוזים | Comparing Bitcoins and Oranges | Comparing Bitcoins and Oranges |
13 | Key Bitcoin Price Levels for the Week Ahead | רמת מחיר Bitcoin מפתח לשבוע קדימה | Key Bitcoin Price Levels for the Week Ahead | Key Bitcoin Price Levels for the Week Ahead |
14 | The 2014 Bitcoin War Has Started— And it is Bo... | מלחמת הביטקוין 2014 החלה - וזה בובי לי שירה את... | The 2014 Bitcoin War Has Started— And it is Bo... | The 2014 Bitcoin War Has Started— And it is Bo... |
15 | Two new Bitcoin ASIC Miners : Scam or Real? | שני כורים חדשים של ביטקוין: | Two new Bitcoin ASIC Miners : Scam or Real? | Two new Bitcoin ASIC Miners : Scam or Real? |
16 | KnCMiner Announces 2TH/s Bitcoin ASIC Miner | KNCMiner מכריז 2/S Bitcoin ASIC כורה | KnCMiner Announces 2TH/s Bitcoin ASIC Miner | KnCMiner Announces 2TH/s Bitcoin ASIC Miner |
17 | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax | נ.ס: אל תיפול בפח של מיינר הוקס | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax | PSA: Do Not Fall for the Apple Bitcoin Miner Hoax |
18 | Breaking News: HashCows Mining Pool Hacked | חדשות חמות: | Breaking News: | Breaking News: HashCows Mining Pool Hacked |
18 | Breaking News: HashCows Mining Pool Hacked | בריכת הכרייה של "הסכ" נפרצה | HashCows Mining Pool Hacked | Breaking News: HashCows Mining Pool Hacked |
19 | Primecoin Cloud Mining using Digital Ocean – T... | כריית ענן פרימקוין באמצעות האוקיינוס הדיגיטלי ... | Primecoin Cloud Mining using Digital Ocean – T... | Primecoin Cloud Mining using Digital Ocean – T... |
while 1 : 1
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-6-349c1ec5b685> in <module>() ----> 1 while 1 : 1 KeyboardInterrupt: