!pip install spacy
Requirement already satisfied: spacy in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (3.1.1) Requirement already satisfied: preshed<3.1.0,>=3.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (3.0.5) Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (4.59.0) Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (1.7.4) Requirement already satisfied: setuptools in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (54.1.1) Requirement already satisfied: requests<3.0.0,>=2.13.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (2.25.1)
WARNING: You are using pip version 21.1.2; however, version 21.2.4 is available.
Requirement already satisfied: cymem<2.1.0,>=2.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (2.0.5) Requirement already satisfied: jinja2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (2.11.3) Requirement already satisfied: srsly<3.0.0,>=2.4.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (2.4.1) Requirement already satisfied: catalogue<2.1.0,>=2.0.4 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (2.0.4) Requirement already satisfied: blis<0.8.0,>=0.4.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (0.7.4) Requirement already satisfied: thinc<8.1.0,>=8.0.8 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (8.0.8) Requirement already satisfied: numpy>=1.15.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (1.20.2) Requirement already satisfied: pathy>=0.3.5 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (0.4.0) Requirement already satisfied: packaging>=20.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (20.9) Requirement already satisfied: typer<0.4.0,>=0.3.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (0.3.2) Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.7 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (3.0.8) Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (1.0.5) Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy) (0.8.2) Requirement already satisfied: pyparsing>=2.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from packaging>=20.0->spacy) (2.4.7) Requirement already satisfied: smart-open<4.0.0,>=2.2.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from pathy>=0.3.5->spacy) (3.0.0) Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (1.26.3) Requirement already satisfied: certifi>=2017.4.17 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (2020.12.5) Requirement already satisfied: idna<3,>=2.5 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (2.10) Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy) (4.0.0) Requirement already satisfied: click<7.2.0,>=7.1.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from typer<0.4.0,>=0.3.0->spacy) (7.1.2) Requirement already satisfied: MarkupSafe>=0.23 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from jinja2->spacy) (1.1.1)
You should consider upgrading via the 'c:\users\wma22\appdata\local\programs\python\python39\python.exe -m pip install --upgrade pip' command.
!python -m spacy download en_core_web_sm
Collecting en-core-web-sm==3.1.0
2021-09-14 12:51:21.698439: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll WARNING: You are using pip version 21.1.2; however, version 21.2.4 is available. You should consider upgrading via the 'C:\Users\wma22\AppData\Local\Programs\Python\Python39\python.exe -m pip install --upgrade pip' command.
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl (13.6 MB) Requirement already satisfied: spacy<3.2.0,>=3.1.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from en-core-web-sm==3.1.0) (3.1.1) Requirement already satisfied: srsly<3.0.0,>=2.4.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.4.1) Requirement already satisfied: typer<0.4.0,>=0.3.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.3.2) Requirement already satisfied: numpy>=1.15.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.20.2) Requirement already satisfied: blis<0.8.0,>=0.4.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.7.4) Requirement already satisfied: requests<3.0.0,>=2.13.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.25.1) Requirement already satisfied: catalogue<2.1.0,>=2.0.4 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.4) Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.7.4) Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.7 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.8) Requirement already satisfied: packaging>=20.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (20.9) Requirement already satisfied: cymem<2.1.0,>=2.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.5) Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.8.2) Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.0.5) Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (4.59.0) Requirement already satisfied: pathy>=0.3.5 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.4.0) Requirement already satisfied: thinc<8.1.0,>=8.0.8 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (8.0.8) Requirement already satisfied: setuptools in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (54.1.1) Requirement already satisfied: jinja2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.11.3) Requirement already satisfied: preshed<3.1.0,>=3.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.5) Requirement already satisfied: pyparsing>=2.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from packaging>=20.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.4.7) Requirement already satisfied: smart-open<4.0.0,>=2.2.0 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.0) Requirement already satisfied: urllib3<1.27,>=1.21.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.26.3) Requirement already satisfied: certifi>=2017.4.17 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2020.12.5) Requirement already satisfied: chardet<5,>=3.0.2 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (4.0.0) Requirement already satisfied: idna<3,>=2.5 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.10) Requirement already satisfied: click<7.2.0,>=7.1.1 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from typer<0.4.0,>=0.3.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (7.1.2) Requirement already satisfied: MarkupSafe>=0.23 in c:\users\wma22\appdata\local\programs\python\python39\lib\site-packages (from jinja2->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.1.1) [+] Download and installation successful You can now load the package via spacy.load('en_core_web_sm')
import spacy
INFO:tensorflow:Enabling eager execution INFO:tensorflow:Enabling v2 tensorshape INFO:tensorflow:Enabling resource variables INFO:tensorflow:Enabling tensor equality INFO:tensorflow:Enabling control flow v2
nlp = spacy.load("en_core_web_sm")
with open ("data/wiki_us.txt", "r") as f:
text = f.read()
print (text)
The United States of America (U.S.A. or USA), commonly known as the United States (U.S. or US) or America, is a country primarily located in North America. It consists of 50 states, a federal district, five major unincorporated territories, 326 Indian reservations, and some minor possessions.[j] At 3.8 million square miles (9.8 million square kilometers), it is the world's third- or fourth-largest country by total area.[d] The United States shares significant land borders with Canada to the north and Mexico to the south, as well as limited maritime borders with the Bahamas, Cuba, and Russia.[22] With a population of more than 331 million people, it is the third most populous country in the world. The national capital is Washington, D.C., and the most populous city is New York. Paleo-Indians migrated from Siberia to the North American mainland at least 12,000 years ago, and European colonization began in the 16th century. The United States emerged from the thirteen British colonies established along the East Coast. Disputes over taxation and political representation with Great Britain led to the American Revolutionary War (1775–1783), which established independence. In the late 18th century, the U.S. began expanding across North America, gradually obtaining new territories, sometimes through war, frequently displacing Native Americans, and admitting new states; by 1848, the United States spanned the continent. Slavery was legal in the southern United States until the second half of the 19th century when the American Civil War led to its abolition. The Spanish–American War and World War I established the U.S. as a world power, a status confirmed by the outcome of World War II. During the Cold War, the United States fought the Korean War and the Vietnam War but avoided direct military conflict with the Soviet Union. The two superpowers competed in the Space Race, culminating in the 1969 spaceflight that first landed humans on the Moon. The Soviet Union's dissolution in 1991 ended the Cold War, leaving the United States as the world's sole superpower. The United States is a federal republic and a representative democracy with three separate branches of government, including a bicameral legislature. It is a founding member of the United Nations, World Bank, International Monetary Fund, Organization of American States, NATO, and other international organizations. It is a permanent member of the United Nations Security Council. Considered a melting pot of cultures and ethnicities, its population has been profoundly shaped by centuries of immigration. The country ranks high in international measures of economic freedom, quality of life, education, and human rights, and has low levels of perceived corruption. However, the country has received criticism concerning inequality related to race, wealth and income, the use of capital punishment, high incarceration rates, and lack of universal health care. The United States is a highly developed country, accounts for approximately a quarter of global GDP, and is the world's largest economy. By value, the United States is the world's largest importer and the second-largest exporter of goods. Although its population is only 4.2% of the world's total, it holds 29.4% of the total wealth in the world, the largest share held by any country. Making up more than a third of global military spending, it is the foremost military power in the world; and it is a leading political, cultural, and scientific force internationally.[23]
doc = nlp(text)
print (doc)
The United States of America (U.S.A. or USA), commonly known as the United States (U.S. or US) or America, is a country primarily located in North America. It consists of 50 states, a federal district, five major unincorporated territories, 326 Indian reservations, and some minor possessions.[j] At 3.8 million square miles (9.8 million square kilometers), it is the world's third- or fourth-largest country by total area.[d] The United States shares significant land borders with Canada to the north and Mexico to the south, as well as limited maritime borders with the Bahamas, Cuba, and Russia.[22] With a population of more than 331 million people, it is the third most populous country in the world. The national capital is Washington, D.C., and the most populous city is New York. Paleo-Indians migrated from Siberia to the North American mainland at least 12,000 years ago, and European colonization began in the 16th century. The United States emerged from the thirteen British colonies established along the East Coast. Disputes over taxation and political representation with Great Britain led to the American Revolutionary War (1775–1783), which established independence. In the late 18th century, the U.S. began expanding across North America, gradually obtaining new territories, sometimes through war, frequently displacing Native Americans, and admitting new states; by 1848, the United States spanned the continent. Slavery was legal in the southern United States until the second half of the 19th century when the American Civil War led to its abolition. The Spanish–American War and World War I established the U.S. as a world power, a status confirmed by the outcome of World War II. During the Cold War, the United States fought the Korean War and the Vietnam War but avoided direct military conflict with the Soviet Union. The two superpowers competed in the Space Race, culminating in the 1969 spaceflight that first landed humans on the Moon. The Soviet Union's dissolution in 1991 ended the Cold War, leaving the United States as the world's sole superpower. The United States is a federal republic and a representative democracy with three separate branches of government, including a bicameral legislature. It is a founding member of the United Nations, World Bank, International Monetary Fund, Organization of American States, NATO, and other international organizations. It is a permanent member of the United Nations Security Council. Considered a melting pot of cultures and ethnicities, its population has been profoundly shaped by centuries of immigration. The country ranks high in international measures of economic freedom, quality of life, education, and human rights, and has low levels of perceived corruption. However, the country has received criticism concerning inequality related to race, wealth and income, the use of capital punishment, high incarceration rates, and lack of universal health care. The United States is a highly developed country, accounts for approximately a quarter of global GDP, and is the world's largest economy. By value, the United States is the world's largest importer and the second-largest exporter of goods. Although its population is only 4.2% of the world's total, it holds 29.4% of the total wealth in the world, the largest share held by any country. Making up more than a third of global military spending, it is the foremost military power in the world; and it is a leading political, cultural, and scientific force internationally.[23]
print (len(text))
print (len(doc))
3525 652
for token in text[0:10]:
print (token)
T h e U n i t e d
for token in doc[:10]:
print (token)
The United States of America ( U.S.A. or USA )
for token in text.split()[:10]:
print (token)
The United States of America (U.S.A. or USA), commonly known
for sent in doc.sents:
print (sent)
The United States of America (U.S.A. or USA), commonly known as the United States (U.S. or US) or America, is a country primarily located in North America. It consists of 50 states, a federal district, five major unincorporated territories, 326 Indian reservations, and some minor possessions.[j] At 3.8 million square miles (9.8 million square kilometers), it is the world's third- or fourth-largest country by total area.[d] The United States shares significant land borders with Canada to the north and Mexico to the south, as well as limited maritime borders with the Bahamas, Cuba, and Russia.[22] With a population of more than 331 million people, it is the third most populous country in the world. The national capital is Washington, D.C., and the most populous city is New York. Paleo-Indians migrated from Siberia to the North American mainland at least 12,000 years ago, and European colonization began in the 16th century. The United States emerged from the thirteen British colonies established along the East Coast. Disputes over taxation and political representation with Great Britain led to the American Revolutionary War (1775–1783), which established independence. In the late 18th century, the U.S. began expanding across North America, gradually obtaining new territories, sometimes through war, frequently displacing Native Americans, and admitting new states; by 1848, the United States spanned the continent. Slavery was legal in the southern United States until the second half of the 19th century when the American Civil War led to its abolition. The Spanish–American War and World War I established the U.S. as a world power, a status confirmed by the outcome of World War II. During the Cold War, the United States fought the Korean War and the Vietnam War but avoided direct military conflict with the Soviet Union. The two superpowers competed in the Space Race, culminating in the 1969 spaceflight that first landed humans on the Moon. The Soviet Union's dissolution in 1991 ended the Cold War, leaving the United States as the world's sole superpower. The United States is a federal republic and a representative democracy with three separate branches of government, including a bicameral legislature. It is a founding member of the United Nations, World Bank, International Monetary Fund, Organization of American States, NATO, and other international organizations. It is a permanent member of the United Nations Security Council. Considered a melting pot of cultures and ethnicities, its population has been profoundly shaped by centuries of immigration. The country ranks high in international measures of economic freedom, quality of life, education, and human rights, and has low levels of perceived corruption. However, the country has received criticism concerning inequality related to race, wealth and income, the use of capital punishment, high incarceration rates, and lack of universal health care. The United States is a highly developed country, accounts for approximately a quarter of global GDP, and is the world's largest economy. By value, the United States is the world's largest importer and the second-largest exporter of goods. Although its population is only 4.2% of the world's total, it holds 29.4% of the total wealth in the world, the largest share held by any country. Making up more than a third of global military spending, it is the foremost military power in the world; and it is a leading political, cultural, and scientific force internationally.[23]
sentence1 = doc.sents[0]
print (sentence1)
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-14-925a537c08a9> in <module> ----> 1 sentence1 = doc.sents[0] 2 print (sentence1) TypeError: 'generator' object is not subscriptable
sentence1 = list(doc.sents)[0]
print (sentence1)
The United States of America (U.S.A. or USA), commonly known as the United States (U.S. or US) or America, is a country primarily located in North America.
for token in doc[:10]:
print (token)
The United States of America ( U.S.A. or USA )
token2 = sentence1[2]
print (token2)
States
token2.text
'States'
token2.left_edge
The
token2.right_edge
America
token2.ent_type
384
token2.ent_type_
'GPE'
token2.ent_iob_
'I'
token2.lemma_
'States'
sentence1[12].lemma_
'know'
print (sentence1[12])
known
token2.morph
NounType=Prop|Number=Sing
sentence1[12].morph
Aspect=Perf|Tense=Past|VerbForm=Part
token2.pos_
'PROPN'
token2.dep_
'nsubj'
token2.lang_
'en'
text = "Mike enjoys playing football."
doc2 = nlp(text)
print (doc2)
Mike enjoys playing football.
for token in doc2:
print (token.text, token.pos_, token.dep_)
Mike PROPN nsubj enjoys VERB ROOT playing VERB xcomp football NOUN dobj . PUNCT punct
from spacy import displacy
displacy.render(doc2, style="dep")
for ent in doc.ents:
print (ent.text, ent.label_)
The United States of America GPE U.S.A. GPE USA GPE the United States GPE U.S. GPE US GPE America GPE North America LOC 50 CARDINAL five CARDINAL 326 CARDINAL Indian NORP 3.8 million square miles QUANTITY 9.8 million square kilometers QUANTITY fourth ORDINAL The United States GPE Canada GPE Mexico GPE Bahamas GPE Cuba GPE more than 331 million CARDINAL third ORDINAL Washington GPE D.C. GPE New York GPE Paleo-Indians NORP Siberia LOC North American NORP at least 12,000 years ago DATE European NORP the 16th century DATE The United States GPE thirteen CARDINAL British NORP the East Coast LOC Great Britain GPE the American Revolutionary War ORG the late 18th century DATE U.S. GPE North America LOC Native Americans NORP 1848 DATE the United States GPE United States GPE the second half of the 19th century DATE the American Civil War EVENT The Spanish–American War and World War EVENT U.S. GPE World War II EVENT the Cold War EVENT the United States GPE the Korean War EVENT the Vietnam War EVENT the Soviet Union GPE two CARDINAL the Space Race FAC 1969 DATE first ORDINAL The Soviet Union's GPE 1991 DATE the Cold War EVENT the United States GPE The United States GPE three CARDINAL the United Nations ORG World Bank ORG International Monetary Fund ORG Organization of American States ORG NATO ORG the United Nations Security Council ORG centuries DATE The United States GPE approximately a quarter DATE the United States GPE second ORDINAL only 4.2% PERCENT 29.4% PERCENT more than a third CARDINAL
displacy.render(doc, style="ent")