Mechanisms notebook

In [25]:
import numpy as np
import pdb
from Mechanisms_bot.src import pubmed_io
import Mechanisms_bot.tweet_mechanism as tweet_mechanism
from Mechanisms_bot.src import abstract_parser 
import importlib
import pickle
import re

Query Pubmed for sentences

Get PMIDs for sentences containing mechanisms and lack of clarity

In [3]:
pubmed_ids = pubmed_io.get_pubmed_ids(99000)
len(pubmed_ids)
Out[3]:
99000

Get the abstracts for those articles

In [4]:
importlib.reload(pubmed_io)
pubmed_abstracts = pubmed_io.get_pubmed_abstracts( pubmed_ids)
len(pubmed_abstracts)
Out[4]:
98536

All of the above in one function

In [ ]:
tweet_mechanism.get_recent_mech_sentences(90000)

Pubmed returns an XML. It is structured PubmedArticleList / PubmedArticle / MedlineCitation. MedlineCitation has children: PMID, and Article

Article has children: Journal, ArticleTitle, and Abstract

Abstract has child: AbstractText

Make sure that the mechanism and lack of clarity are in the same sentence

In [5]:
good_sent = list(map(abstract_parser.get_mech_sent, pubmed_abstracts, pubmed_ids) )
good_sent = [x for x in good_sent if x['mech_sent']]
len(good_sent)
Out[5]:
24945

Save and load, as necessary

In [8]:
with open('good_sent.pickle', 'wb') as pickle_file:
    pickle.dump(good_sent, pickle_file )

Quick stats on sentences

In [2]:
with open('good_sent.pickle', 'rb') as pickle_file:
    good_sent = pickle.load( pickle_file )
In [54]:
unknown = {'understood', 'unclear', 'unknown'}
end_unknown = sum([1 for x in good_sent if x['mech_sent'].split(' ')[-1][:-1] in unknown])
print('Number of sentences ending with variant of "unknown": {}'.format(end_unknown) )
Number of sentences ending with variant of "unknown": 19459
In [55]:
however = sum([1 for x in good_sent if re.findall('however', x['mech_sent'].lower()) ])
print('Number of sentences containing "however": {}'.format(however) )
Number of sentences containing "however": 8889
In [56]:
although = sum([1 for x in good_sent if re.findall('although', x['mech_sent'].lower()) ])
print('Number of sentences containing "although": {}'.format(although) )
Number of sentences containing "although": 2822
In [57]:
while_count = sum([1 for x in good_sent if re.findall('while', x['mech_sent'].lower()) ])
print('Number of sentences containing "while": {}'.format(while_count) )
Number of sentences containing "while": 612
In [61]:
sent_length = [len(x['mech_sent']) for x in good_sent]
sent_min = good_sent[np.argmin(sent_length)]
print('Shortest sentence: {0}, PMID: {1}'.format(sent_min['mech_sent'], sent_min['PMID'])  )
Shortest sentence: The mechanism is unclear., PMID: 25602775
In [63]:
# These are long sentences due to punctuation
for i, cur_sent in enumerate(good_sent):
    if cur_sent['PMID'] in {'26526306', '25454993', '26351365', '26648182'}:
        print(i)
1255
2721
4914
14158
16001

A fairly long sentence:

In [47]:
sent_length = [len(x['mech_sent']) for x in good_sent[:1255]]
sent_max = good_sent[np.argmax(sent_length)]
sent_max
Out[47]:
{'PMID': '26663484',
 'mech_sent': "Although an increasing number of studies have identified misregulated miRNAs in the neurodegenerative diseases (NDDs) Alzheimer's disease, Parkinson's disease, Huntington's disease, and amyotrophic lateral sclerosis, which suggests that alterations in the miRNA regulatory pathway could contribute to disease pathogenesis, the molecular mechanisms underlying the pathological implications of misregulated miRNA expression and the regulation of the key genes involved in NDDs remain largely unknown."}

A sample of mechanism sentences:

In [18]:
good_sent[:100]
Out[18]:
[{'PMID': '26752988',
  'mech_sent': 'Lithium and valproate modulate disturbances in intracellular calcium homeostasis implicated in the pathophysiology of bipolar disorder, but the molecular mechanisms are not fully understood.'},
 {'PMID': '26752791',
  'mech_sent': 'Sin Nombre virus, SNV), the mechanism is largely unknown.'},
 {'PMID': '26752716',
  'mech_sent': 'Orchestrated trophoblast differentiation is necessary to establish and maintain a normal pregnancy, however the molecular mechanisms that guide this process remain largely unknown.'},
 {'PMID': '26752685',
  'mech_sent': 'However, the functional consequences of this outside of the HIF pathway remain unclear.'},
 {'PMID': '26752654',
  'mech_sent': 'Yet the physicochemical mechanisms underlying such mineral formation and growth in atheromata remain unknown.'},
 {'PMID': '26752649',
  'mech_sent': 'Although defects in intestinal barrier function are a key pathogenic factor in patients with inflammatory bowel diseases (IBDs), the molecular pathways driving disease-specific alterations of intestinal epithelial cells (IECs) are largely unknown.'},
 {'PMID': '26752420',
  'mech_sent': 'Meanwhile, the pathology of renal impairment in the early phase of CLDs is poorly understood, and animal models to elucidate its mechanisms are needed.'},
 {'PMID': '26752351',
  'mech_sent': 'Despite the extensive research, the exact neuropathological mechanisms remain unknown, which hinders our ability to develop effective treatments.'},
 {'PMID': '26752289',
  'mech_sent': "Mycobacterium fortuitum causes 'mycobacteriosis' in wide range of hosts although the mechanisms remain largely unknown."},
 {'PMID': '26752242',
  'mech_sent': 'Graphene oxide (GO) has great potential for actinide removal due to its extremely high sorption capacity, but the mechanism of sorption remains unclear.'},
 {'PMID': '26752181',
  'mech_sent': 'However, the underlying molecular mechanism is still poorly understood.'},
 {'PMID': '26752119',
  'mech_sent': 'However, the mechanisms of action for SSRIs are unclear and besides neurotransmitter modulation may depend on modulation of the hypothalamic-pituitary-adrenal (HPA) system.'},
 {'PMID': '26752050',
  'mech_sent': 'The molecular mechanisms underlying feed efficiency need to be better understood to improve animal efficiency, a research priority to support a competitive and sustainable livestock production.'},
 {'PMID': '26751966',
  'mech_sent': 'Although a strong dependence of some tumor types on exogenous arginine sources has been reported, the mechanisms of arginine sensing by tumor cells and the impact of changes in arginine availability on translation and cell cycle regulation are not fully understood.'},
 {'PMID': '26751936',
  'mech_sent': 'Evidence indicates that CaSm modulates messenger RNA degradation; however, its target genes and the mechanisms by which CaSm promotes pancreatic cancer remain largely unknown.'},
 {'PMID': '26751847',
  'mech_sent': 'However, the detailed mechanisms of ICAM-1 in breast cancer remain unclear.'},
 {'PMID': '26751842',
  'mech_sent': 'While experimental studies have provided an abundance of information about the properties and mechanisms of action of individual mediators, essential system-level regulatory patterns that determine the time-course of inflammation are not sufficiently understood.'},
 {'PMID': '26751788',
  'mech_sent': 'Genetic analyses in Drosophila have identified conserved signaling pathways controlling size; however, how these pathways control phenotypic diversity is unclear.'},
 {'PMID': '26751691',
  'mech_sent': 'The underlying molecular mechanisms and the physiological significance of such association remain poorly understood.'},
 {'PMID': '26751615',
  'mech_sent': 'However, the underlying mechanisms are not well understood necessitating an appropriate animal model to answer fundamental questions and conduct translational research into optimal interventions.'},
 {'PMID': '26751594',
  'mech_sent': 'The effects and mechanisms of ketogenic diets (KD) are unclear.'},
 {'PMID': '26751515',
  'mech_sent': 'In addition, inhibitory mechanisms-especially local inhibitory mechanisms, which ensure proper spatial and kinetic controls of activation-are not well understood.'},
 {'PMID': '26751467',
  'mech_sent': 'However, previous findings have been inconsistent and the mechanisms responsible remain unclear.'},
 {'PMID': '26751383',
  'mech_sent': 'Women exposed to phthalates during pregnancy are at increased risk for delivering preterm, but the mechanism behind this relationship is unknown.'},
 {'PMID': '26751281',
  'mech_sent': 'Yet the underlying mechanism of the CGM signals is not fully understood due to the complexity of the electrostatic interactions.'},
 {'PMID': '26751268',
  'mech_sent': 'Although the effects of expansins on cell walls and cellulose fibrils are known, the molecular mechanism underlying their biophysical function is poorly understood.'},
 {'PMID': '26751128',
  'mech_sent': 'Although natural organic matter (NOM) is known to affect biological reduction of U(VI) and subsequent reoxidation of U(IV), the underlying mechanisms remain unclear.'},
 {'PMID': '26751127',
  'mech_sent': 'However, the effects on adrenocortical function and the underlying mechanisms are not fully understood.'},
 {'PMID': '26751060',
  'mech_sent': "Identifying patients likely to respond remains a challenge since mobilizations' mechanism(s) of action are unclear."},
 {'PMID': '26750873',
  'mech_sent': 'Betulinic acid (BA) is a naturally occurring pentacyclic triterpene that attenuates vascular diseases and atherosclerosis, but the mechanism by which it stimulates endothelial nitric oxide synthase (eNOS) is unclear.'},
 {'PMID': '26750863',
  'mech_sent': 'Previous studies revealed that patients suffering from depression may also have altered immune responses, but the mechanisms underlying this correlation are unclear.'},
 {'PMID': '26750705',
  'mech_sent': 'However, the mechanism of ASI inhibiting neuroinflammation is unknown.'},
 {'PMID': '26750638',
  'mech_sent': 'Although BRAF(V600E) mutation is associated with adverse clinical outcomes in patients with colorectal cancer (CRC), response and resistance mechanisms for therapeutic BRAF(V600E) inhibitors remains poorly understood.'},
 {'PMID': '26750536',
  'mech_sent': 'Although there is evidence that interleukin-6 (IL-6) can promote gastric cancer progression, the underlying specific mechanisms and the mechanisms of gastric cancer lymphangiogenesis are largely unknown.'},
 {'PMID': '26750456',
  'mech_sent': 'However, the molecular mechanisms of SF-1 in the control of energy balance are largely unknown.'},
 {'PMID': '26750311',
  'mech_sent': 'Interleukin-23 (IL-23) is a pro-inflammatory cytokine required for the pathogenicity of T helper 17 (Th17) cells but the molecular mechanisms governing this process remain unclear.'},
 {'PMID': '26750157',
  'mech_sent': 'However, the cellular mechanism underlying this neuroprotection is still largely unknown.'},
 {'PMID': '26750019',
  'mech_sent': 'It is shown that SDF-1 combined with tissue engineered scaffolds will play a role of multipotent stem cells chemotaxis, however, the exact chemotaxis mechanism has not been fully understood.'},
 {'PMID': '26749462',
  'mech_sent': '6-Gingerol, a potent nutraceutical, has been shown to have antitumor activity in different tumors, although its mechanism of action is not well understood.'},
 {'PMID': '26749424',
  'mech_sent': 'Fibrocytes are collagen-producing leukocytes that accumulate in Scleroderma-associated interstitial lung disease (SSc-ILD) via unknown mechanisms.'},
 {'PMID': '26749315',
  'mech_sent': 'Although the etiology of this illness is not completely understood, FM pain is thought to rely on enhanced pain sensitivity maintained by central mechanisms.'},
 {'PMID': '26749305',
  'mech_sent': 'The mechanism determining the location of these COs is not well understood.'},
 {'PMID': '26749288',
  'mech_sent': 'Platelets promote angiogenesis of endothelial colony forming cells (ECFCs), with the underlying mechanisms not being fully understood.'},
 {'PMID': '26748963',
  'mech_sent': 'Biofilms have been implicated in delayed wound healing, although the mechanisms by which biofilms impair wound healing are poorly understood.'},
 {'PMID': '26748704',
  'mech_sent': 'Whereas Notch pathway components have been studied extensively, the effect of membrane distribution and dynamics of Notch receptors and ligands remains poorly understood.'},
 {'PMID': '26748700',
  'mech_sent': 'It is unclear whether these proteins have access to the complete machinery of the secretory pathway following local synthesis.'},
 {'PMID': '26748643',
  'mech_sent': 'Similarly, the mechanisms of adult lymphangiogenesis are poorly understood and have rarely been studied.'},
 {'PMID': '26748630',
  'mech_sent': 'The intracellular mechanism or how micro- or nano-patterned extracellular signal is ultimately linked to activity of nuclear transcription factors remains unknown.'},
 {'PMID': '26748512',
  'mech_sent': 'OGs may induce ethylene production and fruit ripening, however, the mechanism(s) behind these processes is unknown.'},
 {'PMID': '26748419',
  'mech_sent': 'Human embryonic stem cells (hESCs) are developmentally and clinically relevant for studying the poorly understood molecular mechanisms downstream of these cascades.'},
 {'PMID': '26748418',
  'mech_sent': 'Despite the widespread interest in direct neuronal reprogramming, the mechanisms underpinning fate conversion remain largely unknown.'},
 {'PMID': '26748397',
  'mech_sent': 'Propofol has been shown to exert cardioprotection, but the underlying mechanisms remain incompletely understood.'},
 {'PMID': '26748243',
  'mech_sent': 'While the molecular mechanism of the angiogenesis promoting capacity of TEMs remains unclear.'},
 {'PMID': '26748087',
  'mech_sent': 'While chronic pain is considered by some to be a CNS disease, little is understood about underlying neurobiological mechanisms.'},
 {'PMID': '26748042',
  'mech_sent': "The precise mechanisms linking H2O2 with the oxidation of signalling thiol proteins (phosphates, kinases and transcription factors) are unclear due to these proteins' low reactivity with H2O2 relative to abundant thiol peroxidases such as PRDX."},
 {'PMID': '26747908',
  'mech_sent': 'In addition, the mechanisms underlying the poor ability of carnivorous fish to use dietary carbohydrates as a major energy substrate are poorly understood.'},
 {'PMID': '26747895',
  'mech_sent': 'The strength and duration of nuclear factor-kappa B (NF-κB) signaling is tightly controlled at multiple levels under physiological conditions, but the mechanism underlying constitutive activation of the NF-κB pathway in cancer remains unclear.'},
 {'PMID': '26747779',
  'mech_sent': 'However, the mechanisms underlying LPS-induced ICAM-1 expression in human pulmonary alveolar epithelial cells (HPAEpiCs) remain unclear.'},
 {'PMID': '26747768',
  'mech_sent': 'The rat is one of several species in which a retino-raphe projection has been described; however, the retinal ganglion cell (RGC) types that contribute to this pathway are unknown.'},
 {'PMID': '26747726',
  'mech_sent': 'However, the molecular mechanisms involved are still unclear.'},
 {'PMID': '26747637',
  'mech_sent': 'The mechanisms linking cellular chirality to the large-scale asymmetry of multicellular structures are incompletely understood, and it has been suggested that the chirality of living cells is hardwired in their cytoskeleton.'},
 {'PMID': '26747614',
  'mech_sent': 'However, the detailed functions and mechanism of osteocalcin are not well understood and conflicting results have been obtained from researchers worldwide.'},
 {'PMID': '26747606',
  'mech_sent': 'Although AIP1 is known to be essential for a number of cellular events involving dynamic rearrangement of the actin cytoskeleton, regulatory mechanism of the function of AIP1 is unknown.'},
 {'PMID': '26747508',
  'mech_sent': 'We and others have recently reported that prolonged sitting impairs endothelial function in the leg vasculature; however, the mechanism(s) remain unknown.'},
 {'PMID': '26747403',
  'mech_sent': 'Among these mechanisms, a particular post-translational mechanism, namely the conversion of precursor BDNF into mature BDNF by proteolytic cleavage, was not fully understood.'},
 {'PMID': '26747402',
  'mech_sent': 'Although the underlying mechanisms remain unclear, there is evidence that enteric bacteria play a pivotal role.'},
 {'PMID': '26747300',
  'mech_sent': 'We previously reported the promising effects of dioscin against CCl4-induced liver fibrosis, but its effects and mechanisms on BDL- and DMN-induced liver fibrosis remain unknown.'},
 {'PMID': '26747207',
  'mech_sent': 'Nevertheless, the underlying pathophysiological mechanism of sleep disorders-induced cognitive decline is still largely unknown.'},
 {'PMID': '26747087',
  'mech_sent': 'However, HDAC inhibitors are small-molecule compounds without a specific target, and their mechanism of action is unclear.'},
 {'PMID': '26747071',
  'mech_sent': 'The mechanisms of renewal of neuronal-like cells are unclear.'},
 {'PMID': '26746985',
  'mech_sent': 'However, the functional significance and molecular mechanisms of other classes of non-coding RNAs in the regulation of cerebrovascular endothelial pathophysiology after stroke are completely unknown.'},
 {'PMID': '26746798',
  'mech_sent': 'Although this signaling pathway has been well understood in fission yeast, how the signaling is initiated and thus regulated remains incompletely understood.'},
 {'PMID': '26746666',
  'mech_sent': 'Studies have shown that progesterone enhances functional recovery after ischemic stroke, but the underlying mechanisms are not completely understood.'},
 {'PMID': '26746429',
  'mech_sent': 'However, the exact molecular mechanism is unknown.'},
 {'PMID': '26746337',
  'mech_sent': 'However, the neurological mechanism of BPI remains unclear, which provides a basis for further investigation.'},
 {'PMID': '26745969',
  'mech_sent': 'The mechanisms by which polyphenols exert their action are not fully understood, but it is clear that they have a direct effect through their antioxidant activities.'},
 {'PMID': '26745886',
  'mech_sent': 'The underlying mechanisms for the risk to the intestine remain poorly understood.'},
 {'PMID': '26745856',
  'mech_sent': 'Air pollution has been associated with increased prevalence of type-2 diabetes, however, the mechanisms remain unknown.'},
 {'PMID': '26745853',
  'mech_sent': 'However, the molecular mechanism of FPD/MM remains unknown.'},
 {'PMID': '26745809',
  'mech_sent': 'However, the molecular mechanism by which ethylene decreases root growth via ASA1 is not understood.'},
 {'PMID': '26745805',
  'mech_sent': 'However, the effectiveness of BoNT⁄A for pain alleviation and the mechanisms that mediate the analgesic effects of BoNT⁄A remain unclear.'},
 {'PMID': '26745678',
  'mech_sent': 'Tyrosine kinase inhibitors (TKIs) targeting the epidermal growth factor receptor (EGFR) have shown promising clinical efficacy in non-squamous non-small cell lung cancer (NSCLC); however, resistance is frequently observed in malignant cells, operating through a mechanism that remains largely unknown.'},
 {'PMID': '26745603',
  'mech_sent': 'The specific regulatory mechanism of miR-let-7a to PKM2 in gastric cancer is still unclear.'},
 {'PMID': '26745527',
  'mech_sent': 'Correct localization is essential for lipoprotein function, yet the mechanisms by which this occurs are not yet fully understood.'},
 {'PMID': '26745502',
  'mech_sent': 'However, the mechanisms by which H. pylori induces gastric cancer are not completely understood.'},
 {'PMID': '26745484',
  'mech_sent': 'CONCLUSIONS These results suggest that reduced integrity of the endothelial wall, as conferred by ADAMTS variants, together with inflammatory processes and defective vascular remodeling plays an important role in CA pathogenesis, although the mechanism of action remains unknown.'},
 {'PMID': '26745481',
  'mech_sent': 'Also, the pathophysiological mechanism of sound perception in patients with VPT remains poorly understood.'},
 {'PMID': '26745327',
  'mech_sent': 'The mechanisms underlying sexual reproduction and sex ratio determination remains unclear in turbot, a flatfish of great commercial value.'},
 {'PMID': '26745299',
  'mech_sent': 'The potential for hydraulic fracturing to induce contaminant pathways in groundwater is not well understood since gas wells are completed while isolating the water table and the gas-bearing reservoirs lay thousands of feet below the water table.'},
 {'PMID': '26745038',
  'mech_sent': 'The mechanism that explains how insulin resistance is associated with CRC is complex and not fully understood.'},
 {'PMID': '26745029',
  'mech_sent': 'Among halogen species, the sources and emission mechanisms of inorganic iodine compounds in the polar boundary layer remain unknown.'},
 {'PMID': '26744595',
  'mech_sent': 'However, the causes of asthma and the underlying mechanisms are not fully understood.'},
 {'PMID': '26744531',
  'mech_sent': 'However, the underlying mechanisms are poorly understood.'},
 {'PMID': '26744469',
  'mech_sent': 'While it is common for IBS to develop following the resolution of infectious gastroenteritis (then termed Post-Infectious-IBS), the mechanisms remain incompletely understood.'},
 {'PMID': '26744468',
  'mech_sent': 'Mice with whole-body FXR deficiency develop liver tumors spontaneously, but the underlying mechanism is unclear.'},
 {'PMID': '26744420',
  'mech_sent': 'Although PTIP is a unique component of the mixed-lineage leukemia 3 (MLL3)/MLL4 chromatin-modifying complex, the mechanisms for how PTIP promotes transcription remain unclear.'},
 {'PMID': '26744383',
  'mech_sent': 'Nevertheless, whether HMGB1 can modulate osteoblast proliferation and migration is poorly understood, as well as the intracellular signaling pathways that are involved in this process.'},
 {'PMID': '26744317',
  'mech_sent': 'Although breast cancer is one of the most common malignancies, the molecular mechanisms underlying its development and progression are not fully understood.'},
 {'PMID': '26744233',
  'mech_sent': 'During embryogenesis, a rise in intracellular Ca(2+) is known to be a widespread trigger for directing stem cells towards a specific tissue fate, but the precise Ca(2+) signalling mechanisms involved in achieving these pleiotropic effects are still poorly understood.'},
 {'PMID': '26744217',
  'mech_sent': 'The biochemical and expression data together indicate that Bayberry surface glycerolipids are synthesized by a previously unknown pathway for TAG synthesis that is related to cutin biosynthesis.'}]

Tweeting!

In [10]:
importlib.reload(tweet_mechanism)
new_sent, tweeted_sentences = tweet_mechanism.main();
INFO:Mechanisms_bot.tweet_mechanism:Getting recent papers
INFO:Mechanisms_bot.tweet_mechanism:Got 100 recent PubMed IDs
INFO:Mechanisms_bot.tweet_mechanism:Getting abstracts
INFO:Mechanisms_bot.tweet_mechanism:Downloaded 100 abstracts
INFO:Mechanisms_bot.tweet_mechanism:Found 22 mechanisms sentences

In [ ]: