Quality Check of BiGo Larvae Experiment¶
In [ ]:
In [23]:
#directory on greenbird
ls /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq
FCD2CA8.tar filtered_BS_CgF_TTAGGC_L007_R1.fastq.gz filtered_BS_CgF_TTAGGC_L007_R2.fastq.gz filtered_BS_CgLarv_T1D3_TGACCA_L007_R1.fastq.gz filtered_BS_CgLarv_T1D3_TGACCA_L007_R2.fastq.gz filtered_BS_CgLarv_T1D5_ACAGTG_L007_R1.fastq.gz filtered_BS_CgLarv_T1D5_ACAGTG_L007_R2.fastq.gz filtered_BS_CgLarv_T3D5_CAGATC_L007_R1.fastq.gz filtered_BS_CgLarv_T3D5_CAGATC_L007_R2.fastq.gz filtered_BS_CgM1_ACTTGA_L007_R1.fastq.gz filtered_BS_CgM1_ACTTGA_L007_R2.fastq.gz filtered_BS_CgM3_GATCAG_L007_R1.fastq.gz filtered_BS_CgM3_GATCAG_L007_R2.fastq.gz filtered_Bs_CgLarve_T3D3_GCCAAT_L007_R1.fastq.gz filtered_Bs_CgLarve_T3D3_GCCAAT_L007_R2.fastq.gz
In [32]:
cd Shared/Apps/bsmap-2.73/
/Users/Shared/Apps/bsmap-2.73
In [34]:
! /Users/Shared/Apps/bsmap-2.73/bsmap
BSMAP v2.73
Usage: bsmap [options]
-a <str> query a file, FASTA/FASTQ/BAM format
-d <str> reference sequences file, FASTA format
-o <str> output alignment file, BSP/SAM/BAM format, if omitted, the output will be written to STDOUT in SAM format.
Options for alignment:
-s <int> seed size, default=16(WGBS mode), 12(RRBS mode). min=8, max=16.
-v <float> if this value is between 0 and 1, it's interpreted as the mismatch rate w.r.t to the read length.
otherwise it's interpreted as the maximum number of mismatches allowed on a read, <=15.
example: -v 5 (max #mismatches = 5), -v 0.1 (max #mismatches = read_length * 10%)
default=0.08.
-g <int> gap size, BSMAP only allow 1 continuous gap (insert or deletion) with up to 3 nucleotides
default=0
-w <int> maximum number of equal best hits to count, <=1000
-B <int> start from the Nth read or read pair, default: 1
-E <int> end at the Nth read or read pair, default: 4,294,967,295
-I <int> index interval, default=4
-k <float> set the cut-off ratio for over-represented kmers, default=5e-07
example: -k 1e-6 means the top 0.0001% over-represented kmer will be skipped in alignment
-p <int> number of processors to use, default=8
-D <str> activating RRBS mapping mode and set restriction enzyme digestion sites.
digestion position marked by '-', example: -D C-CGG for MspI digestion.
default: none (whole genome shotgun bisulfite mapping mode)
-S <int> seed for random number generation used in selecting multiple hits
other seed values generate pseudo random number based on read index number, to allow reproducible mapping results.
default=0. (get seed from system clock, mapping results not resproducible.)
-n [0,1] set mapping strand information. default: 0
-n 0: only map to 2 forward strands, i.e. BSW(++) and BSC(-+),
for PE sequencing, map read#1 to ++ and -+, read#2 to +- and --.
-n 1: map SE or PE reads to all 4 strands, i.e. ++, +-, -+, --
-M <str> set alignment information for the additional nucleotide transition.
<str> is in the form of two different nucleotides N1N2,
indicating N1 in the reads could be mapped to N2 in the reference sequences.
default: -M TC, corresponds to C=>U(T) transition in bisulfite conversion.
example: -M GA could be used to detect A=>I(G) transition in RNA editing.
Options for trimming:
-q <int> quality threshold in trimming, 0-40, default=0 (no trim)
-z <int> base quality, default=33 [Illumina is using 64, Sanger Institute is using 33]
-f <int> filter low-quality reads containing >n Ns, default=5
-A <str> 3-end adapter sequence, default: none (no trim)
-L <int> map the first N nucleotides of the read, default:144 (map the whole read).
Options for reporting:
-r [0,1] how to report repeat hits, 0=none(unique hit/pair only); 1=random one, default:1.
-R print corresponding reference sequences in SAM output, default=off
-u report unmapped reads, default=off
-H do not print header information in SAM format output
Options for pair-end alignment:
-b <str> query b file
-m <int> minimal insert size allowed, default=28
-x <int> maximal insert size allowed, default=500
-h help
BSMAP and Methratio¶
Female¶
In [41]:
#F
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgF_TTAGGC_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgF_TTAGGC_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_F.sam -p 8
BSMAP v2.73 Start at: Tue Oct 1 15:12:31 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 19 secs passed total_kmers: 43046721 Create seed table. 34 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgF_TTAGGC_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgF_TTAGGC_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_F.sam (format: SAM) Thread #4: 323336 read pairs finished. 76 secs passed Thread #2: 50000 read pairs finished. 101 secs passed Thread #0: 150000 read pairs finished. 104 secs passed Thread #6: 100000 read pairs finished. 104 secs passed Thread #1: 200000 read pairs finished. 105 secs passed Thread #3: 300000 read pairs finished. 106 secs passed Thread #5: 250000 read pairs finished. 106 secs passed Total number of aligned reads: pairs: 180566 (56%) single a: 48357 (15%) single b: 37390 (12%) Done. Finished at Tue Oct 1 15:14:18 2013 Total time consumed: 107 secs
In [3]:
#methratio on female
!python /Users/Shared/Apps/bsmap-2.74/methratio.py -d /Volumes/web/cnidarian/oyster.v9.fa -u -z -g -o /Volumes/web/cnidarian/BiGo_lar_F_methratio_v9_A.txt -s /Users/Shared/Apps/bsmap-2.74/samtools /Volumes/web/cnidarian/BiGo_lar_F.sam
@ Wed Oct 2 12:26:36 2013: reading reference /Volumes/web/cnidarian/oyster.v9.fa ... @ Wed Oct 2 12:27:19 2013: reading /Volumes/web/cnidarian/BiGo_lar_F.sam ... [samopen] SAM header is present: 11969 sequences. @ Wed Oct 2 12:27:41 2013: combining CpG methylation from both strands ... @ Wed Oct 2 12:28:25 2013: writing /Volumes/web/cnidarian/BiGo_lar_F_methratio_v9_A.txt ... @ Wed Oct 2 12:31:18 2013: done. total 351170 valid mappings, 2879485 covered cytosines, average coverage: 1.09 fold.
In [2]:
from pandas import *
# read data from data file into a pandas DataFrame
F = read_table("http://eagle.fish.washington.edu/cnidarian/BiGo_lar_F_methratio_v9_A.txt", # name of the data file
# sep=",", # what character separates each column?
na_values=["", " "]) # what values should be considered "blank" values?
In [7]:
!head /Volumes/web/cnidarian/BiGo_lar_F_methratio_v9_A.txt
chr pos strand context ratio eff_CT_count C_count CT_count rev_G_count rev_GA_count CI_lower CI_upper C10211 59 + TGCTG 0.000 1.00 0 1 0 0 0.000 0.793 C10211 70 + GACAA 0.000 1.00 0 1 0 0 0.000 0.793 C10211 75 + GACGT 0.000 1.00 0 1 1 1 0.000 0.793 C10211 79 + TTCAC 0.000 1.00 0 1 0 0 0.000 0.793 C10211 81 + CACCT 0.000 1.00 0 1 0 0 0.000 0.793 C10211 82 + ACCTT 0.000 1.00 0 1 0 0 0.000 0.793 C10211 99 + TTCCT 0.000 1.00 0 1 0 0 0.000 0.793 C10211 100 + TCCTG 0.000 1.00 0 1 0 0 0.000 0.793 C10211 106 + AACTG 0.000 1.00 0 1 0 0 0.000 0.793
In [14]:
!wc /Volumes/web/cnidarian/BiGo_lar_F_methratio_v9_A.txt
2879486 34553830 167118205 /Volumes/web/cnidarian/BiGo_lar_F_methratio_v9_A.txt
In [15]:
F
Out[15]:
<class 'pandas.core.frame.DataFrame'> Int64Index: 2879485 entries, 0 to 2879484 Data columns: chr 2879485 non-null values pos 2879485 non-null values strand 2879485 non-null values context 2879483 non-null values ratio 2871586 non-null values eff_CT_count 2879485 non-null values C_count 2879485 non-null values CT_count 2879485 non-null values rev_G_count 2879485 non-null values rev_GA_count 2879485 non-null values CI_lower 2871586 non-null values CI_upper 2871586 non-null values dtypes: float64(4), int64(5), object(3)
In [13]:
F['CT_count'].hist(bins=100);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
plt.axis([0, 120, 0, 1000])
Out[13]:
[0, 120, 0, 1000]
Lets do some datachecks in R¶
read in F
f<-read.table("http://eagle.fish.washington.edu/cnidarian/BiGo_lar_F_methratio_v9_A.txt", header=T, sep="\t
filter for CG and 5 fold
F_cg<-f[grepl(".{2}CG.{1}",f$context) & f$CT_count >= 5, ]
only 401 obs.
Somthing is funny- on 3k 5 fold? and even then # do not add up
TID3 Larvae¶
In [39]:
#TID3
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D3_TGACCA_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D3_TGACCA_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_T1D3.sam -p 8
BSMAP v2.73 Start at: Tue Oct 1 13:33:06 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 8 secs passed total_kmers: 43046721 Create seed table. 23 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D3_TGACCA_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D3_TGACCA_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_T1D3.sam (format: SAM) Thread #7: 100000 read pairs finished. 120 secs passed Thread #3: 150000 read pairs finished. 121 secs passed Thread #0: 50000 read pairs finished. 122 secs passed Thread #5: 200000 read pairs finished. 123 secs passed Thread #6: 250000 read pairs finished. 124 secs passed Thread #1: 300000 read pairs finished. 125 secs passed Thread #2: 350000 read pairs finished. 126 secs passed Thread #4: 400000 read pairs finished. 127 secs passed Thread #7: 450000 read pairs finished. 212 secs passed Thread #3: 500000 read pairs finished. 213 secs passed Thread #0: 550000 read pairs finished. 214 secs passed Thread #2: 750000 read pairs finished. 220 secs passed Thread #1: 700000 read pairs finished. 220 secs passed Thread #4: 800000 read pairs finished. 221 secs passed Thread #6: 650000 read pairs finished. 224 secs passed Thread #5: 600000 read pairs finished. 231 secs passed Thread #7: 850000 read pairs finished. 313 secs passed Thread #3: 900000 read pairs finished. 315 secs passed Thread #0: 950000 read pairs finished. 316 secs passed Thread #2: 1000000 read pairs finished. 321 secs passed Thread #1: 1050000 read pairs finished. 323 secs passed Thread #4: 1100000 read pairs finished. 325 secs passed Thread #6: 1150000 read pairs finished. 327 secs passed Thread #5: 1200000 read pairs finished. 334 secs passed Thread #7: 1250000 read pairs finished. 415 secs passed Thread #3: 1300000 read pairs finished. 418 secs passed Thread #0: 1350000 read pairs finished. 418 secs passed Thread #2: 1400000 read pairs finished. 424 secs passed Thread #1: 1450000 read pairs finished. 426 secs passed Thread #4: 1500000 read pairs finished. 428 secs passed Thread #6: 1550000 read pairs finished. 430 secs passed Thread #5: 1600000 read pairs finished. 438 secs passed Thread #7: 1650000 read pairs finished. 520 secs passed Thread #0: 1750000 read pairs finished. 524 secs passed Thread #3: 1700000 read pairs finished. 525 secs passed Thread #2: 1800000 read pairs finished. 528 secs passed Thread #1: 1850000 read pairs finished. 529 secs passed Thread #4: 1900000 read pairs finished. 532 secs passed Thread #6: 1950000 read pairs finished. 534 secs passed Thread #5: 2000000 read pairs finished. 542 secs passed Thread #7: 2050000 read pairs finished. 621 secs passed Thread #0: 2100000 read pairs finished. 624 secs passed Thread #3: 2150000 read pairs finished. 625 secs passed Thread #2: 2200000 read pairs finished. 628 secs passed Thread #6: 2350000 read pairs finished. 638 secs passed Thread #1: 2250000 read pairs finished. 639 secs passed Thread #4: 2300000 read pairs finished. 642 secs passed Thread #5: 2400000 read pairs finished. 644 secs passed Thread #7: 2450000 read pairs finished. 722 secs passed Thread #0: 2500000 read pairs finished. 726 secs passed Thread #3: 2550000 read pairs finished. 728 secs passed Thread #2: 2600000 read pairs finished. 730 secs passed Thread #6: 2650000 read pairs finished. 739 secs passed Thread #1: 2700000 read pairs finished. 741 secs passed Thread #4: 2750000 read pairs finished. 744 secs passed Thread #5: 2800000 read pairs finished. 748 secs passed Thread #7: 2850000 read pairs finished. 824 secs passed Thread #0: 2900000 read pairs finished. 827 secs passed Thread #3: 2950000 read pairs finished. 828 secs passed Thread #2: 3000000 read pairs finished. 831 secs passed Thread #6: 3050000 read pairs finished. 840 secs passed Thread #1: 3100000 read pairs finished. 841 secs passed Thread #4: 3150000 read pairs finished. 843 secs passed Thread #5: 3200000 read pairs finished. 845 secs passed Thread #3: 3343987 read pairs finished. 860 secs passed Thread #7: 3250000 read pairs finished. 861 secs passed Thread #0: 3300000 read pairs finished. 862 secs passed Total number of aligned reads: pairs: 1639918 (49%) single a: 709577 (21%) single b: 541078 (16%) Done. Finished at Tue Oct 1 13:47:34 2013 Total time consumed: 868 secs
In [4]:
#TID3 methratio
!python /Users/Shared/Apps/bsmap-2.74/methratio.py -d /Volumes/web/cnidarian/oyster.v9.fa -u -z -g -o /Volumes/web/cnidarian/BiGo_lar_TID3_methratio_v9_A.txt -s /Users/Shared/Apps/bsmap-2.74/samtools /Volumes/web/cnidarian/BiGo_lar_T1D3.sam
@ Wed Oct 2 12:37:26 2013: reading reference /Volumes/web/cnidarian/oyster.v9.fa ... @ Wed Oct 2 12:37:59 2013: reading /Volumes/web/cnidarian/BiGo_lar_T1D3.sam ... [samopen] SAM header is present: 11969 sequences. @ Wed Oct 2 12:41:29 2013: combining CpG methylation from both strands ... @ Wed Oct 2 12:42:09 2013: writing /Volumes/web/cnidarian/BiGo_lar_TID3_methratio_v9_A.txt ... @ Wed Oct 2 12:51:24 2013: done. total 3716891 valid mappings, 27890024 covered cytosines, average coverage: 1.34 fold.
In [17]:
from pandas import *
# read data from data file into a pandas DataFrame
TID3 = read_table("http://eagle.fish.washington.edu/cnidarian/BiGo_lar_TID3_methratio_v9_A.txt", # name of the data file
# sep=",", # what character separates each column?
na_values=["", " "]) # what values should be considered "blank" values?
In [31]:
TID3['CT_count'].hist(bins=100);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
plt.axis([0, 200, 0, 1000])
#not sure what this does
#where ('CT_count') < 100
Out[31]:
[0, 200, 0, 1000]
T1D5¶
In [40]:
#T1D5
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D5_ACAGTG_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D5_ACAGTG_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_T1D5.sam -p 8
BSMAP v2.73 Start at: Tue Oct 1 13:47:35 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 8 secs passed total_kmers: 43046721 Create seed table. 24 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D5_ACAGTG_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T1D5_ACAGTG_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_T1D5.sam (format: SAM) Thread #7: 100000 read pairs finished. 128 secs passed Thread #3: 150000 read pairs finished. 129 secs passed Thread #5: 50000 read pairs finished. 130 secs passed Thread #4: 200000 read pairs finished. 131 secs passed Thread #6: 250000 read pairs finished. 132 secs passed Thread #2: 300000 read pairs finished. 133 secs passed Thread #1: 350000 read pairs finished. 134 secs passed Thread #0: 400000 read pairs finished. 134 secs passed Thread #7: 450000 read pairs finished. 228 secs passed Thread #4: 600000 read pairs finished. 232 secs passed Thread #6: 650000 read pairs finished. 233 secs passed Thread #5: 550000 read pairs finished. 234 secs passed Thread #2: 700000 read pairs finished. 235 secs passed Thread #1: 750000 read pairs finished. 236 secs passed Thread #0: 800000 read pairs finished. 236 secs passed Thread #3: 500000 read pairs finished. 243 secs passed Thread #7: 850000 read pairs finished. 329 secs passed Thread #4: 900000 read pairs finished. 335 secs passed Thread #6: 950000 read pairs finished. 336 secs passed Thread #5: 1000000 read pairs finished. 336 secs passed Thread #2: 1050000 read pairs finished. 337 secs passed Thread #1: 1100000 read pairs finished. 337 secs passed Thread #0: 1150000 read pairs finished. 339 secs passed Thread #3: 1200000 read pairs finished. 345 secs passed Thread #7: 1250000 read pairs finished. 431 secs passed Thread #4: 1300000 read pairs finished. 438 secs passed Thread #2: 1450000 read pairs finished. 442 secs passed Thread #6: 1350000 read pairs finished. 442 secs passed Thread #1: 1500000 read pairs finished. 443 secs passed Thread #5: 1400000 read pairs finished. 443 secs passed Thread #0: 1550000 read pairs finished. 444 secs passed Thread #3: 1600000 read pairs finished. 448 secs passed Thread #7: 1650000 read pairs finished. 533 secs passed Thread #4: 1700000 read pairs finished. 541 secs passed Thread #2: 1750000 read pairs finished. 544 secs passed Thread #6: 1800000 read pairs finished. 546 secs passed Thread #0: 1950000 read pairs finished. 549 secs passed Thread #5: 1900000 read pairs finished. 551 secs passed Thread #3: 2000000 read pairs finished. 553 secs passed Thread #1: 1850000 read pairs finished. 558 secs passed Thread #7: 2050000 read pairs finished. 635 secs passed Thread #4: 2100000 read pairs finished. 643 secs passed Thread #2: 2150000 read pairs finished. 647 secs passed Thread #6: 2200000 read pairs finished. 648 secs passed Thread #0: 2250000 read pairs finished. 652 secs passed Thread #5: 2300000 read pairs finished. 654 secs passed Thread #3: 2350000 read pairs finished. 655 secs passed Thread #1: 2400000 read pairs finished. 661 secs passed Thread #3: 2709822 read pairs finished. 674 secs passed Thread #7: 2450000 read pairs finished. 717 secs passed Thread #4: 2500000 read pairs finished. 721 secs passed Thread #2: 2550000 read pairs finished. 723 secs passed Thread #6: 2600000 read pairs finished. 724 secs passed Thread #0: 2650000 read pairs finished. 724 secs passed Thread #5: 2700000 read pairs finished. 725 secs passed Total number of aligned reads: pairs: 1142766 (42%) single a: 764422 (28%) single b: 567963 (21%) Done. Finished at Tue Oct 1 13:59:46 2013 Total time consumed: 731 secs
In [5]:
#TID5 methratio
!python /Users/Shared/Apps/bsmap-2.74/methratio.py -d /Volumes/web/cnidarian/oyster.v9.fa -u -z -g -o /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt -s /Users/Shared/Apps/bsmap-2.74/samtools /Volumes/web/cnidarian/BiGo_lar_T1D5.sam
@ Wed Oct 2 12:54:31 2013: reading reference /Volumes/web/cnidarian/oyster.v9.fa ... @ Wed Oct 2 12:55:04 2013: reading /Volumes/web/cnidarian/BiGo_lar_T1D5.sam ... [samopen] SAM header is present: 11969 sequences. @ Wed Oct 2 12:57:52 2013: combining CpG methylation from both strands ... @ Wed Oct 2 12:58:33 2013: writing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt ... @ Wed Oct 2 13:07:53 2013: done. total 2964457 valid mappings, 24242913 covered cytosines, average coverage: 1.26 fold.
T3D3¶
In [1]:
#T3D3
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_Bs_CgLarve_T3D3_GCCAAT_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_Bs_CgLarve_T3D3_GCCAAT_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_T3D3.sam -p 8
BSMAP v2.73 Start at: Wed Oct 2 07:01:57 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 15 secs passed total_kmers: 43046721 Create seed table. 29 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_Bs_CgLarve_T3D3_GCCAAT_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_Bs_CgLarve_T3D3_GCCAAT_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_T3D3.sam (format: SAM) Thread #2: 50000 read pairs finished. 122 secs passed Thread #3: 150000 read pairs finished. 122 secs passed Thread #5: 100000 read pairs finished. 123 secs passed Thread #1: 250000 read pairs finished. 123 secs passed Thread #4: 300000 read pairs finished. 125 secs passed Thread #6: 200000 read pairs finished. 125 secs passed Thread #0: 350000 read pairs finished. 126 secs passed Thread #7: 400000 read pairs finished. 127 secs passed Thread #2: 450000 read pairs finished. 215 secs passed Thread #3: 500000 read pairs finished. 217 secs passed Thread #5: 550000 read pairs finished. 218 secs passed Thread #1: 600000 read pairs finished. 219 secs passed Thread #4: 650000 read pairs finished. 220 secs passed Thread #0: 750000 read pairs finished. 221 secs passed Thread #7: 800000 read pairs finished. 223 secs passed Thread #6: 700000 read pairs finished. 224 secs passed Thread #2: 850000 read pairs finished. 304 secs passed Thread #3: 900000 read pairs finished. 307 secs passed Thread #5: 950000 read pairs finished. 307 secs passed Thread #7: 1150000 read pairs finished. 317 secs passed Thread #6: 1200000 read pairs finished. 319 secs passed Thread #0: 1100000 read pairs finished. 319 secs passed Thread #1: 1000000 read pairs finished. 322 secs passed Thread #4: 1050000 read pairs finished. 324 secs passed Thread #2: 1250000 read pairs finished. 396 secs passed Thread #3: 1300000 read pairs finished. 400 secs passed Thread #5: 1350000 read pairs finished. 401 secs passed Thread #7: 1400000 read pairs finished. 411 secs passed Thread #0: 1500000 read pairs finished. 415 secs passed Thread #6: 1450000 read pairs finished. 416 secs passed Thread #1: 1550000 read pairs finished. 417 secs passed Thread #4: 1600000 read pairs finished. 419 secs passed Thread #2: 1650000 read pairs finished. 494 secs passed Thread #3: 1700000 read pairs finished. 498 secs passed Thread #5: 1750000 read pairs finished. 499 secs passed Thread #7: 1800000 read pairs finished. 509 secs passed Thread #0: 1850000 read pairs finished. 513 secs passed Thread #1: 1950000 read pairs finished. 517 secs passed Thread #4: 2000000 read pairs finished. 518 secs passed Thread #6: 1900000 read pairs finished. 519 secs passed Thread #2: 2050000 read pairs finished. 592 secs passed Thread #3: 2100000 read pairs finished. 597 secs passed Thread #5: 2150000 read pairs finished. 597 secs passed Thread #7: 2200000 read pairs finished. 608 secs passed Thread #0: 2250000 read pairs finished. 611 secs passed Thread #1: 2300000 read pairs finished. 615 secs passed Thread #4: 2350000 read pairs finished. 617 secs passed Thread #6: 2400000 read pairs finished. 621 secs passed Thread #2: 2450000 read pairs finished. 690 secs passed Thread #3: 2500000 read pairs finished. 696 secs passed Thread #5: 2550000 read pairs finished. 697 secs passed Thread #7: 2600000 read pairs finished. 707 secs passed Thread #0: 2650000 read pairs finished. 710 secs passed Thread #1: 2700000 read pairs finished. 713 secs passed Thread #4: 2750000 read pairs finished. 716 secs passed Thread #6: 2800000 read pairs finished. 722 secs passed Thread #2: 2850000 read pairs finished. 787 secs passed Thread #5: 2950000 read pairs finished. 799 secs passed Thread #3: 2900000 read pairs finished. 800 secs passed Thread #7: 3000000 read pairs finished. 807 secs passed Thread #0: 3050000 read pairs finished. 808 secs passed Thread #1: 3100000 read pairs finished. 811 secs passed Thread #4: 3150000 read pairs finished. 813 secs passed Thread #6: 3200000 read pairs finished. 822 secs passed Thread #2: 3250000 read pairs finished. 884 secs passed Thread #5: 3300000 read pairs finished. 897 secs passed Thread #3: 3350000 read pairs finished. 899 secs passed Thread #7: 3400000 read pairs finished. 907 secs passed Thread #0: 3450000 read pairs finished. 907 secs passed Thread #1: 3500000 read pairs finished. 910 secs passed Thread #4: 3550000 read pairs finished. 913 secs passed Thread #6: 3600000 read pairs finished. 924 secs passed Thread #2: 3650000 read pairs finished. 980 secs passed Thread #5: 3700000 read pairs finished. 993 secs passed Thread #3: 3750000 read pairs finished. 995 secs passed Thread #7: 3800000 read pairs finished. 1002 secs passed Thread #0: 3850000 read pairs finished. 1003 secs passed Thread #1: 3900000 read pairs finished. 1015 secs passed Thread #4: 3950000 read pairs finished. 1024 secs passed Thread #6: 4000000 read pairs finished. 1032 secs passed Thread #2: 4050000 read pairs finished. 1081 secs passed Thread #5: 4100000 read pairs finished. 1092 secs passed Thread #3: 4150000 read pairs finished. 1093 secs passed Thread #7: 4200000 read pairs finished. 1100 secs passed Thread #0: 4250000 read pairs finished. 1101 secs passed Thread #1: 4300000 read pairs finished. 1114 secs passed Thread #4: 4350000 read pairs finished. 1123 secs passed Thread #6: 4400000 read pairs finished. 1135 secs passed Thread #2: 4450000 read pairs finished. 1181 secs passed Thread #5: 4500000 read pairs finished. 1191 secs passed Thread #3: 4550000 read pairs finished. 1192 secs passed Thread #7: 4600000 read pairs finished. 1199 secs passed Thread #0: 4650000 read pairs finished. 1200 secs passed Thread #1: 4700000 read pairs finished. 1212 secs passed Thread #4: 4750000 read pairs finished. 1222 secs passed Thread #6: 4800000 read pairs finished. 1236 secs passed Thread #2: 4850000 read pairs finished. 1282 secs passed Thread #3: 4950000 read pairs finished. 1292 secs passed Thread #5: 4900000 read pairs finished. 1292 secs passed Thread #7: 5000000 read pairs finished. 1297 secs passed Thread #0: 5050000 read pairs finished. 1299 secs passed Thread #1: 5100000 read pairs finished. 1310 secs passed Thread #4: 5150000 read pairs finished. 1321 secs passed Thread #6: 5200000 read pairs finished. 1338 secs passed Thread #2: 5250000 read pairs finished. 1380 secs passed Thread #3: 5300000 read pairs finished. 1390 secs passed Thread #5: 5350000 read pairs finished. 1392 secs passed Thread #7: 5400000 read pairs finished. 1398 secs passed Thread #0: 5450000 read pairs finished. 1399 secs passed Thread #1: 5500000 read pairs finished. 1410 secs passed Thread #4: 5550000 read pairs finished. 1420 secs passed Thread #6: 5600000 read pairs finished. 1433 secs passed Thread #0: 5837666 read pairs finished. 1449 secs passed Thread #2: 5650000 read pairs finished. 1451 secs passed Thread #3: 5700000 read pairs finished. 1453 secs passed Thread #5: 5750000 read pairs finished. 1454 secs passed Thread #7: 5800000 read pairs finished. 1455 secs passed Total number of aligned reads: pairs: 3130107 (54%) single a: 1140569 (20%) single b: 880396 (15%) Done. Finished at Wed Oct 2 07:26:12 2013 Total time consumed: 1455 secs
In [6]:
#T3D3 methratio
!python /Users/Shared/Apps/bsmap-2.74/methratio.py -d /Volumes/web/cnidarian/oyster.v9.fa -u -z -g -o /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_v9_A.txt -s /Users/Shared/Apps/bsmap-2.74/samtools /Volumes/web/cnidarian/BiGo_lar_T3D3.sam
@ Wed Oct 2 13:07:54 2013: reading reference /Volumes/web/cnidarian/oyster.v9.fa ... @ Wed Oct 2 13:08:27 2013: reading /Volumes/web/cnidarian/BiGo_lar_T3D3.sam ... [samopen] SAM header is present: 11969 sequences. @ Wed Oct 2 13:15:01 2013: combining CpG methylation from both strands ... @ Wed Oct 2 13:15:41 2013: writing /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_v9_A.txt ... @ Wed Oct 2 13:29:25 2013: done. total 6941630 valid mappings, 45580028 covered cytosines, average coverage: 1.55 fold.
T3D5¶
In [2]:
#T3D5
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T3D5_CAGATC_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T3D5_CAGATC_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_T3D5.sam -p 8
BSMAP v2.73 Start at: Wed Oct 2 07:26:12 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 8 secs passed total_kmers: 43046721 Create seed table. 24 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T3D5_CAGATC_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgLarv_T3D5_CAGATC_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_T3D5.sam (format: SAM) Thread #5: 100000 read pairs finished. 120 secs passed Thread #0: 150000 read pairs finished. 120 secs passed Thread #3: 50000 read pairs finished. 121 secs passed Thread #6: 200000 read pairs finished. 121 secs passed Thread #4: 250000 read pairs finished. 122 secs passed Thread #7: 300000 read pairs finished. 123 secs passed Thread #1: 350000 read pairs finished. 124 secs passed Thread #2: 400000 read pairs finished. 125 secs passed Thread #5: 450000 read pairs finished. 216 secs passed Thread #0: 500000 read pairs finished. 216 secs passed Thread #3: 550000 read pairs finished. 218 secs passed Thread #6: 600000 read pairs finished. 218 secs passed Thread #4: 650000 read pairs finished. 220 secs passed Thread #7: 700000 read pairs finished. 220 secs passed Thread #1: 750000 read pairs finished. 227 secs passed Thread #2: 800000 read pairs finished. 237 secs passed Thread #0: 900000 read pairs finished. 316 secs passed Thread #3: 950000 read pairs finished. 316 secs passed Thread #6: 1000000 read pairs finished. 316 secs passed Thread #4: 1050000 read pairs finished. 317 secs passed Thread #7: 1100000 read pairs finished. 317 secs passed Thread #5: 850000 read pairs finished. 319 secs passed Thread #1: 1150000 read pairs finished. 324 secs passed Thread #2: 1200000 read pairs finished. 334 secs passed Thread #0: 1250000 read pairs finished. 413 secs passed Thread #3: 1300000 read pairs finished. 414 secs passed Thread #6: 1350000 read pairs finished. 415 secs passed Thread #4: 1400000 read pairs finished. 416 secs passed Thread #7: 1450000 read pairs finished. 419 secs passed Thread #5: 1500000 read pairs finished. 419 secs passed Thread #1: 1550000 read pairs finished. 423 secs passed Thread #2: 1600000 read pairs finished. 433 secs passed Thread #0: 1650000 read pairs finished. 511 secs passed Thread #3: 1700000 read pairs finished. 512 secs passed Thread #6: 1750000 read pairs finished. 513 secs passed Thread #4: 1800000 read pairs finished. 514 secs passed Thread #7: 1850000 read pairs finished. 518 secs passed Thread #5: 1900000 read pairs finished. 520 secs passed Thread #1: 1950000 read pairs finished. 523 secs passed Thread #2: 2000000 read pairs finished. 532 secs passed Thread #0: 2050000 read pairs finished. 609 secs passed Thread #3: 2100000 read pairs finished. 611 secs passed Thread #6: 2150000 read pairs finished. 611 secs passed Thread #4: 2200000 read pairs finished. 618 secs passed Thread #5: 2300000 read pairs finished. 619 secs passed Thread #7: 2250000 read pairs finished. 620 secs passed Thread #1: 2350000 read pairs finished. 621 secs passed Thread #2: 2400000 read pairs finished. 631 secs passed Thread #0: 2450000 read pairs finished. 707 secs passed Thread #3: 2500000 read pairs finished. 710 secs passed Thread #6: 2550000 read pairs finished. 711 secs passed Thread #4: 2600000 read pairs finished. 716 secs passed Thread #5: 2650000 read pairs finished. 719 secs passed Thread #7: 2700000 read pairs finished. 720 secs passed Thread #1: 2750000 read pairs finished. 721 secs passed Thread #2: 2800000 read pairs finished. 730 secs passed Thread #0: 2850000 read pairs finished. 803 secs passed Thread #3: 2900000 read pairs finished. 805 secs passed Thread #6: 2950000 read pairs finished. 814 secs passed Thread #1: 3150000 read pairs finished. 818 secs passed Thread #7: 3100000 read pairs finished. 819 secs passed Thread #5: 3050000 read pairs finished. 822 secs passed Thread #4: 3000000 read pairs finished. 826 secs passed Thread #2: 3200000 read pairs finished. 828 secs passed Thread #0: 3250000 read pairs finished. 901 secs passed Thread #3: 3300000 read pairs finished. 905 secs passed Thread #6: 3350000 read pairs finished. 913 secs passed Thread #1: 3400000 read pairs finished. 918 secs passed Thread #7: 3450000 read pairs finished. 919 secs passed Thread #5: 3500000 read pairs finished. 921 secs passed Thread #4: 3550000 read pairs finished. 925 secs passed Thread #2: 3600000 read pairs finished. 927 secs passed Thread #0: 3650000 read pairs finished. 1000 secs passed Thread #3: 3700000 read pairs finished. 1007 secs passed Thread #6: 3750000 read pairs finished. 1014 secs passed Thread #1: 3800000 read pairs finished. 1018 secs passed Thread #7: 3850000 read pairs finished. 1024 secs passed Thread #5: 3900000 read pairs finished. 1025 secs passed Thread #4: 3950000 read pairs finished. 1025 secs passed Thread #2: 4000000 read pairs finished. 1026 secs passed Thread #0: 4050000 read pairs finished. 1094 secs passed Thread #3: 4100000 read pairs finished. 1101 secs passed Thread #6: 4150000 read pairs finished. 1107 secs passed Thread #1: 4200000 read pairs finished. 1111 secs passed Thread #7: 4250000 read pairs finished. 1115 secs passed Thread #5: 4300000 read pairs finished. 1116 secs passed Thread #4: 4350000 read pairs finished. 1117 secs passed Thread #2: 4400000 read pairs finished. 1117 secs passed Thread #0: 4421486 read pairs finished. 1118 secs passed Total number of aligned reads: pairs: 2414419 (55%) single a: 819661 (19%) single b: 637662 (14%) Done. Finished at Wed Oct 2 07:44:50 2013 Total time consumed: 1118 secs
In [7]:
#T3D5 methratio
!python /Users/Shared/Apps/bsmap-2.74/methratio.py -d /Volumes/web/cnidarian/oyster.v9.fa -u -z -g -o /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt -s /Users/Shared/Apps/bsmap-2.74/samtools /Volumes/web/cnidarian/BiGo_lar_T3D5.sam
@ Wed Oct 2 13:29:26 2013: reading reference /Volumes/web/cnidarian/oyster.v9.fa ... @ Wed Oct 2 13:30:10 2013: reading /Volumes/web/cnidarian/BiGo_lar_T3D5.sam ... [samopen] SAM header is present: 11969 sequences. @ Wed Oct 2 13:35:11 2013: combining CpG methylation from both strands ... @ Wed Oct 2 13:35:52 2013: writing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt ... @ Wed Oct 2 13:47:24 2013: done. total 5275651 valid mappings, 37156094 covered cytosines, average coverage: 1.45 fold.
In [14]:
!wc /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt
37156095 445873112 2157638420 /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt
M1¶
In [3]:
#M1
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgM1_ACTTGA_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgM1_ACTTGA_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_M1.sam -p 8
BSMAP v2.73 Start at: Wed Oct 2 07:44:50 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 8 secs passed total_kmers: 43046721 Create seed table. 24 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgM1_ACTTGA_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgM1_ACTTGA_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_M1.sam (format: SAM) Thread #0: 100000 read pairs finished. 122 secs passed Thread #1: 50000 read pairs finished. 122 secs passed Thread #7: 200000 read pairs finished. 122 secs passed Thread #2: 150000 read pairs finished. 123 secs passed Thread #6: 250000 read pairs finished. 123 secs passed Thread #3: 300000 read pairs finished. 124 secs passed Thread #5: 350000 read pairs finished. 124 secs passed Thread #4: 400000 read pairs finished. 125 secs passed Thread #0: 450000 read pairs finished. 216 secs passed Thread #1: 500000 read pairs finished. 218 secs passed Thread #7: 550000 read pairs finished. 220 secs passed Thread #2: 600000 read pairs finished. 222 secs passed Thread #6: 650000 read pairs finished. 223 secs passed Thread #3: 700000 read pairs finished. 224 secs passed Thread #5: 750000 read pairs finished. 225 secs passed Thread #4: 800000 read pairs finished. 225 secs passed Thread #0: 850000 read pairs finished. 312 secs passed Thread #1: 900000 read pairs finished. 313 secs passed Thread #5: 1150000 read pairs finished. 322 secs passed Thread #4: 1200000 read pairs finished. 323 secs passed Thread #3: 1100000 read pairs finished. 324 secs passed Thread #6: 1050000 read pairs finished. 326 secs passed Thread #7: 950000 read pairs finished. 327 secs passed Thread #2: 1000000 read pairs finished. 330 secs passed Thread #0: 1250000 read pairs finished. 410 secs passed Thread #1: 1300000 read pairs finished. 411 secs passed Thread #5: 1350000 read pairs finished. 421 secs passed Thread #4: 1400000 read pairs finished. 422 secs passed Thread #3: 1450000 read pairs finished. 423 secs passed Thread #6: 1500000 read pairs finished. 426 secs passed Thread #7: 1550000 read pairs finished. 427 secs passed Thread #2: 1600000 read pairs finished. 430 secs passed Thread #0: 1650000 read pairs finished. 510 secs passed Thread #1: 1700000 read pairs finished. 510 secs passed Thread #5: 1750000 read pairs finished. 519 secs passed Thread #4: 1800000 read pairs finished. 522 secs passed Thread #3: 1850000 read pairs finished. 524 secs passed Thread #6: 1900000 read pairs finished. 526 secs passed Thread #7: 1950000 read pairs finished. 527 secs passed Thread #2: 2000000 read pairs finished. 530 secs passed Thread #0: 2050000 read pairs finished. 609 secs passed Thread #1: 2100000 read pairs finished. 610 secs passed Thread #5: 2150000 read pairs finished. 619 secs passed Thread #4: 2200000 read pairs finished. 621 secs passed Thread #3: 2250000 read pairs finished. 623 secs passed Thread #6: 2300000 read pairs finished. 627 secs passed Thread #7: 2350000 read pairs finished. 628 secs passed Thread #2: 2400000 read pairs finished. 632 secs passed Thread #0: 2450000 read pairs finished. 709 secs passed Thread #1: 2500000 read pairs finished. 710 secs passed Thread #5: 2550000 read pairs finished. 717 secs passed Thread #4: 2600000 read pairs finished. 720 secs passed Thread #3: 2650000 read pairs finished. 722 secs passed Thread #6: 2700000 read pairs finished. 726 secs passed Thread #7: 2750000 read pairs finished. 731 secs passed Thread #2: 2800000 read pairs finished. 735 secs passed Thread #1: 2900000 read pairs finished. 810 secs passed Thread #0: 2850000 read pairs finished. 811 secs passed Thread #5: 2950000 read pairs finished. 816 secs passed Thread #4: 3000000 read pairs finished. 819 secs passed Thread #3: 3050000 read pairs finished. 822 secs passed Thread #6: 3100000 read pairs finished. 825 secs passed Thread #7: 3150000 read pairs finished. 831 secs passed Thread #2: 3200000 read pairs finished. 836 secs passed Thread #1: 3250000 read pairs finished. 910 secs passed Thread #0: 3300000 read pairs finished. 912 secs passed Thread #5: 3350000 read pairs finished. 916 secs passed Thread #4: 3400000 read pairs finished. 919 secs passed Thread #3: 3450000 read pairs finished. 921 secs passed Thread #6: 3500000 read pairs finished. 927 secs passed Thread #7: 3550000 read pairs finished. 929 secs passed Thread #2: 3600000 read pairs finished. 935 secs passed Thread #1: 3650000 read pairs finished. 1006 secs passed Thread #0: 3700000 read pairs finished. 1015 secs passed Thread #3: 3850000 read pairs finished. 1022 secs passed Thread #4: 3800000 read pairs finished. 1023 secs passed Thread #5: 3750000 read pairs finished. 1026 secs passed Thread #6: 3900000 read pairs finished. 1026 secs passed Thread #7: 3950000 read pairs finished. 1027 secs passed Thread #2: 4000000 read pairs finished. 1033 secs passed Thread #1: 4050000 read pairs finished. 1104 secs passed Thread #0: 4100000 read pairs finished. 1115 secs passed Thread #3: 4150000 read pairs finished. 1122 secs passed Thread #4: 4200000 read pairs finished. 1123 secs passed Thread #5: 4250000 read pairs finished. 1126 secs passed Thread #6: 4300000 read pairs finished. 1128 secs passed Thread #7: 4350000 read pairs finished. 1128 secs passed Thread #2: 4400000 read pairs finished. 1136 secs passed Thread #1: 4450000 read pairs finished. 1202 secs passed Thread #0: 4500000 read pairs finished. 1213 secs passed Thread #3: 4550000 read pairs finished. 1220 secs passed Thread #4: 4600000 read pairs finished. 1223 secs passed Thread #5: 4650000 read pairs finished. 1226 secs passed Thread #6: 4700000 read pairs finished. 1228 secs passed Thread #7: 4750000 read pairs finished. 1229 secs passed Thread #2: 4800000 read pairs finished. 1236 secs passed Thread #1: 4850000 read pairs finished. 1302 secs passed Thread #0: 4900000 read pairs finished. 1314 secs passed Thread #3: 4950000 read pairs finished. 1320 secs passed Thread #4: 5000000 read pairs finished. 1322 secs passed Thread #5: 5050000 read pairs finished. 1326 secs passed Thread #6: 5100000 read pairs finished. 1329 secs passed Thread #7: 5150000 read pairs finished. 1330 secs passed Thread #2: 5200000 read pairs finished. 1336 secs passed Thread #1: 5250000 read pairs finished. 1393 secs passed Thread #0: 5300000 read pairs finished. 1401 secs passed Thread #7: 5543163 read pairs finished. 1403 secs passed Thread #3: 5350000 read pairs finished. 1403 secs passed Thread #4: 5400000 read pairs finished. 1404 secs passed Thread #5: 5450000 read pairs finished. 1405 secs passed Thread #6: 5500000 read pairs finished. 1405 secs passed Total number of aligned reads: pairs: 2600662 (47%) single a: 1090658 (20%) single b: 894893 (16%) Done. Finished at Wed Oct 2 08:08:15 2013 Total time consumed: 1405 secs
In [8]:
#M1 methratio
!python /Users/Shared/Apps/bsmap-2.74/methratio.py -d /Volumes/web/cnidarian/oyster.v9.fa -u -z -g -o /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt -s /Users/Shared/Apps/bsmap-2.74/samtools /Volumes/web/cnidarian/BiGo_lar_M1.sam
@ Wed Oct 2 13:47:24 2013: reading reference /Volumes/web/cnidarian/oyster.v9.fa ... @ Wed Oct 2 13:47:58 2013: reading /Volumes/web/cnidarian/BiGo_lar_M1.sam ... [samopen] SAM header is present: 11969 sequences. @ Wed Oct 2 13:53:31 2013: combining CpG methylation from both strands ... @ Wed Oct 2 13:54:11 2013: writing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt ... @ Wed Oct 2 14:07:48 2013: done. total 5799674 valid mappings, 37975071 covered cytosines, average coverage: 1.57 fold.
M3¶
In [4]:
#M3
! /Users/Shared/Apps/bsmap-2.73/bsmap -a /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgM3_GATCAG_L007_R1.fastq.gz -b /Volumes/NGS\ Drive/NGS\ Raw\ Data/Cg_larvae_BSseq/filtered_BS_CgM3_GATCAG_L007_R2.fastq.gz -d /Volumes/web/cnidarian/oyster.v9.fa -o /Volumes/web/cnidarian/BiGo_lar_M3.sam -p 8
BSMAP v2.73 Start at: Wed Oct 2 08:08:16 2013 Input reference file: /Volumes/web/cnidarian/oyster.v9.fa (format: FASTA) Load in 11969 db seqs, total size 558601156 bp. 8 secs passed total_kmers: 43046721 Create seed table. 24 secs passed max number of mismatches: read_length * 8% max gap size: 0 kmer cut-off ratio:5e-07 max multi-hits: 100 max Ns: 5 seed size: 16 index interval: 4 quality cutoff: 0 base quality char: '!' min fragment size:28 max fragemt size:500 start from read #1 end at read #4294967295 additional alignment: T in reads => C in reference mapping strand (read_1): ++,-+ mapping strand (read_2): +-,-- Pair-end alignment(8 threads) Input read file #1: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgM3_GATCAG_L007_R1.fastq.gz (format: gzipped FASTQ) Input read file #2: /Volumes/NGS Drive/NGS Raw Data/Cg_larvae_BSseq/filtered_BS_CgM3_GATCAG_L007_R2.fastq.gz (format: gzipped FASTQ) Output file: /Volumes/web/cnidarian/BiGo_lar_M3.sam (format: SAM) Thread #1: 100000 read pairs finished. 121 secs passed Thread #0: 150000 read pairs finished. 122 secs passed Thread #2: 200000 read pairs finished. 122 secs passed Thread #4: 250000 read pairs finished. 123 secs passed Thread #5: 300000 read pairs finished. 123 secs passed Thread #3: 50000 read pairs finished. 124 secs passed Thread #6: 350000 read pairs finished. 124 secs passed Thread #7: 400000 read pairs finished. 124 secs passed Thread #1: 450000 read pairs finished. 216 secs passed Thread #0: 500000 read pairs finished. 218 secs passed Thread #2: 550000 read pairs finished. 219 secs passed Thread #4: 600000 read pairs finished. 220 secs passed Thread #5: 650000 read pairs finished. 221 secs passed Thread #6: 750000 read pairs finished. 224 secs passed Thread #7: 800000 read pairs finished. 225 secs passed Thread #3: 700000 read pairs finished. 226 secs passed Thread #1: 850000 read pairs finished. 309 secs passed Thread #0: 900000 read pairs finished. 312 secs passed Thread #7: 1150000 read pairs finished. 322 secs passed Thread #6: 1100000 read pairs finished. 323 secs passed Thread #5: 1050000 read pairs finished. 324 secs passed Thread #3: 1200000 read pairs finished. 324 secs passed Thread #2: 950000 read pairs finished. 325 secs passed Thread #4: 1000000 read pairs finished. 328 secs passed Thread #1: 1250000 read pairs finished. 406 secs passed Thread #0: 1300000 read pairs finished. 409 secs passed Thread #7: 1350000 read pairs finished. 420 secs passed Thread #6: 1400000 read pairs finished. 421 secs passed Thread #5: 1450000 read pairs finished. 422 secs passed Thread #2: 1550000 read pairs finished. 424 secs passed Thread #3: 1500000 read pairs finished. 426 secs passed Thread #4: 1600000 read pairs finished. 427 secs passed Thread #1: 1650000 read pairs finished. 504 secs passed Thread #0: 1700000 read pairs finished. 507 secs passed Thread #7: 1750000 read pairs finished. 518 secs passed Thread #6: 1800000 read pairs finished. 521 secs passed Thread #5: 1850000 read pairs finished. 523 secs passed Thread #2: 1900000 read pairs finished. 524 secs passed Thread #4: 2000000 read pairs finished. 526 secs passed Thread #3: 1950000 read pairs finished. 527 secs passed Thread #1: 2050000 read pairs finished. 602 secs passed Thread #0: 2100000 read pairs finished. 605 secs passed Thread #7: 2150000 read pairs finished. 617 secs passed Thread #6: 2200000 read pairs finished. 619 secs passed Thread #5: 2250000 read pairs finished. 621 secs passed Thread #2: 2300000 read pairs finished. 622 secs passed Thread #4: 2350000 read pairs finished. 625 secs passed Thread #3: 2400000 read pairs finished. 629 secs passed Thread #1: 2450000 read pairs finished. 700 secs passed Thread #0: 2500000 read pairs finished. 704 secs passed Thread #7: 2550000 read pairs finished. 715 secs passed Thread #6: 2600000 read pairs finished. 718 secs passed Thread #5: 2650000 read pairs finished. 719 secs passed Thread #2: 2700000 read pairs finished. 720 secs passed Thread #4: 2750000 read pairs finished. 726 secs passed Thread #3: 2800000 read pairs finished. 734 secs passed Thread #1: 2850000 read pairs finished. 800 secs passed Thread #0: 2900000 read pairs finished. 803 secs passed Thread #7: 2950000 read pairs finished. 814 secs passed Thread #6: 3000000 read pairs finished. 817 secs passed Thread #5: 3050000 read pairs finished. 818 secs passed Thread #2: 3100000 read pairs finished. 819 secs passed Thread #4: 3150000 read pairs finished. 826 secs passed Thread #3: 3200000 read pairs finished. 835 secs passed Thread #1: 3250000 read pairs finished. 899 secs passed Thread #0: 3300000 read pairs finished. 902 secs passed Thread #7: 3350000 read pairs finished. 914 secs passed Thread #6: 3400000 read pairs finished. 916 secs passed Thread #5: 3450000 read pairs finished. 917 secs passed Thread #2: 3500000 read pairs finished. 917 secs passed Thread #4: 3550000 read pairs finished. 924 secs passed Thread #3: 3600000 read pairs finished. 935 secs passed Thread #1: 3650000 read pairs finished. 993 secs passed Thread #0: 3700000 read pairs finished. 1004 secs passed Thread #2: 3900000 read pairs finished. 1015 secs passed Thread #5: 3850000 read pairs finished. 1016 secs passed Thread #6: 3800000 read pairs finished. 1020 secs passed Thread #4: 3950000 read pairs finished. 1021 secs passed Thread #7: 3750000 read pairs finished. 1023 secs passed Thread #3: 4000000 read pairs finished. 1033 secs passed Thread #1: 4050000 read pairs finished. 1090 secs passed Thread #0: 4100000 read pairs finished. 1103 secs passed Thread #2: 4150000 read pairs finished. 1115 secs passed Thread #5: 4200000 read pairs finished. 1116 secs passed Thread #6: 4250000 read pairs finished. 1120 secs passed Thread #4: 4300000 read pairs finished. 1120 secs passed Thread #7: 4350000 read pairs finished. 1122 secs passed Thread #3: 4400000 read pairs finished. 1135 secs passed Thread #1: 4450000 read pairs finished. 1187 secs passed Thread #0: 4500000 read pairs finished. 1201 secs passed Thread #2: 4550000 read pairs finished. 1213 secs passed Thread #5: 4600000 read pairs finished. 1216 secs passed Thread #4: 4700000 read pairs finished. 1220 secs passed Thread #7: 4750000 read pairs finished. 1221 secs passed Thread #6: 4650000 read pairs finished. 1222 secs passed Thread #3: 4800000 read pairs finished. 1235 secs passed Thread #1: 4850000 read pairs finished. 1285 secs passed Thread #0: 4900000 read pairs finished. 1299 secs passed Thread #2: 4950000 read pairs finished. 1311 secs passed Thread #5: 5000000 read pairs finished. 1314 secs passed Thread #4: 5050000 read pairs finished. 1319 secs passed Thread #7: 5100000 read pairs finished. 1321 secs passed Thread #6: 5150000 read pairs finished. 1322 secs passed Thread #3: 5200000 read pairs finished. 1337 secs passed Thread #6: 5526741 read pairs finished. 1370 secs passed Thread #1: 5250000 read pairs finished. 1375 secs passed Thread #0: 5300000 read pairs finished. 1382 secs passed Thread #2: 5350000 read pairs finished. 1386 secs passed Thread #5: 5400000 read pairs finished. 1387 secs passed Thread #4: 5450000 read pairs finished. 1388 secs passed Thread #7: 5500000 read pairs finished. 1389 secs passed Total number of aligned reads: pairs: 3032870 (55%) single a: 996127 (18%) single b: 789107 (14%) Done. Finished at Wed Oct 2 08:31:25 2013 Total time consumed: 1389 secs
In [33]:
!wc /Volumes/web/cnidarian/BiGo_lar_M3_methratio_v9_A.txt
42369118 508429391 2460930058 /Volumes/web/cnidarian/BiGo_lar_M3_methratio_v9_A.txt
Mapping Summary¶
Uploading methratio files to SQLShare¶
Parsing files for characterization
In [21]:
!python /Users/sr320/sqlshare-pythonclient/tools/singleupload.py -h
Usage: singleupload.py [options] <filename>
Options:
-h, --help show this help message and exit
-u USERNAME, --user=USERNAME
SQLshare user name
-p PASSWORD, --password=PASSWORD
SQLshare password
-d DATASETNAME, --datasetname=DATASETNAME
Dataset name (defaults to filename if not supplied
F¶
Uploading Female data
python /Users/sr320/sqlshare-pythonclient/tools/singleupload.py -d BiGo_lar_F /Volumes/web/cnidarian/BiGo_lar_F_methratio_v9_A.txt
In [66]:
#2.8 million lines
#!!!!!! 3447 loci 5x coverage
# 401 CG
M3¶
In [35]:
#M3 methratio output has 42 million lines
#!!!!!!!! 41 million lines have CT_count <5 ---- 97%
#
In [ ]:
#19 M - strand
#23 m + strand
In [39]:
from pandas import *
# read data from data file into a pandas DataFrame
m3 = read_table("http://eagle.fish.washington.edu/cnidarian/BiGo_lar_M3_methratio_v9_A.txt", # name of the data file
# sep=",", # what character separates each column?
na_values=["", " "]) # what values should be considered "blank" values?
In [75]:
m3['CT_count'].hist(bins=5000);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
plt.axis([0, 10, 0, 10000000])
Out[75]:
[0, 10, 0, 10000000]
In [78]:
m3['CT_count'].hist(bins=5000);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
plt.axis([0, 100, 0, 2000])
Out[78]:
[0, 100, 0, 2000]
In [ ]:
#3010 CG loci have >20 coverage
In [59]:
!head /Volumes/web/cnidarian/BiGo_lar_M3_methratio_v9_A.txt
chr pos strand context ratio eff_CT_count C_count CT_count rev_G_count rev_GA_count CI_lower CI_upper C10005 78 - TAGGG 0.000 1.00 0 1 0 0 0.000 0.793 C10005 79 - AGGGA 0.000 1.00 0 1 0 0 0.000 0.793 C10005 80 - GGGAA 0.000 1.00 0 1 0 0 0.000 0.793 C10005 86 - ATGTG 0.000 1.00 0 1 0 0 0.000 0.793 C10005 88 - GTGGG 0.000 1.00 0 1 0 0 0.000 0.793 C10005 89 - TGGGT 0.000 1.00 0 1 0 0 0.000 0.793 C10005 90 - GGGTT 0.000 1.00 0 1 0 0 0.000 0.793 C10005 94 - TTGTT 0.000 1.00 0 1 0 0 0.000 0.793 C10005 100 - AAGTC 0.000 1.00 0 1 0 0 0.000 0.793
In [ ]:
In [ ]:
#Mac's gill sample
#methratio output had 91 million lines
#69 million lines have CT_count <5 --- 75%
In [ ]:
#Claire's sperm data
#methratio output had 126 million lines (after ratio NA removed) vs 127/22
#22 million lines have CT_count <5 --- 17%
for context ...
In [36]:
#Creating GFF of CG with 5x
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGO_lar_M3] where context like '__CG_' and CT_Count >= 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
In [37]:
!head /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
In [38]:
!wc /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
182754 1644786 9561122 /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
In [53]:
#scratch
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGO_lar_M3] where context like '__CT_' and CT_Count >= 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CT.txt
In [54]:
!wc /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CT.txt
131803 1186227 6885390 /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CT.txt
In [ ]:
#confirming CNs add up
#CG 182k
#CC 72k
#CA 240k
#CT 131k
# - 625k
#^ that would have just been positive strand.
# yes add up
In [60]:
from pandas import *
# read data from data file into a pandas DataFrame
m3CG = read_table("http://eagle.fish.washington.edu/cnidarian/BiGo_lar_M3_methratio_CG.txt", # name of the data file
# sep=",", # what character separates each column?
na_values=["", " "]) # what values should be considered "blank" values?
In [61]:
!head /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
In [62]:
!wc /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
182754 1644786 9561122 /Volumes/web/cnidarian/BiGo_lar_M3_methratio_CG.txt
In [85]:
m3CG['score'].hist(bins=500);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
plt.axis([0, 1, 0, 10000])
Out[85]:
[0, 1, 0, 10000]
In [86]:
#worried about score...
cleaning
SELECT [chr]
, CAST([pos] as INTEGER) as [pos]
, [strand]
, [context]
, CAST([ratio] as FLOAT) as [ratio]
, CAST([eff_CT_count] as INTEGER) as [eff_CT_count]
, CAST([C_count] as INTEGER) as [C_count]
, CAST([CT_count] as INTEGER) as [CT_count]
, CAST([rev_G_count] as INTEGER) as [rev_G_count]
, CAST([CI_lower] as FLOAT) as [CI_lower]
, CAST([CI_upper] as FLOAT) as [CI_upper]
FROM [sr320@washington.edu].[BiGo_lar_M3]
WHERE [ratio] <> 'NA'
and
context like '__CG_'
and
CT_count >= 5
In [9]:
from pandas import *
# read data from data file into a pandas DataFrame
m3c = read_csv("http://eagle.fish.washington.edu/cnidarian/clean_BiGo_lar_M3_cg5.csv", # name of the data file
sep=",", # what character separates each column?
na_values=["", " "]) # what values should be considered "blank" values?
In [98]:
m3c['ratio'].hist(bins=4);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
#plt.axis([0, 1, 0, 10000])
Out[98]:
<matplotlib.axes.AxesSubplot at 0x10b30ced0>
In [92]:
# 136751 CG loci have methylation ratio of 0 @ 5 coverage
# 12,270 between 0-0.25
# 7572 0.25-.5
# 7323 0.5-0.75
# 18559 >0.75
# 46k have non 0 ratios
In [116]:
m3c['ratio'].hist(bins=10);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
plt.axis([0, 1, 0, 20000])
plt.xlabel('Methylation(%)', fontsize=20)
plt.ylabel('count', fontsize= 20)
Out[116]:
<matplotlib.text.Text at 0x1091a3d50>
In [10]:
m3c['ratio'].hist(bins=100);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
#plt.axis([0, 1, 0, 20000])
plt.xlabel('Methylation(%)', fontsize=20)
plt.ylabel('count', fontsize= 20)
plt.title('Sperm (M3)', fontsize= 20)
Out[10]:
<matplotlib.text.Text at 0x106928250>
In [11]:
m3c['ratio'].hist(bins=10);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
#plt.axis([0, 1, 0, 20000])
plt.xlabel('Methylation(%)', fontsize=20)
plt.ylabel('count', fontsize= 20)
plt.title('Sperm (M3)', fontsize= 20)
Out[11]:
<matplotlib.text.Text at 0x106962610>
In [104]:
12270 + 7572 + 7323 + 18559
Out[104]:
45724
In [ ]:
In [105]:
45724 + 136751
Out[105]:
182475
In [ ]:
#3010 CG have > 20x coverage
T3D3¶
In [68]:
!wc /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_v9_A.txt
45580029 546960327 2647413384 /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_v9_A.txt
In [ ]:
#45 Million lines
In [2]:
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGO_lar_T3D3] where context like '__CG_' and CT_Count >= 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_CG.txt
In [3]:
!head /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_CG.txt
In [4]:
!wc /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_CG.txt
185149 1666341 9685366 /Volumes/web/cnidarian/BiGo_lar_T3D3_methratio_CG.txt
In [5]:
from pandas import *
# read data from data file into a pandas DataFrame
T3D3cg = read_table("http://eagle.fish.washington.edu/cnidarian/BiGo_lar_T3D3_methratio_CG.txt", # name of the data file
#sep=",", # what character separates each column?
na_values=["", " "]) # what values should be considered "blank" values?
In [6]:
T3D3cg['score'].hist(bins=100);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
#plt.axis([0, 1, 0, 20000])
plt.xlabel('Methylation(%)', fontsize=20)
plt.ylabel('count', fontsize= 20)
plt.title('Larvae (T3D3)', fontsize= 20)
Out[6]:
<matplotlib.text.Text at 0x10062a450>
In [12]:
T3D3cg['score'].hist(bins=10);
#Axis limits are changed using the axis([xmin, xmax, ymin, ymax]) function.
#plt.axis([0, 1, 0, 20000])
plt.xlabel('Methylation(%)', fontsize=20)
plt.ylabel('count', fontsize= 20)
plt.title('Larvae (T3D3)', fontsize= 20)
Out[12]:
<matplotlib.text.Text at 0x10737acd0>
little R¶
In [122]:
#37 million lines in methratio file
T3D3_cg5<-T3D3[grepl(".{2}CG.{1}",T3D3$context) & T3D3$CT_count >= 5, ]
In [124]:
#107k CG
#42k CC
#129k CA
#74k CT
In [125]:
#checking out methylation
T3D3_Mcg5<-T3D3[grepl(".{2}Cg.{1}",T3D3$context) & T3D3$CT_count >= 5 & T3D3$ratio > 0, ]
T3D5¶
In [18]:
!wc /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt
37156095 445873112 2157638420 /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt
In [17]:
!python /Users/sr320/sqlshare-pythonclient/tools/singleupload.py -d BiGo_lar_T3D5 /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt
processing chunk line 0 to 1912976 (9.52025008202 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 3BECDFF8... processing chunk line 1912976 to 3706333 (261.892167091 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 7B0BB388... processing chunk line 3706333 to 5497779 (504.070580006 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 22A7DEA7... processing chunk line 5497779 to 7289476 (753.540595055 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 06FF5C23... processing chunk line 7289476 to 9086570 (997.756511927 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 3069725D... processing chunk line 9086570 to 10881117 (1248.36775589 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing EA65F8BA... processing chunk line 10881117 to 12674445 (1486.49684811 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 4FD169CA... processing chunk line 12674445 to 14466732 (1725.51903009 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 7A016A37... processing chunk line 14466732 to 16281788 (1963.86586499 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 5A352782... processing chunk line 16281788 to 18098482 (2205.82230592 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing C93E2393... processing chunk line 18098482 to 19898210 (2449.84771109 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 58094237... processing chunk line 19898210 to 21694100 (2689.89153194 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 5E8B622E... processing chunk line 21694100 to 23480398 (2929.38493204 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 740C919C... processing chunk line 23480398 to 25254501 (3171.81918502 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing B98CE17D... processing chunk line 25254501 to 27019270 (3396.80474091 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing B5185682... processing chunk line 27019270 to 28819421 (3633.14284706 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 60E8AE53... processing chunk line 28819421 to 30639892 (3902.0498631 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 9C4B4B69... processing chunk line 30639892 to 32461586 (4185.20643806 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing B16C5515... processing chunk line 32461586 to 34282834 (4439.85324693 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing FC7459C3... processing chunk line 34282834 to 36107047 (4672.67855406 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 782C7711... processing chunk line 36107047 to 37156095 (4912.63782191 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_v9_A.txt... parsing 7B28E2B0... finished BiGo_lar_T3D5
In [ ]:
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGO_lar_T3D5] where context like '__CG_' and CT_Count > 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_T3D5_methratio_CG.txt
M1¶
In [1]:
!wc /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt
37975072 455700841 2204748730 /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt
In [12]:
#M1
!python /Users/sr320/sqlshare-pythonclient/tools/singleupload.py -d BiGo_lar_M1 /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt
processing chunk line 0 to 1914203 (2.65032696724 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 4942AF32... processing chunk line 1914203 to 3707769 (241.177401066 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 4F965951... processing chunk line 3707769 to 5499087 (474.987231016 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 9D822C1B... processing chunk line 5499087 to 7290548 (710.098507881 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 3164AA12... processing chunk line 7290548 to 9087335 (956.481744051 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 8874D24D... processing chunk line 9087335 to 10882268 (1189.40272784 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing AB2EDFCC... processing chunk line 10882268 to 12676537 (1426.55913591 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 70F4400E... processing chunk line 12676537 to 14469534 (1676.53050089 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing E3185686... processing chunk line 14469534 to 16282655 (1915.30709291 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing E48EF49E... processing chunk line 16282655 to 18100457 (2160.12649894 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 076C6727... processing chunk line 18100457 to 19902928 (2399.14282203 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 6D1BF13D... processing chunk line 19902928 to 21697108 (2636.89273787 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 1737036D... processing chunk line 21697108 to 23488733 (2880.17217803 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 650E9F3D... processing chunk line 23488733 to 25267525 (3120.61586404 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing F4EB0EF1... processing chunk line 25267525 to 27036869 (3364.64700294 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 2B2DB663... processing chunk line 27036869 to 28818585 (3603.17266893 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 9392D856... processing chunk line 28818585 to 30638898 (3842.82249403 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing E1A83D16... processing chunk line 30638898 to 32459779 (4084.22851491 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 4449465C... processing chunk line 32459779 to 34282522 (4328.33319592 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 0DB6E88F... processing chunk line 34282522 to 36106900 (4575.06277585 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 4DE8FE90... processing chunk line 36106900 to 37928931 (4827.68771791 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing A757D52E... processing chunk line 37928931 to 37975072 (5070.4182899 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_M1_methratio_v9_A.txt... parsing 073F8705... finished BiGo_lar_M1
In [13]:
#Creating GFF of CG 5x
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGo_lar_M1] where context like '__CG_' and CT_Count > 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_M1_methratio_CG.txt
TID3¶
In [2]:
!wc /Volumes/web/cnidarian/BiGo_lar_TID3_methratio_v9_A.txt
27890025 334680276 1619321169 /Volumes/web/cnidarian/BiGo_lar_TID3_methratio_v9_A.txt
In [8]:
#Creating GFF of CG 5x
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGO_lar_T1D3] where context like '__CG_' and CT_Count > 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_T1D3_methratio_CG.txt
T1D5¶
In [3]:
!wc /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt
24242914 290914957 1407430012 /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt
In [9]:
#uploading T1D5
!python /Users/sr320/sqlshare-pythonclient/tools/singleupload.py -d BiGo_lar_T1D5 /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt
processing chunk line 0 to 1871173 (2.50428009033 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 90802450... processing chunk line 1871173 to 3663442 (247.303070068 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 028BBD67... processing chunk line 3663442 to 5458933 (476.486818075 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 28D87D98... processing chunk line 5458933 to 7253223 (710.011958122 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 99A8901A... processing chunk line 7253223 to 9046005 (958.407089949 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 02D3125D... processing chunk line 9046005 to 10857553 (1194.479213 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 1124948D... processing chunk line 10857553 to 12667016 (1437.00030994 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 47A00721... processing chunk line 12667016 to 14463026 (1684.63456011 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing F731C24C... processing chunk line 14463026 to 16243026 (1919.55414295 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 29C0AEA6... processing chunk line 16243026 to 18009217 (2155.419595 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing FB34A640... processing chunk line 18009217 to 19828442 (2394.01386905 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing EFAFC190... processing chunk line 19828442 to 21651312 (2637.01132607 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing F0E3FA81... processing chunk line 21651312 to 23474703 (2880.31882596 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 45F713FD... processing chunk line 23474703 to 24242914 (3119.95345712 s elapsed) pushing /Volumes/web/cnidarian/BiGo_lar_TID5_methratio_v9_A.txt... parsing 89D514E4... finished BiGo_lar_T1D5
In [11]:
#Creating GFF of CG
!python /Users/sr320/sqlshare-pythonclient/tools/fetchdata.py -s "SELECT chr as seqname,'methratio' as source,'CpG' as feature, pos as start, pos + 1 as [end], ratio as score, strand, '.' as frame, '.' as attribute FROM [sr320@washington.edu].[BiGo_lar_T1D5] where context like '__CG_' and CT_Count > 5" -f tsv -o /Volumes/web/cnidarian/BiGo_lar_T1D5_methratio_CG.txt
misc¶
In [16]:
from IPython.display import HTML
HTML('<iframe src=https://www.evernote.com/shard/s74/sh/051a489f-2851-4320-a613-ee1402417b30/ef6a40b2af6e77ae6c03fa0dda46b527 width=700 height=350></iframe>')
Out[16]:
In [ ]: