Присоединяйтесь к чату воркшопа:

https://clck.ru/9yRpd

Подходите за материалами (на флешках)

Один день из жизни нагрузочного тестировщика

Алексей Лавренюк, Яндекс

Процесс нагрузочного тестирования:

  • Анализ работы сервиса:
    • архитектура
    • нагрузка (планируемая или существующая)
  • Подготовка стрельб
    • настройка тестового стенда
    • подготовка патронов
  • Стрельбы
  • Анализ результатов

Из вышеперечисленного сегодня мы займемся анализом входной нагрузки и анализом результатов стрельб.

Задачка 0: готовим тестовое окружение

Логи, с которыми мы будем работать -- на флешках:

  • access_log_Jul95 -- логи веб-сервера NASA Kennedy Space Center за июль 1995 года
  • phout_1 и phout_2 -- логи стрельб Яндекс.Танком

Страничка с описанием логов NASA: http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html

Если будете использовать jupyter, запустите его в той папке, куда скачали логи.

Проверьте, что библиотеки имортируются:

import numpy as np    # numpy
import pandas as pd   # pandas

# альтернативная библиотека для визуализации
# https://github.com/mwaskom/seaborn
import seaborn as sns

# подключаем графику прямо в браузер
%matplotlib inline

Если нет -- самое время их поставить.

sudo -H pip3 install numpy pandas seaborn

Если вы под ubuntu, вероятно, для сборки этих библиотек вам понадобятся:

sudo apt install build-essential python-dev gfortran

Парсим access.log

In [ ]:
# немного о jupyter. vim-like, помощь, автодополнение
In [3]:
import numpy as np
?np.linspace
In [4]:
%%bash
# можно исполнять bash-команды прямо отсюда

head ./access_log_Jul95
199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245
unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.0" 200 3985
199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] "GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0" 200 4085
burger.letters.com - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/countdown/liftoff.html HTTP/1.0" 304 0
199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0" 200 4179
burger.letters.com - - [01/Jul/1995:00:00:12 -0400] "GET /images/NASA-logosmall.gif HTTP/1.0" 304 0
burger.letters.com - - [01/Jul/1995:00:00:12 -0400] "GET /shuttle/countdown/video/livevideo.gif HTTP/1.0" 200 0
205.212.115.106 - - [01/Jul/1995:00:00:12 -0400] "GET /shuttle/countdown/countdown.html HTTP/1.0" 200 3985
d104.aa.net - - [01/Jul/1995:00:00:13 -0400] "GET /shuttle/countdown/ HTTP/1.0" 200 3985
129.94.144.152 - - [01/Jul/1995:00:00:13 -0400] "GET / HTTP/1.0" 200 7074
In [5]:
# мы на PyCon, поэтому будем работать с помощью преимущественно в Python

# читаем первую строку:
with open("./access_log_Jul95") as datafile:
    log_line = datafile.readline()
print(log_line)
199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245

Задачка 1: парсим access.log

Задание Написать функцию parse_line, которая принимает на вход строку из лога и возвращает массив полей:

  • hostname - имя хоста, например 199.72.81.55
  • date - дата в виде строки, например 01/Jul/1995:00:00:01 -0400
  • method - метод, указанный в запросе, например GET
  • url - url, указанный в запросе, например /history/apollo/
  • response_code - код ответа, например 200
  • size - размер ответа, например 6245

вспомогательные материалы

заготовка кода

with open("./access_log_Jul95") as datafile:
    log_line = datafile.readline()

def parse_line(line):
    # ваш код тут
    return []

print(parse_line(log_line))
In [29]:
# andrey-yantsen
def parse_line_t(line):
    ip, _, _, dt, tz, method, path, _, responce_code, size = line.split(' ')
    return ip, dt + tz, method.replace('"', ''), path, responce_code, size

def parse_line_t(line):
    ip, _, _, dt, tz, method, path, _, responce_code, size = line.split(' ')
    return ip, (dt + tz).replace('[', '').replace(']', ''), method.replace('"', ''), path, responce_code, size.strip()

print(parse_line_t(log_line))
('199.72.81.55', '01/Jul/1995:00:00:01-0400', 'GET', '/history/apollo/', '200', '6245')



































































In [8]:
# вариант с регэксом

import re

# регэкс скомпилим заранее
regex = re.compile('([\w\-\.]+) - - \[(.*?)\] "([A-Z]+) (\S+)\s+(\S+)" (\d+) (\d+)')
def parse_line_re(line):
    return regex.match(line).groups()

parse_line_re(log_line)
Out[8]:
('199.72.81.55',
 '01/Jul/1995:00:00:01 -0400',
 'GET',
 '/history/apollo/',
 'HTTP/1.0',
 '200',
 '6245')
In [9]:
# вариант без регэкспа: постепенно откусываем поля по известным разделителям

def parse_line_split(line):
    host, line = line.split(" ", 1)
    _, line = line.split("[", 1)
    date, line = line.split("]", 1)
    _, line = line.split('"', 1)
    req, line = line.split('"', 1)
    code, timing = line.strip("\r\n ").split()
    return [host, date] + req.split(" ") + [code, timing]

parse_line_split(log_line)
Out[9]:
['199.72.81.55',
 '01/Jul/1995:00:00:01 -0400',
 'GET',
 '/history/apollo/',
 'HTTP/1.0',
 '200',
 '6245']
In [31]:
%timeit parse_line_re(log_line)
%timeit parse_line_split(log_line)

%timeit parse_line_t(log_line)
100000 loops, best of 3: 2.57 µs per loop
100000 loops, best of 3: 3.74 µs per loop
The slowest run took 4.26 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 1.91 µs per loop

Тестируем на нашем логе

In [16]:
def test_parser_on_file(parser):
    with open("./access_log_Jul95") as datafile:
        for line in datafile:
            try:
                parser(line)
            except Exception as e:
                # если парсер не сработает, напечатаем строку,
                # на которой он сломался, и выйдем
                print(line)
                raise e
In [17]:
# сначала попробуем на функции, которая ничего не делает
# python 2 работает по-другому =)
test_parser_on_file(lambda l: l)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-17-30e604a0ff59> in <module>()
      1 # сначала попробуем на функции, которая ничего не делает
      2 # python 2 работает по-другому =)
----> 3 test_parser_on_file(lambda l: l)

<ipython-input-16-0b4ad3a93aa3> in test_parser_on_file(parser)
      1 def test_parser_on_file(parser):
      2     with open("./access_log_Jul95") as datafile:
----> 3         for line in datafile:
      4             try:
      5                 parser(line)

/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/codecs.py in decode(self, input, final)
    319         # decode input (taking the buffer into account)
    320         data = self.buffer + input
--> 321         (result, consumed) = self._buffer_decode(data, self.errors, final)
    322         # keep undecoded input until the next call
    323         self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 3690: invalid start byte



































































In [18]:
%%bash
# посмотрим, что же это за 0x80

hexdump ./access_log_Jul95 | grep -m 1 ' 80 '
0c86e60 6e 74 64 6f 77 6e 2f 64 79 20 80 3f 73 68 75 74
In [19]:
%%bash

xxd ./access_log_Jul95 | grep '0c86e60' -C 2 -m 1
0c86e40: 323a 3234 3a34 3120 2d30 3430 305d 2022  2:24:41 -0400] "
0c86e50: 4745 5420 2f73 6875 7474 6c65 2f63 6f75  GET /shuttle/cou
0c86e60: 6e74 646f 776e 2f64 7920 803f 7368 7574  ntdown/dy .?shut
In [20]:
# пофиксим читалку файла
def test_parser_on_file(parser):
    with open("./access_log_Jul95", encoding="ascii", errors="replace") as datafile:
        for line in datafile:
            try:
                parser(line)
            except Exception as e:
                # если парсер не сработает, напечатаем строку,
                # на которой он сломался, и выйдем
                print(line)
                raise e

test_parser_on_file(lambda l: l)
In [21]:
# протестируем вариант с регэкспом
test_parser_on_file(parse_line_re)
dd15-062.compuserve.com - - [01/Jul/1995:00:01:12 -0400] "GET /news/sci.space.shuttle/archive/sci-space-shuttle-22-apr-1995-40.txt HTTP/1.0" 404 -

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-21-803443080bf3> in <module>()
      1 # протестируем вариант с регэкспом
----> 2 test_parser_on_file(parse_line_re)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      9                 # на которой он сломался, и выйдем
     10                 print(line)
---> 11                 raise e
     12 
     13 test_parser_on_file(lambda l: l)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      4         for line in datafile:
      5             try:
----> 6                 parser(line)
      7             except Exception as e:
      8                 # если парсер не сработает, напечатаем строку,

<ipython-input-8-d7c9d614de00> in parse_line_re(line)
      6 regex = re.compile('([\w\-\.]+) - - \[(.*?)\] "([A-Z]+) (\S+)\s+(\S+)" (\d+) (\d+)')
      7 def parse_line_re(line):
----> 8     return regex.match(line).groups()
      9 
     10 parse_line_re(log_line)

AttributeError: 'NoneType' object has no attribute 'groups'
In [22]:
test_parser_on_file(parse_line_t)
pipe6.nyc.pipeline.com - - [01/Jul/1995:00:22:43 -0400] "GET /shuttle/missions/sts-71/movies/sts-71-mir-dock.mpg" 200 946425

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-22-bbfa984a1552> in <module>()
----> 1 test_parser_on_file(parse_line_t)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      9                 # на которой он сломался, и выйдем
     10                 print(line)
---> 11                 raise e
     12 
     13 test_parser_on_file(lambda l: l)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      4         for line in datafile:
      5             try:
----> 6                 parser(line)
      7             except Exception as e:
      8                 # если парсер не сработает, напечатаем строку,

<ipython-input-13-313c0dc71145> in parse_line_t(line)
      1 # andrey-yantsen
      2 def parse_line_t(line):
----> 3     ip, _, _, dt, tz, method, path, _, responce_code, size = line.split(' ')
      4     return ip, dt + tz, method.replace('"', ''), path, responce_code, size
      5 print(parse_line(log_line))

ValueError: not enough values to unpack (expected 10, got 9)



































































In [23]:
# если сервер возвращает ошибку, то размера нет
# добавим поддержку для отсутствующего размера
regex = re.compile('([\w\-\.]+) - - \[(.*?)\] "([A-Z]+) (\S+)\s+(\S+)" (\d+) (-|\d+)')
def parse_line_re(line):
    return regex.match(line).groups()

# и проверим, что все работает на этой строке
parse_line_re('dd15-062.compuserve.com - - [01/Jul/1995:00:01:12 -0400] "GET /news/sci.space.shuttle/archive/sci-space-shuttle-22-apr-1995-40.txt HTTP/1.0" 404 -')

# кстати, если бы мы парсили не регэкспом, а вручную, мы могли бы в случае отсутствия значения поставить там 
# 0 или None, и потом нам было бы удобнее работать с такими данными
Out[23]:
('dd15-062.compuserve.com',
 '01/Jul/1995:00:01:12 -0400',
 'GET',
 '/news/sci.space.shuttle/archive/sci-space-shuttle-22-apr-1995-40.txt',
 'HTTP/1.0',
 '404',
 '-')
In [24]:
# попробуем еще раз
test_parser_on_file(parse_line_re)
pipe6.nyc.pipeline.com - - [01/Jul/1995:00:22:43 -0400] "GET /shuttle/missions/sts-71/movies/sts-71-mir-dock.mpg" 200 946425

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-24-efe7c782512d> in <module>()
      1 # попробуем еще раз
----> 2 test_parser_on_file(parse_line_re)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      9                 # на которой он сломался, и выйдем
     10                 print(line)
---> 11                 raise e
     12 
     13 test_parser_on_file(lambda l: l)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      4         for line in datafile:
      5             try:
----> 6                 parser(line)
      7             except Exception as e:
      8                 # если парсер не сработает, напечатаем строку,

<ipython-input-23-4a826dae5b89> in parse_line_re(line)
      3 regex = re.compile('([\w\-\.]+) - - \[(.*?)\] "([A-Z]+) (\S+)\s+(\S+)" (\d+) (-|\d+)')
      4 def parse_line_re(line):
----> 5     return regex.match(line).groups()
      6 
      7 # и проверим, что все работает на этой строке

AttributeError: 'NoneType' object has no attribute 'groups'
In [25]:
# учитываем возможное отсутствие протокола в запросе
regex = re.compile('([\w\-\.]+) - - \[(.*?)\] "([A-Z]+) (\S+)(?: (\S+))?" (\d+) (-|\d+)')
def parse_line_re(line):
    return regex.match(line).groups()

parse_line_re('pipe6.nyc.pipeline.com - - [01/Jul/1995:00:22:43 -0400] "GET /shuttle/missions/sts-71/movies/sts-71-mir-dock.mpg" 200 946425')
Out[25]:
('pipe6.nyc.pipeline.com',
 '01/Jul/1995:00:22:43 -0400',
 'GET',
 '/shuttle/missions/sts-71/movies/sts-71-mir-dock.mpg',
 None,
 '200',
 '946425')
In [27]:
# посмотрим все необработанные строки (опасно, но я пробовал)
regex = re.compile('([\w\-\[email protected]]+) - - \[(.*?)\] "([A-Z]+) (\S+)(?:\s+(\S+))?" (\d+) (-|\d+)')
def parse_line_re(line):
    try:
        return regex.match(line).groups()
    except AttributeError:
        print(line)
        # не будем бросать тут эксепшн, просто продолжим
        raise
        return None
test_parser_on_file(parse_line_re)
204.120.229.63 - - [01/Jul/1995:04:29:05 -0400] "GET /history/history.html                                                 hqpao/hqpao_home.html HTTP/1.0" 200 1502

204.120.229.63 - - [01/Jul/1995:04:29:05 -0400] "GET /history/history.html                                                 hqpao/hqpao_home.html HTTP/1.0" 200 1502

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-27-a74df5e447b3> in <module>()
      9         raise
     10         return None
---> 11 test_parser_on_file(parse_line_re)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      9                 # на которой он сломался, и выйдем
     10                 print(line)
---> 11                 raise e
     12 
     13 test_parser_on_file(lambda l: l)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      4         for line in datafile:
      5             try:
----> 6                 parser(line)
      7             except Exception as e:
      8                 # если парсер не сработает, напечатаем строку,

<ipython-input-27-a74df5e447b3> in parse_line_re(line)
      3 def parse_line_re(line):
      4     try:
----> 5         return regex.match(line).groups()
      6     except AttributeError:
      7         print(line)

AttributeError: 'NoneType' object has no attribute 'groups'

Задачка 2: починка парсера

Задание Починить парсер хотя бы для тех ответов, которые обработал сервер

  • нераспаршенными должны остаться только те ответы, на которые сервер выдал код 400 (bad request)
  • предположим, что сервер читает url до первого пробельного символа, все остальное выкидывает

Заготовка кода

Можно либо поменять регулярное выражение, либо использовать свою функцию.

import re

def test_parser_on_file(parser):
    with open("./access_log_Jul95", encoding="ascii", errors="replace") as datafile:
        for line in datafile:
            try:
                parser(line)
            except Exception as e:
                # если парсер не сработает, напечатаем строку,
                # на которой он сломался, и выйдем
                print(line)
                raise e

regex = re.compile('([\w\-\[email protected]]+) - - \[(.*?)\] "([A-Z]+) (\S+)(?:\s+(\S+))?" (\d+) (-|\d+)')
def parse_line_re(line):
    try:
        return regex.match(line).groups()
    except AttributeError:
        print(line)
        return None

test_parser_on_file(parse_line_re)
In [34]:
#briskly
import re
def parse_line(line):
    res = re.match(r"(.*)- - (\[.*\]) \"(\w+) (.+)( (.+))?\" (\d+) (\d+|-)", line)
    if res is None:
        print(line)
        return
    return res.groups()

def parser_log(fn):
    for line in open(fn, encoding="ascii", errors="replace"):
        res = parse_line(line)
        if res:
            pass

parser_log("access_log_Jul95")
klothos.crl.research.digital.com - - [10/Jul/1995:16:45:50 -0400] "" 400 -

firewall.dfw.ibm.com - - [20/Jul/1995:07:34:34 -0400] "1/history/apollo/images/" 400 -

firewall.dfw.ibm.com - - [20/Jul/1995:07:53:24 -0400] "1/history/apollo/images/" 400 -

128.159.122.20 - - [20/Jul/1995:15:28:50 -0400] "k��tx��tG��t���" 400 -

128.159.122.20 - - [24/Jul/1995:13:52:50 -0400] "k��tx��tG��t���" 400 -

alyssa.p
In [28]:
# beloborodov
def parse_line(line):
    return re.search('(.*) - - \[(.*)\] "(.*) (\/.*) HTTP/1.0" (.*) (.*)', line).groups()
test_parser_on_file(parse_line)
pipe6.nyc.pipeline.com - - [01/Jul/1995:00:22:43 -0400] "GET /shuttle/missions/sts-71/movies/sts-71-mir-dock.mpg" 200 946425

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-28-55ac332b59f3> in <module>()
      2 def parse_line(line):
      3     return re.search('(.*) - - \[(.*)\] "(.*) (\/.*) HTTP/1.0" (.*) (.*)', line).groups()
----> 4 test_parser_on_file(parse_line)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      9                 # на которой он сломался, и выйдем
     10                 print(line)
---> 11                 raise e
     12 
     13 test_parser_on_file(lambda l: l)

<ipython-input-20-db0fa5390d1a> in test_parser_on_file(parser)
      4         for line in datafile:
      5             try:
----> 6                 parser(line)
      7             except Exception as e:
      8                 # если парсер не сработает, напечатаем строку,

<ipython-input-28-55ac332b59f3> in parse_line(line)
      1 # beloborodov
      2 def parse_line(line):
----> 3     return re.search('(.*) - - \[(.*)\] "(.*) (\/.*) HTTP/1.0" (.*) (.*)', line).groups()
      4 test_parser_on_file(parse_line)

AttributeError: 'NoneType' object has no attribute 'groups'



































































что получилось у меня

  • запросом считаем то, что после метода и до первого пробельного символа, далее выкидываем все до последней кавычки
  • учитываем то, что после метода перед урлом может быть не один пробел
  • добавил все символы, которые встречались в хостнеймах. Можно было просто ограничить хостнейм первым пробельным символом, но мне было интересно, какие бывают хостнеймы
In [35]:
regex = re.compile('([\w\-\[email protected]\'*,:/#&]+) - - \[(.*?)\] "([A-Z]+)\s+(\S+).*" (\d+) (-|\d+)')
def parse_line_re(line):
    try:
        return regex.match(line).groups()
    except AttributeError:
        print(line)
        # не кидаем эксепшн, посмотрим все нераспаршенные строки
        return None
test_parser_on_file(parse_line_re)
klothos.crl.research.digital.com - - [10/Jul/1995:16:45:50 -0400] "" 400 -

firewall.dfw.ibm.com - - [20/Jul/1995:07:34:34 -0400] "1/history/apollo/images/" 400 -

firewall.dfw.ibm.com - - [20/Jul/1995:07:53:24 -0400] "1/history/apollo/images/" 400 -

128.159.122.20 - - [20/Jul/1995:15:28:50 -0400] "k��tx��tG��t���" 400 -

128.159.122.20 - - [24/Jul/1995:13:52:50 -0400] "k��tx��tG��t���" 400 -

alyssa.p

Считаем урлы

In [36]:
# убираем печать ненайденных урлов, чтобы не мешались

regex = re.compile('([\w\-\[email protected]\'*,:/#&]+) - - \[(.*?)\] "([A-Z]+)\s+(\S+).*" (\d+) (-|\d+)')
def parse_line_re(line):
    try:
        return regex.match(line).groups()
    except AttributeError:
        return None
In [37]:
from collections import defaultdict

# нас интересуют урлы без параметров, параметры могут быть и через хеш, и через вопрос
# отделим их с помощью регэкспа
delim = re.compile("[#?]")

counts = defaultdict(int)
with open("./access_log_Jul95", encoding="ascii", errors="replace") as datafile:
    for line in datafile:
        fields = parse_line_re(line)
        if fields:
            uri = delim.split(fields[3])[0]
            counts[uri] += 1
counts
Out[37]:
defaultdict(int,
            {'/history/apollo/apollo-13/images/70HC418.GIF': 616,
             '/shuttle/missions/sts-71/news/ksc-status-06-07-95.txt': 4,
             '/shuttle/missions/sts-51/': 18,
             '/images/home_igloo.gif': 2,
             '/history/aoillo/': 1,
             '/shuttle/movies/astronauts.mpg': 39,
             '/history/mercury/mercury-capsules.txt': 6,
             '/news/sci.space.news/1552': 2,
             '/HISTORY/APOLLO/APOLLO-13/APOLLO13.HTML': 2,
             '/shuttle/missions/sts-2/movies/': 11,
             '/shuttle/missions/sts-71/images/KSC-95EC-0873.jpeg': 1,
             '/11/history/gemini/gemini-4/news/': 1,
             '/images/lc39a.gif': 669,
             '/home/guided-tour.html': 1,
             '/procurement/midrange/rfo/consamp.txt': 15,
             '/shuttle/missions/status/r91-61': 1,
             '/shuttle/technology/sts-newsref/stsover-missions': 1,
             '/statistics/1993/Nov/Nov93_byte.gif': 1,
             '/shuttle/missions/sts-63/sts-63-patch-small.gif': 1936,
             '/icon/q_mark3.gif': 1,
             '/shuttle/missions/sts-41/sts-41-info.html': 6,
             '/shuttle/missions/missions.html;': 1,
             '/shuttle/missions/61-c/news': 7,
             '/shuttle/coutdown': 2,
             '/history/apollo/apollo-10/news/': 36,
             '/shuttle/missions/61-c/sounds/': 3,
             '/shuttle/missions/41-c/images/84HC210.GIF': 6,
             '/facts/wais-readme.html': 2,
             '/software/webadmin/astronauts.perl': 4,
             '/shuttle/missions/sts-28/sts-28-patch.jpg': 11,
             '/history/gemini/gemini-viii/': 5,
             '/shuttle/missions/sts-43/sounds/': 1,
             '/shuttle/missions/sts-46/news/': 1,
             '/software.html': 3,
             '/history/apollo/apollo-1/docs/': 433,
             '/shuttle/shuttle/missions/sts-70/mission-sts-70.html': 3,
             '/index.html': 2,
             '/shuttle/missions/sts-48/sts-48-info.html': 5,
             '/shuttle/missions/sts-40/sts-40-patch-small.gif': 171,
             '/shuttle/missions/sts-2/': 24,
             '/history/apollo/appollo-13info.html': 1,
             '/shuttle/missions/sts-72/sts-72-patch.jpg': 41,
             '/cgi-bin/imagemap': 1,
             '/shuttle/technology/sts-newsref/sts-cron.html': 701,
             '/shuttle/missions/sts-67/images/k95p0394.jpg': 2,
             '/shuttle/resources/orbiters/orbiters-logo.gif': 8545,
             '/facilities/vpf.html': 60,
             '/shuttle/missions/status/r91-68': 1,
             '/apollo.html': 2,
             '/history/apollo/apollo-11/sounds': 1,
             '/mdss/srqa/../MDSS.html': 5,
             '/software/winvn/bluemarb.gif': 5814,
             '/images/cdrom-1-95/img0019.jpg': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0875.jpg': 66,
             '/history/apollo/apollo-13/news/': 1432,
             '/elv/uplink.htm%3EThis': 1,
             '/shuttle/missions/sts-70/mission-sts70.html': 2,
             '/peripherals/graphics/peripherals_bar.gif': 1,
             '/images/NASA-logos': 5,
             '/history/gemini/gemini-4.html': 1,
             '/persons/astronauts/a-to-d/BurschDW.txt': 4,
             '/history/apollo/a-001/a-001.html': 140,
             '/classes/net/www/content/image/jpeg.class': 2,
             '/shuttle/missions/sts-65/news/sts-65-mcc-08.txt': 1,
             '/software/winvn/userguide/winvn34.gif': 23,
             '/base-ops/procurement/kscbus.htm': 14,
             '/images/getstats.gif': 65,
             '/shuttle/missions/sts-71/sts-71-day-04-highlights.html': 542,
             '/history/apollo/apollo-12apollo-13.html': 1,
             '/shuttle/missions/sts-5/sts-5-patch.jpg': 22,
             '/history/apollo/apollo14/apollo14.html': 1,
             '/history/apollo/apollo-1/apollo-1.html': 2574,
             '/facilities/press.html': 112,
             '/shuttle/missions/sts-31/movies/': 4,
             '/cgi-bin/imagemap/video/livevideo2.gif': 2,
             '/elv/uncons.htm': 409,
             '/shuttle/missions/sts-71/images/KSC-95EC-0423.gif': 1520,
             '/~www/pics/welcome.gif': 1,
             '/procurement/midrange/abstract/pneuabs.txt': 35,
             '/msfc/who.html': 1,
             '/shuttle/missions/41-d/images/84HC191.GIF': 9,
             '/mdss/ped/acs/cmu2.gif': 14,
             '/payloads/processing/vpart5as.gif': 1,
             '/shuttle/missions/41-g/images/': 14,
             '/shuttle/missions/sts-70/mission-sts-72.html': 2,
             '/job.com': 1,
             '/shuttle/missions/sts-67/sts-67-patch-small.gif': 2209,
             '/shuttle/missions/sts-70/images/DSC-95EC-0002': 1,
             '/shuttle/Countdown': 1,
             '/skylab/skylab.html': 2,
             '/biomed/soils/gif/legend.gif': 3,
             '/shuttle/missions/sts-51/news/': 2,
             '/shuttle/missions/sts-66/news/sts-66-pocc-16.txt': 4,
             '/shuttle/missions/sts-63/images/k95p0264.jpg': 23,
             '/shuttle/missions/sts-65/news/sts-65-pocc-09.txt': 1,
             '/elv/elvpage.htw': 2,
             '/payloads/processing/ppart5a.gif': 1,
             '/shuttle/missions/news/1993/h01.29.93': 2,
             '/history/apollo/apollo-17/images/721200.GIF': 112,
             '/shuttle/missions/sts-70/movies/sts-70-launch-press.mpg': 588,
             '/shuttle/missions/sts-38/news': 5,
             '/shuttle/missions/sts-49/news/': 9,
             '/history/apollo/apollo-13/apllo-13.html': 1,
             '/missions.html': 1,
             '/shuttle/countdown/images/cdtclock.gif': 8662,
             '/shuttle/missions/51-l/images/86HC119.GIF': 407,
             '/shuttle/missions/sts-47/movies/': 5,
             '/shuttle/missions/51-i/51-i-info.html': 6,
             '/www/software': 2,
             '/payloads/processing/vpart3s.gif': 2,
             '/history/apollo/apollo-16/images/': 107,
             '/biomed/vegetation/gif/y255255000.gif': 1,
             '/history/gemini/gemini-viii/gemini-viii.html': 297,
             '/shuttle/technology/sts-newsref/operatio.txt': 19,
             '/shuttle/countdown/http://stentor.ca/': 1,
             '/history/skylab/skylab-2.html': 395,
             '/shuttle/missions/sts-73/sts-73-info.html': 292,
             '/history/apollo/sa-6/news/': 5,
             '/shuttle/missions/51-b/news/': 5,
             '/shuttle/missions/sts-38/images/90HC399.GIF': 1,
             '/payloads/processing/paylproc.gif': 49,
             '/history/gemini/images/GT-8.gif': 2,
             '/shuttle/missions/sts-66/news/sts-66-pocc-04.txt': 3,
             '/cgi-bin/imagemap/DSC-95EC-0001.gif': 2,
             '/shuttle/missions/4I-d/mission-4I-d.html': 1,
             '/shuttle/missions/51-a/sounds/': 2,
             '/shuttle/missions/sts-42/mission-sts-52.html': 1,
             '/SOFTWARE/WINVN.HTML': 1,
             '/shuttle/missions/sts-48/images/': 4,
             '/shuttle/missions/sts-68/news/sts-68-mcc-10.txt': 3,
             '/history/apollo/apollo-10/apollo-10.html': 922,
             '/shuttle/missions/sts-39/images/91HC372.GIF': 4,
             '/shuttle/missions/sts-63/news/sts-63-mcc-03.txt': 3,
             '/shuttle/missions/sts-28/images/': 7,
             '/persons': 11,
             '/statistics/1995/Jul/Jul95.html': 304,
             '/shuttle/missions/sts-33/sts-33-patch-small.gif': 212,
             '/software/winvn/userguide/3_7.htm': 8,
             '/shuttle/missions/sts-29/sts-29-info.html': 5,
             '/facts/faq04.html': 931,
             '/shuttle/news/sci.space.news/164': 1,
             '/software/winvn/userguide/3_1_1_6.htm': 8,
             '/statistics/1995/Jan/Jan95.html': 4,
             '/shuttle/missions/sts-61/movies/hub2.mpg': 4,
             '/HISTORY/APOLLO/APPOLO-13/APPOLO-13.HTML': 1,
             '/s...ters/enterprise.html': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0870.jpg': 121,
             '/shuttle/missions/sts-61/mcc07.txt': 1,
             '/shuttlecountdown': 1,
             '/history/apollo/apollo-13/images': 2,
             '/sts-59': 1,
             '/shuttle/missions/astro.gif': 1,
             '/shuttle/technolgoy/sts-nesref/stsref.html': 1,
             '/shuttle/missions/status/index': 3,
             '/persons/astronauts/e-to-h/GutierrezSM.txt': 9,
             '/shuttle/missions/sts-70/images/KSC-95EC-1054.jpg': 34,
             '/shuttle/technology/sts-newsref/sts-acronyms.html': 200,
             '/history/apollo/apollo-13/apollo-13.info.html': 5,
             '/sts-70/landing/orbiter/': 1,
             '/hist/apollo/apollo-13/apollo-13.html': 2,
             '/shuttle/missions/sts-9/mission-sts-9.html': 371,
             '/shuttle/missions/status/r91-16': 1,
             '/shuttle/missions/status/r153-92.ksc': 1,
             '/history/apollo/apollo-11/images/69HC635.GIF': 81,
             '/mission/': 1,
             '/shuttle/missions/sts-69/movies/': 214,
             '/shuttle/missions/sts-71/news/sts-71-mcc-08.txt': 107,
             '/shuttle/missions/61-c/docs/': 4,
             '/history/apollo/apollo-13-info.html': 21,
             '/shuttle/mission/mission.hmml': 1,
             '/shuttle/missions/status/r92-80': 1,
             '/shuttle/missions/sts-54/news/': 3,
             '/ic': 1,
             '/shuttle/countdown/liftoff.htnl': 1,
             '/shuttle/missions/sts-4/images/': 19,
             '/shuttle/missions/sts-70/news/N95-29-New-MCC-Briefing.txt': 133,
             '/history/apollo/apollo-6/images/': 39,
             '/shuttle/missions/sts-71/pixs/nasa.jpg': 1,
             '/history/appolo/appolo-13/appolo-13.html': 4,
             '/shuttle/missions/sts-77/': 86,
             '/history/apollo/apollo-1/images/apollo-1-crew.gif': 242,
             '/shuttle/missions/sts-30/mission-sts-30.html': 242,
             '/msfc/crew/wannabe.html': 11,
             '/shuttle/missions/sts-66/news/sts-66-mcc-18.txt': 2,
             '/shuttle/cpuntdown/liftoff.html': 4,
             '/history/apollo/apollo~13/apollo~13.html': 1,
             '/software/winvn/release.txt': 22,
             '/Brief': 1,
             '/shuttle/missions/sts-5/movies/': 4,
             '/shuttle/countdown/lps/consoles.html': 3,
             '/history/apollo/apollo-13/apollo-13.htm/': 7,
             '/finance': 1,
             '/shuttle/missions/sts-69/mission-sts-69.html': 6968,
             '/shuttle/missions/sts-70/images/KSC-95EC-1017.txt': 25,
             '/histoy/apollo/apollo-13/apollo-13-info.html': 1,
             '/biomed/threat/animals.html': 6,
             '/images/lcc-logo.gif': 19,
             '/image': 1,
             '/facts/http:www.disney.com': 1,
             '/shuttle/missions/sts-63/news/ksc-status-02-01-95.txt': 6,
             '/msfc/description/starview.html': 6,
             '/payloads/images/acts.gif': 4,
             '/history/apollo/apollo-15/images/71HC1159.GIF': 42,
             '/history/apollow-13/apollo-13.html': 1,
             '/news/nasa.nasamail.p/166': 1,
             '/shuttle/missions/sts-70/images/classes/net/www/content/image/jpeg.class': 2,
             '/utbin/cdt-main.pl': 2,
             '/history/apollo/apollo-13/apollo-13info.html': 7,
             '/software//viewers/icons/menu.xbm': 1,
             '/statistics/1994/Sep/Sep94_reverse_domains.html': 4,
             '/HTTP/1.0': 2,
             '/software/winvn/userguide/3_2_2.htm': 7,
             '/shuttle/missions/sts-31/images/90HC316.GIF': 5,
             '/shuttle/missions/51-f/movies/': 5,
             '/biomet/climate/gif': 1,
             '/shuttle/missions/sts-71/status/': 1,
             '/shuttle/missions/sts-71/images.html': 3,
             '/shuttle/missions/sts-43/sts-43-patch-small.gif': 155,
             '/shuttle/resources/orbiters/h': 1,
             '/htbin/imagemap/Sep94stats_r': 2,
             '/shuttle/missions/sts-70/sts-70-crew.gif': 130,
             '/shuttle/missions/sts-71/movies/sts-71-launch.mpg': 1983,
             '/history/apollo/apollo-13/.html': 1,
             '/shuttle/missions/sts-71/news/sts-71-mcc-02.txt': 50,
             '/procurement/procurment.html': 2,
             '/history/mercury/ma-6/ma-6-patch-small.gif': 487,
             '/software/winvn/userguide/2_1_1.htm': 15,
             '/shuttle/missions/sts-59/news/sts-59-mcc-06.txt': 1,
             '/images/cm-map-1.eps': 20,
             '/shuttle/missions/sts-52/docs/': 1,
             '/msfc/astro_home3.gif': 492,
             '/history/skylab/skylab.html': 1496,
             '/shuttle/missions/sts-70/news/sts-70-mcc-02.txt': 102,
             '/shuttle/missions/sts-66/news': 12,
             '/shuttle/missions/sts-25/': 1,
             '/persons/astronauts/a-to-d/conradCC.txt': 52,
             '/shuttle/missions/sts-63/movies/movies.html': 1,
             '/shuttle/missions/sts-71/movies/crew-suitup.mpg': 94,
             '/news/sci.space.news/1371': 2,
             '/shuttle/missions/sts-71/images/KSC-95EC-0747.jpg': 253,
             '/history/apollo/apollo-17/apollo-17-info.html': 517,
             '/shuttle/missions/sts-66/nission-sts-66.html': 1,
             '/shuttle/missions/status/r91-2': 2,
             '/software/winvn/userguide/3_1_1_2.htm': 14,
             '/shuttle/missions/sts-37/news': 9,
             '/history/apollo/apollo-17/docs/': 95,
             '/news/sci.space.news/archive/sci-space-news-23-oct-1993-17.txt': 9,
             '/facts/announce.html': 28,
             '/msfc/description/instruments/uitlogo_tiny.gif': 44,
             '/history/apollo/apollo-13/apollo-13.html."': 1,
             '/shuttle/missions/61-a/': 23,
             '/procurement/midrange/notices/equip/mts.htm': 52,
             '/news/sci.space.news/1988': 12,
             '/news/sci.space.news/937': 1,
             '/History/Apollo/apollo13/apollo.html': 1,
             '/shuttle/sts-69/missions/sts-69.html': 1,
             '/history/apollo/apollo-8/images/68HC870.GIF': 76,
             '/shuttle/shuttle.html': 2,
             '/statistics/1995/bkup/Jan95_full.html': 10,
             '/history/apollo/sa-2/sounds/': 3,
             '/elv/SCOUT/sampexs.jpg': 327,
             '/cgi-bin/geturlstats.pl': 153,
             '/elv/SCOUT/scprev.bak': 1,
             '/shuttle/missions/sts-70/images/KSC-95EC-1019.gif': 193,
             '/persons/astronauts/q-to-t/SpringerRC.txt': 9,
             '/payloads/schedules/mps/': 9,
             '/history/skylab/skylab-station.txt': 346,
             '/shuttle/missions.html/': 1,
             '/persons/dec/': 5,
             '/shuttle/missions/status/r90-152': 6,
             '/history/apollo/apollo-11/apollo-11.html': 5004,
             '/shuttle/missions/sts-34/sts-34-patch-small.gif': 325,
             '/ksc.html.com:80/': 1,
             '/finance/other.htm': 40,
             '/history/apollo/apollo-4/sounds/': 37,
             '/ksc.htmml': 1,
             '/history/apollo/apollo-14/images/71HC71.GIF': 66,
             '/history/apollo/apollo13/apollo13html.': 1,
             '/news/sci.space.news/1366': 2,
             '/shuttle/countdown/video/countclock1.gif': 1,
             '/images/butts/forms.gif': 1,
             '/molastro2_home.html': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0948.gif': 126,
             '/pub/winvn/': 8,
             '/shuttle/missions/sts-78/sts-78-patch.jpg': 70,
             '/shuttle/missions/sts-3/movies/': 4,
             '/shuttle/missions/sts-71/images/KSC-95EC-0589.gif': 494,
             '/history/apollo/apollo-8/images/': 160,
             '/news/nasa.nasamail.p/622': 7,
             '/software/winvn/winvn.htms': 1,
             '/history/apollo-11/apollo-11.html': 5,
             '/software/winvnwinvn.html': 2,
             '/procurement/midrange/notices/equip/rfo27.htm': 53,
             '/shuttle/missions/sts-71/images/KSC-95EC-0913.txt': 133,
             '/procurement/clause/k236-103.htm': 1,
             '/shuttle/missions/sts-64/images/sts-64d.gif': 3,
             '/software/winvn/faq/WINVNFAQ.html': 46,
             '/software/winvn/faq/WINVNFAQ-I-2.html': 77,
             '/PTS/thin_strip2.gif': 7,
             '/history/skylab/skylab.jpg': 214,
             '/htbin/imagemap/Apr95stats_b': 2,
             '/history/apollo/apollo-13/sounds/a13_001.wav': 795,
             '/software/webadmin/stsref_toc_backup.perl': 1,
             '/news/sci.space.news/2139': 1,
             '/shuttle/missions/sts-44/': 15,
             '/shuttle/tecxhnology/sts-newsref': 1,
             '/shuttle/missions/41-c/41-c-patch.jpg': 20,
             '/history/apollo/sa-4/docs/': 5,
             '/shuttle/missions/sts-60/docs/': 6,
             '/shuttle/missions/sts-30/images/89HC643.GIF': 1,
             '/shuttle/missions/sts-66/news/sts-66-pocc-19.txt': 5,
             '/news/sci.space.news/869': 1,
             '/statistics/1994/Dec/Dec94_reverse_domains.html': 2,
             '/shuttle/missions/41-b/41-b-info.html': 14,
             '/shuttle/technology/sts-newsref/sts_eclss.html': 178,
             '/history/apollo/apollo-11/images/69HC806.GIF': 45,
             '/Apollo/': 2,
             '/shuttle/missions/41-d/images/84HC421.GIF': 14,
             '/history/apollo/images/': 98,
             '/shuttle/missions/sts-26/sts-26-patch.jpg': 16,
             '/shuttle/missions/sts-51l/images/images.html': 1,
             '/history/apollo/apollo-4/images/apollo-4.jpg': 44,
             '/shuttle/missions/sts-61/sounds/': 12,
             '/facts/faq01.html': 160,
             '/www/software/winvn/winvn.html': 2,
             '/history/apollo/images/moonwalk.gif': 21,
             '/statistics/1995/Mar/Mar95_byte.gif': 4,
             '/software/techdoc/td1.gif': 3,
             '/shuttle/missions/news/1992/h05.01.92': 1,
             '/shuttle/missions/51-a/images/84HC547.GIF': 5,
             '/shuttle/missions/51-i/images/': 12,
             '/shuttle/missions/sts-77/news/': 59,
             '/history/apollo/apollo-9/images/69HC110.GIF': 33,
             '/history/gemini/images/GT-6.gif': 2,
             '/history/appolo/apollo-13/appolo-13.html': 1,
             '/shuttle/missions/41-d/images/84HC290.GIF': 6,
             '/shuttle/missions/sts-71/images/KSC-95EC-0910.jpg': 1,
             '/shuttle/missions/sts-missions-flown.txt': 275,
             '/ping': 1,
             '/history/apollo/flight': 2,
             '/mercury-logo.gif': 2,
             '/shuttle/missions/sts-27/movies/': 3,
             '/shuttle/missions/sts-70/images/KSC-95EC-0515.txt': 110,
             '/history/history.html': 11818,
             '/HISTORY/APOLLO/APOLLO-13.APOLLO-13.HTML': 2,
             '/office/nsp/': 3,
             '/shuttle/missions/sts-44/news/': 6,
             '/history/apollo/apollo-11/': 615,
             '/software/winvn/userguide/winvn20.gif': 37,
             '/shuttle/missions/sts-71/images/index71.jpg': 24,
             '/cgi-bin/mail_launchteam.pl': 2,
             '/shuttle/technolo': 1,
             '/nasa/nasa_subjects/nasa_subjectpage.html': 3,
             '/htbin/imagemap/Nov94stats_b': 3,
             '/persons/astronauts/a-to-d/ChaffeeRB.txt': 56,
             '/shuttle/missions/sts-7/mission-sts-7.html': 310,
             '/%7Edowns/home.html': 2,
             '/history/apollo/pad-abort-test-1/movies/': 30,
             '/daily/esc/': 3,
             '/pub/PC/win/': 3,
             '/history/apollo/apollo-sa.html': 643,
             '/shuttle/missions/sts-73/sts-73-patch.jpg': 77,
             '/images/cdrom-1-95/img0037.jpg': 1,
             '/history/apollo/apollo-1/apollo-1-info.html': 1213,
             '/facts/acron2.txt': 24,
             '/shuttle/missions/sts-47/news/': 8,
             '/facilities/': 15,
             '/history/gemini/gemini-11.html': 1,
             '/facilities/mlp.html': 1524,
             '/shuttle/missions/sts-66/news/sts-66-pocc-09.txt': 2,
             '/news/sci.space.shuttle/archive/sci-space-shuttle-27-jan-1995-08.txt': 5,
             '/payloads/documents/e1365.pdf': 19,
             '/msfc/description/ultraviolet.html': 1,
             '/shuttle/missions/sts-74/sts-74-patch-small.gif': 1586,
             '/resources/orbiters/atlantis.html': 1,
             '/shuttle/missions/51-l/51-l-crew.gif': 326,
             '/shuttle/missions/sts-41/news': 5,
             '/netscape.gif': 1,
             '/shuttle/resources/orbiters/orbiter-logo.gif': 4,
             '/persons/astronauts/u-to-z/YoungJW.txt': 363,
             '/software/winvn/userguide/A_1.htm': 15,
             '/payloads/processing/vpart4s.gif': 1,
             '/shuttle/countdown/lps/images/C-11-12.gif': 105,
             '/shuttle/missions/sts-71/sts-71-press-kit.txt': 1033,
             '/biomed/wwwicons/purple.gif': 36,
             '/shuttle/missions/sts-66/news/sts-66-mcc-04.txt': 3,
             '/payloads/org/cv-kvr.gif': 1,
             '/shuttle/missions/sts-59/news/sts-59-mcc-07.txt': 2,
             '/payloads/schedules/test/pics.pdf': 9,
             '/i/': 1,
             '/mdss': 40,
             '/payloads/processing/ppart12as.gif': 4,
             '/countdown.html': 1,
             '/shuttle/missions/sts-71/movies/sts-71-hand-shake.mpg': 635,
             '/history/apollo/%20apollo-13apollo-13.html': 1,
             '/shuttle/missions/51-d/news/': 5,
             '/shuttle/missions/news/1993/h03.03.93': 2,
             '/news/sci.space.news/754': 3,
             '/news/sci.space.news/2097': 1,
             '/history/aplollo': 1,
             '/space/': 1,
             '/shuttle/countdown/lps/images/BKUP-INT.gif': 108,
             '/images/USA-logosmall.gi': 6,
             '/images/postit.gif': 68,
             '/shuttle/missions/sts-67/images/k95p0390.jpg': 8,
             '/shuttle/missions/sts-65/docs/': 6,
             '/history/apollo-13/apollo.html': 1,
             '/shuttle/missions/sts-66/news/sts-66-pocc-18.txt': 4,
             '/news/sci.space.shuttle/archive/sci-space-shuttle-9-aug-1994-62.txt': 24,
             '/news/sci.space.news/archive/sci-space-news-20-feb-1995-71.txt': 6,
             '/history/apollo/apollo-11/apollo-11-info.html': 898,
             '/shuttle/missions/sts-31/sts-31-patch-small.gif': 179,
             '/news/sci.space.news/1245': 8,
             '/facilities/aerial.gif': 1,
             '/nasa/history/': 2,
             '/shuttle/countdown/irftoft.html': 3,
             '/biomed/threat/gif/eag7fin.gif': 1,
             '/shuttle/missions/sts-70/images/KSC-95EC-1016.jpg': 331,
             '/shuttle/missions/sts-5/sts-5-patch-small.gif': 263,
             '/shuttle/missions/news/1992/h05.06.92': 1,
             '/shuttle/missions/sts-70/mission-sts-70.htm/': 1,
             '/history/apollo/sa-7/images/': 6,
             '/history/apollo/apollo-15/images/71HC519.GIF': 40,
             '/shuttle/countdown/images/KSC-83.JPG': 1,
             '/news/sci.space.news/1393': 1,
             '/procurement/clause/k236111a.htm': 1,
             '/shuttle/missions/status/r92-88': 1,
             '/shuttle/orbites/': 1,
             '/BROWSER': 1,
             '/GIF/ksc-logo-small.gif': 1,
             '/shuttle/missions/sts-71/countdown': 1,
             '/images/kscarea-big.gif': 2,
             '/shuttle/missions/sts-65/news/sts-65-pocc-20.txt': 1,
             '/shuttle/missions/sts-67/images/KSC-95EC-0394.txt': 5,
             '/mdss/cia': 1,
             '/shuttle/missions/sts-70/tv': 2,
             '/shuttle/countdown/lps/images/C-9-10-large.gif': 18,
             '/software/webadmin/payload2.perl': 9,
             '/history/apollo/apollo-17/images/72HC181.GIF': 80,
             '/history/apollo/sa-7/sa-7.html': 58,
             '/shuttle/missions/sts-33/news': 4,
             '/news/sci.space.news/1116': 1,
             '/shuttle/missions/sts-52/sts-52-press-kit.txt': 11,
             '/shuttle/missions/sts-1/images/81HC315.GIF': 32,
             '/SOFTWARE/WINUIM.HTML': 2,
             '/histroy/apollo/apollo-13/apollo-13.html': 5,
             '/shuttle/missions/sts-76/docs/': 13,
             '/penthousemag.com': 1,
             '/shuttle/missions/sts-53/movies/sts-53-launch.mpg': 658,
             '/shuttle/resources/orbiters/columbia.html': 2091,
             '/elv/FACILITIES/cape.htm': 1,
             '/news/sci.space.news/1472': 2,
             '/shuttle/missions/51-f/51-f-patch-small.gif': 213,
             '/shuttle/missions/sts-61/images/': 30,
             '/shuttle/sts-28/': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0885.txt': 1,
             '/persons/astronauts/m-to-p/ParkerRA.txt': 7,
             '/shuttle/missions/sts-55/sts-55-press-kit.txt': 11,
             '/shuttle/missions/sts-59/sounds/': 4,
             '/persons/astronauts/a-to-d/BakerES.txt': 8,
             '/persons/astronauts/i-to-l/LeeMC.txt': 1,
             '/shuttle/missions/sts-70/sts-70-day-10-highlights.html': 68,
             '/history/apollo/apollo-13.htm': 8,
             '/shuttle/video/lifevideo.jpge': 1,
             '/eta.shtml': 1,
             '/biomed/fire/gif/fire9.gif': 1,
             '/nasa.html': 3,
             '/shuttle/missions/sts-43/mission-sts-43.html': 146,
             '/biomed/vegetation/gif/scpcfinmed.gif': 1,
             '/elv/SCOUT/scout.gif': 1493,
             '/shuttle/missions/sts-65/movies/': 12,
             '/software/winvn.html': 9,
             '/msfc/visitor/visitors.html': 2,
             '/software/winvn/faq/WINVNFAQ-V-1.html': 18,
             '/software/winvn/userguide/winvn35.gif': 2,
             '/shuttle/missions/sts-44/sts-44-patch-small.gif': 156,
             '/shuttle/missions/sts-69/test/test1.gif': 1,
             '/elv/ATLAS_CENTAUR/goess.jpg': 366,
             '/shuttle/missions/51-j/51-j-info.html': 10,
             '/payloads/org/cg.html': 2,
             '/history/apollo/pad-abort-test-2/news/': 4,
             '/news/sci.space.news/1402': 1,
             '/shuttle/missions/status/r88-171': 3,
             '/shuttle/missions/sts-3/images/82HC247.GIF': 8,
             '/history/apollo/apollo-13/ap0ll0-13.html': 2,
             '/shuttle/missions/sts-71/news/ksc-status-06-08-95.txt': 5,
             '/shuttle/missions/sts-54/': 20,
             '/procurement/midrange/notices/equip/equip.htm': 44,
             '/biomed/threat/gif/ws2pcfinmed1.gif': 1,
             '/shuttle/missions/sts-57/news/': 21,
             '/history/gemini/gemini-3/gemini-3.html': 471,
             '/payloads/cm-systems.html': 267,
             '/shuttle/missions/sts-34/news/': 18,
             '/shuttle/missions/sts-39/sts-39-patch-small.gif': 192,
             '/software/winvn/userguide/winvn24.gif': 4,
             '/history/apollo/apollo-16/apollo-16.html': 606,
             '/cgi-bin/imagemap/countdown70': 19085,
             '/shuttle/missions/sts-65/news/': 9,
             '/shuttle/missions/sts-67/images/KSC-95EC-0391.gif': 113,
             '/images/crawlerway.gif': 26,
             '/news/sci.space.news/705': 7,
             '/statistics/1995/Feb/Feb95_reverse_domains.html': 3,
             '/software/winvn/userguide/winvn17.gif': 1,
             '/shuttle/countdown/countdown71.html': 1,
             '/images/kscmap-logo.gif': 194,
             '/shuttle/resources/orbiters/atlantis.gif': 232,
             '/shuttle/missions/51-c/': 19,
             '/elv/ATLAS_CENTAUR/acsuns.jpg': 369,
             '/shuttle/missions/sts-73/news/sts-73-mir-01.txt': 184,
             '/shuttle/missions/sts-49/sts-49-patch-small.gif': 428,
             '/persons/astronauts/a-to-d/BoldenCF.txt': 61,
             '/software/webadmin/astronauts.old': 1,
             '/shuttle/missions/sts-1/news/': 54,
             '/history/apollo/apollo-13/images/70hc467.gif': 1,
             '/biomed/threat/': 3,
             '/shuttle/technology/sts-newsref/centers.html': 392,
             '/history/appolo-13/apollo-13.html': 1,
             '/persons/astronauts/u-to-z/VossJE.txt': 44,
             '/shuttle/missions/sts-71/images/images.html': 15897,
             '/images/kscmap.gif': 325,
             '/payloads/org/cs-ped.gif': 2,
             '/history/apollo/apollo-overview.txt': 19,
             '/procurement/midrange/notices/equip/ifb29.htm': 51,
             '/mdss/srqa/qpa/s_qpa.gif': 27,
             '/history/apollo/apollo-10/movies/': 54,
             '/software/winvn/userguide/1_3.htm': 23,
             '/Harvest/gatherers/WWW/tmp/index.html': 1,
             '/shuttle/missions/sts-7/sts-7-patch.jpg': 24,
             '/payloads/processing/ppart11c.gif': 1,
             '/finance/tour.gif': 513,
             '/news': 3,
             '/shuttle/countdown/lps/sts-71-s-5-i.mpg': 2,
             '/persons/astronauts/e-to-h/gibsonEG.txt': 18,
             '/shuttle/news/sci.space.news/50': 13,
             '/icons/movie.xbm': 2925,
             '/shuttle/missions/sts-70/images/KSC-95EC-1053.gif': 24,
             '/software//viewers/': 1,
             '/biomed/history/gif/historyfinmed.gif': 9,
             '/shuttle/missions/status/r103-92.ksc': 1,
             '/history/apollo/apollo-14/apollo-14.html': 962,
             '/shuttle/missions/sts-64/images/sts-64a.gif': 7,
             '/iuma.com': 3,
             '/shuttle/missions/sts-68/news/sts-68-mcc-09.txt': 3,
             '/shuttle/missions/sts-63/sts-63-patch-smallgif': 5,
             '/shuttle/technology/sts-newsref/102_operatns.txt': 3,
             '/~pccomp/graphics/sfeat.gif': 3,
             '/shuttle/missions/sts-56/sounds/': 4,
             '/shuttle/missions/sts-69/mission-sts-69/html': 2,
             '/shuttle/missions-69/news/': 1,
             '/software/winvn/userguide/previous.gif': 4,
             '/shuttle/countdown/video/': 116,
             '/shuttle/missions/sts-71/mission-sts-71.html': 16717,
             '/history/apollo/apollo.htm': 2,
             '/payloads/org/cv-lod.gif': 2,
             '/shuttle/missions/sts-30/news': 4,
             '/shuttle/missions/sts-74/': 247,
             '/facilities/letf.html': 31,
             '/shuttle/missions/51-d/51-d-patch-small.gif': 259,
             '/payloads/level4/canod.mpg': 2,
             '/facilities/oc.html': 174,
             '/software/techdoc/td-2.3.html': 1,
             '/history/apollo/sa-1/sa-1-info.html': 64,
             '/Harvest/brokers/queryhelp.html': 1,
             '/history/mercury/ma-9/ma-9-patch.gif': 31,
             '/procurement/clause/k236-115.htm': 2,
             '/news/sci.space.news/2566': 1,
             '/shuttle/missions/sts-70/images/KSC-95EC-1013.jpg': 373,
             '/shuttle/countdown/video/lifevideo.jpge': 1,
             '/biomed/vegetation/saltmrsh.html': 1,
             '/statistics/1995/Mar/Mar95_reverse_domains.html': 12,
             '/~www/pics/ssuthumb.gif': 1,
             '/history/early-astronauts.txt': 1083,
             '/shuttle/missions/sts-46/mission-sts-46.html': 144,
             '/statistics/1994/Nov/Nov94_request.gif': 4,
             '/shuttle/missions/missions.html': 24865,
             '/shuttle/countdown/lps/images/C-2.gif': 91,
             '/shuttle/missions/sts-71/images/images.html/KSC-95EC-0906.gif': 1,
             '/history/apollo/as-203/as-203-info.html': 29,
             '/htbin/imagemap/Jan95stats_r': 2,
             '/news/sci.space.news/689': 1,
             '/shuttle/missions/sts-3/images/82HC234.GIF': 6,
             '/tiny_um.gif': 1,
             '/software/webadmin/old-sts.perl': 1,
             '/history/apollo/sa-2/images/': 11,
             '/biomed/threat/gif/kscteplantsmed.gif': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0950.txt': 14,
             '/statistics/1994/Dec/Dec94.html': 1,
             '/software/winvn/userguide/3_8_3.htm': 11,
             '/shuttle/missions/sts-71/images/KSC-95EC-0423.jpg': 1365,
             '/microsoft.com/pages': 1,
             '/persons/astronauts/q-to-t/ThorntonKC.txt': 1,
             '/shuttle/missions/sts-30/images/89HC320.GIF': 1,
             '/+': 2,
             '/history/apollo/apollo-2/apollo-2.html': 1,
             '/history/apollo/apollo-14/videos/': 23,
             '/images/vab-medium.gif': 163,
             '/shuttle/missions/sts-29/images/89HC240.GIF': 1,
             '/history/apollo/pad-abort-test-2/sounds/': 3,
             '/msfc/crew/oswald.html': 2,
             '/cgi-bin/imagemap/video/livevideo.gif': 2,
             '/persons/astronauts/q-to-t/SegaRM.txt': 2,
             '/news/sci.space.news/2048': 1,
             '/persons/astronauts/e-to-h/GrabeRJ.txt': 11,
             '/shuttle/missions/sts-66/sts-66-info.html': 30,
             '/history/apollo/a-004/a-004-patch-small.gif': 28,
             '/biomed/program.html': 8,
             '/images/kscmap-tiny.gif': 6920,
             '/apollo/missions/apollo-13/apollo-13-1.html': 2,
             '/shuttle/missions/51-b/51-b-patch.jpg': 11,
             '/persons/astronauts/a-to-d/BakerMA.txt': 4,
             '/sts-70/icons/p-home.gif': 1,
             '/procurement/business/spc.htm': 4,
             '/shuttle/resource/orbites/discovery': 1,
             '/history/apollo/apollo-12/images/69HC1007.GIF': 63,
             '/imagemap.cgi//launch/launch.map': 1,
             '/history/apollo/apollo-9/images/69HC292.GIF': 24,
             '/msfc/crew/gregory.html': 2,
             '/shuttle/countdown/lps/images/AA-ROW.gif': 202,
             '/shuttle/missions/sts-67/images/KSC-95EC-0386.jpg': 20,
             '/shuttle/missions/sts-70/images/KSC-95EC-0649.gif': 332,
             '/shuttle/missions/41-g/images/84HC394.GIF': 10,
             '/payloads/level4': 4,
             '/history/apollo/flight-summary.txt': 2060,
             '/shuttle/missions/61-a/images/': 15,
             '/shuttle/missions/sts-71/sts-71-day-07-highlights.html': 415,
             '/history/gemini/gemini-x/gemini-x-info.html': 52,
             '/shuttle/missions/sts-31/images/90HC477.GIF': 4,
             '/shuttle/missions/sts-65/news/sts-65-pocc-24.txt': 1,
             '/images/STS-5.JPG': 81,
             '/shuttle/missions/sts-35/images/90HC708.GIF': 1,
             '/history/apollo/apollo-13/apollo13.htm': 5,
             '/images/cdrom-1-95/img0006.jpg': 3,
             '/cgi-bin/imagemap/lan': 3,
             '/shuttle/missions/sts-75/sts-75-info.html': 71,
             '/shuttle/missions/sts-70/news/sts-70-mcc-18.txt': 14,
             '/history/apollo/as-201/as-201-patch-small.gif': 518,
             '/history/apollo/apollo-11/sounds/': 403,
             '/shuttle/missions/sts-6/mission-sts-6.html': 316,
             '/shuttle/missions/sts-64/images/sts-64e.gif': 13,
             '/shuttle/missions/sts-65/news/sts-65-pocc-01.txt': 1,
             '/shuttle/missions/sts-50/images/92HC521.GIF': 1,
             '/cgi-bin/imagemap/pad': 1,
             '/shuttle/missions/sts-73/news/': 310,
             '/Harvest/cgi-bin/HarvestGather.cgi': 3,
             '/shuttle/missions/sts-71/images/KSC-95EC-0903.gif': 65,
             '/history/apollo/apollo-spacecraft.txt': 2459,
             '/history/apollo/sa-2/sa-2.html': 88,
             '/shuttle.html': 2,
             '/images/cm.gif': 63,
             '/images/cdrom-1-95/img0028.jpg': 1,
             '/cgi-bin/findwwwdoc': 1,
             '/misc/showcase/personal_edition/images/milan_banner.gif': 5,
             '/statistics/1994/May/May94_reverse_domains.html': 1,
             '/icons': 1,
             '/shuttle/missions/sts-67/news/sts-67-pocc-18.txt': 1,
             '/shuttle/missions/41-d/news/': 13,
             '/shuttle/missions/sta-71': 1,
             '/shuttle/missions/sts-70/images/KSC-95EC-1020.gif': 189,
             '/news/nasa.nasamail.p/13': 1,
             '/software/winvn/brydon.gif': 22,
             '/history/apollo/apollo-13/news': 1,
             '/~software/winvn': 1,
             '/images/NASA-lo': 1,
             '/news/nasa.nasamail.p/502': 1,
             '/psm/grass.html': 1,
             '/dsc.html': 1,
             '/software/winvn/userguide/2_3_5.htm': 2,
             '/lycos-form.html': 1,
             '/news/nasa.nasamail.p/584': 1,
             '/shuttle/missions/51-l/news': 761,
             '/software//viewers/viewers.html': 1,
             '/news/nasa.nasamail.p/188': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0953.jpg': 2,
             '/history/gemini/gemini-viii/docs/': 1,
             '/history/appollo/appollo-13/appollo-13.htm': 1,
             '/shuttle/missions/sts-70/woodpecker.html': 760,
             '/shuttle/missions/sts-70/images/KSC-95EC-1002.txt': 17,
             '/shuttle/missions/sts-5/news': 2,
             '/htbin/imagemap/Jul95stats_b': 52,
             '/software/winvn/userguide/winvn9.gif': 18,
             '/shuttle/missions/sts-71/images/i': 1,
             '/shuttle/missions/sts-74/sounds/': 37,
             '/msfc/description/description.html': 49,
             '/history/apollo/apollo-16/movies/': 61,
             '/shuttle/missions/51-l': 3,
             '/shuttle/missions/sts-71/images/KSC-95EC-0874.jpg': 135,
             '/history/apollo.htm/...': 1,
             '/shuttle/misssions/sts-67/images/KSC-95EC-0391.gif': 3,
             '/shuttle/missions/sts-53/sts-53-patch.gif': 1,
             '/shuttle/missions/sts-66/news/sts-66-pocc-14.txt': 3,
             '/shuttle/missions/gopher.uni-stuttgart.de': 1,
             '/shuttle/missions/51-l/images/850128.GIF': 502,
             '/shuttle/technology/sts-newsref/sts-oalt.html': 84,
             '/history/gemini/sounds/': 6,
             '/history/apollo/apollo-17/images/72HC941.GIF': 72,
             '/l': 2,
             '/shuttle/missions/sts-75/sts-75-patch.jpg': 13,
             '/history/apollo/apollo.html"': 4,
             '/shuttle/missions/sts-69/sts69.jpg': 67,
             '/shuttle/missions/sts-70/orbit/orbiter/tracksw': 1,
             '/cgi-bin/imagemap/fr.gif': 2,
             '/shuttle/technology/images/sts_spec_6.jpg': 56,
             '/history/apollo/as-201/sounds/': 29,
             '/persons/nasa-cm/uf.gif': 19,
             '/payloads/processing/ppart5s.gif': 6,
             '/shuttle/technology/images/launch_sites_8.jpg': 33,
             '/dig.net/~hed/': 2,
             '/images/ksclogo-medium-scanned.gif': 4,
             '/history/apollo/as-201/as-201.html': 593,
             '/history/apollo/apollo13/sounds': 1,
             '/shuttle/countdown/images/lccteam-large.gif': 2,
             '/shuttle/missions/sts-70/mission-sts-69.html': 1,
             '/shuttle/missions/sts-49/docs/': 1,
             '/facilities/osb.html': 61,
             '/shuttle/missions/sts-59/news/xsar.txt': 1,
             '/software.pub.win3.winvn/': 1,
             '/shuttle/missions/sts-67/images/KSC-95EC-0396.gif': 104,
             '/gpc.html': 1,
             '/statistics/images/stat.gif': 68,
             '/elv/elvpage.htm/elv/TITAN/titan.gif': 1,
             '/shuttle/missions/sts-31/sts-31-patch.jpg': 16,
             '/persons/astronauts/a-to-d/CleaveML.txt': 11,
             '/payloads/processing/vpart4a.gif': 1,
             '/news/sci.space.shuttle': 2,
             '/shuttle/missions/status/r-01.02.93.ksc': 1,
             '/news/sci.space.news/1637': 1,
             '/shuttle/missions/news/1989/h09.18.89': 2,
             '/history/apollo/apollo-14/images/71HC292.GIF': 58,
             '/shuttle/missions/sts-64/news/sts-64-mcc-08.txt': 3,
             '/history/appollo/apollo-13/apollo-13.html': 10,
             '/history/apollo13.html': 1,
             '/shuttle/missions/sts-69/test/test.gif': 1,
             '/shuttle/missions/technology/sts-newsref/stsref-toc.html': 107,
             '/payloads/schedules/mini/sts-71.pdf': 7,
             '/shuttle/countdown/video/6-22-95-pad-sunset.gif': 10,
             '/history/gemini/gemini-vii/sounds/': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0952.gif': 4,
             '/shuttle/technology/sts-newsref/sts-oms.html': 310,
             '/news/sci.space.news/archive/sci-space-news-3-mar-1995-21.txt': 11,
             '/shuttle/missions/sts-26/sts-26-press-kit.txt': 42,
             '/history/apollo/apollo-11/news/': 123,
             '/shuttle/missions/sts-missions-flown.txt~': 11,
             '/images/STS-7.JPG': 26,
             '/shuttle/missions/sts-71/sts-71-day-06-highlights.html': 482,
             '/history/gemini/gemini-3/': 7,
             '/software/winvn/winvn.html/': 5,
             '/facilities/crawlerway.html': 142,
             '/shuttle/missions/61-a/images/85HC430.GIF': 7,
             '/payloads/processing/paylbut1.gif': 1,
             '/apollo/apollo-13': 2,
             '/htbin/cdt': 1,
             '/shuttle/missions/sts-3/images/82HC296.GIF': 7,
             '/shuttle/missions/41-b/images/': 19,
             '/news/nasa.nasamail.p/97': 1,
             '/images/ksc3d.gif': 9,
             '/history/apollo/apollo-13-appolo-13.html': 1,
             '/http:/images.jsc.nasa.gov/html/shuttle.htm': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0882.txt': 36,
             '/shuttle/missions/sts-70/movies/sts-70-crew-suitup.mpg': 83,
             '/shuttle/missions/news/1992/h10.30.92': 1,
             '/shuttle/missions/sts-62/news/': 4,
             '/shuttle/missions/61-c/61-c-patch.jpg': 21,
             '/shuttle/missions/sts-45/news': 3,
             '/shuttle/missions/41-c/images/84HC99.GIF': 6,
             '/news/sci.space.shuttle/archive/sci-space-shuttle-6-feb-1995-73.txt': 1,
             '/shuttle/missions/sts-42/': 12,
             '/payloads/processing/vpart10s.gif': 3,
             '/shuttle/missions/sts-71/images/KSC-95EC.txt': 2,
             '/shuttle/missions/sts-66/news/sts-66-mcc-08.txt': 3,
             '/shuttle/missions/sts-58/news': 2,
             '/shuttle.countdown/': 1,
             '/shuttle/missions/sts-70/images/KSC-95EC-0667.txt': 45,
             '/s': 1,
             '/images/launchpalms.gif': 275,
             '/mdss/': 68,
             '/shuttle/missions/41-d/images/84HC417.GIF': 8,
             '/shuttle/missions/sts-1/movies/': 74,
             '/shuttle/missions/sts-68/': 34,
             '/payloads/processing/cutout0.gif': 1,
             '/news/sci.space.news/2607': 1,
             '/news/sci.space.news/1921': 1,
             '/history/gemini/gemini-ix-a/gemini-ix-a.html': 221,
             '/shuttle/missions/sts-41/sts-41-patch-small.gif': 186,
             '/software/winvn/faq/WINVNFAQ-I-7.html': 95,
             '/shuttle/missions/sts-70/images/KSC-95EC-0575.gif': 88,
             '/history/apollo/as-203/': 14,
             '/history/apollo/apollo-11/sounds/A01106AA.WAV': 287,
             '/history/apollo/apollo-15/images/': 124,
             '/finance/main.htm': 568,
             '/payloads/org/cg-iso.html': 1,
             '/history/apollo/apollo-14/sounds/': 61,
             '/shuttle/technology/sts-newsref/sts_coord.html': 1067,
             '/history/apollo/apollo-14/docs/': 60,
             '/images/p263_240.jpg': 9,
             '/payloads/processing/ppart3as.gif': 2,
             '/shuttle/movies/': 158,
             '/shuttle/technology/images/aft_fuselage_2-small.gif': 1287,
             '/shuttle/missions/51-f/images/85HC290.GIF': 8,
             '/history/apollo/apollo-8/apollo-8.html': 937,
             '/biomed/fire/gif/legend.gif': 4,
             '/shuttle/mission/sts=70/': 1,
             '/nasa_homepage.html': 1,
             '/shuttle/missions/sts-26/mission-sts-26.html': 575,
             '/images/cdrom-1-95/img0077.jpg': 3,
             '/shuttle/missions/sts-5/images/82HC651.GIF': 6,
             '/persons/astronauts/a-to-d/BridgesRD.txt': 6,
             '/shuttle/missions/sts-89/mission-sts-89.html': 7,
             '/statistics/1993/Sep/Sep93.html': 5,
             '/shuttle/missions/sts-72/sts-72-patch-small.gif': 1301,
             '/history/apollo/images/apollo.html': 1,
             '/news/sci.space.news/719': 1,
             '/shuttle/missions/sts-46/images/92HC326.GIF': 2,
             '/statistics/1995/Jan95.html': 1,
             '/software/webadmin/payload.txt': 15,
             '/shuttle/missions/sts-30/news/': 4,
             '/history/apollo/apollo-14/images/71HC277.GIF': 62,
             '/news/sci.space.news/577': 1,
             '/payloads/org/cg-lso.gif': 5,
             '/shuttle/missions/sts-64/images/sts-64c.gif': 3,
             '/shuttle/missions/sts-65/news': 4,
             '/lc39a.gif': 1,
             '/shuttle/missions/sts-68/ksc-upclose.gif': 175,
             '/biomed/soils/gif/soilrmed.gif': 3,
             '/shuttle/missions/sts-66/news/sts-66-ksc-01.txt': 7,
             '/htbin/imagemap/Jan94stats_r': 2,
             '/shuttle/missions/sts-71/news/ksc-status-06-06-95.txt': 6,
             '/shuttle/missions/sts-69/sounds/': 135,
             '/shuttle/missions/sts-37/sts-37-press-kit.txt': 19,
             '/history/apollo/apollo-13/images/70HC431.GIF': 736,
             '/icons/image.xb': 7,
             '/finance//brrow_1t.gif': 170,
             '/biomed/fire/gif/fire1med.gif': 5,
             '/history/apolla-13/apollo-13.html': 1,
             '/facilities/opf.html': 717,
             '/shuttle/missions/sts-30/docs/': 1,
             '/shuttle/missions/sts-76/': 43,
             '/facilities<A': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-091': 1,
             '/shuttle/missions/sts-53/mission-sts-53.html': 148,
             '/shuttle/missions/sts-9/images/83HC751.GIF': 4,
             '/apollo/apollo-13/': 2,
             '/shuttle/missions/sts-67/news/sts-67-pocc-31.txt': 6,
             '/htbin/imagemap/May94stats_b': 2,
             '/history/apollo.html': 27,
             '/shuttle/missions/51-l/images/86HC125.GIF': 421,
             '/shuttle/missions/51-d/images/85HC97.GIF': 2,
             '/ksc..html': 2,
             '/payloads/org/cm.html': 7,
             '/mdss/dcy2.gif': 840,
             '/history/spollo/apollo-13/apollo-13-info.html': 2,
             '/AIAA/': 1,
             '/gtbin/cdt_main.pl': 1,
             '/images/de-logo.gif': 36,
             '/shuttle/technology/images/srb_mod_compare_6.jpg': 29,
             '/images/cdrom-1-95/img0001.jpg': 33,
             '/shuttle/missions/sts-46/sts-46-patch-small.gif': 164,
             '/msfc/crew/lawrence-small.gif': 4,
             '/shuttle/missions/sts-70/mission-sts-': 7,
             '/www.lycos.com': 1,
             '/shuttle/missions/sts-7/docs/': 3,
             '/history/apollo/apollo-7/images/68HC554.GIF': 36,
             '/history/gemini/gemini-xi/images/': 3,
             '/history/sklab/skylab.html': 1,
             '/history/pg24.htm': 2,
             '/msfc/crew/sound_btn.gif': 5,
             '/persons/nasa-cm/smr.gif': 9,
             '/history/apollo/apollo-%2013/apollo-13.html': 3,
             '/payloads/documents/': 32,
             '/software/techdoc/td-REV_.html': 4,
             '/msfc/description/home_btn.gif': 46,
             '/shuttle/technology/sts-newsref/sts-cron-72.html': 40,
             '/history/apollo/pad-abort-test-1/': 12,
             '/shuttle/missions/sts-70/70crew.jpg': 33,
             '/msfc/crew/jernigan.wav': 5,
             '/elv/DOCS/LLV_DATE.HTM': 3,
             '/shuttle/missions/sts-37/mission-sts-37.html': 304,
             '/history/Apollo/apollo-13.html': 1,
             '/shuttle/missions/sts-3/': 33,
             '/winvn/winvn.html': 2,
             '/software/winvn/userguide/3_4.htm': 4,
             '/procurement/business/boc.htm': 5,
             '/shuttle/missions/51-l/51-l-patch.jpg': 242,
             '/history/apollo/apollo-10/apollo-10-patch-small.gif': 857,
             '/shittle/countdown/': 1,
             '/shuttle/countdown/lps/omr/omr.html': 206,
             '/history/apollo/www': 1,
             '/shuttle/missions/sts-70/images/temp/KSC-95EC-0575.jpg': 1,
             '/elv/elvhead3.gif': 1646,
             '/orbit/': 2,
             '/shuttle/missions/sts-63/sts-63-patch-large.gif': 1,
             '/msfc/team/msfc/stephanie-12.GIF': 2,
             '/procurement/midrange/equip.htm': 207,
             '/shuttle/missions/news/1991/h04.02.91': 1,
             '/shuttle/resources/resources.html': 4,
             '/history/appolo/appolo-13.html': 1,
             '/ksc.html/facts/about_ksc.html': 5,
             '/LDAR/LDARhp.html': 9,
             '/history/skylab/skylab-3.html': 246,
             '/shuttle/missions/sts-55/': 16,
             '/shuttle/missions/sts-66/news/sts-66-mcc-19.txt': 3,
             '/payloads/org/cg-iso/nethelp/newslt04/newslt04.htm': 2,
             '/history/apollo/apollo-16/images/72HC277.GIF': 34,
             '/shuttle/missions/sts-31/images/90HC291.GIF': 5,
             '/icon/redball.gif': 42,
             '/news/sci.space.news/1727': 7,
             '/HISTORY/APOLLO/APOLLO-13/APOLLO-13.HTML-': 2,
             '/history/apollo/apollo-13/apollo/13.html': 1,
             '/shuttle/missions/sts-43/images/91HC594.GIF': 2,
             '/home%20page': 1,
             '/software/winvn/userguide/2_1_2.htm': 13,
             '/images/KSC-94EC-412.jpg': 146,
             '/shuttle/technology/images/sts_spec_6-small.gif': 1841,
             '/shuttle/missions/sts-43/sts-43-patch.jpg': 11,
             '/shuttle/missions/status/r89-171': 1,
             '/history/apollo/as-201/images/as-201-launch-small.gif': 13,
             '/shuttle/missions/sts-64/sts-64-crew.txt': 2,
             '/history/apollo/apollo-11/images/69HC761.GIF': 189,
             '/software//winvn/icons/image.xbm': 1,
             '/www/software/winvn/WINVN.HTML': 2,
             '/wxworld/images/IMAGE6pNgmSfc/95072712_00.GIF': 2,
             '/facilities/hq.html': 70,
             '/payloads/org/cg-esd.html': 5,
             '/history/apollo/apollo13/apollo13.htm': 2,
             '/news/sci.space.news/archive/sci-space-news-25-jan-1995-75.txt': 16,
             '/shuttle/missions/sts-76/sts-76-info.html': 54,
             '/history/apollo/apollo-9/apollo-9-info.html': 111,
             '/software/winvn/userguide/winvn3.gif': 7,
             '/news/sci.space.policy/archive/sci-space-policy-24-apr-1995-12.txt': 1,
             '/biomed/climate/gif/f22pcfinmed.gif': 3,
             '/history/apollo/as-202/': 10,
             '/shuttle/missions/51-j/': 31,
             '/sts-70/green/': 1,
             '/software/winvn/wvsmall.gif': 5871,
             '/history/apollo/apollo/-13/apollo-13-info.html': 1,
             '/statistics/1995/Jul/Jul95_daily_byte.gif': 12,
             '/shuttle/missions/sts-65/news/sts-65-pocc-23.txt': 1,
             '/biomed/wwwicons/warning.gif': 32,
             '/shuttle/countdown/lps/sts-71-s-5-i3.mpg': 1,
             '/shuttle/missions/51-l/www.yahoo.com': 1,
             '/sts-70/orbiter/': 1,
             '/shuttle/missions/sts-67/news/sts-67-pocc-20.txt': 1,
             '/persons/astronauts/q-to-t/SullivanKD.txt': 26,
             '/msfc/description/ips-small.gif': 48,
             '/history/apollo/apollo-13/docs/': 1693,
             '/software/winvn/userguide/winvn1.gif': 38,
             '/persons/astronauts/a-to-d/ConradCC.txt': 13,
             '/history/apollo/apollo-12/images/69HC1326.GIF': 40,
             '/SOFTWARE/WINUM/WINUM.HTML': 1,
             '/images/ksclogo-scanned.gif': 15,
             '/shuttle/missions/sts-5/images/82HC557.GIF': 6,
             '/shuttle/missions/sts-67/images/KSC-95EC-0387.txt': 9,
             '/shuttle/missions/sts-67/images/KSC-95EC-0383.gif': 62,
             '/msfc/crew/food.html': 11,
             '/mfsc/astro_home.html': 1,
             '/shuttle/missions/sts-68/sts-68-press-kit.txt': 40,
             '/shuttle/missions/status/r88-113': 1,
             '/news/sci.space.news/archive/sci-space-news-12-apr-1995-61.txt': 12,
             '/shuttle/missions/sts-67/sts-67/images/KSC-95EC-0396-small.gif': 4,
             '/ksc.html/www/shuttle/technology': 1,
             '/news/sci.space.news/2053': 1,
             '/news/sci.space.news/501': 1,
             '/news/sci.space.news/728': 3,
             '/images/rss.gif': 1435,
             '/shuttle/missions/sts-71/images/KSC-95EC-0907.jpg': 242,
             '/software/wunvn/winvn.html': 1,
             '/shuttle/missions/sts-71/images/inside4=icon.gif': 1,
             '/payloads/processing/ppart2a.gif': 1,
             '/msfc/crew/astronaut.html': 7,
             '/bigspot.com': 1,
             '/shuttle/missions/sts-71/images/KSC-95EC-0915.jpg': 1664,
             '/shuttle/': 816,
             '/shuttle/missions/51-f/51-f-info.html': 13,
             '/biomed/wetlands/gif/legend.gif': 4,
             '/software/winvn/faq/WINVNFAQ-IV-2.html': 39,
             '/shuttle/countdown/liftoff.htmll': 1,
             '/history/apollo/apollo_13/apollo13.html': 1,
             '/history/apollo/sa-10/sa-10-patch-small.gif': 3,
             '/shuttle/countdown/lps/images/C-11-12-large.gif': 18,
             '/biomed/threat/gif/perfinmed.gif': 1,
             '/shuttle/missions/sts-74/docs/': 83,
             '/nasa/nasa_hottopics.html': 3,
             '/shuttle/missions/sts-35/sts-35-patch-small.gif': 243,
             '/shuttle/missions/sts-70/images/holes-in-et.gif': 689,
             '/shuttle/missions/sts-67/images/ksc-95ec-0391.gif': 5,
             '/shuttle/missions/sts-71/news/sts-71-mcc-19.txt': 81,
             '/shuttle/missions/sts-68/news/': 15,
             '/shuttle/missions/sts-1/sts-1-crew.jpg': 3,
             '/payloads/processing/vpart7.html': 1,
             '/shuttle/technology/images/srb_mod_compare_1.jpg': 60,
             '/history/appollo/appollo-13/apollo-13-info.html': 1,
             '/history/gemini/movies/': 4,
             '/facilities/www.commerce.com/\\\\www.commerce.com\\cmt': 1,
             '/shuttle/missions/sts-70/images/KSC-95EC-1000.jpg': 70,
             '/shuttle/missions/sts-49/sts-49-launch.gif': 1,
             '/pub/win3/winvn': 20,
             '/history/apollo/apollo-17/images/': 260,
             '/images/ssf.gif': 19,
             '/shuttle/missions/sts-64/news/sts-64-mcc-03.txt': 2,
             '/shuttle/missions/sts-59/news/': 5,
             '/shuttle/missions/sts-60/news': 7,
             ...})
In [38]:
import json
with open("urls.json", "w") as of:
    json.dump(counts, of, indent=2)

Задачка 3: топ 10 урлов

Задание Найти 10 самых посещаемых урлов и число посещений для каждого из них

Заготовка кода

import re
from collections import defaultdict

regex = re.compile('([\w\-\[email protected]\'*,:/#&]+) - - \[(.*?)\] "([A-Z]+)\s+(\S+).*" (\d+) (-|\d+)')
def parse_line_re(line):
    try:
        return regex.match(line).groups()
    except AttributeError:
        return None

# нас интересуют урлы без параметров, параметры могут быть и через хеш, и через вопрос
# отделим их с помощью регэкспа
delim = re.compile("[#?]")

def count_urls():
    counts = defaultdict(int)
    with open("./access_log_Jul95", encoding="ascii", errors="replace") as datafile:
        for line in datafile:
            fields = parse_line_re(line)
            if fields:
                uri = delim.split(fields[3])[0]
                counts[uri] += 1
    return counts

def top_ten():
    # ваш код здесь
    return []
In [41]:
#andrey-yantsen
import json
urls = json.loads(open('urls.json').read())
list(sorted(urls.items(), key=lambda x: x[1], reverse=True))[:10]
Out[41]:
[('/images/NASA-logosmall.gif', 111388),
 ('/images/KSC-logosmall.gif', 89639),
 ('/images/MOSAIC-logosmall.gif', 60468),
 ('/images/USA-logosmall.gif', 60014),
 ('/images/WORLD-logosmall.gif', 59489),
 ('/images/ksclogo-medium.gif', 58802),
 ('/images/launch-logo.gif', 40871),
 ('/shuttle/countdown/', 40279),
 ('/ksc.html', 40232),
 ('/images/ksclogosmall.gif', 33585)]



































































In [42]:
# не оптимально, но нас устроит
list(reversed(sorted(counts.items(), key=lambda x: x[1])))[:10]
Out[42]:
[('/images/NASA-logosmall.gif', 111388),
 ('/images/KSC-logosmall.gif', 89639),
 ('/images/MOSAIC-logosmall.gif', 60468),
 ('/images/USA-logosmall.gif', 60014),
 ('/images/WORLD-logosmall.gif', 59489),
 ('/images/ksclogo-medium.gif', 58802),
 ('/images/launch-logo.gif', 40871),
 ('/shuttle/countdown/', 40279),
 ('/ksc.html', 40232),
 ('/images/ksclogosmall.gif', 33585)]

Подключаем Pandas

In [43]:
import numpy as np    # numpy
import pandas as pd   # pandas

# альтернативная библиотека для визуализации
# https://github.com/mwaskom/seaborn
import seaborn as sns

# подключаем графику прямо в браузер
%matplotlib inline
/usr/local/lib/python3.5/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
  warnings.warn(self.msg_depr % (key, alt_key))
In [44]:
# основное понятие в Pandas -- DataFrame
df = pd.DataFrame([
        ["Stuart Bloom", False, 50],
        ["Sheldon Cooper", True, 2000],
        ["Amy Farrah Fowler", False, 1200],
        ["Radjesh Koothrappali", False, 1000],
        ["Howard Wolowitz", False, 1500],
        ["Bernadette Rostenkowski", True, 2100],
        ["Penny", True, 5000],
        ["Leonard Hofstadter", True, 2500],
    ], columns="name pythonista salary".split())

df
Out[44]:
name pythonista salary
0 Stuart Bloom False 50
1 Sheldon Cooper True 2000
2 Amy Farrah Fowler False 1200
3 Radjesh Koothrappali False 1000
4 Howard Wolowitz False 1500
5 Bernadette Rostenkowski True 2100
6 Penny True 5000
7 Leonard Hofstadter True 2500
In [45]:
# селектим только питонистов
df[df['pythonista']]
Out[45]:
name pythonista salary
1 Sheldon Cooper True 2000
5 Bernadette Rostenkowski True 2100
6 Penny True 5000
7 Leonard Hofstadter True 2500
In [46]:
# как это работает? Индексация по массиву значений.
df['pythonista']
Out[46]:
0    False
1     True
2    False
3    False
4    False
5     True
6     True
7     True
Name: pythonista, dtype: bool
In [47]:
df[[False, False, True, False, False, False, False, False]]
Out[47]:
name pythonista salary
2 Amy Farrah Fowler False 1200
In [48]:
# выберем только интересные нам колонки
df[['pythonista', 'salary']]
Out[48]:
pythonista salary
0 False 50
1 True 2000
2 False 1200
3 False 1000
4 False 1500
5 True 2100
6 True 5000
7 True 2500
In [49]:
print(type(df['pythonista']))
df['pythonista']
<class 'pandas.core.series.Series'>
Out[49]:
0    False
1     True
2    False
3    False
4    False
5     True
6     True
7     True
Name: pythonista, dtype: bool
In [51]:
# можно и так, но надо быть аккуратнее. Если колонка называется, например, size,
# то не сработает -- потому что есть функция с таким же названием
print(type(df.pythonista))
df.pythonista
<class 'pandas.core.series.Series'>
Out[51]:
0    False
1     True
2    False
3    False
4    False
5     True
6     True
7     True
Name: pythonista, dtype: bool
In [50]:
# выбираем одну колонку как датафрейм
print(type(df[['pythonista']]))
df[['pythonista']]
<class 'pandas.core.frame.DataFrame'>
Out[50]:
pythonista
0 False
1 True
2 False
3 False
4 False
5 True
6 True
7 True
In [52]:
# добавим новую колонку, приготовив ее из старой
df['first_name'] = df['name'].apply(lambda x: x.split()[0])
df
Out[52]:
name pythonista salary first_name
0 Stuart Bloom False 50 Stuart
1 Sheldon Cooper True 2000 Sheldon
2 Amy Farrah Fowler False 1200 Amy
3 Radjesh Koothrappali False 1000 Radjesh
4 Howard Wolowitz False 1500 Howard
5 Bernadette Rostenkowski True 2100 Bernadette
6 Penny True 5000 Penny
7 Leonard Hofstadter True 2500 Leonard

Задачка 4: Добавить фамилию

Задание Добавить колонку с фамилией. Если фамилия двойная, в колонке должны быть обе ее части. Если фамилии нет, то в колонке должно быть None

Заготовка кода

df = pd.DataFrame([
        ["Stuart Bloom", False, 50],
        ["Sheldon Cooper", True, 2000],
        ["Amy Farrah Fowler", False, 1200],
        ["Radjesh Koothrappali", False, 1000],
        ["Howard Wolowitz", False, 1500],
        ["Bernadette Rostenkowski", True, 2100],
        ["Penny", True, 5000],
        ["Leonard Hofstadter", True, 2500],
    ], columns="name pythonista salary".split())

# тут ваш код
In [54]:
# andrey-yantsen
df['name'].apply(lambda x: '' if ' ' not in x else x.split(' ')[1])
Out[54]:
0           Bloom
1          Cooper
2          Farrah
3    Koothrappali
4        Wolowitz
5    Rostenkowski
6                
7      Hofstadter
Name: name, dtype: object
In [55]:
def ln(i):                            
    names = i.split()
    try:
        return names[1]
    except IndexError:
        return names[0]
df['name'].apply(ln)
Out[55]:
0           Bloom
1          Cooper
2          Farrah
3    Koothrappali
4        Wolowitz
5    Rostenkowski
6           Penny
7      Hofstadter
Name: name, dtype: object
In [56]:
#khudyakovavi
df['name'].apply(lambda x: None if ' ' not in x else ' '.join(x.split()[1:]))
Out[56]:
0            Bloom
1           Cooper
2    Farrah Fowler
3     Koothrappali
4         Wolowitz
5     Rostenkowski
6             None
7       Hofstadter
Name: name, dtype: object



































































In [57]:
def split_surname(name):
    parts = name.split(" ", 1)
    if len(parts) > 1:
        return parts[1]
    else:
        return None

df['surname'] = df['name'].apply(split_surname)
df
Out[57]:
name pythonista salary first_name surname
0 Stuart Bloom False 50 Stuart Bloom
1 Sheldon Cooper True 2000 Sheldon Cooper
2 Amy Farrah Fowler False 1200 Amy Farrah Fowler
3 Radjesh Koothrappali False 1000 Radjesh Koothrappali
4 Howard Wolowitz False 1500 Howard Wolowitz
5 Bernadette Rostenkowski True 2100 Bernadette Rostenkowski
6 Penny True 5000 Penny None
7 Leonard Hofstadter True 2500 Leonard Hofstadter
In [60]:
# средняя зарплата питонистов и не питонистов
df.groupby('pythonista').mean()
Out[60]:
salary
pythonista
False 937.5
True 2900.0
In [61]:
# Стюарт и Пенни портят всю картину =) Попробуем медиану
df.groupby('pythonista').median()
Out[61]:
salary
pythonista
False 1100
True 2300
In [62]:
# не доверяйте средним значениям за весь тест. Вы можете упустить выбросы, например, пики из за GC.
sns.plt.scatter(df['pythonista'], df['salary'])
Out[62]:
<matplotlib.collections.PathCollection at 0x10db0d860>

Загрузим лог в Pandas

вспомогательные материалы: Генераторы в python

код:

regex = re.compile('([\w\-\[email protected]\'*,:/#&]+) - - \[(.*?)\] "([A-Z]+)\s+(\S+).*" (\d+) (-|\d+)')
delim = re.compile("[#?]")

def parse_log_re(file):
    for line in file:
        try:
            # заодно отделим параметры от урла
            fields = regex.match(line).groups()
            uri_parts = delim.split(fields[3], 1)
            uri = uri_parts[0]
            if len(uri_parts) > 1:
                params = uri_parts[1]
            else:
                params = None
            yield list(fields) + [uri, params]
        except AttributeError:
            pass

columns = "host ts method uri code size url params".split()
with open("./access_log_Jul95", encoding="ascii", errors="replace") as datafile:
    df = pd.DataFrame.from_records(parse_log_re(datafile), columns=columns)
In [63]:
regex = re.compile('([\w\-\[email protected]\'*,:/#&]+) - - \[(.*?)\] "([A-Z]+)\s+(\S+).*" (\d+) (-|\d+)')
delim = re.compile("[#?]")

def parse_log_re(file):
    for line in file:
        try:
            # заодно отделим параметры от урла
            fields = regex.match(line).groups()
            uri_parts = delim.split(fields[3], 1)
            uri = uri_parts[0]
            if len(uri_parts) > 1:
                params = uri_parts[1]
            else:
                params = None
            yield list(fields) + [uri, params]
        except AttributeError:
            pass

columns = "host ts method uri code size url params".split()
with open("./access_log_Jul95", encoding="ascii", errors="replace") as datafile:
    df = pd.DataFrame.from_records(parse_log_re(datafile), columns=columns)
In [64]:
df.head()
Out[64]:
host ts method uri code size url params
0 199.72.81.55 01/Jul/1995:00:00:01 -0400 GET /history/apollo/ 200 6245 /history/apollo/ None
1 unicomp6.unicomp.net 01/Jul/1995:00:00:06 -0400 GET /shuttle/countdown/ 200 3985 /shuttle/countdown/ None
2 199.120.110.21 01/Jul/1995:00:00:09 -0400 GET /shuttle/missions/sts-73/mission-sts-73.html 200 4085 /shuttle/missions/sts-73/mission-sts-73.html None
3 burger.letters.com 01/Jul/1995:00:00:11 -0400 GET /shuttle/countdown/liftoff.html 304 0 /shuttle/countdown/liftoff.html None
4 199.120.110.21 01/Jul/1995:00:00:11 -0400 GET /shuttle/missions/sts-73/sts-73-patch-small.gif 200 4179 /shuttle/missions/sts-73/sts-73-patch-small.gif None
In [65]:
df.dtypes
Out[65]:
host      object
ts        object
method    object
uri       object
code      object
size      object
url       object
params    object
dtype: object
In [67]:
df["size"] = pd.to_numeric(df["size"], errors='coerce')
df["ts"] = pd.to_datetime(df["ts"], format="%d/%b/%Y:%H:%M:%S -0400")
In [68]:
df.dtypes
Out[68]:
host              object
ts        datetime64[ns]
method            object
uri               object
code              object
size             float64
url               object
params            object
dtype: object
In [69]:
df.head()
Out[69]:
host ts method uri code size url params
0 199.72.81.55 1995-07-01 00:00:01 GET /history/apollo/ 200 6245.0 /history/apollo/ None
1 unicomp6.unicomp.net 1995-07-01 00:00:06 GET /shuttle/countdown/ 200 3985.0 /shuttle/countdown/ None
2 199.120.110.21 1995-07-01 00:00:09 GET /shuttle/missions/sts-73/mission-sts-73.html 200 4085.0 /shuttle/missions/sts-73/mission-sts-73.html None
3 burger.letters.com 1995-07-01 00:00:11 GET /shuttle/countdown/liftoff.html 304 0.0 /shuttle/countdown/liftoff.html None
4 199.120.110.21 1995-07-01 00:00:11 GET /shuttle/missions/sts-73/sts-73-patch-small.gif 200 4179.0 /shuttle/missions/sts-73/sts-73-patch-small.gif None
In [70]:
# булевы операции в numpy
a = np.array([True, True, False, False])
b = np.array([True, False, True, False])
print("~a:", ~a)
print("a & b:", a & b)
print("a | b:", a | b)
~a: [False False  True  True]
a & b: [ True False False False]
a | b: [ True  True  True False]
In [71]:
# посмотрим, какие бывают параметры в урлах
df[~df["params"].isnull()]
Out[71]:
host ts method uri code size url params
47 205.189.154.54 1995-07-01 00:01:06 GET /cgi-bin/imagemap/countdown?99,176 302 110.0 /cgi-bin/imagemap/countdown 99,176
89 remote27.compusmart.ab.ca 1995-07-01 00:01:53 GET /cgi-bin/imagemap/countdown?102,174 302 110.0 /cgi-bin/imagemap/countdown 102,174
137 onyx.southwind.net 1995-07-01 00:02:27 GET /cgi-bin/imagemap/countdown?103,146 302 96.0 /cgi-bin/imagemap/countdown 103,146
145 gater3.sematech.org 1995-07-01 00:02:41 GET /cgi-bin/imagemap/countdown?99,173 302 110.0 /cgi-bin/imagemap/countdown 99,173
161 onyx.southwind.net 1995-07-01 00:03:00 GET /cgi-bin/imagemap/countdown?102,210 302 95.0 /cgi-bin/imagemap/countdown 102,210
178 lmsmith.tezcat.com 1995-07-01 00:03:13 GET /cgi-bin/imagemap/countdown?103,175 302 110.0 /cgi-bin/imagemap/countdown 103,175
190 www-a1.proxy.aol.com 1995-07-01 00:03:20 GET /cgi-bin/imagemap/countdown?107,144 302 96.0 /cgi-bin/imagemap/countdown 107,144
211 205.189.154.54 1995-07-01 00:03:45 GET /cgi-bin/imagemap/countdown?108,176 302 110.0 /cgi-bin/imagemap/countdown 108,176
231 www-a1.proxy.aol.com 1995-07-01 00:04:01 GET /cgi-bin/imagemap/countdown?99,113 302 111.0 /cgi-bin/imagemap/countdown 99,113
270 gater3.sematech.org 1995-07-01 00:04:31 GET /cgi-bin/imagemap/countdown?370,274 302 68.0 /cgi-bin/imagemap/countdown 370,274
294 pme607.onramp.awinc.com 1995-07-01 00:04:54 GET /cgi-bin/imagemap/countdown?99,178 302 110.0 /cgi-bin/imagemap/countdown 99,178
302 ix-sd11-26.ix.netcom.com 1995-07-01 00:05:06 GET /cgi-bin/imagemap/countdown?107,144 302 96.0 /cgi-bin/imagemap/countdown 107,144
317 199.166.39.14 1995-07-01 00:05:21 GET /cgi-bin/imagemap/countdown?302,275 302 98.0 /cgi-bin/imagemap/countdown 302,275
376 gater4.sematech.org 1995-07-01 00:06:26 GET /cgi-bin/imagemap/countdown?88,208 302 95.0 /cgi-bin/imagemap/countdown 88,208
409 p06.eznets.canton.oh.us 1995-07-01 00:07:02 GET /cgi-bin/imagemap/countdown?98,110 302 111.0 /cgi-bin/imagemap/countdown 98,110
446 teleman.pr.mcs.net 1995-07-01 00:07:35 GET /cgi-bin/imagemap/countdown?102,172 302 110.0 /cgi-bin/imagemap/countdown 102,172
490 remote27.compusmart.ab.ca 1995-07-01 00:08:22 GET /cgi-bin/imagemap/countdown?382,274 302 68.0 /cgi-bin/imagemap/countdown 382,274
504 ix-sd11-26.ix.netcom.com 1995-07-01 00:08:32 GET /cgi-bin/imagemap/countdown?94,175 302 110.0 /cgi-bin/imagemap/countdown 94,175
512 ppp4.sunrem.com 1995-07-01 00:08:45 GET /cgi-bin/imagemap/countdown?105,143 302 96.0 /cgi-bin/imagemap/countdown 105,143
520 www-a1.proxy.aol.com 1995-07-01 00:08:56 GET /cgi-bin/imagemap/countdown?99,213 302 95.0 /cgi-bin/imagemap/countdown 99,213
537 ppp-mia-53.shadow.net 1995-07-01 00:09:18 GET /cgi-bin/imagemap/countdown?99,150 302 96.0 /cgi-bin/imagemap/countdown 99,150
542 199.120.91.6 1995-07-01 00:09:25 GET /cgi-bin/imagemap/countdown?373,278 302 68.0 /cgi-bin/imagemap/countdown 373,278
558 net-1-141.eden.com 1995-07-01 00:09:47 GET /cgi-bin/imagemap/countdown?376,273 302 68.0 /cgi-bin/imagemap/countdown 376,273
562 p06.eznets.canton.oh.us 1995-07-01 00:09:53 GET /cgi-bin/imagemap/countdown?94,145 302 96.0 /cgi-bin/imagemap/countdown 94,145
594 ppp4.sunrem.com 1995-07-01 00:10:33 GET /cgi-bin/imagemap/countdown?93,176 302 110.0 /cgi-bin/imagemap/countdown 93,176
600 ppp-mia-53.shadow.net 1995-07-01 00:10:38 GET /cgi-bin/imagemap/countdown?103,180 302 110.0 /cgi-bin/imagemap/countdown 103,180
604 p06.eznets.canton.oh.us 1995-07-01 00:10:43 GET /cgi-bin/imagemap/countdown?382,271 302 68.0 /cgi-bin/imagemap/countdown 382,271
642 teleman.pr.mcs.net 1995-07-01 00:11:26 GET /cgi-bin/imagemap/countdown?107,147 302 96.0 /cgi-bin/imagemap/countdown 107,147
646 205.157.131.144 1995-07-01 00:11:27 GET /cgi-bin/imagemap/countdown?92,174 302 110.0 /cgi-bin/imagemap/countdown 92,174
670 ix-dfw13-20.ix.netcom.com 1995-07-01 00:11:44 GET /cgi-bin/imagemap/countdown?95,142 302 96.0 /cgi-bin/imagemap/countdown 95,142
... ... ... ... ... ... ... ... ...
1889169 130.184.141.116 1995-07-28 12:48:35 GET /cgi-bin/imagemap/countdown70?57,182 302 96.0 /cgi-bin/imagemap/countdown70 57,182
1889188 198.213.130.253 1995-07-28 12:48:48 GET /cgi-bin/imagemap/countdown70?74,68 302 72.0 /cgi-bin/imagemap/countdown70 74,68
1889234 internet.datalytics.com 1995-07-28 12:49:37 GET /htbin/wais.pl?51-L 200 316.0 /htbin/wais.pl 51-L
1889312 198.60.146.157 1995-07-28 12:51:13 GET /cgi-bin/imagemap/countdown70?63,154 302 111.0 /cgi-bin/imagemap/countdown70 63,154
1889368 205.168.103.18 1995-07-28 12:52:30 GET /cgi-bin/imagemap/countdown70?295,292 302 85.0 /cgi-bin/imagemap/countdown70 295,292
1889422 205.168.103.18 1995-07-28 12:53:31 GET /cgi-bin/imagemap/countdown70?264,144 302 97.0 /cgi-bin/imagemap/countdown70 264,144
1889440 205.168.103.18 1995-07-28 12:53:47 GET /cgi-bin/imagemap/fr?176,247 302 83.0 /cgi-bin/imagemap/fr 176,247
1889638 pcuvm2.nrl.navy.mil 1995-07-28 12:57:32 GET /htbin/wais.pl?MAHRSI 200 4591.0 /htbin/wais.pl MAHRSI
1889668 fester.sfasu.edu 1995-07-28 12:57:57 GET /cgi-bin/imagemap/countdown70?65,218 302 110.0 /cgi-bin/imagemap/countdown70 65,218
1889802 165.2.141.5 1995-07-28 12:59:55 GET /cgi-bin/imagemap/countdown70?344,288 302 98.0 /cgi-bin/imagemap/countdown70 344,288
1889811 165.2.141.5 1995-07-28 13:00:18 GET /cgi-bin/imagemap/countdown70?55,69 302 72.0 /cgi-bin/imagemap/countdown70 55,69
1889846 165.2.141.5 1995-07-28 13:01:06 GET /cgi-bin/imagemap/countdown70?64,155 302 111.0 /cgi-bin/imagemap/countdown70 64,155
1890092 128.95.78.19 1995-07-28 13:04:51 GET /cgi-bin/imagemap/countdown70?351,284 302 98.0 /cgi-bin/imagemap/countdown70 351,284
1890123 gateway.sgs-thomson.it 1995-07-28 13:05:18 GET /cgi-bin/imagemap/countdown70?400,291 302 68.0 /cgi-bin/imagemap/countdown70 400,291
1890135 gateway.sgs-thomson.it 1995-07-28 13:05:29 GET /cgi-bin/imagemap/countdown70?344,285 302 98.0 /cgi-bin/imagemap/countdown70 344,285
1890418 athena.capital.edu 1995-07-28 13:10:58 GET /htbin/wais.pl?51-L 200 316.0 /htbin/wais.pl 51-L
1890632 129.79.25.97 1995-07-28 13:15:00 GET /htbin/wais.pl?OARE 200 6079.0 /htbin/wais.pl OARE
1890688 dd06-057.compuserve.com 1995-07-28 13:15:51 GET /htbin/wais.pl?IMAX 200 6319.0 /htbin/wais.pl IMAX
1890752 bore.fysik.lu.se 1995-07-28 13:16:47 GET /htbin/wais.pl?SAREX-II 200 7111.0 /htbin/wais.pl SAREX-II
1890765 www-d3.proxy.aol.com 1995-07-28 13:16:53 GET /cgi-bin/imagemap/astrohome?272,285 302 93.0 /cgi-bin/imagemap/astrohome 272,285
1890784 www-d3.proxy.aol.com 1995-07-28 13:17:17 GET /cgi-bin/imagemap/onboard?371,237 302 109.0 /cgi-bin/imagemap/onboard 371,237
1890954 whelanr.east-tenn-st.edu 1995-07-28 13:19:52 GET /cgi-bin/imagemap/countdown70?184,287 302 110.0 /cgi-bin/imagemap/countdown70 184,287
1891074 delarondem.bed.ns.doe.ca 1995-07-28 13:22:12 GET /cgi-bin/imagemap/onboard?103,226 302 91.0 /cgi-bin/imagemap/onboard 103,226
1891180 192.112.239.11 1995-07-28 13:24:42 GET /cgi-bin/imagemap/countdown70?397,276 302 68.0 /cgi-bin/imagemap/countdown70 397,276
1891230 tiger2.ocs.lsu.edu 1995-07-28 13:25:43 GET /cgi-bin/imagemap/countdown70?71,194 302 96.0 /cgi-bin/imagemap/countdown70 71,194
1891247 eng-055.afit.af.mil 1995-07-28 13:26:08 GET /htbin/wais.pl?winvn 200 6212.0 /htbin/wais.pl winvn
1891325 piweba4y.prodigy.com 1995-07-28 13:27:38 GET /cgi-bin/imagemap/countdown70?57,183 302 96.0 /cgi-bin/imagemap/countdown70 57,183
1891360 ppp1-115.ganet.net 1995-07-28 13:28:16 GET /htbin/wais.pl?camp 200 5660.0 /htbin/wais.pl camp
1891468 ntigate.nt.com 1995-07-28 13:29:37 GET /cgi-bin/imagemap/countdown70?11,351 302 102.0 /cgi-bin/imagemap/countdown70 11,351
1891491 ntigate.nt.com 1995-07-28 13:29:53 GET /cgi-bin/imagemap/countdown70?277,289 302 85.0 /cgi-bin/imagemap/countdown70 277,289

46292 rows × 8 columns

In [75]:
# построим распределение размеров ответов

# создаем оси, чтобы можно было задать размер
fig, ax = sns.plt.subplots()
# и задаем размер
fig.set_size_inches(16,10)
# в параметрах указываем оси
df[["size"]].hist(bins=20, ax=ax)
Out[75]:
array([<matplotlib.axes._subplots.AxesSubplot object at 0x15d783710>], dtype=object)
In [76]:
# логарифмический масштаб по оси y
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
df[["size"]].hist(bins=20, ax=ax)

# задаем логарифмический размер
ax.set_yscale("log")

Минизадачка 5: Посмотреть, что за пик

Задание Посмотреть, какие запросы относятся к пику, который мы видим на графике.



































































In [80]:
#butorov
df[df['size'] > 2000000]
Out[80]:
host ts method uri code size url params
162842 ottgate2.bnr.ca 1995-07-03 12:30:07 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
207794 www-b6.proxy.aol.com 1995-07-03 22:02:47 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
265061 disarray.demon.co.uk 1995-07-04 17:20:20 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
301035 poppy.hensa.ac.uk 1995-07-05 07:54:02 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
305026 n164576.ksc.nasa.gov 1995-07-05 08:57:07 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
356013 gateway.cary.ibm.com 1995-07-05 17:21:54 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
406098 163.205.156.16 1995-07-06 10:19:00 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
417781 e659229.boeing.com 1995-07-06 12:11:57 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
477557 ttacs3.ttu.edu 1995-07-06 23:15:56 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
508838 spider.tbe.com 1995-07-07 10:28:56 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
513402 15.254.200.50 1995-07-07 11:16:52 GET /shuttle/countdown/video/livevideo.jpeg 200 2572288.0 /shuttle/countdown/video/livevideo.jpeg None
514706 jcdbs.itsi.disa.mil 1995-07-07 11:29:24 GET /shuttle/countdown/video/livevideo.jpeg 200 2048000.0 /shuttle/countdown/video/livevideo.jpeg None
531316 derec 1995-07-07 14:03:32 GET /shuttle/countdown/video/livevideo.jpeg 200 6823936.0 /shuttle/countdown/video/livevideo.jpeg None
603425 www-b6.proxy.aol.com 1995-07-08 22:24:24 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
603788 www-b6.proxy.aol.com 1995-07-08 22:33:46 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
615109 netspace.net.au 1995-07-09 09:18:44 GET /statistics/1995/Mar/Mar95_reverse_domains.html 200 2889868.0 /statistics/1995/Mar/Mar95_reverse_domains.html None
615164 netspace.net.au 1995-07-09 09:22:14 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
773778 grumpy.ksc.nasa.gov 1995-07-11 17:28:21 GET /mdss/ped/acs/SDP.ps 200 2854989.0 /mdss/ped/acs/SDP.ps None
773872 grumpy.ksc.nasa.gov 1995-07-11 17:29:34 GET /mdss/ped/acs/SDP.ps 200 2854989.0 /mdss/ped/acs/SDP.ps None
981718 acf4.nyu.edu 1995-07-13 14:38:43 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1044273 163.206.89.4 1995-07-14 09:11:29 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1233598 jcj8363.dukepower.com 1995-07-17 13:31:01 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1305650 titan02f 1995-07-18 14:04:54 GET /statistics/1995/Jul/Jul95_reverse_domains.html 200 2661134.0 /statistics/1995/Jul/Jul95_reverse_domains.html None
1305663 titan02f 1995-07-18 14:05:07 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
1559028 wuzza.harvard.edu 1995-07-22 13:08:18 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1617790 chi017.wwa.com 1995-07-24 00:19:37 GET /statistics/1995/bkup/Mar95_full.html 200 2785280.0 /statistics/1995/bkup/Mar95_full.html None
1731514 198.122.135.40 1995-07-25 17:58:56 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1877083 128.206.221.93 1995-07-28 09:13:09 GET /statistics/1995/bkup/Mar95_full.html 200 2957312.0 /statistics/1995/bkup/Mar95_full.html None
In [77]:
df[df["size"] > 2500000]
Out[77]:
host ts method uri code size url params
162842 ottgate2.bnr.ca 1995-07-03 12:30:07 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
207794 www-b6.proxy.aol.com 1995-07-03 22:02:47 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
265061 disarray.demon.co.uk 1995-07-04 17:20:20 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
301035 poppy.hensa.ac.uk 1995-07-05 07:54:02 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
305026 n164576.ksc.nasa.gov 1995-07-05 08:57:07 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
356013 gateway.cary.ibm.com 1995-07-05 17:21:54 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
406098 163.205.156.16 1995-07-06 10:19:00 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
417781 e659229.boeing.com 1995-07-06 12:11:57 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
477557 ttacs3.ttu.edu 1995-07-06 23:15:56 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
508838 spider.tbe.com 1995-07-07 10:28:56 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
513402 15.254.200.50 1995-07-07 11:16:52 GET /shuttle/countdown/video/livevideo.jpeg 200 2572288.0 /shuttle/countdown/video/livevideo.jpeg None
531316 derec 1995-07-07 14:03:32 GET /shuttle/countdown/video/livevideo.jpeg 200 6823936.0 /shuttle/countdown/video/livevideo.jpeg None
603425 www-b6.proxy.aol.com 1995-07-08 22:24:24 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
603788 www-b6.proxy.aol.com 1995-07-08 22:33:46 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
615109 netspace.net.au 1995-07-09 09:18:44 GET /statistics/1995/Mar/Mar95_reverse_domains.html 200 2889868.0 /statistics/1995/Mar/Mar95_reverse_domains.html None
615164 netspace.net.au 1995-07-09 09:22:14 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
773778 grumpy.ksc.nasa.gov 1995-07-11 17:28:21 GET /mdss/ped/acs/SDP.ps 200 2854989.0 /mdss/ped/acs/SDP.ps None
773872 grumpy.ksc.nasa.gov 1995-07-11 17:29:34 GET /mdss/ped/acs/SDP.ps 200 2854989.0 /mdss/ped/acs/SDP.ps None
981718 acf4.nyu.edu 1995-07-13 14:38:43 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1044273 163.206.89.4 1995-07-14 09:11:29 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1233598 jcj8363.dukepower.com 1995-07-17 13:31:01 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1305650 titan02f 1995-07-18 14:04:54 GET /statistics/1995/Jul/Jul95_reverse_domains.html 200 2661134.0 /statistics/1995/Jul/Jul95_reverse_domains.html None
1305663 titan02f 1995-07-18 14:05:07 GET /statistics/1995/Jun/Jun95_reverse_domains.html 200 2973350.0 /statistics/1995/Jun/Jun95_reverse_domains.html None
1559028 wuzza.harvard.edu 1995-07-22 13:08:18 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1617790 chi017.wwa.com 1995-07-24 00:19:37 GET /statistics/1995/bkup/Mar95_full.html 200 2785280.0 /statistics/1995/bkup/Mar95_full.html None
1731514 198.122.135.40 1995-07-25 17:58:56 GET /statistics/1995/bkup/Mar95_full.html 200 3155499.0 /statistics/1995/bkup/Mar95_full.html None
1877083 128.206.221.93 1995-07-28 09:13:09 GET /statistics/1995/bkup/Mar95_full.html 200 2957312.0 /statistics/1995/bkup/Mar95_full.html None
In [79]:
# чтобы сделать логарифмический масштаб по оси x, нам нужны логарифмические бины
# сгенерим их с помощью numpy.logspace
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)

# обратите внимание на параметр bins
df[["size"]].hist(bins=np.logspace(0.1, 10.0, 20), ax=ax)
#df[["size"]].hist(bins=20, ax=ax)


# логарифмический масштаб оси по x
ax.set_xscale("log")

Задачка 6: считаем RPS на сервере

Задание: найти число запросов в секунду для каждой секунды, когда они были

In [81]:
# khudyakovavi
df.groupby('ts').size()
Out[81]:
ts
1995-07-01 00:00:01    1
1995-07-01 00:00:06    1
1995-07-01 00:00:09    1
1995-07-01 00:00:11    2
1995-07-01 00:00:12    3
1995-07-01 00:00:13    2
1995-07-01 00:00:14    3
1995-07-01 00:00:15    3
1995-07-01 00:00:17    2
1995-07-01 00:00:18    1
1995-07-01 00:00:19    2
1995-07-01 00:00:24    1
1995-07-01 00:00:25    1
1995-07-01 00:00:27    1
1995-07-01 00:00:29    1
1995-07-01 00:00:33    1
1995-07-01 00:00:35    1
1995-07-01 00:00:37    1
1995-07-01 00:00:38    1
1995-07-01 00:00:40    1
1995-07-01 00:00:41    5
1995-07-01 00:00:43    1
1995-07-01 00:00:44    1
1995-07-01 00:00:50    1
1995-07-01 00:00:54    1
1995-07-01 00:00:58    1
1995-07-01 00:00:59    2
1995-07-01 00:01:02    1
1995-07-01 00:01:04    3
1995-07-01 00:01:05    1
                      ..
1995-07-28 13:31:46    4
1995-07-28 13:31:47    3
1995-07-28 13:31:49    6
1995-07-28 13:31:50    5
1995-07-28 13:31:51    3
1995-07-28 13:31:52    2
1995-07-28 13:31:53    2
1995-07-28 13:31:54    1
1995-07-28 13:31:55    2
1995-07-28 13:31:57    2
1995-07-28 13:31:58    1
1995-07-28 13:31:59    2
1995-07-28 13:32:00    3
1995-07-28 13:32:01    1
1995-07-28 13:32:03    1
1995-07-28 13:32:04    3
1995-07-28 13:32:06    2
1995-07-28 13:32:11    4
1995-07-28 13:32:12    2
1995-07-28 13:32:13    1
1995-07-28 13:32:14    1
1995-07-28 13:32:16    3
1995-07-28 13:32:17    1
1995-07-28 13:32:18    2
1995-07-28 13:32:19    2
1995-07-28 13:32:20    4
1995-07-28 13:32:21    3
1995-07-28 13:32:22    5
1995-07-28 13:32:23    3
1995-07-28 13:32:25    1
dtype: int64



































































In [82]:
rps = df.groupby("ts")[["method"]].count()
rps.head()
Out[82]:
method
ts
1995-07-01 00:00:01 1
1995-07-01 00:00:06 1
1995-07-01 00:00:09 1
1995-07-01 00:00:11 2
1995-07-01 00:00:12 3
In [83]:
rps.max()
Out[83]:
method    20
dtype: int64
In [84]:
# построим график RPS
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
rps.plot(ax=ax)
Out[84]:
<matplotlib.axes._subplots.AxesSubplot at 0x16011c128>
In [85]:
# добавим скользящее среднее, чтобы сгладить пики
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
#rps.plot(ax=ax)

# rolling сделает нам группы, используя скользящее окно, по которым можно считать разное
roll = rps.rolling(window=600)
roll.mean().plot(ax=ax, c="red")
Out[85]:
<matplotlib.axes._subplots.AxesSubplot at 0x1600f0518>

Творческая задачка 7: исследовать пик

Задание: узнать, с чем связан пик на графике

In [88]:
#vanadium23
from datetime import datetime
df[df['ts'] > datetime(1995, 7, 13)][df['ts'] < datetime(1995, 7, 15)].groupby('uri').count()
/usr/local/lib/python3.5/site-packages/ipykernel/__main__.py:3: UserWarning: Boolean Series key will be reindexed to match DataFrame index.
  app.launch_new_instance()
Out[88]:
host ts method code size url params
uri
/ 3162 3162 3162 3162 3162 3162 0
/%3A//spacelink.msfc.nasa.gov 5 5 5 5 0 5 0
/%3A/spacelink.msfc.nasa.gov 16 16 16 16 0 16 0
/%7Emol 1 1 1 1 0 1 0
/.ksc.html 2 2 2 2 2 2 0
//facilities/spaceport.html 1 1 1 1 1 1 0
//history/history/history.html 1 1 1 1 0 1 0
//ksc.html 1 1 1 1 1 1 0
//shuttle/missions/missions.html 1 1 1 1 1 1 0
/://spacelink.msfc.nasa.gov 24 24 24 24 0 24 0
/:/spacelink.msfc.nasa.gov 1 1 1 1 0 1 0
/BROWSER 1 1 1 1 0 1 0
/GIF/ksc-logo-small.gif 1 1 1 1 0 1 0
/GIF/nasaball.gif 1 1 1 1 0 1 0
/GIF/thin-blue-marble-line.gif 1 1 1 1 0 1 0
/General/Internet/WWW/Grobe.html 1 1 1 1 0 1 0
/HISTORY/APOLLO/ 1 1 1 1 0 1 0
/HISTORY/APOllo/apollo-13/apollo-13.html 1 1 1 1 0 1 0
/HISTORY/APPOLLO/APPOLLO-13 1 1 1 1 0 1 0
/HTBIN/CDT_CLOCK.PL 1 1 1 1 0 1 0
/Images/blueball.gif 1 1 1 1 0 1 0
/KSC.html 1 1 1 1 0 1 0
/LDAR/LDARhp.html 2 2 2 2 0 2 0
/MISC 1 1 1 1 0 1 0
/NASA_home.html 1 1 1 1 0 1 0
/NIS 1 1 1 1 0 1 0
/Organizations/Grumman/Products/Space_Technologies/LEM.html 1 1 1 1 0 1 0
/Organizations/Grumman/Products/Space_Technologies/SpaceStation.html 1 1 1 1 0 1 0
/Pictures/MoonFlag.gif 1 1 1 1 0 1 0
/SDG/Experimental/demoweb/return.gif 1 1 1 1 0 1 0
... ... ... ... ... ... ... ...
/sts-70/icons/b-ascent.gif 1 1 1 1 0 1 0
/sts-70/images/ 1 1 1 1 0 1 0
/sts-70/landing/orbiter/ 1 1 1 1 0 1 0
/sts-70/landing/orbiter/*.html 1 1 1 1 0 1 0
/sts-70/ntv 1 1 1 1 0 1 0
/sts-70/orbit/ 1 1 1 1 0 1 0
/sts-70/orbiter/ 1 1 1 1 0 1 0
/sts-70/shutref/sts-subs.html 1 1 1 1 0 1 0
/sts-70/video 1 1 1 1 0 1 0
/sts-70/video/ 1 1 1 1 0 1 0
/sts-70/video/daily 1 1 1 1 0 1 0
/suttle/countdown/ 1 1 1 1 0 1 0
/welcome.html 18 18 18 18 18 18 0
/whats-new.html 210 210 210 210 210 210 0
/winvn 2 2 2 2 0 2 0
/www/ 1 1 1 1 0 1 0
/~adverts/apple/inet/banner.gif 1 1 1 1 0 1 0
/~adverts/graphics/adbanner.gif 1 1 1 1 0 1 0
/~adverts/graphics/indxlogo.gif 1 1 1 1 0 1 0
/~downs/home.html 5 5 5 5 5 5 0
/~downs/launchup.gif 5 5 5 5 5 5 0
/~pccomp/graphics/sabout.gif 1 1 1 1 0 1 0
/~pccomp/graphics/sfeat.gif 1 1 1 1 0 1 0
/~pccomp/graphics/sfree.gif 1 1 1 1 0 1 0
/~pccomp/graphics/sinsght.gif 1 1 1 1 0 1 0
/~pccomp/graphics/smail.gif 1 1 1 1 0 1 0
/~pccomp/graphics/sone.gif 1 1 1 1 0 1 0
/~pccomp/graphics/spatrol.gif 1 1 1 1 0 1 0
/~pccomp/graphics/swww.gif 1 1 1 1 0 1 0
www.ksc.nasa.gov 1 1 1 1 0 1 0

6192 rows × 7 columns



































































In [89]:
# для начала сравним на одном графике обычный и необычный дни

# возьмем время без даты
df['time'] = df["ts"].dt.time

# и отсеим нужные нам данные
event = df[
    (df['ts'] > pd.Timestamp("1995-07-13 00:00:00")) &
    (df['ts'] < pd.Timestamp("1995-07-14 00:00:00"))
]
no_event = df[
    (df['ts'] > pd.Timestamp("1995-07-14 00:00:00")) &
    (df['ts'] < pd.Timestamp("1995-07-15 00:00:00"))
]

# оси
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)

# скользящее среднее для дня с событием
event_rps = event.groupby("time")[["method"]].count()
event_roll = event_rps.rolling(window=600)
event_roll.mean().plot(ax=ax, c="red")

# скользящее среднее для дня без события
no_event_rps = no_event.groupby("time")[["method"]].count()
no_event_roll = no_event_rps.rolling(window=600)
no_event_roll.mean().plot(ax=ax, c="green")
Out[89]:
<matplotlib.axes._subplots.AxesSubplot at 0x15ffe52b0>
In [90]:
# урлы, на которые ходили в четверг, но не ходили в пятницу
set(event['url']) - set(no_event['url'])
Out[90]:
{'/BROWSER',
 '/General/Internet/WWW/Grobe.html',
 '/HISTORY/APOLLO/',
 '/HTBIN/CDT_CLOCK.PL',
 '/Images/blueball.gif',
 '/MISC',
 '/NASA_home.html',
 '/NIS',
 '/Organizations/Grumman/Products/Space_Technologies/LEM.html',
 '/Organizations/Grumman/Products/Space_Technologies/SpaceStation.html',
 '/Pictures/MoonFlag.gif',
 '/SHUTTLE/COUNTDOWN/',
 '/Shuttle',
 '/Shuttle/Countdown',
 '/astro-2.msfc.nasa.gov',
 '/base-ops/procurement/kscbus.htm',
 '/biomed/gif/aerpcfinmed.gif',
 '/biomed/intro.html',
 '/biomed/text/launch.doc',
 '/cgi-bin/imagemap/KSC-95EC-0823.jpg',
 '/cgi-bin/imagemap/count70.gif',
 '/cgi-bin/imagemap/lo-org',
 '/cgi-bin/imagemap/thunder.atms.purdue.edu',
 '/cgi-bin/imagemap/weatstat/weatinfo.htm',
 '/cgi-bin/imagemap/~tfish/oregon.map',
 '/countdown',
 '/countdown/',
 '/cwis/organizations/kucia/uroulette/jayhawk.gif',
 '/cwis/organizations/kucia/uroulette/press',
 '/cwis/organizations/kucia/uroulette/sig.gif',
 '/elv',
 '/elv/',
 '/elv/ATLAS_CENTAUR/goes_lau.gif',
 '/elv/SCOUT/elvhead2.gif',
 '/elv/SCOUT/sampex.jpg',
 '/elv/uplink.htm>This',
 '/facilities/cdsc.html',
 '/facilities/cif.html',
 '/facilities/phsf.html',
 '/facts/faq08.txt',
 '/facts/internet/DoesNotExist.gif',
 '/facts/internet/filename.GIF',
 '/facts/internet/html-primer.html',
 '/facts/internet/url-primer.html',
 '/facts/wais-readme.html',
 '/finance',
 '/finance/',
 '/finance/gvtlinks.htm',
 '/gov/shuttle/countdown',
 '/history.apollo/apollo-13-info.html.',
 '/history.html',
 '/history/.history.html',
 '/history/apollo-11/apollo-11.html',
 '/history/apollo-13/apollo-13.html',
 '/history/apollo/-apollo-13/apollo-13.html',
 '/history/apollo/a-001/images/',
 '/history/apollo/a-002/',
 '/history/apollo/a-002/a-002-info.html',
 '/history/apollo/a-002/a-002-patch-small.gif',
 '/history/apollo/a-002/a-002.html',
 '/history/apollo/a-002/docs/',
 '/history/apollo/a-002/images/',
 '/history/apollo/a-002/movies/',
 '/history/apollo/a-002/news/',
 '/history/apollo/a-003/a-003-info.html',
 '/history/apollo/a-003/a-003-patch-small.gif',
 '/history/apollo/a-003/a-003.html',
 '/history/apollo/a-004/docs/',
 '/history/apollo/a-004/images/',
 '/history/apollo/apollo-1',
 '/history/apollo/apollo-1.html',
 '/history/apollo/apollo-10/apollo-10-patch.jpg',
 '/history/apollo/apollo-11/apollo-10.html',
 '/history/apollo/apollo-12/apollo-12-patch.jpg',
 '/history/apollo/apollo-12/docs/',
 '/history/apollo/apollo-12/images/69HC1007.GIF',
 '/history/apollo/apollo-12/images/69HC1324.GIF',
 '/history/apollo/apollo-12/images/69HC1326.GIF',
 '/history/apollo/apollo-12/images/69HC1344.GIF',
 '/history/apollo/apollo-12/images/77HC85.GIF',
 '/history/apollo/apollo-12/news/',
 '/history/apollo/apollo-13',
 '/history/apollo/apollo-13-patch-small.gif',
 '/history/apollo/apollo-13/apoll',
 '/history/apollo/apollo-13/apollo-13-1nfo.html.',
 '/history/apollo/apollo-13/apollo-13-info.html.',
 '/history/apollo/apollo-13/apollo-13.htmp',
 '/history/apollo/apollo-13/apollo-13/html',
 '/history/apollo/apollo-13/apollo-13html',
 '/history/apollo/apollo-13/apollo-13info.htm',
 '/history/apollo/apollo-13/apollo-13info.html',
 '/history/apollo/apollo-13/apollo13-info.html',
 '/history/apollo/apollo-13/apollo13.html',
 '/history/apollo/apollo-14',
 '/history/apollo/apollo-14-patch-small.gif',
 '/history/apollo/apollo-15/',
 '/history/apollo/apollo-15/apollo-15-patch.jpg',
 '/history/apollo/apollo-15/videos/',
 '/history/apollo/apollo-16/',
 '/history/apollo/apollo-16/apollo-16-patch.jpg',
 '/history/apollo/apollo-16/docs/',
 '/history/apollo/apollo-16/news/',
 '/history/apollo/apollo-18/apollo-18.html',
 '/history/apollo/apollo-2/apollo-2.html',
 '/history/apollo/apollo-3/apollo-3.html',
 '/history/apollo/apollo-4/',
 '/history/apollo/apollo-6/',
 '/history/apollo/apollo-6/docs/',
 '/history/apollo/apollo-6/movies/',
 '/history/apollo/apollo-6/news/',
 '/history/apollo/apollo-6/sounds/',
 '/history/apollo/apollo-6/videos/',
 '/history/apollo/apollo-7/apollo7.html',
 '/history/apollo/apollo-8/apollo8.html',
 '/history/apollo/apollo-8/images/68HC678.GIF',
 '/history/apollo/apollo-8/images/68HC731.GIF',
 '/history/apollo/apollo-8/images/68HC870.GIF',
 '/history/apollo/apollo-8/images/69HC2.GIF',
 '/history/apollo/apollo-8/images/69HC21.GIF',
 '/history/apollo/apollo-9/docs/',
 '/history/apollo/apollo-9/images/68HC839.GIF',
 '/history/apollo/apollo-9/images/69HC110.GIF',
 '/history/apollo/apollo-overview.txt',
 '/history/apollo/apollo/-13/apollo-13html',
 '/history/apollo/apollo13',
 '/history/apollo/apolo-13/apollo-13.html',
 '/history/apollo/as-201/as-201-patch.jpg',
 '/history/apollo/as-201/images/as-201-launch-small.gif',
 '/history/apollo/images/670800.GIF',
 '/history/apollo/images/AT110.GIF',
 '/history/apollo/images/abort-test.jpeg',
 '/history/apollo/images/lem.gif',
 '/history/apollo/images/moonwalk.gif',
 '/history/apollo/pad-abort-test-2/images/',
 '/history/apollo/publications/',
 '/history/apollo/publications/sp-350/',
 '/history/apollo/sa-1/docs/',
 '/history/apollo/sa-1/images/',
 '/history/apollo/sa-1/news/',
 '/history/apollo/sa-2/images/',
 '/history/apollo/sa-2/sa-2-info.html',
 '/history/apollo/sa-2/sa-2-patch-small.gif',
 '/history/apollo/sa-3/images/',
 '/history/apollo/sa-3/sa-3-info.html',
 '/history/apollo/sa-3/sa-3-patch-small.gif',
 '/history/apollo/sa-4/images/',
 '/history/apollo/sa-4/sa-4-info.html',
 '/history/apollo/sa-4/sa-4-patch-small.gif',
 '/history/apollo/sa-5/images/',
 '/history/apollo/sa-5/sa-5-info.html',
 '/history/apollo/sa-5/sa-5-patch-small.gif',
 '/history/apollo/sa-6/images/',
 '/history/apollo/sa-7/',
 '/history/apollo/sa-7/images/',
 '/history/apollo/sa-7/sa-7-info.html',
 '/history/apollo/sa-7/sa-7-patch-small.gif',
 '/history/apollo/sa-8/images/',
 '/history/apollo/sa-8/sa-8-info.html',
 '/history/apollo/sa-9/images/',
 '/history/apollo/sa-9/sa-9-info.html',
 '/history/apollo/sa-9/sa-9-patch-small.gif',
 '/history/appolo/apollo-13.html',
 '/history/appolo/apollo-13/appolo-13.html',
 '/history/gemini/gemini-1/gemini-1-patch.jpg',
 '/history/gemini/gemini-x/gemini-x-info.html',
 '/history\\apollo',
 '/htbin/',
 '/htbin/cdt_clock/.pl',
 '/htbin/cdt_klock.pl',
 '/htbin/cdt_main.pl129.120.20.198',
 '/htbin/ctd_clock.pl',
 '/htbin/imagemap/Jun95stats_b',
 '/htbin/imagemap/cm-org',
 '/htbin/launch',
 '/htbin/odt_main.pl',
 '/htbin/radio',
 '/icon/blueball.gif',
 '/icon/redball.gif',
 '/icons/im',
 '/imagemap.cgi//sts-70/home.map',
 '/images/NASA-logosmall.if',
 '/images/cdrom-1-95/img0014.jpg',
 '/images/cm-map-1.jpg',
 '/images/cm-map-3.eps',
 '/images/cm-map.gif',
 '/images/cmmap-small.gif',
 '/images/cmmap1.gif',
 '/images/elv.jpg',
 '/images/imagemaps/stats.map',
 '/images/ius.gif',
 '/images/landingsmall.gif',
 '/images/lps.gif',
 '/images/p263_150.jpg',
 '/images/shuttle',
 '/images/stats.gif',
 '/kistory/apollo/apollo-13/apollo-13html',
 '/ksc',
 '/ksc.htnl',
 '/launch/countdown/',
 '/logistics',
 '/logistics/',
 '/mdss/ped/acs/mdlogo.gif',
 '/mdss/ped/acs/s_md-1.gif',
 '/mdss/srqa/../MDSS.html',
 '/mdss/srqa/qpa/qpa.html',
 '/mdss/srqa/qpa/s_qpa.gif',
 '/mdss/srqa/qpa/tools.html',
 '/microsoft.com/',
 '/missions/missions.html',
 '/msfc/astro_home.htm',
 '/msfc/crew/astro2patch.gif',
 '/msfc/crew/food.html',
 '/msfc/crew/meatball.gif',
 '/msfc/crew/oswald-small.gif',
 '/msfc/crew/oswald.html',
 '/msfc/crew/patches.html',
 '/msfc/crew/stspatch.gif',
 '/msfc/crew/training.html',
 '/msfc/crew/trivia.html',
 '/msfc/crew/wannabe.html',
 '/msfc/description/starview.gif',
 '/msfc/description/starview.html',
 '/msfc/team/msfc/stephanie-12.GIF',
 '/nasa.html',
 '/news/nasa.nasamail.p/332',
 '/news/nasa.nasamail.p/361',
 '/news/nasa.nasamail.p/449',
 '/news/nasa.nasamail.p/486',
 '/news/nasa.nasamail.p/499',
 '/news/nasa.nasamail.p/571',
 '/news/sci.space.news/1040',
 '/news/sci.space.news/1095',
 '/news/sci.space.news/1371',
 '/news/sci.space.news/1392',
 '/news/sci.space.news/1412',
 '/news/sci.space.news/1552',
 '/news/sci.space.news/1805',
 '/news/sci.space.news/1812',
 '/news/sci.space.news/1992',
 '/news/sci.space.news/2055',
 '/news/sci.space.news/2284',
 '/news/sci.space.news/2362',
 '/news/sci.space.news/2383',
 '/news/sci.space.news/2513',
 '/news/sci.space.news/2558',
 '/news/sci.space.news/2559',
 '/news/sci.space.news/2566',
 '/news/sci.space.news/415',
 '/news/sci.space.news/426',
 '/news/sci.space.news/754',
 '/news/sci.space.news/855',
 '/news/sci.space.news/878',
 '/news/sci.space.news/962',
 '/news/sci.space.news/965',
 '/news/sci.space.news/982',
 '/news/sci.space.news/archive/sci-space-news-1-feb-1995-80.txt',
 '/news/sci.space.news/archive/sci-space-news-12-apr-1994-01.txt',
 '/news/sci.space.news/archive/sci-space-news-12-apr-1995-61.txt',
 '/news/sci.space.news/archive/sci-space-news-15-mar-1994-35.txt',
 '/news/sci.space.news/archive/sci-space-news-19-mar-1995-25.txt',
 '/news/sci.space.news/archive/sci-space-news-19-mar-1995-80.txt',
 '/news/sci.space.news/archive/sci-space-news-21-feb-1994-92.txt',
 '/news/sci.space.news/archive/sci-space-news-25-jan-1995-17.txt',
 '/news/sci.space.news/archive/sci-space-news-25-jan-1995-33.txt',
 '/news/sci.space.news/archive/sci-space-news-25-jan-1995-45.txt',
 '/news/sci.space.news/archive/sci-space-news-25-jan-1995-83.txt',
 '/news/sci.space.news/archive/sci-space-news-25-jan-1995-86.txt',
 '/news/sci.space.news/archive/sci-space-news-25-mar-1995-30.txt',
 '/news/sci.space.news/archive/sci-space-news-26-feb-1994-07.txt',
 '/news/sci.space.news/archive/sci-space-news-29-mar-1994-28.txt',
 '/news/sci.space.news/archive/sci-space-news-29-mar-1994-88.txt',
 '/news/sci.space.shuttle/archive/sci-space-shuttle-22-apr-1995-40.txt',
 '/ntv',
 '/ntv/',
 '/pao/fact-sheets/countdown.pdf',
 '/payloads/documents/',
 '/payloads/documents/e1365.pdf',
 '/payloads/documents/e6058.pdf',
 '/payloads/documents/p5904.pdf',
 '/payloads/org/cm-org.gif',
 '/payloads/processing/ppart7.html',
 '/payloads/processing/ppart7as.gif',
 '/payloads/processing/ppart7bs.gif',
 '/payloads/processing/ppart7cs.gif',
 '/payloads/processing/ppart7s.gif',
 '/payloads/schedules/manifest/',
 '/payloads/schedules/mini/',
 '/payloads/schedules/mini/sts-71-MMS.pdf',
 '/payloads/schedules/mps/',
 '/payloads/schedules/pics/',
 '/payloads/schedules/pics/test/ci1_jd.eps',
 '/persons/astronauts',
 '/persons/astronauts/a-to-d/',
 '/persons/astronauts/a-to-d/AllenJP.txt',
 '/persons/astronauts/a-to-d/BakerES.txt',
 '/persons/astronauts/a-to-d/DurranceST.html',
 '/persons/astronauts/i-to-l/LoungeJM.txt',
 '/persons/astronauts/m-to-p/pogueWR.txt',
 '/persons/astronauts/q-to-t/ThorntonWE.txt',
 '/persons/astronauts/thagar.html',
 '/persons/astronauts/u-to-z/VangenSD.html',
 '/persons/astronauts/u-to-z/weitzPJ.txt',
 '/persons/nasa-cm/jmd.jpg',
 '/procurement/business/oportun.htm',
 '/procurement/clause/k236-115.htm',
 '/procurement/midrange/rfo/antenrfo.txt',
 '/pub/winvn/',
 '/schuttle/missions/missions.html',
 '/shuttle.countdown',
 '/shuttle.countdown/',
 '/shuttle/Countdown',
 '/shuttle/count70.gif',
 '/shuttle/countdow.n/video/livevideo.gif',
 '/shuttle/countdown/./video/livevideo.gif',
 '/shuttle/countdown/./video/livevideo.jpeg',
 '/shuttle/countdown/count2.html',
 '/shuttle/countdown/images/cdtclock-large.gif',
 '/shuttle/countdown/lps/images/AB-ROW-large.gif',
 '/shuttle/countdown/lps/images/C-2-large.gif',
 '/shuttle/countdown/lps/mpeg.html',
 '/shuttle/countdown/lps/sts-71-s-5-2l-c2-i4-12.mpg',
 '/shuttle/countdown/news',
 '/shuttle/countdown/video',
 '/shuttle/countdown/video/countclock.gif',
 '/shuttle/countdown/video/countclock.jpeg',
 '/shuttle/countdown/video/countclock2.gif',
 '/shuttle/countdown/video/discovery-pad-prelaunch-7-13-95.jpeg',
 '/shuttle/countdown/video/lifevideo.jpeg',
 '/shuttle/countdown/video/lifevideo.jpge',
 '/shuttle/countdown/video/livevideo2.jpeg',
 '/shuttle/countdown/video/tempvideo.gif',
 '/shuttle/countdown/video/xxx.gif',
 '/shuttle/countdown/video/xxx2.gif',
 '/shuttle/countdown70',
 '/shuttle/gifs',
 '/shuttle/images',
 '/shuttle/mission/41-d/mission-41-d.html',
 '/shuttle/mission/4l-d/mission-4l-d.html',
 '/shuttle/mission/facilities',
 '/shuttle/mission/facilities/',
 '/shuttle/mission/facilities/lc39a.html',
 '/shuttle/mission/mission.html',
 '/shuttle/mission/sts-70/mission-sts-70.html',
 '/shuttle/mission/sts-71/images/images.html',
 '/shuttle/missions.html/',
 '/shuttle/missions//',
 '/shuttle/missions/41-b/images/841100.GIF',
 '/shuttle/missions/41-c/images/',
 '/shuttle/missions/41-c/images/840407.GIF',
 '/shuttle/missions/41-c/images/84HC179.GIF',
 '/shuttle/missions/41-c/images/84HC99.GIF',
 '/shuttle/missions/51-L/mission-51-1.html',
 '/shuttle/missions/51-a/51-a-info.html',
 '/shuttle/missions/51-b/images/85HC160.GIF',
 '/shuttle/missions/51-c/images/84HC643.GIF',
 '/shuttle/missions/51-d/51-d-info.html',
 '/shuttle/missions/51-d/docs/',
 '/shuttle/missions/51-d/images/85HC106.GIF',
 '/shuttle/missions/51-d/movies/',
 '/shuttle/missions/51-d/news',
 '/shuttle/missions/51-d/news/',
 '/shuttle/missions/51-d/sounds/',
 '/shuttle/missions/51-g/51-g-info.html',
 '/shuttle/missions/51-g/docs/',
 '/shuttle/missions/51-g/news/',
 '/shuttle/missions/51-i/images/85HC316.GIF',
 '/shuttle/missions/51-i/images/85HC346.GIF',
 '/shuttle/missions/51-j/images/85HC372.GIF',
 '/shuttle/missions/html/',
 '/shuttle/missions/images',
 '/shuttle/missions/mission-sts-71.html',
 '/shuttle/missions/missions.htnl',
 '/shuttle/missions/news/',
 '/shuttle/missions/news/1990/h07.31.90',
 '/shuttle/missions/news/1991/h04.23.91',
 '/shuttle/missions/news/1992/h10.26.92',
 '/shuttle/missions/news/1993/h01.27.93',
 '/shuttle/missions/news/1994/',
 '/shuttle/missions/news/1994/n94-1',
 '/shuttle/missions/s',
 '/shuttle/missions/status/r102-90.ksc',
 '/shuttle/missions/status/r167-92.ksc',
 '/shuttle/missions/status/r48-90.ksc',
 '/shuttle/missions/status/r79-92.ksc',
 '/shuttle/missions/status/r88-113',
 '/shuttle/missions/status/r88-171',
 '/shuttle/missions/status/r89-101',
 '/shuttle/missions/status/r89-2',
 '/shuttle/missions/status/r91-10',
 '/shuttle/missions/status/r91-129',
 '/shuttle/missions/status/r91-44',
 '/shuttle/missions/status/r92-201',
 '/shuttle/missions/status/r92-31',
 '/shuttle/missions/status/r92-42',
 '/shuttle/missions/status/r92-60',
 '/shuttle/missions/status/r92-67',
 '/shuttle/missions/status/r93-30',
 '/shuttle/missions/sts-1/images/81HC315.GIF',
 '/shuttle/missions/sts-1/images/81HC320.GIF',
 '/shuttle/missions/sts-1/news',
 '/shuttle/missions/sts-1/news/',
 '/shuttle/missions/sts-10/mission-sts-10.html',
 '/shuttle/missions/sts-11/mission-sts-11.html',
 '/shuttle/missions/sts-12/mission-sts-12.html',
 '/shuttle/missions/sts-13/mission-sts-13.html',
 '/shuttle/missions/sts-1992-mission-summary.txt',
 '/shuttle/missions/sts-26/',
 '/shuttle/missions/sts-35/',
 '/shuttle/missions/sts-35/sts-35-patch.jpg',
 '/shuttle/missions/sts-37/movies/',
 '/shuttle/missions/sts-37/news',
 '/shuttle/missions/sts-37/news/',
 '/shuttle/missions/sts-37/sts-37-info.html',
 '/shuttle/missions/sts-38/sts-38-patch.jpg',
 '/shuttle/missions/sts-39/sts-39-patch.jpg',
 '/shuttle/missions/sts-4/news',
 '/shuttle/missions/sts-4/news/',
 '/shuttle/missions/sts-40/sts-40-patch.jpg',
 '/shuttle/missions/sts-41/sts-41-press-kit.txt',
 '/shuttle/missions/sts-42/movies/',
 '/shuttle/missions/sts-43/movies/',
 '/shuttle/missions/sts-44/images/91HC806.GIF',
 '/shuttle/missions/sts-44/sts-44-patch.jpg',
 '/shuttle/missions/sts-46/',
 '/shuttle/missions/sts-46/images/',
 '/shuttle/missions/sts-46/images/92HC326.GIF',
 '/shuttle/missions/sts-46/images/92HC553.GIF',
 '/shuttle/missions/sts-46/images/92HC556.GIF',
 '/shuttle/missions/sts-46/images/92HC561.GIF',
 '/shuttle/missions/sts-46/movies/',
 '/shuttle/missions/sts-46/sts-46-info.html',
 '/shuttle/missions/sts-46/sts-46-patch.jpg',
 '/shuttle/missions/sts-47/images/',
 '/shuttle/missions/sts-47/images/92HC648.GIF',
 '/shuttle/missions/sts-47/movies/',
 '/shuttle/missions/sts-47/news/',
 '/shuttle/missions/sts-47/sts-47-patch.jpg',
 '/shuttle/missions/sts-47/sts-47-press-kit.txt',
 '/shuttle/missions/sts-48/sts-48-patch.jpg',
 '/shuttle/missions/sts-49/news',
 '/shuttle/missions/sts-49/news/',
 '/shuttle/missions/sts-5/images/',
 '/shuttle/missions/sts-5/images/82HC557.GIF',
 '/shuttle/missions/sts-5/images/82HC651.GIF',
 '/shuttle/missions/sts-51l/movies/movies.html',
 '/shuttle/missions/sts-52/sts-52-patch.jpg',
 '/shuttle/missions/sts-52/sts-52-press-kit.txt',
 '/shuttle/missions/sts-53/sts-53-patch.jpg',
 '/shuttle/missions/sts-54/movies/',
 '/shuttle/missions/sts-55/sts-55-patch.jpg',
 '/shuttle/missions/sts-56/mission-sts-56.html>',
 '/shuttle/missions/sts-57/news',
 '/shuttle/missions/sts-57/news/',
 '/shuttle/missions/sts-58/',
 '/shuttle/missions/sts-58/sts-58-patch.jpg',
 '/shuttle/missions/sts-59/',
 '/shuttle/missions/sts-59/sts-59-patch.jpg',
 '/shuttle/missions/sts-60/images/',
 '/shuttle/missions/sts-60/sounds/',
 '/shuttle/missions/sts-61/mcc07.txt',
 '/shuttle/missions/sts-62/sts-62-info.html',
 '/shuttle/missions/sts-63/docs/',
 '/shuttle/missions/sts-63/images/k95p0264.jpg',
 '/shuttle/missions/sts-63/images/k95p0270.jpg',
 '/shuttle/missions/sts-63/news/sts-63-mcc-03.txt',
 '/shuttle/missions/sts-63/news/sts-63-mcc-17.txt',
 '/shuttle/missions/sts-64/news',
 '/shuttle/missions/sts-64/news/',
 '/shuttle/missions/sts-64/news/sts-64-mcc-01.txt',
 '/shuttle/missions/sts-64/news/sts-64-mcc-23.txt',
 '/shuttle/missions/sts-65',
 '/shuttle/missions/sts-65/',
 '/shuttle/missions/sts-65/news/sts-65-mcc-22.txt',
 '/shuttle/missions/sts-66/sounds/',
 '/shuttle/missions/sts-67',
 '/shuttle/missions/sts-67/images/KSC-95EC-0385.gif',
 '/shuttle/missions/sts-67/images/KSC-95EC-0386.gif',
 '/shuttle/missions/sts-67/images/KSC-95EC-0391.txt',
 '/shuttle/missions/sts-67/sounds/',
 '/shuttle/missions/sts-67/sounds/GrunsfeldJM.wav',
 '/shuttle/missions/sts-67/sts-67-patch.jpg',
 '/shuttle/missions/sts-67/sts-67/images/KSC-95EC-0396-small.gif',
 '/shuttle/missions/sts-69/mission-sts-69.htmlhttp://hamburg.bda.de:800/bda/index_germany.html',
 '/shuttle/missions/sts-70/http',
 '/shuttle/missions/sts-70/images/70LAUNCH.txt',
 '/shuttle/missions/sts-70/mission-sts-70',
 '/shuttle/missions/sts-70/mission-sts-71.html',
 '/shuttle/missions/sts-70/movies/sts-70-launch-big.mpg',
 '/shuttle/missions/sts-70/movies/\x7f',
 '/shuttle/missions/sts-70/sts-70-links.perl',
 '/shuttle/missions/sts-71.html',
 '/shuttle/missions/sts-71/Images/Images.htmL',
 '/shuttle/missions/sts-71/images/KSC-95EC-0443.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0544.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0869.gif',
 '/shuttle/missions/sts-71/images/KSC-95EC-0870.jpg',
 '/shuttle/missions/sts-71/images/KSC-95EC-0871.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0873.gif',
 '/shuttle/missions/sts-71/images/KSC-95EC-0874.gif',
 '/shuttle/missions/sts-71/images/KSC-95EC-0874.jpg',
 '/shuttle/missions/sts-71/images/KSC-95EC-0875.gif',
 '/shuttle/missions/sts-71/images/KSC-95EC-0891.gif',
 '/shuttle/missions/sts-71/images/KSC-95EC-0902.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0907.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0913.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0946.txt',
 '/shuttle/missions/sts-71/images/KSC-95EC-0951.txt',
 '/shuttle/missions/sts-71/images/captions.txt',
 '/shuttle/missions/sts-71/images/captions1.txt',
 '/shuttle/missions/sts-71/images/captions2.txt',
 '/shuttle/missions/sts-71/images/images.html/',
 '/shuttle/missions/sts-71/movies/crew-arrival.mpg',
 '/shuttle/missions/sts-71/news/ksc-status-06-07-95.txt',
 '/shuttle/missions/sts-71/news/ksc-status-06-26-95.txt',
 '/shuttle/missions/sts-71/news/ksc-status-06-27-95.txt',
 '/shuttle/missions/sts-71/news/mcc21.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-08.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-09.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-10.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-11.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-13.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-14.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-15.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-16.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-17.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-18.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-19.txt',
 '/shuttle/missions/sts-71/news/sts-71-mcc-20.txt',
 '/shuttle/missions/sts-71/shuttle.txt',
 '/shuttle/missions/sts-71/sts-71-crew.JPG',
 '/shuttle/missions/sts-71/sts-71-patch-logo.gif',
 '/shuttle/missions/sts-71/sts-71-patch.html',
 '/shuttle/missions/sts-75/movies/',
 '/shuttle/missions/sts-76/',
 '/shuttle/missions/sts-76/images/',
 '/shuttle/missions/sts-76/news',
 '/shuttle/missions/sts-76/news/',
 '/shuttle/missions/sts-76/sts-76-patch.jpg',
 '/shuttle/missions/sts-8/',
 '/shuttle/missions/sts-8/images/',
 '/shuttle/missions/sts-8/images/83HC600.GIF',
 '/shuttle/missions/sts-89/mission-sts-89.html',
 '/shuttle/missions/sts-9/',
 '/shuttle/missions/sts-9/images/',
 '/shuttle/missions/sts-9/images/83HC742.GIF',
 '/shuttle/missions/sts-9/images/83HC751.GIF',
 '/shuttle/missions/sts-XX/sts-XX-info.html',
 '/shuttle/missions/sts-XX/sts-XX-press-kit.txt',
 '/shuttle/missions/sts-ksc-landings.txt',
 '/shuttle/missions/sts-missions.txt',
 '/shuttle/missions/sts-test.txt',
 '/shuttle/missions/sts70/',
 '/shuttle/missoins/missions.html',
 '/shuttle/missons',
 '/shuttle/missons/sts-71',
 '/shuttle/missons/sts-71/images',
 '/shuttle/missons/sts-71/images/images.html',
 '/shuttle/movies/sts-70-tdrs-processing.mpg',
 '/shuttle/news',
 '/shuttle/news/',
 '/shuttle/news/nasa.nasamail.p/35',
 '/shuttle/news/sci.space.news/164',
 '/shuttle/resources',
 '/shuttle/resources/orbiters',
 '/shuttle/resources/orbiters/endeavor.html',
 '/shuttle/shuttle/sts-71',
 '/shuttle/sightings/',
 '/shuttle/technology/images/sts-comm-small.gif',
 '/shuttle/technology/images/sts_events-small.gif',
 '/shuttle/technology/sts-newsref/09_ov_ma.txt',
 '/shuttle/technology/sts-newsref/23_ov_gn.txt',
 '/shuttle/technology/sts-newsref/sts-comm.html',
 '/shuttle/technology/sts-newsref/sts-wcl.html',
 '/shuttle/technology/sts-newsref/sts_events.html',
 '/shuttle/technology/sts-newsref/tps_mods.html',
 '/shuttle/video/lifevideo.jpge',
 '/shuttlecountdown',
 '/sightings/',
 '/software/webadmin/payload.perl',
 '/software/webadmin/sts-tech.bad',
 '/software/webadmin/sts-tech.good',
 '/software/winvn/brydon.gif',
 '/software/winvn/faq/WINVNFAQ-I-1.html',
 '/software/winvn/faq/WINVNFAQ-I-10.html',
 '/software/winvn/faq/WINVNFAQ-I.html',
 '/software/winvn/faq/WINVNFAQ-II-1.html',
 '/software/winvn/faq/WINVNFAQ-II-4.html',
 '/software/winvn/faq/WINVNFAQ-III-2.html',
 '/software/winvn/faq/WINVNFAQ-III-4.html',
 '/software/winvn/faq/WINVNFAQ-V-1.html',
 '/software/winvn/faq/WINVNFAQ-V-3.html',
 '/software/winvn/faq/WINVNFAQ-V-5.html',
 '/software/winvn/faq/WINVNFAQ-V-6.html',
 '/software/winvn/faq/WINVNFAQ-V-7.html',
 '/software/winvn/faq/WINVNFAQ-V.html',
 '/software/winvn/userguide/2_2_2.htm',
 '/software/winvn/userguide/2_2_4.htm',
 '/software/winvn/userguide/2_3_4.htm',
 '/software/winvn/userguide/winvn16.gif',
 '/software/winvn/winvn/html',
 '/statistics/1993/Dec/Dec93.html',
 '/statistics/1993/Dec/Dec93_byte.gif',
 '/statistics/1993/Dec/Dec93_request.gif',
 '/statistics/1995/Jun/Jun95_country_byte.gif',
 '/statistics/1995/Jun/Jun95_daily_byte.gif',
 '/statistics/1995/bkup/Apr95_full.html',
 '/sts-70/',
 '/sts-70/ccc/realtime.html',
 '/sts-70/demos',
 '/sts-70/demos/',
 '/sts-70/ntv',
 '/sts-70/orbiter/',
 '/suttle/countdown/',
 'www.ksc.nasa.gov'}
In [91]:
# посмотрим на разницу в топ100
visits_event = event.groupby('url')[['method']].count().sort_values(by='method', ascending=False)
visits_no_event = no_event.groupby('url')[['method']].count().sort_values(by='method', ascending=False)
set(visits_event[:100].index) - set(visits_no_event[:100].index)
Out[91]:
{'/facilities/lcc.html',
 '/history/apollo/apollo-13/images/',
 '/icons/movie.xbm',
 '/images/launch-small.gif',
 '/shuttle/countdown/count.html',
 '/shuttle/countdown/images/cdtclock.gif',
 '/shuttle/countdown/video/livevideo.gif',
 '/shuttle/missions/',
 '/shuttle/missions/sts-70/images/KSC-95EC-0667.jpg',
 '/shuttle/missions/sts-70/images/KSC-95EC-0852.jpg',
 '/shuttle/missions/sts-70/images/index70.gif',
 '/shuttle/missions/sts-70/sts-70-info.html',
 '/shuttle/missions/sts-70/sts-70-patch.jpg',
 '/shuttle/missions/sts-70/sts-70-press-kit.txt',
 '/shuttle/technology/images/tal_abort_2-small.gif',
 '/shuttle/technology/sts-newsref/sts-lcc.html'}

Анализируем результаты стрельб Танком

In [92]:
# функция для импорта сырого отчета phantom

phout_columns = [
    'time', 'tag', 'interval_real',
    'connect_time', 'send_time',
    'latency', 'receive_time',
    'interval_event', 'size_out',
    'size_in', 'net_code', 'proto_code']

def read_phout(filename):
    data = pd.read_csv(
        filename, sep='\t', names=phout_columns)
    
    # хитрые манипуляции с колонками
    
    # в логе есть время отправки и время выполнения запроса. Суммируем их и получается
    # время получения ответа - будем группировать по нему (заводим новую колонку)
    data['ts'] = data.time + data.interval_real / 1000000
    # округляем до секунды
    data['receive_sec'] = data.ts.astype(int)
    # и индексируем по этой секунде
    data.set_index(['receive_sec'], inplace=True)
    # для удобства посчитаем время ответа в миллисекундах
    data['rt_ms'] = data.interval_real / 1000
    
    return data
In [93]:
phout1 = read_phout("phout_1.log")
phout2 = read_phout("phout_2.log")
In [94]:
# графики RPS
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
phout1.groupby(level=0).count().time.rolling(window=10).mean().plot(title="RPS 1", ax=ax)

fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
phout2.groupby(level=0).count().time.rolling(window=10).mean().plot(title="RPS 2", ax=ax)
Out[94]:
<matplotlib.axes._subplots.AxesSubplot at 0x16402ad68>
In [95]:
# группируем по индексу, считаем медиану, среднее и максимум и рисуем график
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
phout1.groupby(level=0).rt_ms.agg([np.mean, np.median, np.max]).plot(ax=ax)
Out[95]:
<matplotlib.axes._subplots.AxesSubplot at 0x1767f0a90>
In [96]:
# медиана и среднее
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
phout1 = phout1[2000:-1000]
phout1.groupby(level=0).rt_ms.agg([np.mean, np.median]).plot(ax=ax)
Out[96]:
<matplotlib.axes._subplots.AxesSubplot at 0x17b5af2e8>
In [97]:
# сглаженные медиана и среднее
fig, ax = sns.plt.subplots()
fig.set_size_inches(16,10)
phout1 = phout1[2000:-1000]
phout1.groupby(level=0).rt_ms.agg([np.mean, np.median]).rolling(window=10).mean().plot(ax=ax)
Out[97]:
<matplotlib.axes._subplots.AxesSubplot at 0x1820afc18>