项目原作者 ArtrixTech, 项目地址 BoomMine, 原文档发表在知乎 @Artrix, 助教进行了文档的改进,和代码形式上的交互式重构。
这个笔记本主要是为了让大家了解一个小型的 Python + OpenCV 项目是如何完成的,以便于以后进行更复杂的软件工程项目。
通过对比 notebook 和 GitHub 上面的项目,大家也可以知道如何从交互式编程练习过渡到实际的 Python 脚本开发。
一直对着命令行编程也累了,我们下载扫雷游戏 Minesweeper Arbiter 来玩一玩吧。
如果你从前没有玩过扫雷,可以看看国内最大的扫雷爱好者论坛扫雷网的基础入门教程:
简单玩几局后,就可以考虑如何设计一个扫雷机器人,来帮助我们完成这个游戏啦~
在去做一件事情之前最重要的是什么?是将要做的这件事情在心中搭建一个步骤框架。只有这样,才能保证在去做这件事的过程中,尽可能的做到深思熟虑,使得最终有个好的结果。我们写程序也要尽可能做到在正式开始开发之前,在心中有个大致的思路。
对于本项目而言,大致的开发过程是这样的:
好啦,既然我们有了个思路,那就撸起袖子大力干!
其实对于本项目而言,窗体截取是一个逻辑上简单,实现起来却相当麻烦的部分,而且还是必不可少的部分。笔者通过 Spy++ 得到了以下两点信息:
class_name = "TMain" # ms_arbiter.exe的主窗体类别
title_name = "Minesweeper Arbiter" # ms_arbiter.exe的主窗体名称
注意到了么?主窗体的名称后面有个空格。正是这个空格让笔者困扰了一会儿,只有加上这个空格,win32gui 才能够正常的获取到窗体的句柄。(提示:在助教的机器上不加空格反而有效)
运行 ms_arbiter.exe
, 本项目采用了 win32gui 来获取窗体的位置信息,具体代码如下:
import win32gui
class_name = "TMain" # ms_arbiter.exe的主窗体类别
title_name = "Minesweeper Arbiter" # ms_arbiter.exe的主窗体名称
hwnd = win32gui.FindWindow(class_name, title_name) # Handle to A Window
if hwnd:
left, top, right, bottom = win32gui.GetWindowRect(hwnd)
print("Window find.")
print("left: " + str(left))
print("top: " + str(top))
print("right: " + str(right))
print("bottom: " + str(bottom))
else:
print("Window not find.")
你可以移动窗体的位置,重复运行上面的代码,看得到的位置数据是否有变化。
通过以上代码,我们得到了窗体相对于整块屏幕的位置。
之后我们需要通过 PIL 来进行扫雷界面的棋盘截取:
from PIL import ImageGrab
# 微调参数,得到棋盘的相对位置,仅在我的电脑环境下测试通过
# 这一步如果需要重复运行,需要把上面的代码也一起重复运行 (参数更新)
left += 15
top += 111
right -= 15
bottom -= 31
# 棋盘截取
rect = (left, top, right, bottom)
img = ImageGrab.grab().crop(rect) # 扫雷界面不能被遮挡
img.show() # 通过快照的位置调整上面的四个参数,实际调用中没有这一行
# 得到棋盘的宽度和高度
width = right - left
height = bottom - top
其实有着更好的寻找扫雷棋盘相对位置的方法,但是这里希望大家进行人为参数寻找。
这种无法由算法给出,需要人类给定的参数叫作“超参数”,这一概念后面还会见到。
在进行雷块分割之前,我们事先需要了解雷块的尺寸以及它的边框大小。经过笔者的测量,在 ms_arbiter
下,每一个雷块的尺寸为 16px*16px.
block_width, block_height = 16, 16
blocks_x = int((right - left) / block_width) # 宽度上块的个数,高级为 30
blocks_y = int((bottom - top) / block_height) # 高度上块的个数,高级为 15
print(str(blocks_x) + "," + str(blocks_y))
之后,我们建立一个二维数组用于存储每一个雷块的图像,并且进行图像分割,保存在之前建立的数组中。
def crop_block(hole_img, x, y):
# 基于棋盘进行单个块的截取
x1, y1 = x * block_width, y * block_height
x2, y2 = x1 + block_width, y1 + block_height
return hole_img.crop((x1, y1, x2, y2))
blocks_img = [[0 for i in range(blocks_y)] for i in range(blocks_x)]
for y in range(blocks_y):
for x in range(blocks_x):
blocks_img[x][y] = crop_block(img, x, y)
将整个图像获取、分割的部分封装成一个库,随时调用就OK啦~在笔者的实现中,笔者将这一部分封装成了imageProcess.py
, 其中函数 get_frame()
用于完成上述的图像获取、分割过程:
import win32gui
import numpy
from PIL import ImageGrab
import cv2
block_width, block_height = 16, 16
def crop_block(hole_img, x, y):
x1, y1 = x * block_width, y * block_height
x2, y2 = x1 + block_width, y1 + block_height
return hole_img.crop((x1, y1, x2, y2))
def pil_to_cv(img):
return cv2.cvtColor(numpy.asarray(img), cv2.COLOR_RGB2BGR)
def get_frame():
class_name = "TMain"
title_name = "Minesweeper Arbiter"
hwnd = win32gui.FindWindow(class_name, title_name)
if hwnd:
left, top, right, bottom = win32gui.GetWindowRect(hwnd)
left += 15
top += 111
right -= 15
bottom -= 31
width = right - left
height = bottom - top
rect = (left, top, right, bottom)
img = ImageGrab.grab().crop(rect)
blocks_x = int((right - left) / block_width)
blocks_y = int((bottom - top) / block_height)
blocks_img = [[0 for i in range(blocks_y)] for i in range(blocks_x)]
for y in range(blocks_y):
for x in range(blocks_x):
blocks_img[x][y] = crop_block(img, x, y)
return img, blocks_img, (blocks_x, blocks_y), (width, height), (left, top, right, bottom)
return -1
img = get_frame() # 单元测试该方法
# img[0].show() # 显示整个棋盘
img[1][0][0].show() # 显示指定的 [1][x][y] 块,可人为点击改变块的状态
同样地,我们这里采用了笨笨的分割方法,如果学习过特征提取,我们能够使用更好的方法进行图像分割。在本节练习中,我们将利用图像颜色特征的知识对雷块的属性进行识别。
我们现在来设计一个类(Class),下面是已经写好的源码,你能不能试着直接读懂它们:
class BoomMine:
__inited = False
blocks_x, blocks_y = -1, -1
width, height = -1, -1
img_cv, img = -1, -1
blocks_img = [[-1 for i in range(blocks_y)] for i in range(blocks_x)]
blocks_num = [[-3 for i in range(blocks_y)] for i in range(blocks_x)]
blocks_is_mine = [[0 for i in range(blocks_y)] for i in range(blocks_x)]
is_new_start = True
next_steps = []
@staticmethod # Python 内置函数 staticmethod()
def rgb_to_bgr(rgb):
return rgb[2], rgb[1], rgb[0]
@staticmethod # 该方法不强制要求传递参数,无需实例化
def equal(arr1, arr2):
if arr1[0] == arr2[0] and arr1[1] == arr2[1] and arr1[2] == arr2[2]:
return True
return False
def is_in_form(self, location):
x, y = location[0], location[1]
if x < self.left or x > self.right or y < self.top or y > self.bottom:
return False
return True
def iterate_blocks_image(self, func):
if self.__inited:
for y in range(self.blocks_y):
for x in range(self.blocks_x):
# args are: self, [0]singleBlockImage, [1]location(as an array)
func(self, self.blocks_img[x][y], (x, y))
def iterate_blocks_number(self, func):
if self.__inited:
for y in range(self.blocks_y):
for x in range(self.blocks_x):
# args are: self, [0]singleBlockNumber, [1]location(as an array)
func(self, self.blocks_num[x][y], (x, y))
def analyze_block(self, block, location):
x, y = location[0], location[1]
now_num = self.blocks_num[x][y]
# if 1:
if not now_num == -2 and not 0 < now_num < 9:
block = imageProcess.pil_to_cv(block)
block_color = block[8, 8]
# -1:Not opened
# -2:Opened but blank
# -3:Un initialized
# Opened
if self.equal(block_color, self.rgb_to_bgr((192, 192, 192))):
if not self.equal(block[8, 1], self.rgb_to_bgr((255, 255, 255))):
self.blocks_num[x][y] = -2
self.is_started = True
else:
self.blocks_num[x][y] = -1
elif self.equal(block_color, self.rgb_to_bgr((0, 0, 255))):
self.blocks_num[x][y] = 1
elif self.equal(block_color, self.rgb_to_bgr((0, 128, 0))):
self.blocks_num[x][y] = 2
elif self.equal(block_color, self.rgb_to_bgr((255, 0, 0))):
self.blocks_num[x][y] = 3
elif self.equal(block_color, self.rgb_to_bgr((0, 0, 128))):
self.blocks_num[x][y] = 4
elif self.equal(block_color, self.rgb_to_bgr((128, 0, 0))):
self.blocks_num[x][y] = 5
elif self.equal(block_color, self.rgb_to_bgr((0, 128, 128))):
self.blocks_num[x][y] = 6
elif self.equal(block_color, self.rgb_to_bgr((0, 0, 0))):
if self.equal(block[6, 6], self.rgb_to_bgr((255, 255, 255))):
# Is mine
self.blocks_num[x][y] = 9
elif self.equal(block[5, 8], self.rgb_to_bgr((255, 0, 0))):
# Is flag
self.blocks_num[x][y] = 0
else:
self.blocks_num[x][y] = 7
elif self.equal(block_color, self.rgb_to_bgr((128, 128, 128))):
self.blocks_num[x][y] = 8
else:
self.blocks_num[x][y] = -3
self.is_mine_form = False
if self.blocks_num[x][y] == -3 or not self.blocks_num[x][y] == -1:
self.is_new_start = False
def detect_mine(self, block, location):
def generate_kernel(k, k_width, k_height, block_location):
ls = []
loc_x, loc_y = block_location[0], block_location[1]
for now_y in range(k_height):
for now_x in range(k_width):
if k[now_y][now_x]:
rel_x, rel_y = now_x - 1, now_y - 1
ls.append((loc_y + rel_y, loc_x + rel_x))
return ls
def count_unopen_blocks(blocks):
count = 0
for single_block in blocks:
if self.blocks_num[single_block[1]][single_block[0]] == -1:
count += 1
return count
def mark_as_mine(blocks):
for single_block in blocks:
if self.blocks_num[single_block[1]][single_block[0]] == -1:
self.blocks_is_mine[single_block[1]][single_block[0]] = 1
x, y = location[0], location[1]
if self.blocks_num[x][y] > 0:
kernel_width, kernel_height = 3, 3
# Kernel mode:[Row][Col]
kernel = [[1, 1, 1], [1, 1, 1], [1, 1, 1]]
# Left border
if x == 0:
for i in range(kernel_height):
kernel[i][0] = 0
# Right border
if x == self.blocks_x - 1:
for i in range(kernel_height):
kernel[i][kernel_width - 1] = 0
# Top border
if y == 0:
for i in range(kernel_width):
kernel[0][i] = 0
# Bottom border
if y == self.blocks_y - 1:
for i in range(kernel_width):
kernel[kernel_height - 1][i] = 0
# Generate the search map
to_visit = generate_kernel(kernel, kernel_width, kernel_height, location)
unopen_blocks = count_unopen_blocks(to_visit)
if unopen_blocks == self.blocks_num[x][y]:
mark_as_mine(to_visit)
def detect_to_click_block(self, block, location):
def generate_kernel(k, k_width, k_height, block_location):
ls = []
loc_x, loc_y = block_location[0], block_location[1]
for now_y in range(k_height):
for now_x in range(k_width):
if k[now_y][now_x]:
rel_x, rel_y = now_x - 1, now_y - 1
ls.append((loc_y + rel_y, loc_x + rel_x))
return ls
def count_mines(blocks):
count = 0
for single_block in blocks:
if self.blocks_is_mine[single_block[1]][single_block[0]] == 1:
count += 1
return count
def mark_to_click_block(blocks):
for single_block in blocks:
# Not Mine
if not self.blocks_is_mine[single_block[1]][single_block[0]] == 1:
# Click-able
if self.blocks_num[single_block[1]][single_block[0]] == -1:
# Source Syntax: [y][x] - Converted
if not (single_block[1], single_block[0]) in self.next_steps:
self.next_steps.append((single_block[1], single_block[0]))
x, y = location[0], location[1]
if block > 0:
kernel_width, kernel_height = 3, 3
# Kernel mode:[Row][Col]
kernel = [[1, 1, 1], [1, 1, 1], [1, 1, 1]]
# Left border
if x == 0:
for i in range(kernel_height):
kernel[i][0] = 0
# Right border
if x == self.blocks_x - 1:
for i in range(kernel_height):
kernel[i][kernel_width - 1] = 0
# Top border
if y == 0:
for i in range(kernel_width):
kernel[0][i] = 0
# Bottom border
if y == self.blocks_y - 1:
for i in range(kernel_width):
kernel[kernel_height - 1][i] = 0
# Generate the search map
to_visit = generate_kernel(kernel, kernel_width, kernel_height, location)
mines_count = count_mines(to_visit)
if mines_count == block:
mark_to_click_block(to_visit)
def rel_loc_to_real(self, block_rel_location):
return self.left + 16 * block_rel_location[0] + 8, self.top + 16 * block_rel_location[1] + 8
def __init__(self):
self.next_steps = []
self.left = 0
self.top = 0
self.right = 0
self.bottom = 0
self.continue_random_click = False
self.is_mine_form = True
self.is_started = False
self.have_solve = False
if self.process_once():
self.__inited = True
def show_map(self):
if self.__inited:
for y in range(self.blocks_y):
line = ""
for x in range(self.blocks_x):
if self.blocks_num[x][y] == -1:
line += " "
else:
line += str(self.blocks_num[x][y]) + " "
print(line)
def show_mine(self):
if self.__inited:
for y in range(self.blocks_y):
line = ""
for x in range(self.blocks_x):
if self.blocks_is_mine[x][y] == 0:
line += " "
else:
line += str(self.blocks_is_mine[x][y]) + " "
print(line)
def process_once(self):
# Initialize
result = imageProcess.get_frame()
if result == -1:
print("Minesweeper Arbiter Window Not Detected!")
return False
self.img, self.blocks_img, form_size, img_size, form_location = result
self.blocks_num = [[-1 for i in range(self.blocks_y)] for i in range(self.blocks_x)]
self.blocks_is_mine = [[0 for i in range(self.blocks_y)] for i in range(self.blocks_x)]
self.next_steps = []
self.is_new_start = True
self.is_mine_form = True
self.blocks_x, self.blocks_y = form_size[0], form_size[1]
self.width, self.height = img_size[0], img_size[1]
self.img_cv = imageProcess.pil_to_cv(self.img)
self.left, self.top, self.right, self.bottom = form_location
# Analyze the number of blocks
self.iterate_blocks_image(BoomMine.analyze_block)
# Mark all mines
self.iterate_blocks_number(BoomMine.detect_mine)
# Calculate where to click
self.iterate_blocks_number(BoomMine.detect_to_click_block)
self.have_solve = False
if len(self.next_steps) > 0:
self.have_solve = True
if self.is_in_form(mouseOperation.get_mouse_point()):
for to_click in self.next_steps:
on_screen_location = self.rel_loc_to_real(to_click)
mouseOperation.mouse_move(on_screen_location[0], on_screen_location[1])
mouseOperation.mouse_click()
# If your computer's performance is not high, enable this:
# time.sleep(0.001)
if not self.have_solve and self.is_mine_form:
rand_location = (random.randint(0, self.blocks_x - 1), random.randint(0, self.blocks_y - 1))
rand_x, rand_y = rand_location[0], rand_location[1]
iter_times = 0
if len(self.blocks_is_mine) > 0:
while self.blocks_is_mine[rand_x][rand_y] or not self.blocks_num[rand_x][rand_y] == -1 and iter_times < 50:
rand_location = (random.randint(0, self.blocks_x - 1), random.randint(0, self.blocks_y - 1))
rand_x, rand_y = rand_location[0], rand_location[1]
iter_times += 1
screen_location = self.rel_loc_to_real((rand_location[0], rand_location[1]))
if self.is_in_form(mouseOperation.get_mouse_point()):
mouseOperation.mouse_move(screen_location[0], screen_location[1])
mouseOperation.mouse_click()
else:
self.is_mine_form = False
cv2.imshow("Sweeper Screenshot", self.img_cv)
if cv2.waitKey(1) & 0xFF == ord('q'):
return False
return True
我们先不进行单元测试,先集成测试看一下整体效果:
import random
import cv2
import time
import mouseOperation # 鼠标操作,无需知道细节
import imageProcess # 这个文件我们之前实现了所有功能
miner = BoomMine()
while 1:
try:
miner.process_once()
except:
pass
# miner.show_map()
# print(miner.next_steps)
这个项目在一些机器上可能存在着 Bug, 试一下你能不能理解代码,找到出错的原因。
简单起见,这里使用了一种比较暴力的扫雷逻辑:
在实现它之后,你可以使用动态规划或者搜索的方式,改进这个扫雷算法,根据你的期望重构这个项目,发布到你的 GitHub 上面。