Python OpenCV Computer Vision: 6~7

Original: OpenCV Computer Vision with Python

License: CC BY-NC-SA 4.0

Translator: Flying Dragon

This article comes from [ApacheCN Computer Vision Translation Collection] , using post-translation editing (MTPE) process to improve efficiency as much as possible.

When others say you have no bottom line, you'd better not; when others say you have done something, you'd better really do it.

Appendix A: Integration with Pygame

This appendix shows how to set up the Pygame library in an OpenCV application and how to use Pygame for window management. In addition, the appendix provides an overview of other features of Pygame and some resources for learning Pygame.

---

Notice

All the completed code for this chapter can be downloaded from my website .

---

Install Pygame

Suppose we have already set up Python according to one of the methods described in Chapter 1, "Setting Up OpenCV". Depending on our existing setup, we can install Pygame in one of the following ways:

• Windows with 32-bit Python: Download and install Pygame 1.9.1 from the following location .

• Windows with 64-bit Python: Download and install the Pygame 1.9.2 preview from the following location .

• Mac with Macports: Open Terminal and run the following command:

  
  $ sudo port install py27-game
  
  

• Mac with Homebrew: Open Terminal and run the following commands to install Pygame's dependencies, then install Pygame itself

  
  $ brew install sdl sdl_image sdl_mixer sdl_ttf smpeg portmidi
  $ /usr/local/share/python/pip install \
  > hg+http://bitbucket.org/pygame/pygame
  
  

• Ubuntu and its derivatives: Open Terminal and run the following command:

  
  $ sudo apt-get install python-pygame
  
  

• Other Unix-like systems: Pygame is available in standard repositories on many systems. Typical package names include pygame, pygame27, py-game, py27-game, python-pygame,and python27-pygame.

Now, Pygame should be ready to use.

Documentation and Tutorials

Pygame's API documentation and some tutorials can be found online at :

Al Sweigart's Making Games with Python and Pygame is a cookbook for recreating several classic games in Pygame 1.9.1\. A free electronic copy is available online at the following website . Or download the PDF file from the website below .

derivationManager.WindowManager

As described in Chapter 2, "Handling Cameras, Files, and GUIs", our object-oriented design allows us to easily switch OpenCV's HighGUI window manager to another window manager, such as Pygame. To do this, we just need to subclass our managers.WindowManagerclass and override four methods: createWindow(), show(), destroyWindow()and processEvents(). Also, we need to import some new dependencies.

To continue, we'll need the files from Chapter 2, "Working with Cameras, Files, and the GUI managers.py," and the files from Chapter 4, "Tracking Faces with Haar Cascades utils.py. " In utils.py, we only need one function isGray(), which we implement in Chapter 4, "Face Tracking with Haar Cascades". Let's edit managers.pyto add the following imports:

import pygame
import utils

Also in managers.py, WindowManagersomewhere after the execution, we want to add PygameWindowManagera new subclass named:

class PygameWindowManager(WindowManager):
    def createWindow(self):
        pygame.display.init()
        pygame.display.set_caption(self._windowName)
        self._isWindowCreated = True
    def show(self, frame):
        # Find the frame's dimensions in (w, h) format.
        frameSize = frame.shape[1::-1]
        # Convert the frame to RGB, which Pygame requires.
        if utils.isGray(frame):
            conversionType = cv2.COLOR_GRAY2RGB
        else:
            conversionType = cv2.COLOR_BGR2RGB
        rgbFrame = cv2.cvtColor(frame, conversionType)
        # Convert the frame to Pygame's Surface type.
        pygameFrame = pygame.image.frombuffer(
            rgbFrame.tostring(), frameSize, 'RGB')
        # Resize the window to match the frame.
        displaySurface = pygame.display.set_mode(frameSize)
        # Blit and display the frame.
        displaySurface.blit(pygameFrame, (0, 0))
        pygame.display.flip()
    def destroyWindow(self):
        pygame.display.quit()
        self._isWindowCreated = False
    def processEvents(self):
        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN and \
                    self.keypressCallback is not None:
                self.keypressCallback(event.key)
            elif event.type == pygame.QUIT:
                self.destroyWindow()
                return

Note that we use two Pygame modules: pygame.displayand pygame.event.

Call pygame.display.init()to create a window, call to pygame.display.quit()destroy a window. Repeated calls display.init()have no effect, since Pygame only works with single-window applications. The drawing surface type of the Pygame window is pygame.Surface. To get Surfacea reference to it, we can call pygame.display.get_surface()or pygame.display.set_mode(). The latter function modifies the attributes of the entity before returning it Surface. One Surfaceentity has a blit()method that takes Surfaceas parameters another and a coordinate pair where the latter Surfaceshould be "whitened" (painted) onto the first. When we're done updating the window for the current frame Surface, we should call pygame.display.flip()it to display it.

pygame.event.get()Events such as can be polled by calling keypresses, which returns a list of all events that have occurred since the last call. Each event is of type pygame.event.Event, and has an attribute type, which indicates the category of the event, such as pygame.KEYDOWNpressing a key, and pygame.QUIT indicating that the close button of the window was clicked. Depending on typethe value of , Eventthe entity may have other properties, for example, KEYDOWNevent's key(ASCII key code).

There is some overhead by converting between OpenCV's image format and Pygame's format on a per-frame basis WindowManager, as opposed to using HighGUI's basics . However, normal window closing behavior is provided, whereas base does not.PygameWindowManagerSurfacePygameWindowManagerWindowManager

modify application

Let's cameo.pymodify the file to use PygameWindowManagerinstead of WindowManager. cameo.pyFind the following line in :

from managers import WindowManager, CaptureManager

replace it with:

from managers import PygameWindowManager as WindowManager, \
                     CaptureManager

that's all! Now, cameo.pywith a Pygame window, the window should close when the standard "close" button is clicked.

Further use of Pygame

We only used some basic functions of the pygame.displayand module. pygame.eventPygame provides many more features, including:

• Draw 2D geometry
• draw text
• Manage grouping of drawable AI entities (sprites)
• Capture various input events related to windows, keyboard, mouse and joystick/gamepad
• Create custom events
• Play and synthesize sounds and music

For example, Pygame might be an appropriate backend for a game using computer vision, while HighGUI is not.

Summarize

By now we should have an application that uses OpenCV to capture (and possibly manipulate) images while using Pygame to display images and capture events. Starting with this basic integration example, you may want to extend PygameWindowManagerto wrap other Pygame functionality, or you may want to create another WindowManagersubclass to wrap another library.

Appendix B: Generating Haar Cascades for Custom Targets

This appendix shows how to generate a Haar cascade XML file, such as the one used in Chapter 4, "Tracking Faces Using a Haar Cascade". By generating our own cascade files, we can potentially track any pattern or object, not just faces. However, good results may not come soon. We had to carefully collect images, configure script parameters, perform actual tests and iterate. Significant labor and processing time may be involved.

Collect positive and negative training images

Do you know the teaching method of flashcards? Here's a way to teach words and recognition skills to young children. The teacher showed the class a series of pictures and said the following:

"It's a cow. Mo! It's a horse. Neigh!"

Cascading files are generated in a similar way to the flashcard pedagogy. To learn how to recognize cows, the computer needs positive training images pre-identified as cows and negative training images pre-identified as "not cows . " As trainers, our first step is to collect these two sets of images.

When deciding how many positive training images to use, we need to consider the various ways in which the user views objects. Ideally, the simplest case is where the target is a 2D pattern that is always on a flat surface. In this case, a single positive training image may suffice. However, in other cases, hundreds or even thousands of training images may be required. Let's say the target is the flag of your country. When printed on a document, the appearance of a logo may be predictable, but when printed on fabric blowing in the wind, the appearance of the logo can vary greatly. Natural 3D objects such as human faces may have a larger range of appearances. Ideally, our set of frontal training images should represent many variations that our camera might capture. Optionally, any of our frontal training images can contain multiple instances of the object.

For our negative training set, we need a large number of images that do not contain any instances of objects, but do contain other things that the camera might capture. For example, if a flag is our target, then our negative training set might include photos of the sky in various weather conditions. (The sky is not a flag, but is often seen behind it.) Don't assume too much, though. If the camera's environment is unpredictable and objects appear in many settings, use a wide variety of negative training images. Consider building a common set of environment images that you can reuse across multiple training scenarios. ,

Find the training executable

To make cascade training as automatic as possible, OpenCV provides two executables. Their names and locations depend on the operating system and OpenCV-specific settings, as described in the following two sections.

on windows

The two executables on Windows are called ONopencv_createsamples.exeand ONopencv_traincascade.exe. They are not pre-built. Instead, they only exist if you compile OpenCV from source. Depending on the compilation method you chose in Chapter 1, "Setting Up OpenCV", their parent folder is one of the following folders:

• MinGW：<unzip_destination>\bin
• Visual Studio 或 Visual C++ Express：<unzip_destination>\bin\Release

If you want to add the executable's folder to the system's Pathvariables, refer to the information box in the "Choosing on Windows XP, Windows Vista, Windows 7 and Windows 8" section of Chapter 1, "Setting Up OpenCV". illustrate. Otherwise, note the full path to the executable as we will need it when running them.

On Mac, Ubuntu and other Unix-like systems

The two executables on Mac, Ubuntu, and other Unix-like systems are called opencv_createsamplesand opencv_traincascade. Their parent folder is one of the following folders, depending on your system and the method chosen in Chapter 1, "Setting Up OpenCV":

• Mac with MacPorts:/opt/local/bin
• Mac with Homebrew: /opt/local/binor/opt/local/sbin
• Ubuntu with Apt:/usr/bin
• Ubuntu using my custom install script:/usr/local/bin
• Other Unix-like systems: /usr/binand/usr/local/bin

Except on Macs with Homebrew, by default the executable's folder should be in PATH. For Homebrew, if you want to add the relevant folders to it PATH, see the instructions in the second part of "Using Homebrew with an off-the-shelf package (no depth camera support)" in Chapter 1, "Setting Up OpenCV". Otherwise, note the full path to the executable, as we'll need that when running them.

Create training set and cascade

Hereafter, we refer to these two executables as <opencv_createsamples>and <opencv_traincascade>. Remember to substitute the path and filename appropriate for your system and setup.

These executables have certain data files as input and output. Here's a typical way to generate these data files:

1. Manually create a text file describing the set of negative training images. We will call this file <negative_description>.
2. Manually create a text file describing the set of frontal training images. We will call this file <positive_description>.
3. Run with <negative_description>and as arguments . This executable will create a binary file describing the training data. We will refer to the latter file as .<positive_description><opencv_createsamples><binary_description>
4. <binary_description>run as an argument <opencv_traincascade>. This executable creates a binary concatenation file, which we'll call <cascade>.

We can choose the actual name and path of <negative_description>, <positive_description>, <binary_description>and .<cascade>

Now, let us understand the three steps in detail.

create<negative_description>

<negative_description>is a text file listing relative paths to all negative training images. Paths should be separated by newlines. For example, suppose we have the following directory structure, <negative_description>where negative/desc.txt:

negative
    desc.txt
    images
        negative 0.png
        negative 1.png

Then, negative/desc.txtthe content could be as follows:

"images/negative 0.png"
"images/negative 1.png"

For a small number of images, we can manually write such a file. For large numbers of images, we should use the command line instead to find relative paths that match a certain pattern and output those matches to a file. Continuing with our example, we can generate by running the following command in the Windows Command Prompt negative/desc.txt:

> cd negative
> forfiles /m images\*.png /c "cmd /c echo @relpath" > desc.txt

Note that in this case, relative paths are of the form .\images\negative 0.png, which is acceptable.

Alternatively, in a Unix-like shell, such as Terminal on Mac or Ubuntu, we can run the following command:

$ cd negative
$ find images/*.png | sed -e "s/^/\"/g;s/$/\"/g" > desc.txt

create<positive_description>

Needed to use if we have multiple frontal training images <positive_description>. Otherwise, continue to the next section. <positive_description>is a text file listing relative paths to all active training images. After each path <positive_description>is also included a series of numbers indicating how many object instances were found in the image and which sub-rectangles contained those object instances. For each sub-rectangle, the numbers are in the following order: x, y, width and height. Consider the following example:

"images/positive 0.png"  1  120 160 40 40
"images/positive 1.png"  2  200 120 40 60  80 60 20 20

Here, images/positive 0.pngcontains an instance of the target in a subrectangle with the upper left corner (120, 160)and the lower right corner (160, 200). Also, images/positive 1.pnginclude two instances of the target. An instance is in a subrectangle whose upper-left corner is (200, 120), and whose lower-right corner is (240, 180). The other instance is in a subrectangle with the upper left corner (80, 60)and the lower right corner (100, 80).

To create such a file, we can <negative_description>start generating a list of image paths in the same way as with . We then had to manually add data about target instances based on expert (human) analysis of the images.

<opencv_createsamples>Created by running<binary_description>

Assuming we have multiple frontal training images, therefore, we created <positive_description>, which can now be generated by running <binary_description>:

$ <opencv_createsamples> -vec <binary_description> -info <positive_description> -bg <negative_description>

Also, if we have a positive training image, which we'll call <positive_image>, we should instead run the following command:

$ <opencv_createsamples> -vec <binary_description> -image <positive_image> -bg <negative_description>

For other <opencv_createsamples>flags (optional), see the official documentation .

<opencv_traincascade>Created by running<cascade>

Finally, we can generate by running <cascade>:

$ <opencv_traincascade> -data <cascade> -vec <binary_description> -bg <negative_description>

See the official documentation for information about <opencv_traincascade>other (optional) flags .

---

hint

vocalize

For good luck, <opencv_traincascade>make a mocking sound at runtime. For example, say "Moo!" if the positive training image is of a cow.

---

test and improve<cascade>

<cascade>is an XML file compatible with OpenCV's CascadeClassifierclass constructors. For CascadeClassifieran example of how to use it, see the FaceTrackerimplementation in Chapter 4, "Face Tracking with Haar Cascades". . By copying and modifying FaceTrackerand Cameo, you should be able to create a simple test app that draws a rectangle around a tracked custom target instance.

Maybe the first time you try cascade training, you won't get reliable tracking results. To improve training performance, do the following:

• Consider making the classification problem more specific. For example, cascades may be easier to train bald, shaven, male face without glassesthan plain cascades. faceLater, as your results improve, you can try to broaden the problem again.
• Collect more training images, more!
• Make sure <negative_description>to include all negative training images and only negative training images.
• Make sure <positive_description>to include all positive training images and only positive training images.
• Make sure <positive_description>the subrectangle specified in is correct.
• See and try the optional flags with <opencv_createsamples>and . <opencv_traincascade>These flags are described in the official documentation on this page .

Good luck finding images!

Summarize

We have discussed CascadeClassifierthe data and executables used to generate OpenCV-compatible concatenated files. Now you can start collecting images of your favorite things and training a classifier for them!