Series Article Directory

foreword

Some common semantic segmentation camera demos on the blog often have only one frame of the camera image when running on ubuntu18.4, but it seems to be able to run normally on windows. For these codes, I have not been able to find out the specifics of the problem The reason is only suspected to be caused by system differences. Friends who know the cause and solution are welcome to leave a message in the comment area for discussion.

The following shows a demo that can run on ubuntu18.4. The specific effects that can be achieved are:

1. Real-time continuous display of semantically segmented image screens

2. Can be mounted on the designated position on the car

3. The viewing angle can be freely converted in the three-axis coordinate system to realize the display of any viewing angle such as horizontal/bird's eye view

4. Define the corresponding field angle of view and combine the cameras of multiple modules to achieve the surround view effect

5. Store the collected data screen

1. Demo module description

1. Define the corresponding field angle of view, combine the cameras of multiple modules, and achieve the surround view effect requires "360/field angle of view" cameras, here is the effect of three cameras

 #-------------------------- 添加语义分割相机--------------------------#
        sensor_queue = Queue()
        sem_bp = blueprint_library.find('sensor.camera.semantic_segmentation')
        sem_bp.set_attribute("image_size_x",f"{IM_WIDTH}")
        sem_bp.set_attribute("image_size_y",f"{IM_HEIGHT}")
        sem_bp.set_attribute("fov",str(60))
        #场视角，需要组成环视需要“360/场视角”个摄像头

2. The angle of view can be freely converted in the three-axis coordinate system to realize the display of any angle of view such as horizontal/bird's eye view:


        sem01 = world.spawn_actor(sem_bp,carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=60 )),attach_to=ego_vehicle)
        sem01.listen(lambda data: sensor_callback(data, sensor_queue, "sem_01"))
        sensor_list.append(sem01)
        sem02 = world.spawn_actor(sem_bp,carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=0)),attach_to=ego_vehicle)

carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=60 ))

location is the translation position relative to the center of the vehicle chassis, and the adjustment can realize the installation of the camera at any position on the vehicle

Rotation is the rotational position relative to the center of the vehicle chassis

Yaw (yaw): the y-axis of the Euler angle vector
Pitch (pitch): the x-axis of the Euler angle vector
Roll (roll): the z-axis of the Euler angle vector

Here the default Pitch=0 Roll=0, adjust it to achieve any viewing angle display

For the description of Yaw-Pitch-Roll, you can read this blog: Python rotation matrix and Euler angle mutual conversion

3. Continue to add the camera module

    sem02.listen(lambda data: sensor_callback(data, sensor_queue, "sem_02"))
        sensor_list.append(sem02) 
        sem03 = world.spawn_actor(sem_bp,carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=-60)),attach_to=ego_vehicle)
        sem03.listen(lambda data: sensor_callback(data, sensor_queue, "sem_03"))
        sensor_list.append(sem03)   
        
        #-------------------------- 语义分割相机设置完毕 --------------------------#

4. Camera screen combination and visual display


            w_frame = world.get_snapshot().frame
            print("\nWorld's frame: %d" % w_frame)
            try:
                sems = []
                for i in range (0, len(sensor_list)):
                    s_frame, s_name, s_data = sensor_queue.get(True, 1.0)
                    print("    Frame: %d   Sensor: %s" % (s_frame, s_name))
                    sensor_type = s_name.split('_')[0]
                    if sensor_type == 'sem':
                        sems.append(process_semantic(s_data))    

                   
                # 仅用来可视化 可注释
                rgb=np.concatenate(sems, axis=1)[...,:3] # 合并图像
                
                cv2.imshow('vizs', visualize_data(rgb))
                cv2.waitKey(100)
                if rgb is None or args.save_path is not None:
                    # 检查是否有各自传感器的文件夹
                    mkdir_folder(args.save_path)
                    filename = args.save_path +'rgb/'+str(w_frame)+'.png'
                    cv2.imwrite(filename, np.array(rgb[...,::-1]))
                
            except Empty:
                print("    Some of the sensor information is missed")

5. Store data

 if rgb is None or args.save_path is not None:
                    # 检查是否有各自传感器的文件夹
                    mkdir_folder(args.save_path)
                    filename = args.save_path +'rgb/'+str(w_frame)+'.png'
                    cv2.imwrite(filename, np.array(rgb[...,::-1]))

Two, the overall code


import glob
import os
import sys
import time

try:
    sys.path.append(glob.glob('../carla/dist/carla-*%d.%d-%s.egg' % (
        sys.version_info.major,
        sys.version_info.minor,
        'win-amd64' if os.name == 'nt' else 'linux-x86_64'))[0])
except IndexError:
    pass

import carla
import random
import numpy as np
import cv2
from queue import Queue, Empty
import random
random.seed(50)

# args
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--host', metavar='H',    default='127.0.0.1', help='IP of the host server (default: 127.0.0.1)')
parser.add_argument('--port', '-p',           default=2000, type=int, help='TCP port to listen to (default: 2000)')
parser.add_argument('--tm_port',              default=8000, type=int, help='Traffic Manager Port (default: 8000)')
parser.add_argument('--ego-spawn', type=list, default=None, help='[x,y] in world coordinate')
parser.add_argument('--save-path',            default='存储路径', help='Synchronous mode execution')
args = parser.parse_args()

# 图片大小可自行修改
IM_WIDTH = 500
IM_HEIGHT= 500


actor_list, sensor_list = [], []
sensor_type = ['sem']
def main(args):
    # We start creating the client
    client = carla.Client(args.host, args.port)
    client.set_timeout(5.0)
    
    world = client.get_world()
    #world = client.load_world('Town04')
  
    
    
    blueprint_library = world.get_blueprint_library()
    try:
        original_settings = world.get_settings()
        settings = world.get_settings()

        # We set CARLA syncronous mode
        settings.fixed_delta_seconds = 0.05
        settings.synchronous_mode = True
        world.apply_settings(settings)
        spectator = world.get_spectator()

        # 手动规定
        # transform_vehicle = carla.Transform(carla.Location(0, 10, 0), carla.Rotation(0, 0, 0))
        # 自动选择
        transform_vehicle = random.choice(world.get_map().get_spawn_points())
        ego_vehicle = world.spawn_actor(random.choice(blueprint_library.filter("model3")), transform_vehicle)
        actor_list.append(ego_vehicle)

        #-------------------------- 添加语义分割相机--------------------------#
        sensor_queue = Queue()
        sem_bp = blueprint_library.find('sensor.camera.semantic_segmentation')
        sem_bp.set_attribute("image_size_x",f"{IM_WIDTH}")
        sem_bp.set_attribute("image_size_y",f"{IM_HEIGHT}")
        sem_bp.set_attribute("fov",str(60))
        #场视角，需要组成环视需要“360/场视角”个摄像头，这里展示两个摄像头的效果

        # Yaw（偏航）：欧拉角向量的y轴
        # Pitch（俯仰）：欧拉角向量的x轴
        # Roll（翻滚）： 欧拉角向量的z轴
        # sem_transform = carla.Transform(carla.Location(0,0,10),carla.Rotation(0,0,0))
        sem01 = world.spawn_actor(sem_bp,carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=60 )),attach_to=ego_vehicle)
        sem01.listen(lambda data: sensor_callback(data, sensor_queue, "sem_01"))
        sensor_list.append(sem01)
        sem02 = world.spawn_actor(sem_bp,carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=0)),attach_to=ego_vehicle)
        sem02.listen(lambda data: sensor_callback(data, sensor_queue, "sem_02"))
        sensor_list.append(sem02) 
        sem03 = world.spawn_actor(sem_bp,carla.Transform(carla.Location(0,0,1.8),carla.Rotation(yaw=-60)),attach_to=ego_vehicle)
        sem03.listen(lambda data: sensor_callback(data, sensor_queue, "sem_03"))
        sensor_list.append(sem03)   
        
        #-------------------------- 设置完毕 --------------------------#

        # 设置traffic manager
        tm = client.get_trafficmanager(args.tm_port)
        tm.set_synchronous_mode(True)
        # 是否忽略红绿灯
        # tm.ignore_lights_percentage(ego_vehicle, 100)
        # 如果限速30km/h -> 30*(1-10%)=27km/h
        tm.global_percentage_speed_difference(10.0)
        ego_vehicle.set_autopilot(True, tm.get_port())

        while True:
            # Tick the server
            world.tick()
            
            # 将CARLA界面摄像头跟随车动
            loc = ego_vehicle.get_transform().location
            spectator.set_transform(carla.Transform(carla.Location(x=loc.x,y=loc.y,z=35),carla.Rotation(yaw=0,pitch=-90,roll=0)))

            w_frame = world.get_snapshot().frame
            print("\nWorld's frame: %d" % w_frame)
            try:
                sems = []
                for i in range (0, len(sensor_list)):
                    s_frame, s_name, s_data = sensor_queue.get(True, 1.0)
                    print("    Frame: %d   Sensor: %s" % (s_frame, s_name))
                    sensor_type = s_name.split('_')[0]
                    if sensor_type == 'sem':
                        sems.append(process_semantic(s_data))    

                   
                # 仅用来可视化 可注释
                rgb=np.concatenate(sems, axis=1)[...,:3] # 合并图像
                
                cv2.imshow('vizs', visualize_data(rgb))
                cv2.waitKey(100)
                if rgb is None or args.save_path is not None:
                    # 检查是否有各自传感器的文件夹
                    mkdir_folder(args.save_path)
                    filename = args.save_path +'rgb/'+str(w_frame)+'.png'
                    cv2.imwrite(filename, np.array(rgb[...,::-1]))
                
            except Empty:
                print("    Some of the sensor information is missed")

    finally:
        world.apply_settings(original_settings)
        tm.set_synchronous_mode(False)
        for sensor in sensor_list:
            sensor.destroy()
        for actor in actor_list:
            actor.destroy()
        print("All cleaned up!")

def mkdir_folder(path):
    for s_type in sensor_type:
        if not os.path.isdir(os.path.join(path, s_type)):
            os.makedirs(os.path.join(path, s_type))
    return True

def sensor_callback(sensor_data, sensor_queue, sensor_name):
    # Do stuff with the sensor_data data like save it to disk
    # Then you just need to add to the queue
    sensor_queue.put((sensor_data.frame, sensor_name, sensor_data))

# modify from world on rail code
def visualize_data(rgb,  text_args=(cv2.FONT_HERSHEY_SIMPLEX, 0.3, (255,255,255), 1)):
    canvas = np.array(rgb[...,::-1])
    return canvas

# modify from manual control
def process_semantic(image):
    image.convert(carla.ColorConverter.CityScapesPalette)
    array = np.frombuffer(image.raw_data, dtype=np.dtype("uint8"))
    array = np.reshape(array, (image.height, image.width, 4))
    array = array[:, :, :3]
    return array

if __name__ == "__main__":
    try:
        main(args)
    except KeyboardInterrupt:
        print(' - Exited by user.')

3. Effect display

carla segmentation semantic segmentation camera operation recording

4. Expansion: What is Semantic Segmentation

Image semantic segmentation (semantic segmentation), literally means to let the computer segment according to the semantics of the image, for example, let the computer output the right image when inputting the left image below. Semantics in speech recognition refers to the meaning of speech. In the image field, semantics refers to the content of the image and the understanding of the meaning of the picture. For example, the semantics of the left picture is three people riding three bicycles; the meaning of segmentation is from the pixel Different objects in the picture are segmented by angle, and each pixel in the original picture is labeled. For example, pink in the right picture represents a person, and green represents a bicycle.

Semantic Segmentation Current Applications

At present, the application fields of semantic segmentation mainly include:

geographic information system
driverless car
Medical Image Analysis
robotics and other fields

Geographic Information System : By training the neural network, the machine can input satellite remote sensing images, automatically identify roads, rivers, crops, buildings, etc., and label each pixel in the image. (The left side of the figure below is the satellite remote sensing image, the middle is the real label, and the right side is the label result predicted by the neural network. It can be seen that as the training deepens, the prediction accuracy rate continues to improve. Use the ResNet FCN network for training)

**Unmanned vehicle driving: **Semantic segmentation is also the core algorithm technology of unmanned vehicle driving. After the vehicle camera or lidar detects the image and inputs it into the neural network, the background computer can automatically segment and classify the image to avoid pedestrians. and vehicles and other obstacles.

**Medical image analysis:** With the rise of artificial intelligence, the combination of neural network and medical diagnosis has become a research hotspot, and intelligent medical research has gradually matured. In the field of intelligent medical care, the main applications of semantic segmentation are tumor image segmentation, dental caries diagnosis, etc. (The following pictures are dental caries diagnosis, head CT scan emergency care diagnosis assistance and lung cancer diagnosis assistance)
Expand knowledge link: Semantic segmentation of computer vision

CARLA--Vehicles add segmentation semantic segmentation camera-display and store data [super detailed]--[Getting Started-2]