Virtual Avoid Particle Game

Team GAME Section: Virtual Avoid Particle Game

The program launching process along with parameter settings are all simplified and set up on the Jupyter Notebook Environment.

• Open the 01_03_pong_game.ipynb Jupyter Notebook.
• Let’s experience an augmented reality game using Intel RealSense as a team.
• Make sure your display is connected to your Jetson Nano!

(The Jetson Board used for these examples are => Jetson Nano)

---

• 01_03_pong_game.ipynb

• Running the cell code.

  Ctrl + Enter

• When the code runs completely, you will see a window like the one below on your display.

How To Play

• The rules of the game are as follows.

  1. Players can control the green bar left and right.

  2. When the blue ball hits the wall and bounces, your score increases by one.

  3. If the green bar does not receive the bouncing ball, the score is reset and the game starts again from the beginning.

  4. Bounce the ball as long as possible to get a high score.

Code Analysis

• The game code consists of the following.

import cv2
import numpy as np
import pyrealsense2 as rs

# Initialize the RealSense camera
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
pipeline.start(config)

# Game elements and properties
paddle_width = 100
paddle_height = 10
paddle_speed = 10
paddle_color = (0, 255, 0)

ball_radius = 10
ball_speed_x = 5
ball_speed_y = 5
ball_color = (255, 0, 0)

paddle_x = 320 - paddle_width // 2
paddle_y = 460

ball_x = 320
ball_y = 240

ball_direction_x = 1
ball_direction_y = 1

score = 0

# Initialize hand tracking EMA parameters
alpha = 0.2  # EMA smoothing factor, adjust as needed
ema_hand_position = (0, 0)

while True:
    # Wait for a new frame
    frames = pipeline.wait_for_frames()
    depth_frame = frames.get_depth_frame()
    color_frame = frames.get_color_frame()

    if not depth_frame or not color_frame:
        continue

    # Convert RealSense frames to numpy arrays
    depth_image = np.asanyarray(depth_frame.get_data())
    color_image = np.asanyarray(color_frame.get_data())

    # Flip the color image and depth image horizontally (left-right flip)
    color_image = cv2.flip(color_image, 1)
    depth_image = cv2.flip(depth_image, 1)

    # Convert color image to grayscale
    gray_image = cv2.cvtColor(color_image, cv2.COLOR_BGR2GRAY)

    # Threshold the grayscale image to segment the hand region
    _, thresh = cv2.threshold(gray_image, 120, 255, cv2.THRESH_BINARY)

    # Find contours of the hand
    contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    # Get the position of the hand (centroid)
    hand_position = (0, 0)
    if len(contours) > 0:
        contour = max(contours, key=cv2.contourArea)
        M = cv2.moments(contour)
        if M["m00"] != 0:
            cx = int(M["m10"] / M["m00"])
            cy = int(M["m01"] / M["m00"])
            hand_position = (cx, cy)

    # Apply exponential moving average (EMA) to smooth the hand position
    ema_hand_position = tuple(alpha * np.array(hand_position) + (1 - alpha) * np.array(ema_hand_position))

    # Update paddle position based on hand position
    paddle_x = int(ema_hand_position[0]) - paddle_width // 2
    paddle_x = max(paddle_x, 0)
    paddle_x = min(paddle_x, 640 - paddle_width)

    # Update ball position
    ball_x += ball_speed_x * ball_direction_x
    ball_y += ball_speed_y * ball_direction_y

    # Check collision with walls
    if ball_x <= 0 or ball_x >= 640:
        ball_direction_x *= -1

    if ball_y <= 0:
        ball_direction_y *= -1

    # Check collision with paddle
    if ball_y >= paddle_y - ball_radius and paddle_x <= ball_x <= paddle_x + paddle_width:
        ball_direction_y *= -1
        score += 1

    # Check if ball goes out of bounds (reset ball and score)
    if ball_y > 480:
        ball_x = 320
        ball_y = 240
        score = 0

    # Draw paddle and ball on the color image
    cv2.rectangle(color_image, (paddle_x, paddle_y), (paddle_x + paddle_width, paddle_y + paddle_height), paddle_color, -1)
    cv2.circle(color_image, (ball_x, ball_y), ball_radius, ball_color, -1)

    # Display the color image with game elements and score
    cv2.putText(color_image, "Score: " + str(score), (20, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255), 2)
    cv2.imshow("Virtual Pong", color_image)

    # Exit the loop when the 'q' key is pressed
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release resources
pipeline.stop()
cv2.destroyAllWindows()

• Analyze how the code is progressing by team.