The Challenge
One
of the most important aspects of operating an autonomous vehicle is
understanding the surrounding environment in order to make safe
decisions. Udacity and Didi Chuxing are partnering together to provide
incentive for students to come up with the best way to detect obstacles
using camera and LIDAR data. This challenge will allow for pedestrian,
vehicle, and general obstacle detection that is useful to both human
drivers and self-driving car systems.
Competitors will need to
process LIDAR and Camera frames to output a set of obstacles, removing
noise and environmental returns. Participants will be able to build on
the large body of work that has been put into the Kitti datasets and
challenges, using existing techniques and their own novel approaches to
improve the current state-of-the-art.
Specifically, students will be competing against each other in the
Kitti Object Detection Evaluation Benchmark.
While a current leaderboard exists for academic publications, Udacity
and Didi will be hosting our own leaderboard specifically for this
challenge, and we will be using the standard object detection
development kit that enables us to evaluate approaches as they are done
in academia and industry.
New datasets for both testing and training will be released in a format that adheres to the
Kitti standard,
and participants will be able to use all of the associated tools to
process and evaluate their own approaches. While Udacity is currently
producing datasets for the challenge, all participants can get started
today using the existing Kitti data.
Round 1 - Vehicles
The
first round will provide data collected from sensors on a moving car,
and competitors must identify distance and estimated orientation of
multiple stationary and moving obstacles.
Round 2 - Vehicles, Pedestrians
The second round will also challenge participants with identifying pedestrians.
Data/Inputs
Training data will mirror the
Kitti Datasets,
enabling all Kitti data to be used by competitors in order to train and
refine their models. Below is a detailed description of the data
available, retrieved from the
Kitti Raw Data site.
The dataset comprises the following information, captured and synchronized at 10 Hz:
- Raw (unsynced+unrectified) and processed (synced+rectified) grayscale stereo sequences (0.5 Megapixels, stored in png format)
- Raw (unsynced+unrectified) and processed (synced+rectified) color stereo sequences (0.5 Megapixels, stored in png format)
- 3D Velodyne point clouds (100k points per frame, stored as binary float matrix)
- 3D GPS/IMU data (location, speed, acceleration, meta information, stored as text file)
- Calibration (Camera, Camera-to-GPS/IMU, Camera-to-Velodyne, stored as text file)
- 3D object tracklet labels (cars, trucks, trams, pedestrians, cyclists, stored as xml file)
Here,
"unsynced+unrectified" refers to the raw input frames where images are
distorted and the frame indices do not correspond, while
"synced+rectified" refers to the processed data where images have been
rectified and undistorted and where the data frame numbers correspond
across all sensor streams. For both settings, files with timestamps are
provided. Most people require only the "synced+rectified" version of the
files.
Requirements
Using the given data, competitors must:
- Automatically
detect and locate obstacles in 3D space to inform the driver/SDC system
(e.g. using deep learning and classification approaches)
- Fuse detection output from camera and LIDAR sensors
- Remove noise and environment false detections
Evaluation and Judging Criteria
Student submissions will be automatically evaluated using the method put forth by Kitti in their
CVPR 2012 publication,
which uses the PASCAL criteria for object detection and orientation
estimation performance. Specifically, we will be using the “Moderate”
evaluation parameters for ranking, which are specified on the
Object Detection Evaluation Benchmark page.
For ranking purposes, we will be using the “Object Detection and
Orientation Estimation Evaluation” with the “Moderate” difficulty.
Currently, Deep MANTA holds the “Moderate” record of 89.73% performance.
Extensive documentation on getting started with the dataset format and
the evaluation procedure is available on the
Kitti object detection evaluation page, and specifically within the Development Kit which is available
here.
Prizes
First Place – US$100,000 Cash Prize
Second Place – US$3,000 Cash Prize
Third Place – US$1,500 Cash Prize
Top 5 Teams
– Airfare and hotel accomodation for two representatives from each team
to attend the award ceremony in Silicon Valley from their place of
residence, and chance to run code on the Udacity self-driving car.
Timeline
The
submission deadlines are as noted below. All start dates start at 12:00
AM PST and end-dates/deadlines end at 11:59 PM PST on the noted dates.
March 8 – March 21 — Competitor and Team Registration
Competition platform opens for account and team registration
Competitors can register with Udacity accounts or create a new account
Team leaders can recruit team members through Forum
March 22 – April 21 — Round 1
Data set for Round 1 is released
New user and team registration closes on April 21
April 22 – April 30 — Round 1 Evaluation
Top 75 teams will be asked to submit runnable code
Code will be spot-checked to prevent fraudulent submissions
Of that group the Top 50 qualified teams will progress to next round
May 1 – May 31 — Round 2
Data set for Round 2 is released
Teams will no longer be able to add or remove members after May 21
Jun 1 – Jun 14 — Finalist Evaluation
Top teams required to submit identity verification documents and runnable code
Code will be evaluated and output compared against scores on final leaderboard
Top 5 teams will be invited to attend final award ceremony at Udacity headquarters in Mountain View, California
Jun 15 – July 12 — Travel arrangements
5 week break for teams to arrange visas and travel that Udacity will help with
Jul 12 — Final Award Ceremony
Top
5 teams present their solutions to a panel of Udacity and DiDi
executives and have chance to run their code on Udacity’s self-driving
car
Rules
In addition to
Contest Rules, the following administrative rules shall be applied to the Contest:
Eligibility
See the
Contest Rules for all terms of eligiblity and any restrictions.
Teams
- No limit on the number of members per team but all team members must meet the eligibility requirements to participate and win.
- Team
name shall not comprise any terms that violate intellectual property or
rights of privacy or publicity or other applicable laws and
regulations.
- Each team will have an appointed team captain authorized to receive communications.
- Participant registration will close on April 21 11:59 PM PST.
- Team leaders can invite new members until May 21 11:59 PM PST.
- Participants are not allowed to join more than one team at the same time.
Submissions
- Solution must run in the provided ROS framework (Ubuntu 14.04, ROS Indigo)
- Submission
will be considered ineligible if it was developed using code containing
or depending on software that is not approved by the Open Source
Initiative, or a license that prohibits commercial use.
- Code must demonstrably run in real time. This means at least 10Hz on our evaluation platform with a Titan X and I7.
- Each
submission will take 1-2 minutes to submit and process, to prevent
teams from using automatic submissions to guess the solution and
artificially inflate their scores.
- All code submitted will be open-sourced, and there should be no expectation of maintaining exclusive IP over submitted code.
- No hand-labelling of test dataset allowed.
Prizes
- To qualify as finalist and to receive any prizes teams must:
- Submit
runnable code after close Round 1 that will be spot-checked to ensure
that teams that progress to the final round have not violated any
Contest rules.
- Submit runnable code (with documentation and
description of resources/dependencies required to run the solution) with
reproducible results at the close of Round 2. If running code results
in output and score that differs significantly from the last submission
and leaderboard score, that team will be disqualified.
- Submit
completed, signed and returned documentation required by Udacity,
including but not limited to, Affidavit of Eligibility, release of
liability (except where prohibited), publicity release form, and a
completed IRS W-9 or W-8BEN form, from each team member at the close of
Round 2.
- Prizes shall be awarded equally and pro rata
to all registered team members who meet eligibility requirements and
complete Participant Release forms.
- Udacity cannot provide any guarantees that visas will be available from all jurisdictions and to all individuals.
- No cash value for the travel expense subsidy will be awarded to teams that do not travel to attend the final award ceremony.
- “Top 5 teams” includes the first place team.
- At the end of Round 1, teams with the top 50 scores will move on to Round 2.
- At
the end of Round 2, in the event that teams have tying scores, the
run-time of the code submitted to Udacity will be used to determine the
winning team.
Terms & Conditions
For complete set of Terms & Conditions see
here.
