Gun violence in the United States: Impact, Trends, and Defensive Use

2019-08-23 Leave a comment

This is the first post in a series about Reducing Gun Violence in the United States.

In 2017, there were almost 40,000 firearm-related deaths.

To put this into perspective, this is just slightly more than the number of deaths due to motor vehicle crashes (around 37,000).

In order to really understand the scope of the problem of gun violence in the United States, let’s start by looking at the data itself. It is important to know that the data will not tell us why there is so much gun violence, but it will provide context.

For those who want to see the highlights without going through the data, skip right to the conclusions at the bottom of this post.
The data in this post comes from several different sources, which I’ve linked in the references section at the bottom of this post for those who want to see the data for themselves.

Now, let’s dive into how gun violence shows up in the United States.

Firearm violence data

Those roughly 40,000 deaths due to firearms can be broken down into several categories (source CDC Wonder):

Suicide: 60% (23,854)
Homicide: 37% (14,542)
Legal Intervention (typically due to law enforcement activities): 1.4% (538)
Unintentional: 1.2% (495)
Unknown: 0.8% (338)

Slicing this data another way, we can look at the number of people admitted to hospitals with non-fatal gunshot wounds, and how they got there. Note how more than three-quarters of these people were admitted due to assaults (source: CDC Wonder):

Assault: 76% (88,702)
Unintentional: 18% (21,219)
Intentional self-inflicted (failed suicide attempt): 4% (4,357)
Legal intervention: 2% (2,136)

Let’s take a look at the rate of firearm homicides and suicides overall. The number of suicides due to firearms has been significantly bigger than the number of homicides for almost twenty years, with a big increase in suicides since 2006.

Firearm homicide and suicide rates per 100,000 people in the US, 1999-2017 (source: CDC Wonder)

Men have a much greater rate of being involved in firearm violence than women; more than 90% of people killing others with guns are men, and almost 90% of suicide victims are men.

Fractions of firearm homicide and suicide per gender in the US, 2017 (sources: CDC Wonder & FBI Crime in the US)

Three of the biggest racial groups that experience firearm violence in the US are whites, blacks, and Native Americans. There is a much larger rate of homicide (both as offenders and victims) among blacks, though the suicide rates are more evenly divided though slightly higher for whites:

Firearm homicide offenders, homicide victims, and suicide victims per race per 100,000 people in the US, 2017 (sources: CDC Wonder & FBI Crime in the US)

There is a rise in suicide rates as people age:

Ages of suicide victims per 100,000 people in the US, 1999-2017 (sources: CDC Wonder & FBI Crime in the US)

However, the number of Americans involved in homicides by age looks very different. Victims of gun homicide peak around age 21, and those pulling the trigger on those guns also peak around ages 18-20.

Ages of firearm homicide victims per 100,000 people in the US, 1999-2017 (sources: CDC Wonder & FBI Crime in the US)

Ages of firearm homicide offenders per 100,000 people in the US, 1999-2017 (sources: CDC Wonder & FBI Crime in the US)

Looking at regional effects, it’s pretty clear that the rate of homicide is greatest in the Southeast and Midwestern states. The states in red have the highest firearm homicide rates, with the states in light green having the lowest firearm homicide rates:

Firearm homicide rates per state (sources: CDC NHSS and US Census)

And, if we look at firearm suicides, some of the Northern Plains states have the highest rates, likewise with some other states in the interior:

Firearm suicide rates per state (sources: CDC NHSS and US Census)

The highest rates of firearm suicide tend to be in rural and non-metro areas. Firearm homicides are generally highest in large metro areas, but then don’t shrink as dramatically in rural areas as some people would expect:

Firearm homicide and suicide rates per area, 2017 (sources: CDC NHSS and US Census)

Gun Ownership

Let’s now look at data related to gun ownership.

About four in ten US adults say that they live in a house with a gun (source Pew research):

Who are these gun owners?

Breaking down owners among gender and race, it’s more common for whites and men to own guns (source Pew research):

39% of men
22% of women

36% of whites
24% of blacks
15% of hispanics

White men report that almost half of them (48%) own a gun, and the average age for an American to become a gun owner is at age 22.

Let’s look at the educational level of gun owners. The educational level differs significantly only among whites (source Pew research):

40% have a high-school diploma or less
42% have completed some college
26% have a bachelors degree or higher

We can also look at political affiliation: twice as many people who identify as republicans own guns as those who identify as democrats (source Pew research).

Where do gun owners live?

Looking at where gun owners live (source Pew research):

46% of rural residents own a gun
28% of suburban residents own a gun
18% of urban residents own a gun

How about geographical concentration? (source Pew research):

18% of residents in the Northeast of the country own a gun
36% of residents in the South own a gun
32% of residents in the Midwest own a gun
31% of residents in the West own a gun

Why do people own a gun, and what kinds do they own?

People give different reasons for owning a gun, and most people give multiple reasons (source Pew research):

67% gun owners cite self-defense
38% gun owners cite hunting
30% gun owners cite sport shooting

Similar numbers appear for the types of guns that people own (source Pew research):

62% of gun owners own a handgun or pistol
22% of gun owners own a rifle
16% of gun owners own a shotgun

How many guns do gun owners own? Two-thirds of gun owners own multiple guns, and 29% of gun owners own five or more guns (source Pew research).

Gun ownership over time

Gun ownership over time has been gradually decreasing. The following chart shows the percentage of people in the US who own at least one gun (source NORC GSS):

Though the number of people who own handguns or pistols fluctuates without a clear pattern (source NORC GSS):

Defensive gun use, Gun Access, and Public Safety

Incidents of gun violence are spread across the United States (source: Gun Violence Archive):

… while reports of guns used in self-defense are relatively small, and they’re generally located in southern states where gun ownership is more common (source: Gun Violence Archive):

This data is confirmed by Pew Research, which finds that guns are used for self-defense in less than 1% of crimes. This is contrasting with data saying that 67% of gun owners own guns for self-defense.

Another two particularly interesting data points to note are:

4% of people using guns in self-defense actually injured themselves
4% of people not using guns in self-defense also injured themselves

… meaning that there isn’t a benefit to using a gun in self-defense, statistically – you’re just as likely to get hurt as if you didn’t use a gun.

What’s the legality of using a gun in self-defense?

Only 43% of the self-reported use of guns in self-defense are actually deemed to be legal by criminal court judges. In other words, most people who report using guns for self-defense aren’t legally allowed to do what they did. (source: Gun Use in the US)

Certainly, some use of self-defense is legally justifiable, just not all of it.

What about anger and impulsivity?

9% of US adults (4 million people) have a history of angry and impulsive behavior and have access to firearms in their home.

Those adults who own six or more guns are four times more likely to carry a gun and show angry and impulsive behavior than those who own just one gun (source: Anger / Impulsive)

Conclusions

Firearm Violence Impact

40,000 people a year die from firearm violence
- 60% – almost 24,000 – are suicides
- almost 40% – about 14,000 – are homicides
Suicide rates have been steadily increasing since 2006
Men are nine times as likely to be involved in homicide or suicide as women
Suicide risk rises with age
Homicide risk (both being an offender and victim) peaks around age 20
Northern plains and some Southern states have the highest rates of suicide
- Higher rates in rural and non-urban areas
Southern and Midwestern states have the highest rates of homicide
- No strong relationship between homicide rates and population density (urban vs rural)

Gun Ownership

Four out of ten US adults live in a home with a gun
The most common group of people who identify themselves as gun owners are white men who have not completed college, live in rural areas, and are republicans
The majority of gun owners own handguns for self-defense
Gun ownership has been decreasing over time to 32% of the population

Defensive gun use

Guns are used in self-defense for less than 1% of crimes
- And 4% of those times, the person using the gun for self-defense hurts themselves
- That’s the same fraction of people that hurt themselves in self-defense without a gun
Most people who use guns for self-defense do so in a way that courts view as illegal
9% of US adults own a gun and have a history of angry and impulsive behavior
Adults who own six or more guns are four times more likely to carry a gun in public and show angry and impulsive behavior than those who just own one gun

Next up: Gun Markets, Ownership, and Violence Risk

References

CDC Wonder – a public resource of health-related datasets for CDC staff, public health departments, researchers, and the general public. The year 2017 is the most recent year for which comprehensive, compiled data exists for firearm violence
FBI Crime in the US 2017 – a statistical compilation of offense, arrest, and police employee data reported by law enforcement agencies voluntarily participating in the FBI’s Uniform Crime Reporting (UCR) Program
CDC National Vital Statistics System (NVSS) – a public resource for official vital statistics
US Census Bureau – providing aggregate data for public use
Pew Research – published “America’s Complex Relationship With Guns”
National Opinion Research Center’s General Social Survey (NORC GSS) – a public report for policymakers and scholars to give an unbiased and clear perspective on what Americans think about general social topics
Gun Violence Archive – a free 2019 dataset providing public access to accurate information about gun-related violence in the United States
Gun use in the United States – a survey of guns used in self-defense and evaluation by criminal judges
Anger / Impulsive – a national report on the co‐occurrence of impulsive angry behavior and possessing or carrying a gun among adults

Autonomous vehicle technology: System Integration with ROS

2018-08-13 Leave a comment

The self-driving systems in autonomous vehicles have four major components: sensors, perception software, planning software, and control software. Within each of the software components, multiple modules execute concurrently to process vast amounts of data collected from the sensor systems. This processing eventually results in control outputs which steer, accelerate, and brake the vehicle at the right moments to get the vehicle safely and efficiently to its destination while providing a pleasant ride for its passengers.

Each of these systems in an autonomous vehicle operate with input from upstream components: for example, the lane detection module might input data from a camera; the localization system might input data from a lidar and gps module; and the vehicle behavior prediction module might input data from the lane detection, traffic light detection, traffic sign classification, and object detection and tracking systems. An example data flow graph between all of the software modules in an autonomous vehicle might look like this:

Self Driving vehicle Architecture

Wiring up all of the various inputs and outputs from each module, given the different types of data and refresh rates provided by each module, is non-trivial and time-consuming. Using a software framework specifically designed for this, the Robot Operating System (or ROS for short), can make this challenge significantly easier, by abstracting away hardware details, managing data flow between software modules, and hosting common re-usable code.

Together with several other team members, I implemented a completely autonomous vehicle software system (including perception, planning, and control) using ROS, which was then used to run on a real-life Lincoln MKZ hybrid vehicle to autonomously drive around a test lot.

Exploring my implementation

All of the code and resources used in this project are available in our team’s Github repository. Enjoy!

Technologies Used

C++
Python
ROS
Flask
Keras
Tensorflow
NumPy
SciPy
Python Imaging Library
OpenCV

System Integration with ROS

Autonomous Vehicle Systems

Within an autonomous vehicle, each vehicle component has a specific task:

Sensors are the hardware components that gather raw data about the environment. They include LIDAR, RADAR, cameras, and even GPS sensors.

Perception systems (including detection and localization components) process raw sensor data into meaningful structured information about the environment around a vehicle. The detection components are responsible for lane line detection, traffic sign and light detection and classification, object detection and tracking and free space detection. Localization components use sensor and map data to determine the vehicle’s precise location.

Planning systems use the output from the perception systems to create driving behaviors and to plan short and long-term paths for the vehicle to follow. Planning components include high-level route planning from start to destination on a map, prediction of what other objects on the road might do, behavior planning to decide what specific actions the vehicle should take next, and trajectory generation which plots the precise path the vehicle should follow.

Control systems ensure that the vehicle follows the path given by the planning system and send commands to the vehicle’s hardware for safe driving. These include sending acceleration, braking, and steering commands.

Information generally flows from top to bottom, starting with sensor systems, then moving through perception, planning, and finally to control systems.

System integration

ROS is an open-source robotics framework which solves many of the challenges of communication and modularity that come with integrating all of these vehicle systems. It is widely used in industry and academia and provides libraries and tools for common tasks involving transferring data between modules, dealing with sensor and control actuator interaction, logging, error handling, process scheduling, and interfacing with common third-party libraries. Visualization, simulation, and data analysis tools are also provided, which are incredibly useful for building and debugging a completely autonomous vehicle. Altogether, ROS lets autonomous vehicle engineers focus on the algorithms they wish to implement for each module rather than spending time on the “glue” that holds the entire system together.

An autonomous vehicle system in ROS is built by writing perception, planning, and control algorithms into modules called nodes, which are containers for algorithms. Nodes are designed to be lightweight and focused one specific task; each sensor, perception component, planning component and actuator may have its own node. A special node, called the ROS master, manages all nodes and facilitates communication between them. It allows nodes to identify each other, communicate, and to listen for and request parameters, which are ways to tune each node’s behavior quickly and easily. An autonomous vehicle may have hundreds of nodes, depending on the complexity of the vehicle being controlled.

Here is a graph of the nodes in a moderately complex robot:

SDC Many ROS nodes

Nodes share data with each other by using message passing on topics, whose messages are being read by one or more nodes. For example, a lidar node may write a new sensor measurement to a lidar topic, and the message will be read by all nodes which subscribe to that topic, for example, a free-space detection node and a localization node. ROS comes with many different message types, which are useful for sending data typically associated with autonomous vehicles: sensor measurements, steering wheel angles, images, point clouds, and custom message types that developers can create.

Here is a small graph of nodes and the types of messages that they send:

SDC Node communication

Sometimes, having a system which can listen for requests and respond to them is useful; ROS supports services which do precisely this. Services operate in much the way that a typical web service waits for a request, processes the input, computes a response, then sends it back to the requestor. For example, a camera node would publish many images per second on a camera topic (for any subscribers who may be listening), but if a behavior planner only wants to process an image from a camera every few seconds, then a camera service would be useful to return the current camera image that is available.

Training Dataset
A training dataset was prepared by combining simulator data with data provided by other vehicles which ran on the test track.

SDC ROS Service

Implementation

Together with several other team members, I implemented a completely autonomous vehicle software system (including perception, path planning, and control) using the ROS framework, which was then used to run on a real-life Lincoln MKZ hybrid vehicle to autonomously drive around a test lot.

The real vehicle has a Dataspeed drive-by-wire (DBW) interface for throttle, brake, and steering control, a forward-facing camera for traffic light detection, and LIDAR for localization. Autonomous vehicle software is run on a Linux PC with ROS and a TitanX GPU for TensorFlow processing.

Development proceeded using a local simulator, after which successful testing proceeded to a real-life vehicle in a test lot.

System Architecture

SDC System Integration ROS Graph

The autonomous vehicle system starts by loading mapped route base waypoints (locations to follow on the test track) in the planning system’s Waypoint Loader and setting an overall maximum speed guard for each waypoint. This initial setup is used to protect for safe operation in the test lot.

The system then starts receiving the vehicle’s sensor data: current pose from LIDAR localization, current speed, DBW enable switch and camera images.

The perception system’s Traffic Light Detection Node processes camera images to detect traffic lights, and to decide if and where the vehicle needs to stop at an upcoming waypoint location.

The planning system’s Waypoint Updater Node plans the driving path target speed profile by setting upcoming waypoints with associated target speeds. This includes smoothly accelerating up to the target maximum speed and slowing down to stop at detected red lights. This is accomplished by using a smooth speed profile planner using a Jerk Minimizing Trajectory (JMT) following dynamic red light stopping locations.

The control system’s Waypoint Follower sets target linear speed (from the planned waypoint target speeds) and target angular velocity to steer toward the waypoint path.

The control system’s Drive-By-Wire Node computes throttle, brake, and steering commands using PID feedback control for throttle and brake, and a kinematic bicycle model yaw control for steering, both of which are then low-pass filtered to eliminate rapid back-and-forth control changes. These commands are sent to the Dataspeed drive-by-wire system to actuate the vehicle’s pedals and steering wheel.

Traffic light detection

A teammate worked on this module; their description of their work is adapted here.

The traffic light detection node subscribes to the following topics:

image_color provides an image stream from the vehicle’s camera. These images are used to determine the color of upcoming traffic lights
current_pose provides the vehicle’s current position
base_waypoints provides a complete list of waypoints of the vehicle’s course
vehicle/traffic_lights provides the (x, y, z) coordinates of all traffic lights. This helps to acquire an accurate ground truth data source for the traffic light classifier by sending the current color state of all traffic lights in the simulator for testing purposes, before running on the real vehicle

The node publishes the index of the waypoint for the nearest upcoming red traffic light’s stop line to the traffic_waypoint topic.

SDC System Integration Traffic Light Detector ROS graph

The camera image data is used to classify the color of the upcoming traffic light using a classifier.

Traffic Light Classifier

Architecture

A classifier was built using transfer learning on a ResNet50 convolutional neural net architecture. This pre-trained Keras model was previously trained on the ImageNet dataset. The first 150 layers of the model were frozen, and the top layers were removed. One average pooling layer, one fully connected layer with ReLU activation, two dropout layers, and one softmax loss layer with four classes (Green, None, Red, Yellow) were added. An Adam optimizer was used during the training with a low learning rate (0.00001).

Training Dataset

A training dataset was prepared by combining simulator data with data provided by other vehicles which ran on the test track.

Preprocessing

To pre-process input data, training images were resized to (224,224) which is the input to ResNet50 architecture. Next, the images were randomly adjusted to images seen the camera in true driving conditions, by:

rotation by a random angle (+/- 5 degrees)
random horizontal flipping
random zooming (+/- 0.3)

Traffic Light Detection State Machine

The determination of the color of the traffic light is made based on a state machine which uses the previous light state information to determine the allowed light states and compare it with the classified light state. Each predicted state has to be predicted several of times before it becomes trusted as the correct state of the traffic light. If the classified light state does not match with any of the allowed light states, then the light state does not change.

Traffic Light Classifier State Machine

Closest Waypoint

The base waypoints provide the complete list of waypoints for the course. Using the vehicle’s current location and the coordinates for the traffic lights, the nearest visible traffic light ahead of the vehicle is identified. The closest waypoint index is published to the waypoint updater node if the vehicle is approaching nearest light.

Waypoint Updater

A teammate worked on this module; their description of their work is described here.

The waypoint updater node has two main tasks:

Waypoint preparation – Load the waypoints from the /base_waypoints topic and prepare them for use during planning
Planning
- Use pose messages from /current_pose topic to identify the location of the closest waypoint in front of the vehicle
- Use messages from /traffic_waypoint topic to decide whether the vehicle should speed up, maintain current speed, or come to a stop. Also, it should calculate an appropriate trajectory to accomplish this. A set of waypoints and velocities is then published to the /final_waypoints topic for the waypoint follower to consume

Trajectory Velocity Calculations

Jerk minimizing trajectory planning is used to produce speedup and slowdown profiles that can be merged at any point based on changing traffic signal status without exceeding 10m/s^2 of acceleration or 10 m/s^3 of jerk, which would cause passenger discomfort.

Loading the Waypoints

The waypoints are loaded into a custom structure which stores the waypoint data, the distance of the waypoint (through preceding waypoints) from the start of the track, the maximum velocity allowed for that waypoint, and the velocity, acceleration, jerk, and elapsed time variables used in the jerk minimizing trajectory planning calculations.

The vehicle coordinates of each of the waypoints are also loaded into a kd-tree to enable finding the closest waypoint to the vehicle. This search is used as a backup method to find the closest waypoint. Tests have shown the kd-tree to be very fast and usable for up to at least 11,000 waypoints; however, it is generally only used by the waypoint_updater at startup.

Once the waypoints have been loaded and at least one pose message has been received, the updater loop begins. The loop is set to run at approximately 30 Hz.

Trajectory Planning

The first stage uses the most recent pose data to locate the closest waypoint ahead of the vehicle on the waypoint list. Since the vehicle is expected to follow the waypoints, a local search of waypoints ahead and behind the waypoint identified in the last cycle is conducted to try to find the vehicle location as quickly as possible.

A check is then made to see if a traffic_light message has been received from the traffic light detection system since the last cycle. Messages are stored and only updated at the start of each cycle to prevent race conditions when the traffic signal information changes during an ongoing cycle.

The vehicle can be in one of five states:

Stopped – the vehicle is not moving and is waiting at a red or yellow light
Creeping – the vehicle is moving at a steady minimum velocity up to a stop line
Speedup – the vehicle is accelerating toward the default cruising velocity
Maintainspeed – the vehicle is maintaining constant default cruising speed
Slowdown – the vehicle is decelerating to stop just behind the stop line

If the vehicle is moving (in speedup,maintainspeed, or slowdown states), the minimum stopping distances are calculated to prevent exceeding maximum desired jerk and maximum allowed jerk using the current velocity and acceleration of the vehicle as starting conditions. This remains an area of investigation, as a simple, fast and accurate method to determine the minimum distance required to prevent exceeding a specific jerk level has not been achieved. The current method is functional but is thought to overestimate the necessary distance. This is a critical issue, as it is key in the decision as to whether the vehicle can stop for a signal.

During tests on in simulator, it was found that the minimum stopping distance calculated at higher speeds (~ 60 m for 60 km/hr) to meet maximum jerk limits occasionally resulted in not being able to stop for a red traffic light which was detected within 60m of the line. At 40 km/hr the minimum stopping distance was below 30m which could still result in the vehicle going through a yellow light but was much less likely to result in moving through a red light. The slower speed also gave more time for the traffic light classifier to operate.

If the distance to a signaling light is between the two calculated stopping distances, the time required to complete the slowdown is optimized to produce the lowest jerk along the trajectory, and the slowdown trajectory is calculated and published. If the distance to the traffic light is greater than the larger stopping distance, then the vehicle will continue on its current plan. If the distance to the traffic light is less than the minimum calculated stopping distance, then the planner will proceed past the signal – this should correspond to driving through a yellow light.

A decelerating message is published when the planner has put the vehicle in slowdown or stopped to assist the DBW Node with braking control.

In the event that the vehicle is in slowdown or stopped, and the distance to the next signaling traffic light increases beyond the calculated stopping distance, a new trajectory is calculated to accelerate the vehicle to the default velocity. Maximum desired jerk is used as the criteria to calculate the distance and time over which this trajectory should be completed. This algorithm is not optimal, but overestimating the minimum distance required to accelerate without exceeding a specific jerk limit is not as critical as being able to stop in time for a stop signal.

A special creeping state was used for the condition where the vehicle is stopped a short distance from a stop signal. If the vehicle is within the creeping setback from the light, the planner will just use the minimum moving velocity on waypoints until the vehicle gets to the traffic light buffer setback, where velocity is returned to zero. This prevents short acceleration/braking cycles at the light, simulating a driver just releasing the brake to ease forward.

The upcoming waypoints are published to the /final_waypoints topic. It was found that using 40 waypoints in the simulator was sufficient for use by the waypoint follower.

The calculation time required in each cycle was low enough in the conditions tested that no algorithm was necessary to account for latency when publishing the waypoints.

Waypoint Follower

A teammate worked on this module; their description of their work is described here.

In order to drive along the planned path consisting of waypoints with associated target speeds and yaw angles, an implementation of the Pure Pursuit algorithm is used.

This algorithm receives the upcoming waypoints from the Waypoint Updater and publishes a target linear velocity and angular velocity for the DBW Node to control throttle, brake, and steering actuators.

Pure Pursuit Algorithm Overview

The Pure Pursuit algorithm finds the radius of a circular path that is tangent to the current orientation of the vehicle and crosses through a point on the target driving path some distance ahead (called “the look-ahead point”), then calculates the corresponding target angular velocity to follow it. As the vehicle drives forward and steers along this arc, the look-ahead point continues to be pushed further ahead so the vehicle gradually approaches and straightens along the path.

The amount of look-ahead distance alters the sharpness of the steering angles, where a short look ahead causes larger steering angles but can follow sharp turns, and a large look ahead causes smaller steering angles but may cut the corners of the driving path. The look-ahead distance is variable in proportion to the vehicle speed.

The target linear velocity from this algorithm passes through the target velocity associated with the closest waypoint.

Autoware Open-source Pure Pursuit Library

The Autoware open-source ROS module called “Waypoint Follower” has been included to perform the pure pursuit algorithm. However, the base C++ algorithm had some areas that needed further improvement:

Tuned the judgment thresholds for distance and relative angle thresholds to keep more continuous steering adjustments to prevent ping-ponging around the target path
Perform closest waypoint search to pick up the correct waypoint speed targets when the vehicle has driven past the initial waypoints in the list (in case of lag or Waypoint Follower updating at a faster rate than Waypoint Updater)
Perform look-ahead waypoint search starting from the closest waypoint instead of the initial waypoint in the list to prevent searching far behind the vehicle
Tune the minimum look-ahead distance to be able to steer around tighter turns (such as the test lot)
Fix the possibility for negative target velocity command at high lateral velocities

Drive-By-Wire Node

The autonomous vehicle for which this system was written uses a Dataspeed drive-by-wire interface. The interface accepts a variety of control commands, including (but not limited to):

throttle – for pedal position
brake – for braking torque (or pedal position)
steering – for steering wheel angle

The job of the DBW control node in this software is to publish appropriate values to these three control command interfaces, based on input from upstream message sources:

twist_cmd – target linear and angular velocity published by the waypoint follower
current_velocity – current linear velocity published by the vehicle’s sensor system
dbw_enabled – flag indicating if the drive-by-wire system is currently engaged
is_decelerating – flag indicating if the waypoint updater is attempting to decelerate the vehicle

ROS Node

The main DBW ROS node performs the following setup upon being created:

accepts a number of vehicle parameters from the vehicle configuration
implements the dbw_node ROS topic publishers and subscribers
the four ROS topic subscribers (twist_cmd, current_velocity, dbw_enabled, is_decelerating) assign various instance variables, used by the Control class, once extracted from the topic messages
creates a Controller instance to manage the specific vehicle control
enters a loop which provides the most recent data from topic subscribers to the Controller instance

The loop executes at a target rate of 50Hz (any lower than this and the vehicle will automatically disable the DBW control interface for safety). The loop checks if the DBW node is enabled, and all necessary data is available for the Controller, then hands the appropriate values (current and target linear velocity, target angular velocity, and whether the vehicle is attempting to decelerate) to the Controller. Once the Controller returns throttle, brake, and steering commands, these are published on the corresponding ROS interfaces.

Controller

A Controller class manages the computation of throttle, brake, and steering control values. The controller has two main components: speed control and steering control.

The Controller, upon initialization, sets up a Yaw Controller instance for computing steering measurements, as well as three Low Pass Filter instances for throttle, brake, and steering.

Speed control

At each control request, the following steps are performed:

Compute the timestep from the last control request to the current one
Compute the linear velocity error (the difference between target and current linear velocity)
Reset PI control integrators if the vehicle is stopped (has a zero target and current linear velocity); more on the integrators later

Next, the raw throttle and brake values are computed. The basic design:

adds variable throttle if the vehicle is accelerating or vehicle is slowing down but not significantly enough to release the throttle entirely
adds variable braking if the vehicle is traveling too fast relative to the target speed (and simply releasing throttle will not slow down fast enough)
adds constant braking if the vehicle is slowing down to a stop

Once the raw throttle and braking values are computed, the raw braking value is sent through a low pass filter to prevent rapid braking spikes. If the resulting value is too small (below 10Nm), the braking value is reduced to zero; else, the throttle is reduced to zero. This is to prevent the brake and throttle from actuating at the same time. Finally, the throttle value is sent through a separate low pass filter to prevent rapid throttle spikes.

Steering control

The target linear velocity, target angular velocity, and current linear velocity are sent into the Yaw controller. This controller computes a nominal steering angle based on a simple kinematic bicycle model. Finally, this steering value is sent through its own low pass filter to smooth out final steering commands.

Visualization Tools

A teammate worked on this module; their description of their work is described here.

Some visualization tools were built for monitoring data while running the simulation using the ROS tools RQT and RViz.

RQT

Here is a sample of the RQT view including dynamic parameters, xy position on the track, throttle, speed, and braking values.

SDC Visualization RQT

RViz (3D scene view)

To visualize the vehicle’s position relative to the waypoints and traffic lights in an RViz 3D scene, a visualization node is used to publish a /visualization_marker_array topic to populate the objects and a /visualization_basewp_path topic to populate a path line.

The visualization node publishes when the parameter vis_enabled == True. When using the actual vehicle, the visualization node is not launched to reduce bandwidth.

Here is a sample view of the RViz 3D scene view:

SDC Visualization RViz

Testing

Testing the fully integrated system proceeded in multiple ways. First, the traffic light classifier was tested and debugged with real-world camera images. Once complete, the fully integrated system was run locally against a vehicle simulation on a highway track and a test lot. The data visualization and analysis tools helped to find issues and correct them as they appeared.

This video shows a successful run of the vehicle in the simulator:

Results

For the final test, the fully integrated system was loaded onto a real vehicle to run in a simulated test lot. After several testing and debugging iterations, the final test was a complete success. The vehicle was piloted up to a traffic light, where it stopped for a red light, then proceeded on a green. It drove very smoothly with appropriate acceleration around the test track and followed the route around the test lot without difficulty.

Autonomous Vehicle Technology: Semantic Segmentation for Scene Understanding

2018-02-11 Leave a comment

A huge portion of the challenge in building a self-driving car is environment perception. Autonomous vehicles may use many different types of inputs to help them perceive their environment and make decisions about how to navigate. The field of computer vision includes techniques to allow a self-driving car to perceive its environment simply by looking at inputs from cameras. Cameras have a much higher spatial resolution than radar and lidar, and while raw camera images themselves are two-dimensional, their higher resolution often allows for inference of the depth of objects in a scene. Plus, cameras are much less expensive than radar and lidar sensors, giving them a huge advantage in current self-driving car perception systems. In the future, it is even possible that self-driving cars will be outfitted simply with a suite of cameras and intelligent software to interpret the images, much like a human does with its two eyes and a brain.

Semantic segmentation helps when asking the question “where is an object in a given image?”, which is a technique which is incredibly important in the field of scene understanding. Standard convolutional neural networks (which start with convolutional layers, followed by fully connected layers, followed by a softmax or other activation function) are great for classifying objects in an image. However, if we need to identify where in an image an object exists, we need a slightly different architecture. For example, if we want to highlight the road in a video stream, this kind of task applies.

Road Identified 1

This repository contains a software pipeline which identifies the sections of an image which represent the road in images from a front-facing vehicle camera. The following techniques are used:

Start with a pre-trained VGG model, used for image classification
Remove the final fully connected layers
Add 1×1 convolutions, upsampling, and skip layers
Optimize the network with inference optimization techniques
Retrain the network on labeled images from the KITTI Road Detection dataset

Exploring my implementation

All of the code and resources used in this project are available in my Github repository. Enjoy!

Technologies used

Python
Tensorflow

Scene understanding

Scene understanding is important to an autonomous vehicle’s ability to perceive its environment.

One method in scene understanding is to train multiple decoders on the same encoder; for example, one decoder for semantic segmentation, one for depth perception, etc. In this way, the same network can be used for multiple purposes. This project focuses solely on semantic segmentation.

Techniques for semantic segmentation

Fully Convolutional Networks

Fully convolutional networks, or FCNs, are powerful tools in semantic segmantation tasks. (Several other techniques have since improved upon FCNs: SegNet, Dialated Convolutions, DeepLab, RefineNet, PSPNet, Large Kernel Matters to name a few.) FCNs incorporate three main features beyond that of standard convolutional networks:

Fully-connected layers are replaced by 1×1 convolutional layers, to preserve spatial information that would otherwise be lost
Upsampling through the use of transpose convolutional layers
Skip connections, which allow the network to use information from multiple resolutions to more precisely identify desired pixels

Fully Convolutional Network Architecture

Structurally, a fully convolutional network is comprised of an encoder and a decoder.

The encoder is a series of standard convolutional layers, the goal of which is to extract features from an image, as in a traditional convolutional neural network. Often, encoders for fully convolutional networks are taken from VGG or ResNet, being pre-trained on ImageNet (another example of the power of transfer learning, another project I worked on.)

The decoder upscales the output of the encoder to be the same resolution as the original input, resulting in prediction or “segmentation” of each pixel in the original image. This happens through the use of transpose convolutional layers. However, even though the decoder returns the output in the original dimensions, some information about the “big picture” of the image (no pun intended) is lost due to the feature extraction in the encoder. To retain this information, skip connections are used, which add values from the pooling layers in the encoder to the output of the corresponding sized decoder transpose convolutional layers.

Performance enhancements

Because semantic segmentation performance on state of the art autonomous vehicle hardware may not be able to process a video stream in real-time, various techniques can be used to speed up inference by using less processing and memory bandwidth.

Freezing graphs – by converting variables in a Tensorflow graph into constants once trained, memory costs decrease and model deployment can be simplified
Fusion – by combining adjacent network nodes without forks, operations which would previous have used multiple tensors and processor executions can be reduced into one
Quantization – by reducing precision of floating point constants to integers, memory and processing time can be saved
Machine code optimization – by compiling the various system startup and load routines into a binary, overhead in inference is greatly reduced

Network architecture for semantic segmentation

A modified version of the impressive VGG16 neural network image classification pipeline is used as a starting point. The pipeline takes a pre-trained fully-convolutional network based on Berkeley’s FCN-8 network and adds skip layers.

From the originally inputted layer, the input, keep probability, and layers 3, 4, and 7 are extracted for further use.

Next, 1×1 convolutions are constructed from layers 3, 4, and 7 in an encoding step. Skip layers are inserted by adding the 1×1 convolutions from layers 3 and 4. Layers 3, 4, and 7 are deconvolved in reverse order to complete the final piece of the decoding step.

An Adam optimizer is used to minimize the softmax cross-entropy between the logits created by the network and the correct labels for image pixels.

The neural network is trained using a sample of labeled images for a maximum of fifty epochs. A mini-batch size of ten images is used compromise between high memory footprint and smooth network convergence. The training step has an early terminator which does not continue to train the network if total training loss does not decrease for three subsequent epochs.

Finally, a separate held-out sample of test images are run through the final neural network classifier for evaluation.

Results

Overall, the semantic segmentation network designed works well. The road pixels are highlighted in the test images with close to a human level of accuracy, with an occassional windshield or sidewalk highlighted as a road, and some road areas with shadows are missed.

Some example images segmented by the pipeline:

Road Identified 1

Road Identified 2

Road Identified 3

Road Identified 4

Future improvements

Use the Cityscapes dataset for more images to train a network that can classify more than simply road / non-road pixels
Augment input images by flipping on the horizontal axis to improve network generalization
Implement another segmentation implementation such as SegNet, Dialated Convolutions, DeepLab, RefineNet, PSPNet, or Large Kernel Matters (see this page for a review)
Apply trained classifier to a video stream

Autonomous Vehicle Technology: Path Planning

2018-01-22 Leave a comment

Path planning systems enable autonomous vehicles to generate safe, drivable trajectories to get from one location to another. Given information about its environment from computer vision and sensor fusion (such as vehicle detection and localization systems) and a destination, a path planner will produce speed and turning commands for control systems to actuate.

Highway path

A path planner includes three main components: a vehicle predictor, a behavior planner, and a trajectory generator. Vehicle prediction involves estimating what other vehicles in the local environment might do next. Behavior planning decides what action to take next, given the goal and the estimates from the vehicle prediction step. Trajectory generation computes an actual path to follow based on the output from the behavior planning step.

I created a vehicle path planner implementation which safely navigates a simulated vehicle around a virtual highway with other traffic. The vehicle attempts to keep as close to the speed limit as possible, while maintaining safe driving distances from other vehicles and the side of the road, and providing a comfortable riding environment for any passengers inside. This may involve switching lanes, passing vehicles, speeding up and slowing down.

Exploring my implementation

All of the code and resources used in this project are available in my Github repository. Enjoy!

Technologies Used

C++
uWebSockets
Eigen
Cubic Spline Interpolation

Path planning

Path Planning steps

Search and Cost Functions

The fundamental problem in path planning is to find the optimal path from the start to the goal, given a map of the world, locations of all objects in the map, a starting location, a goal location, and a cost function. The “cost” in this case refers to the penalty involved in taking a particular path; for example, driving primarily on two lane 25 MPH roads to navigate between New York City and Los Angeles would have a high cost; most of the driving instead should occur on Interstate highways, which would have a much lower cost. Various actions that the vehicle can take incur different costs; left turns, right turns, driving forward, driving backward, stopping, and other actions will all incur costs, which may vary depending on the current speed or location of the vehicle.

Various algorithms exist for searching for a path; grid– and interval-based search algorithms (such as A*) are popular, and many other algorithms involving rewards or probabilistic sampling exist as well.

In reality, because the environment that an autonomous vehicle drives in is non-deterministic (other vehicles, road obstacles, and traffic markers may appear without warning), a path planner may choose to use a dynamic programming-based approach to plan for the optimal path to follow given multiple possible starting locations (called a policy).

Prediction

Prediction systems estimate what other vehicles will do in a given environment in the near future. For example, when a vehicle is detected by the perception systems, the prediction system would be able to answer “where will the vehicle to be in five seconds?” This happens by taking as input a map of the world and data from sensor fusion, and generating the future state of all other vehicles and moving objects in the local environment. Prediction systems generate multi-modal probability distributions, which means that the estimates they create may have multiple likely outcomes. Information about what vehicles generally do at certain locations in the world, coupled with the current and recent past behavior of a vehicle, inform the prediction to make certain outcomes more or less likely. For example, a car slowing down as it approaches a turn increases the likelihood of a turn.

Prediction algorithms can be model-based (where process models of vehicles are informed by actual observations) or data-driven (using, for example, a machine learning model to extract insight from raw sensor fusion data). Both algorithms have strengths: model-based approaches use constraints encoded in allowable traffic patterns and road conditions, while data-driven approaches may learn subtle nuances of behavior which may be difficult to model. Hybrid approaches, which use model- and data-based algorithms, are often the best to use. For example, the Gaussian Naive Bayes classifier is often used with appropriate features which are highly representative of specific trajectories to predict vehicle behavior.

Behavior Planning

Behavior planning answers the question of “what do to next” in an autonomous vehicle. Behavior planners use inputs from a map, route to destination, and localization and prediction information, and output a high-level driving instruction which is sent to the trajectory generator. Behavior planners do not execute as frequently as other autonomous vehicle systems (such as a sensor fusion system), partially because they require a large volume of input to generate a plan, but also because their output plans will not change as frequently as outputs from other systems. A behavior planner acts as a navigator, whose output might be “speed up”, or “get in the right lane”, including behaviors which are feasible, safe, legal, and efficient.

Behavior planners often use finite state machines, which encode a fixed set of possible states and allowable transitions between states. For example, when driving on a highway, the states might include “stay in lane”, “change to right lane”, or “get ready for left lane change”. Each state has a specific set of behaviors; for example, “stay in lane” requires staying in the center of the lane and matching speed with the vehicle ahead up to the speed limit. Selecting a new state from a given state requires a cost function, which allows for selecting the optimal state for driving to the goal. Cost functions often include as inputs current speed, target speed, current lane, target lane, and distance to goal, among others.

Trajectory generation

A trajectory generator creates a path that the vehicle should follow and the time sequence for following it, ensuring that the path is safe, collision-free, and comfortable for the vehicle and passengers. This happens by using the current vehicle state (location, speed, heading, etc), the destination vehicle state, and constraints on the trajectory (based on a map, physics, traffic, pedestrians, etc). Many methods exists for this: combinatorial, potential field, optimal control, sampling, etc. For example, Hybrid A* is a sampling method, which looks at free space around a vehicle to identify possible collisions, and is great for unstructured environments such as parking lots (without many constraints).

Polynomial trajectory generation works well for highways with more strict and predefined rules about how a vehicle can move on a road. A jerk (rate of change of acceleration) minimizing polynomial solver generates continuous, smooth trajectories that are comfortable for passengers. A jerk minimizing solver may take into account minimum and maximum velocities, minimum and maximum longitudinal (forward and backward) acceleration, maximum lateral (side-to-side) acceleration, and steering angle, among other inputs. Because the behavioral plan typically does not include an exact final state (speed, direction, location, etc), generating several similar concrete final states and computing jerk minimizing trajectories for all of them is typical. Then, the trajectory with the lowest cost which is drivable and does not collide with other objects or road boundaries is chosen. In this case, a cost function might include jerk, distance to other objects, distance to the center of a lane, time to the goal, or many other items.

Implementation

In this project, the goal is to safely navigate around a virtual highway with other traffic that is driving within 10 MPH of the 50 MPH speed limit and may change lanes (as other vehicles often do). The input provided is the vehicle’s localization and sensor fusion data, and there is also a sparse map list of waypoints around the highway. The vehicle should try to go as close as possible to the 50 MPH speed limit, which means passing slower traffic when possible. The vehicle must not collide with other vehicles, and must stay within lane lines unless going from one lane to another. Additionally, the vehicle should not experience total acceleration over 10 m/s^2 nor jerk that is greater than 10 m/s^3.

Vehicle prediction – Other vehicle tracking

The vehicle tracking section of code is responsible for using data from sensor fusion measurements to determine where other vehicles are relative to the controlled vehicle. This includes detecting vehicles directly to the left, ahead, and right and determining how much empty space exists between these vehicles. This information is used for the lane change decision making system.

Behavior planning – Lane change decision making

The lane change decision making system determines if the vehicle should attempt to change lanes if it is currently travellling at less than 90% of the maximum speed (50 MPH) and the vehicle is following less than 50 units behind the vehicle in front of it. Once that determination is made, the vehicle looks to both the left and right for an opening with no other vehicles. If both lanes are open, the lane with a vehicle farthest ahead is chosen for the lane switch.

Behavior planning – Speed control

The speed control system maximizes the speed in the current lane given traffic conditions. If no other vehicles are within 50 units ahead, the system accelerates slowly until almost 50 MPH is reached. Otherwise, the vehicle slows down to match the ahead vehicle’s speed, and maintains a distance from the ahead vehicle equal to the current velocity (a proportional following distance).

To achieve a constant following distance, a PID controller is used with proportional and differential controls turned on.

Trajectory generation

Once a target lane and velocity are computed, a trajectory is calculated several steps into the future. On a first trajectory generation run, the vehicle’s location and previous point using the vehicle’s yaw angle are computed; otherwise, the number of points remaining in the trajectory are obtained. Next, three points evenly spaced down the target lane are generated. These points are converted to x-y coordinates, and they are angle-corrected for the vehicle’s orientation in the environment. Spline interpolation is used for computing additional trajectory points, which when combined with any previous trajectory points from a previous iteration not traveled, to bring the total number of points up to 50. These points are then handed off to the simulator for the vehicle to follow.

Results

Using these components with input data from the simulator and existing map data, this path planner is able to drive the simulated vehicle safely and efficiently around the course, maximizing its speed while avoiding collisions. Simulation runs often approach hundreds of miles before any kind of collision is experienced, and multiple runs of the simulator show this very often due to unsafe and sudden behavior from other vehicles.