Camera Sensor Model

In Chrono:Sensor:ChCameraSensor, the synthetic data is generated via GPU-based ray-tracing. By leveraging hardware accelerated support and the headless rendering capabilities provided by Nvidia Optix Library.

Camera sensor Setup

chrono::ChFrame<double> offset_pose({10, 2, .5}, // Position
Q_from_AngAxis(CH_C_PI, {0, 0, 1})); // Rotation
auto Camera = chrono_types::make_shared<ChCameraSensor>(ground_body, // body camera is attached to
update_rate, // update rate in Hz
offset_pose, // offset pose
image_width, // image width
image_height, // image height
fov, // camera's horizontal field of view
alias_factor, // supersample factor for antialiasing
lens_model); // FOV
Camera->SetName("Camera Sensor");
Camera->SetLag(lag);
Camera->SetCollectionWindow(exposure_time);
// Sensor data access filter
Camera->PushFilter(chrono_types::make_shared<ChFilterRGBA8Access>());
// Add sensor to manager
manager->AddSensor(Camera);

See ChCameraSensor for more details.

The camera setup process will automatically add the Optix Render Filter to the filter list

If the camera supersample_factor is greater than 1, the setup process will adjust the resolution and append the Image Alias Filter to the filter list.

Rendering steps

The Camera sensor in Chrono::sensor uses Optix as the render engine. For each pixel, the engine will shoot out a ray at that direction and find the first object intersects with the ray. By default, the engine uses the physically based BRDF shader for rendering objects. It will spawn additional rays for shadows, reflection, and refraction in a recursive fashion.

The camera update frequency is much slower than the physics. Therefore the ChOptixEngine spawns a thread to perform the rendering, and it will not block the main thread

Each Update (main thread)

  1. Check if there is any camera need to be updated. If there is such camera
    • Update the scene information
    • Push it into render queue
  2. Check if any camera should have the data ready to ship, or wait until they finish.
  3. Continue to the next time step

Rendering thread

  1. Wait until there is a camera in the render queue
  2. Update those camera, clear the render queue, go back to step 1

Filter Graphs

Any number of filters can be append to the list and modify the final result. The filters are executed as the order in filter list. Here are some examples.

Camera animation

The position and rotation of the camera can be easily changed using SetOffsetPose during simulation

Camera->SetOffsetPose(chrono::ChFrame<double>({8, 2, .5}, // Position
Q_from_AngAxis(CH_C_PI, {0, 0, 1}))); // Rotation

Data access

Data will be ready after the lag time. To access

RGBA_ptr = Camera->GetMostRecentBuffer<UserRGBA8BufferPtr>();
if (RGBA_ptr->Buffer) {
unsigned int height = RGBA_ptr->Height;
unsigned int width = RGBA_ptr->Width;
PixelRGBA8 pixel_at_100_100 = RGBA_ptr->Buffer[100 * width + 100];
uint8_t red_channel_at_100_100 = unsigned(pixel_at_100_100.R);
}
ChQuaternion< double > Q_from_AngAxis(double angle, const ChVector< double > &axis)
Get the quaternion from an angle of rotation and an axis, defined in abs coords.
Definition: ChQuaternion.cpp:100
std::shared_ptr< SensorHostRGBA8Buffer > UserRGBA8BufferPtr
pointer to an RGBA image on the host that has been moved for safety and can be given to the user
Definition: ChSensorBuffer.h:88