Class Camera

Inheritance Relationships

Base Type

Derived Types

Class Documentation

class aslam::Camera : public aslam::Sensor

The base camera class provides methods to project/backproject euclidean and homogeneous points.

The actual projection is implemented in the derived classes for euclidean coordinates only; homogeneous coordinates are support by a conversion. The intrinsic parameters are documented in the specialized camera classes.

Subclassed by aslam::Cloneable< Camera, Camera3DLidar >, aslam::Cloneable< Camera, PinholeCamera >, aslam::Cloneable< Camera, UnifiedProjectionCamera >

Constructors/destructors and operators

virtual ~Camera() = default
inline std::ostream &operator<<(std::ostream &out)

Convenience function to print the state using streams.

virtual aslam::Camera *clone() const = 0

Clones the camera instance and returns a pointer to the copy.

inline virtual uint8_t getSensorType() const override

Get sensor type as an integer or as a string.

inline virtual std::string getSensorTypeString() const override
Camera() = delete
Camera(const Eigen::VectorXd &intrinsics, aslam::Distortion::UniquePtr &distortion, const uint32_t image_width, const uint32_t image_height, Type camera_type)

Camera base constructor with distortion.

Parameters

  • [in] intrinsics: Vector containing the intrinsic parameters.

  • [in] distortion: unique_ptr to the distortion model

  • [in] image_width: Image width in pixels.

  • [in] image_height: Image height in pixels.

  • [in] camera_type: CameraType enum value with information which camera model is used by the derived class.

Camera(const Eigen::VectorXd &intrinsics, const uint32_t image_width, const uint32_t image_height, Type camera_type)

Camera base constructor without distortion.

Parameters

  • [in] intrinsics: Vector containing the intrinsic parameters.

  • [in] image_width: Image width in pixels.

  • [in] image_height: Image height in pixels.

  • [in] camera_type: CameraType enum value with information which camera model is used by the derived class.

inline virtual Sensor::Ptr cloneAsSensor() const override
inline Camera(const Camera &other)

Copy constructor for clone operation.

void operator=(const Camera&) = delete
bool isEqualCameraImpl(const Camera &other, const bool verbose = false) const

Compare only the parameters of Camera to the ones of another Camera.

Information about the camera

inline uint32_t imageWidth() const

The width of the image in pixels.

inline uint32_t imageHeight() const

The height of the image in pixels.

inline void setImageWidth(uint32_t image_width)

Set the width of the image in pixels.

inline void setImageHeight(uint32_t image_height)

Set the height of the image in pixels.

virtual void printParameters(std::ostream &out, const std::string &text) const

Print the internal parameters of the camera in a human-readable form Print to the ostream that is passed in.

The text is extra text used by the calling function to distinguish cameras.

virtual int getParameterSize() const = 0

The number of intrinsic parameters.

inline Type getType() const

Returns type of the camera model.

Return

CameraType value representing the camera model used by the derived class.

Methods to project and back-project euclidean points

const ProjectionResult project3(const Eigen::Ref<const Eigen::Vector3d> &point_3d, Eigen::Vector2d *out_keypoint) const

Projects a euclidean point to a 2d image measurement.

Applies the projection (& distortion) models to the point.

Return

Contains information about the success of the projection. Check ProjectionResult for more information.

Parameters

  • [in] point_3d: The point in euclidean coordinates.

  • [out] out_keypoint: The keypoint in image coordinates.

const ProjectionResult project3(const Eigen::Ref<const Eigen::Vector3d> &point_3d, Eigen::Vector2d *out_keypoint, Eigen::Matrix<double, 2, 3> *out_jacobian) const

Projects a euclidean point to a 2d image measurement.

Applies the projection (& distortion) models to the po int.

Return

Contains information about the success of the projection. Check ProjectionResult for more information.

Parameters

  • [in] point_3d: The point in euclidean coordinates.

  • [out] out_keypoint: The keypoint in image coordinates.

  • [out] out_jacobian: The Jacobian wrt. to changes in the euclidean point.

virtual void project3Vectorized(const Eigen::Ref<const Eigen::Matrix3Xd> &points_3d, Eigen::Matrix2Xd *out_keypoints, std::vector<ProjectionResult> *out_results) const

Projects a matrix of euclidean points to 2d image measurements.

Applies the projection (& distortion) models to the points.

This vanilla version just repeatedly calls backProject3. Camera implementers are encouraged to override for efficiency.

Parameters

  • [in] point_3d: The point in euclidean coordinates.

  • [out] out_keypoints: The keypoint in image coordinates.

  • [out] out_results: Contains information about the success of the projections. Check ProjectionResult for more information.

virtual bool backProject3(const Eigen::Ref<const Eigen::Vector2d> &keypoint, Eigen::Vector3d *out_point_3d) const = 0

Compute the 3d bearing vector in euclidean coordinates given a keypoint in image coordinates.

Uses the projection (& distortion) models. The result might be in normalized image plane for some camera implementations but not for the general case.

Return

Was the projection successful?

Parameters

  • [in] keypoint: Keypoint in image coordinates.

  • [out] out_point_3d: Bearing vector in euclidean coordinates

virtual void backProject3Vectorized(const Eigen::Ref<const Eigen::Matrix2Xd> &keypoints, Eigen::Matrix3Xd *out_points_3d, std::vector<unsigned char> *out_success) const

Compute the 3d bearing vectors in euclidean coordinates given a list of keypoints in image coordinates.

Uses the projection (& distortion) models.

This vanilla version just repeatedly calls backProject3. Camera implementers are encouraged to override for efficiency. TODO(schneith): implement efficient backProject3Vectorized

Parameters

  • [in] keypoints: Keypoints in image coordinates.

  • [out] out_point_3ds: Bearing vectors in euclidean coordinates (with z=1 -> non-normalized).

  • [out] out_success: Were the projections successful?

Methods to project and back-project homogeneous points

const ProjectionResult project4(const Eigen::Ref<const Eigen::Vector4d> &point_4d, Eigen::Vector2d *out_keypoint) const

Projects a homogeneous point to a 2d image measurement.

Applies the projection (& distortion) models to the point.

Return

Contains information about the success of the projection. Check ProjectionResult for more information.

Parameters

  • [in] point_4d: The point in homogeneous coordinates.

  • [out] out_keypoint: The keypoint in image coordinates.

const ProjectionResult project4(const Eigen::Ref<const Eigen::Vector4d> &point_4d, Eigen::Vector2d *out_keypoint, Eigen::Matrix<double, 2, 4> *out_jacobian) const

Projects a euclidean point to a 2d image measurement.

Applies the projection (& distortion) models to the point.

Return

Contains information about the success of the projection. Check ProjectionResult for more information.

Parameters

  • [in] point_4d: The point in homogeneous coordinates.

  • [out] out_keypoint: The keypoint in image coordinates.

  • [out] out_jacobian: The Jacobian wrt. to changes in the homogeneous point.

bool backProject4(const Eigen::Ref<const Eigen::Vector2d> &keypoint, Eigen::Vector4d *out_point_4d) const

Compute the 3d bearing vector in homogeneous coordinates given a keypoint in image coordinates.

Uses the projection (& distortion) models.

Return

Was the projection successful?

Parameters

  • [in] keypoint: Keypoint in image coordinates.

  • [out] out_point_3d: Bearing vector in homogeneous coordinates.

Functional methods to project and back-project points

const ProjectionResult project3Functional(const Eigen::Ref<const Eigen::Vector3d> &point_3d, const Eigen::VectorXd *intrinsics_external, const Eigen::VectorXd *distortion_coefficients_external, Eigen::Vector2d *out_keypoint) const

This function projects a point into the image using the intrinsic parameters that are passed in as arguments.

If any of the Jacobians are nonnull, they should be filled in with the Jacobian with respect to small changes in the argument.

Return

Contains information about the success of the projection. Check ProjectionResult for more information.

Parameters

  • [in] point_3d: The point in euclidean coordinates.

  • [in] intrinsics_external: External intrinsic parameter vector. NOTE: If nullptr, use internal intrinsic parameters.

  • [in] distortion_coefficients_external: External distortion parameter vector. Parameter is ignored is no distortion is active. NOTE: If nullptr, use internal distortion parameters.

  • [out] out_keypoint: The keypoint in image coordinates.

virtual const ProjectionResult project3Functional(const Eigen::Ref<const Eigen::Vector3d> &point_3d, const Eigen::VectorXd *intrinsics_external, const Eigen::VectorXd *distortion_coefficients_external, Eigen::Vector2d *out_keypoint, Eigen::Matrix<double, 2, 3> *out_jacobian_point, Eigen::Matrix<double, 2, Eigen::Dynamic> *out_jacobian_intrinsics, Eigen::Matrix<double, 2, Eigen::Dynamic> *out_jacobian_distortion) const = 0

This function projects a point into the image using the intrinsic parameters that are passed in as arguments.

If any of the Jacobians are nonnull, they should be filled in with the Jacobian with respect to small changes in the argument.

Return

Contains information about the success of the projection. Check ProjectionResult for more information.

Parameters

  • [in] point_3d: The point in euclidean coordinates.

  • [in] intrinsics_external: External intrinsic parameter vector. NOTE: If nullptr, use internal intrinsic parameters.

  • [in] distortion_coefficients_external: External distortion parameter vector. Parameter is ignored is no distortion is active. NOTE: If nullptr, use internal distortion parameters.

  • [out] out_keypoint: The keypoint in image coordinates.

  • [out] out_jacobian_point: The Jacobian wrt. to changes in the euclidean point. nullptr: calculation is skipped.

  • [out] out_jacobian_intrinsics: The Jacobian wrt. to changes in the intrinsics. nullptr: calculation is skipped.

  • [out] out_jacobian_distortion: The Jacobian wrt. to changes in the distortion parameters. nullptr: calculation is skipped.

Methods to support rolling shutter cameras

enum LineDelayMode

Values:

enumerator kColumns
enumerator kRows
inline uint64_t getLineDelayNanoSeconds() const

Return the temporal offset of a rolling shutter camera.

Global shutter cameras return zero.

Return

Line delay in nano seconds.

inline void setLineDelayNanoSeconds(uint64_t line_delay_nano_seconds)

Set the temporal offset of a rolling shutter camera.

Parameters

  • [in] line_delay_nano_seconds: Line delay in nano seconds.

inline virtual int64_t rollingShutterDelayNanoSeconds(const Eigen::Vector2d &keypoint) const

The amount of time elapsed between the first row of the image and the keypoint.

For a global shutter camera, this can return Duration(0).

Return

Time elapsed between the first row of the image and the keypoint in nanoseconds.

Parameters

  • [in] keypoint: Keypoint to which the delay should be calculated.

inline virtual int64_t maxRollingShutterDelayNanoSeconds() const

The amount of time elapsed between the first row of the image and the last row of the image.

For a global shutter camera, this can return 0.

inline virtual uint32_t getNumberOfLines() const

returns the number of lines, this is either the number of columns or rows or 1 in case of a global shutter camera.

inline virtual LineDelayMode getLineDelayMode() const

Returns which orientation the rolling shutter effect is occuring.

Methods to support compressed images.

inline bool hasCompressedImages() const

Holds whether this camera receives compressed images.

Return

Returns true for compressed images, false otherwise

inline void setCompressedImages(const bool is_compressed)

Set the whether this camera has compressed images.

Parameters

  • [in] is_compressed: Compressed images.

Methods to test validity and visibility

bool isProjectable3(const Eigen::Ref<const Eigen::Vector3d> &point) const

Can the projection function be run on this point? This doesn’t test if the projected point is visible, only if the projection function can be run without numerical errors or singularities.

bool isProjectable4(const Eigen::Ref<const Eigen::Vector4d> &point) const

Can the projection function be run on this point? This doesn’t test if the projected point is visible, only if the projection function can be run without numerical errors or singularities.

template<typename DerivedKeyPoint>
bool isKeypointVisible(const Eigen::MatrixBase<DerivedKeyPoint> &keypoint) const

Return if a given keypoint is inside the imaging box of the camera.

template<typename DerivedKeyPoint>
bool isKeypointVisibleWithMargin(const Eigen::MatrixBase<DerivedKeyPoint> &keypoint, typename DerivedKeyPoint::Scalar margin) const

Return if a given keypoint is within the specified margin to the boundary of the imaging box of the camera.

Methods to support unit testing.

virtual Eigen::Vector2d createRandomKeypoint() const = 0

Creates a random valid keypoint.

virtual Eigen::Vector3d createRandomVisiblePoint(double depth) const = 0

Creates a random visible point.

Negative depth means random between 0 and 100 meters.

Methods to interface the underlying distortion model.

inline aslam::Distortion *getDistortionMutable()

Returns a pointer to the underlying distortion object.

inline const aslam::Distortion &getDistortion() const

Returns the underlying distortion object.

inline void setDistortion(aslam::Distortion::UniquePtr &distortion)

Set the distortion model.

inline bool hasDistortion() const

Is a distortion model set for this camera.

inline void removeDistortion()

Remove the distortion model from this camera.

Methods to access the intrinsic parameters.

inline const Eigen::VectorXd &getParameters() const

Get the intrinsic parameters (const).

inline double *getParametersMutable()

Get the intrinsic parameters.

inline void setParameters(const Eigen::VectorXd &params)

Set the intrinsic parameters.

Parameters are documented in the specialized camera classes.

virtual bool intrinsicsValid(const Eigen::VectorXd &intrinsics) const = 0

Function to check whether the given intrinsic parameters are valid for this model.

Methods to access the mask.

void setMask(const cv::Mat &mask)

Set the mask.

Masks must be the same size as the image and they follow the same convention as OpenCV: 0 == masked, nonzero == valid.

const cv::Mat &getMask() const

Get the mask.

void clearMask()

Clear the mask.

bool hasMask() const

Does the camera have a mask?

inline bool isMasked(const Eigen::Ref<const Eigen::Vector2d> &keypoint) const

Check if the keypoint is masked.

Factory Methods

template<typename DerivedCamera, typename DerivedDistortion>
static DerivedCamera::Ptr construct(const Eigen::VectorXd &intrinsics, uint32_t imageWidth, uint32_t imageHeight, const Eigen::VectorXd &distortionParameters)

A factory function to create a derived class camera.

This function takes a vectors of intrinsics and distortion parameters and produces a camera.

Return

A new camera based on the template types.

Parameters

  • [in] intrinsics: A vector of projection intrinsic parameters.

  • [in] imageWidth: The width of the image associated with this camera.

  • [in] imageHeight: The height of the image associated with this camera.

  • [in] distortionParameters: The parameters of the distortion object.

Public Types

enum [anonymous]

Values:

enumerator CLASS_SERIALIZATION_VERSION
enum Type

Values:

enumerator kPinhole
enumerator kUnifiedProjection
enumerator kLidar3D

Public Functions

ASLAM_POINTER_TYPEDEFS(Camera)

Protected Attributes

uint64_t line_delay_nanoseconds_

The delay per scanline for a rolling shutter camera in nanoseconds.

uint32_t image_width_

The width of the image.

uint32_t image_height_

The height of the image.

cv::Mat_<uint8_t> mask_

The image mask.

bool is_compressed_

Has compressed images.

Eigen::VectorXd intrinsics_

Parameter vector for the intrinsic parameters of the model.

Type camera_type_

Enum field to store the type of camera model.

aslam::Distortion::UniquePtr distortion_

The distortion for this camera.