GstAnalytics.SegmentationMtd

Fields

Name	Type	Access	Description
id	int	r/w	Instance identifier
meta	GstAnalytics.RelationMeta	r/w	Instance of GstAnalytics.RelationMeta where the analytics-metadata identified by id is stored.

Methods

class	get_mtd_type ()
	get_mask ()
	get_region_count ()
	get_region_id (index)
	get_region_index (id)

Details

class GstAnalytics.SegmentationMtd

This type of metadata holds information on which pixels belongs to a region of the image representing a type of object.

It supports two types of segmentation, semantic or instance:

• Semantic: All objects of the same type have the same id

• Instance: Each instance of an object has a different id

  The results of the segmentation are stored in a Gst.Buffer that has a GstVideo.VideoMeta associated with it. This buffer is stored in the GstAnalytics.SegmentationMtd using GstAnalytics.RelationMeta.add_segmentation_mtd(). The Gst.Buffer containing the segmentation mask is image-like but the color values are arbitrary values, referred by region-id in this API, without meaning beyond specifying that two pixels in the original image with the same values in their corresponding mask value belong to the same region.

  To further describe a region, the GstAnalytics.SegmentationMtd can be associated with other GstAnalytics.Mtd. Since region ids are generated by the segmentation process itself and are not always sequential, we use a map of indexes to region ids starting with 0 without discontinuity which facilitate N-to-N mapping with other GstAnalytics.Mtd. For example it can be associated with GstAnalytics.ClsMtd to describe the class of object matching the pixels of a segmented region.

  Example: Associate Instance Segmentation with Classification

  In the following example the segmentation process will fill segmask with values of 0 for background, 12 for the first region which correspond to a to a strawberry, 7 for the second region that also correspond to a strawberry in the image and 31 for the third region that correspond to a leaf in the image. region_ids is fill during segmentation post-processing

  region_ids: |region-index | region-id | |-------------|———–| | 0 | 0 | | 1 | 12 | | 2 | 7 | | 3 | 31 |

  region_count = 4

  ``` C GstAnalytics.SegmentationMtd segmtd; GstAnalyticsClassificationMtd clsmtd; Gst.Buffer *segmask, *img; int *region_ids; gsize region_count, class_count; float *class_confidence; GQuark *classes;

  … (segmentation filling segmask based on img)

  GstAnalytics.RelationMeta.add_segmentation_mtd (rmeta, segmask, GstAnalytics.SegmentationType.INSTANCE, region_count, region_ids, &segmtd); class_count = region_count;

  … (class-index must match and correspond to region-index) classes [0] = GLib.quark_from_string (“background”); classes [1] = GLib.quark_from_string (“strawberry”); classes [2] = GLib.quark_from_string (“strawberry”); classes [3] = GLib.quark_from_string (“leaf”);

  … (set confidence level for each class associated with a region … where -1.0 mean undefined.) class_confidence [0] = -1.0; class_confidence [1] = 0.6; class_confidence [2] = 0.9; class_confidence [3] = 0.8;

  GstAnalytics.RelationMeta.add_cls_mtd (rmeta, class_count, class_confidence, classes, &clsmtd);

  GstAnalytics.RelationMeta.set_relation (rmeta, GstAnalytics.RelTypes.RELATE_TO, segmtd.id, clsmtd.id); ```

  Example: Associate Semantic Segmentation with Classification Assuming the same context as for Instance Segmentation above but instead a semantic segmentation is performed, therefore region-id-12 and region-id-7 are now represented by the same region-id-12

  region_ids: (here |region-index | region-id | |-------------|———–| | 0 | 0 | | 1 | 12 | | 2 | 31 |

  Code remain the same except that we set all confidence level to undefined (-1.0).

  ``` … (class-index must match and correspond to region-index) classes [0] = GLib.quark_from_string (“background”); classes [1] = GLib.quark_from_string (“strawberry”); classes [2] = GLib.quark_from_string (“leaf”);

  … (set confidence level for each class associated with a region … where -1.0 mean undefined.) class_confidence [0] = -1.0; class_confidence [1] = -1.0; class_confidence [2] = -1.0;

  GstAnalytics.RelationMeta.add_cls_mtd (rmeta, class_count, class_confidence, classes, &clsmtd);

  GstAnalytics.RelationMeta.set_relation (rmeta, GstAnalytics.RelTypes.RELATE_TO, segmtd.id, clsmtd.id); ```

  Example: Retrieving class associated with a segmentation region-id-12 This the typical case for an overlay as we visit the segmentation mask we we find region-id values

  ``` gsize idx; GstAnalytics.SegmentationMtd.get_region_index (&segmtd, &idx, 12); GstAnalytics.RelationMeta.get_direct_related (rmeta, segmtd.id, GstAnalytics.RelTypes.RELATE_TO, gst_analytics_cls_mtd_get_type (), None, &clsmtd);

  GQuark region_class = GstAnalytics.ClsMtd.get_quark (&segmtd, idx) … ```

  Since: 1.26

New in version 1.26.

classmethod get_mtd_type()

Returns:

A #GstAnalyticsMtdType type

Return type:

int

Get an instance of #GstAnalyticsMtdType that represent segmentation metadata type.

New in version 1.26.

get_mask()

Returns:

Segmentation mask data stored in a Gst.Buffer

masks_loc_x:

Left coordinate of the rectangle corresponding to the mask in the image.

masks_loc_y:

Top coordinate of the rectangle corresponding to the mask in the image.

masks_loc_w:

Width of the rectangle corresponding to the mask in the image.

masks_loc_h:

Height of the rectangle corresponding to the mask in the image.

Return type:

(Gst.Buffer, masks_loc_x: int, masks_loc_y: int, masks_loc_w: int, masks_loc_h: int)

Get segmentation mask data.

New in version 1.26.

get_region_count()

Returns:

Number of regions segmented

Return type:

int

Get the regions count.

New in version 1.26.

get_region_id(index)

Parameters:

index (int) – Region index

Returns:

The region ID

Return type:

int

Get id of the region corresponding to index, which should be smaller than the return value of GstAnalytics.SegmentationMtd.get_region_count()

New in version 1.26.

get_region_index(id)

Parameters:

id (int) – Region id

Returns:

True if a region with id exist, otherwise False

index:

Region index

Return type:

(bool, index: int)

Get region index of the region identified by id.

New in version 1.26.