robokudo.annotators.clip_annotator

Attributes

FONT

Classes

ClipAnnotator

A simple CLIP annotator that uses a given vocabulary to annotate all incoming object

Module Contents

robokudo.annotators.clip_annotator.FONT

class robokudo.annotators.clip_annotator.ClipAnnotator(name: str = 'ClipAnnotator', descriptor: ClipAnnotator | None = None)

Bases: robokudo.annotators.core.ThreadedAnnotator

A simple CLIP annotator that uses a given vocabulary to annotate all incoming object hypothesis.

class Descriptor

Bases: robokudo.annotators.core.ThreadedAnnotator.Descriptor

Descriptor for the ClipAnnotator.

class Parameters

Parameters for the Descriptor.

device

Torch device to be used by the annotator.

model_name = 'openai/clip-vit-large-patch14'

Name of the CLIP to use for feature extraction.

vocabulary_template: str = 'A photo of a {}'

A template string that each vocabulary word is inserted into.

vocabulary: typing_extensions.List[str] = ['Milk Container', 'Box of Cornflakes', 'Pancake Batter', 'Instant Coffee', 'Crepe Pan', 'Dish Soap']

The vocabulary used for object classification.

analysis_scope = 'color_image'

The analysis scope of the annotator.

Possible values:

• CASViews.COLOR_IMAGE: Analyze the entire color image at once.

• ObjectHypothesis: Analyze each object hypothesis individually.

filter_fn: typing_extensions.Optional[typing_extensions.Callable[[robokudo.types.scene.ObjectHypothesis], bool]] = None

An optional filter function used to filter object hypotheses before analyzing them.

save_top_k: int

How many annotations to create per analyzed image or hypothesis.

use_softmax: bool = True

Whether to apply softmax to the results of the analyzed image or hypothesis or not.

parameters

The parameters used by this annotator.

clip_model

CLIP model used for feature extraction.

clip_preprocess

CLIP preprocessor used for input preprocessing.

clip_tokenizer

CLIP tokenizer used for text preprocessing.

vocabulary_index: faiss.IndexFlatIP

A faiss index for text features used in similarity search.

_get_image_features(images: typing_extensions.Union[PIL.Image.Image, typing_extensions.List[PIL.Image.Image], numpy.typing.NDArray, typing_extensions.List[numpy.typing.NDArray]]) → numpy.typing.NDArray[numpy.float32]

Get normalized image features for the given image(s).

Parameters:

images – The image(s) to get features from.

Returns:

The image(s) features.

_search_index(features: numpy.typing.NDArray[numpy.float32], k: int, use_softmax: bool, save_top_k: int) → typing_extensions.Tuple[numpy.typing.NDArray[numpy.float32], numpy.typing.NDArray[numpy.int64]]

Search the faiss index using the given features.

Parameters:

• features – The features to search the index with.

• k – The number of candidates to retrieve from the index for each feature.

• use_softmax – Whether to apply a softmax to the results of each feature.

• save_top_k – How many results to keep after applying softmax.

Returns:

A tuple of (similarities, class indices)

_analyze_scene(color_image: numpy.typing.NDArray[numpy.uint8]) → py_trees.common.Status

Analyze the entire color image.

Parameters:

color_image – The color image to analyze.

Returns:

The py_trees status.

_analyze_object_hypotheses(color_image: numpy.typing.NDArray[numpy.uint8]) → py_trees.common.Status

Analyze the object hypothesis using the given color image.

Parameters:

color_image – The color image to use for analyzation.

Returns:

The py_trees status.

compute() → py_trees.common.Status

Compute CLIP features depending on the analysis scope.

Returns:

Whether the computation was successfull or not.