User Manual

0. Table of Contents

• 1. How to Support New Models
  • 1.1 Create Model Class
  • 1.2 Register Model
  • 1.3 Create Configuration
  • 1.4 Enable Model
• 2. Response Schema
• 3. Troubleshooting

1. How to Support New Models

Integrating a custom model requires 4 simple steps:

1. Create a model class in app/models/your_model.py
2. Register it in app/core/registry.py
3. Configure it in configs/auto_labeling/your_model_id.yaml
4. Enable it in configs/models.yaml

1.1 Create Model Class

Your model must inherit from BaseModel and implement three methods:

from . import BaseModel
from app.schemas.shape import Shape

class YourModel(BaseModel):
    def load(self):
        """Load model weights and initialize resources"""
        model_path = self.params.get("model_path")
        self.model = load_your_model(model_path)
    
    def predict(self, image, params):
        """Run inference and return results"""
        results = self.model(image)
        shapes = [Shape(label="...", shape_type="rectangle", points=[...])]
        return {"shapes": shapes, "description": ""}
    
    def unload(self):
        """Free resources on shutdown"""
        del self.model

Key Points:

• Input image is in BGR format (OpenCV style)
• Return a dict with shapes and description fields (see Response Schema)
• Check app/models/ for complete implementation examples

1.2 Register Model

Use the @register_model decorator to register your model class.

...
from app.core.registry import register_model

@register_model("your_model_id")
class YourModel(BaseModel):
    # ... your implementation

You can also register multiple model IDs with the same class:

@register_model(
    "yolo11n", "yolo11s", "yolo11m", "yolo11l", "yolo11x"
)
class YOLO11Detection(BaseModel):
    # ... your implementation

Tip

• The system automatically imports all modules in app/models/ directory (first level only)
• Multiple model IDs can share the same class (e.g., yolo11n and yolo11s both use YOLO11nDetection)

1.3 Create Configuration

Create configs/auto_labeling/your_model_id.yaml:

model_id: your_model_id          # Required: Must be globally unique
display_name: "Your Model Name"  # Required: Displayed in X-AnyLabeling UI
batch_processing_mode: "default" # Optional: "default" or "text_prompt" (default: "default")
capabilities:                    # Optional: Module-scoped capability metadata
  ppocr_pipeline: true           # Example: consumed by PPOCR panel clients

params:                          # Optional: All params are passed to your model's __init__
  model_path: "path/to/weights.pt"
  device: "cuda:0"
  conf_threshold: 0.25
  # Add any custom parameters your model needs

widgets:
  - name: button_run
    value: null
  - name: edit_conf
    value: 0.25          # Widgets with ✅ must have default values
  - name: edit_iou
    value: 0.45
  - name: toggle_preserve_existing_annotations
    value: false
  ...

Capabilities Guidance:

• capabilities is used for module-specific client routing rather than generic auto-labeling UI presentation.
• If a model declares capabilities, clients may hide it from the general Remote-Server dropdown and surface it only in dedicated panels.
• Use clear keys and stable semantics to keep client behavior predictable.

See Widget Reference for details.

1.4 Enable Model

Add to configs/models.yaml:

enabled_models:
  - yolo11n
  - yolo11n_seg
  - your_model_id  # Models will be displayed in this order
  # - yolo11s      # Comment out to disable a model

Note

• Models are displayed in X-AnyLabeling UI in the order listed here
• Comment out any model you don't want to load to save resources

2. Response Schema

Your model's predict() method must return a dictionary with the following fields:

{
    "shapes": [...],      # List of Shape objects (can be empty for caption tasks)
    "description": "...", # Text description (can be empty for detection tasks)
    "replace": None       # Optional: Boolean flag to replace existing annotations (excluded if None)
}

Each Shape object should have the following properties:

Field	Type	Required	Description
label	String	✅ Yes	Category label of the object
shape_type	String	✅ Yes	Type of shape (see supported types below)
points	List	✅ Yes	List of [x, y] coordinates defining the shape vertices
score	Float	No	Confidence score from model inference (default: None)
attributes	Dict	No	Custom object attributes (default: {})
description	String	No	Optional text description for the shape
difficult	Boolean	No	Flag if object is difficult to identify (default: False)
direction	Float	No	Direction in radians, 0-2π (default: 0)
flags	Dict	No	Additional flags or metadata
group_id	Integer	No	ID to group related shapes (e.g., pose keypoints)
kie_linking	List	No	Key Information Extraction linking data (default: [])

Supported Shape Types:

• rectangle: Horizontal bounding box defined by 4 corner points
• polygon: Closed polygon with 3+ vertices
• quadrilateral: Four-sided polygon with 4 vertices
• rotation: Oriented/rotated bounding box
• point: Single point coordinate
• line: Line segment with start and end points
• circle: Circle defined by center and radius point
• linestrip: Connected line segments (polyline)

For detailed shape specifications, see the X-AnyLabeling User Guide and app/schemas/shape.py.

3. Troubleshooting

Error	Solution
Model 'xxx' not registered	Add to _build_registry() in registry.py
Configuration file not found	Check YAML file exists and model_id matches filename
Widget 'edit_conf' requires a default value	Set value: 0.25 in widgets config
Duplicate model_id found	Each model_id must be unique