Vector Database Metrics

Vector database metrics in this repo are for the Milvus metrics path. The ready-to-use exporter maps Milvus core metrics, proxy operation metrics, cache-hit metrics, and queue-latency metrics into AppDynamics custom metrics for the vector database dashboard.

Prerequisites

Ensure that:
  • Milvus is deployed and exposing Prometheus metrics
  • service path is reachable from the AppDynamics scrape path
  • Prometheus-compatible metrics are available at /metrics

Enable Prometheus Scraping for Vector Database

The following are example values from this repo:
  • service: milvus-deployment
  • namespace: milvus
  • port: 9091
  • path: /metrics

Replace these values with the vector database metrics service name and namespace used in the target environment.

Configure Machine Agent Ingestion

Infrastructure Visibility Prometheus monitoring loads the vector database exporter definition through prometheus-config-template.yaml.

To activate this path:

  • include the vector database exporter YAML in the prometheus-exporters ConfigMap
  • ensure that the main include list references that exporter file
  • apply the ConfigMap update and re-run the Infrastructure Visibility Helm upgrade

Before enabling the scrape, update the exporter YAML service discovery fields to the service name and namespace used by your vector database deployment.

Exporter YAML Contract

  • exporter-yamls/vector-database-core-metrics.yaml
  • key source metrics:
    • milvus_querycoord_collection_num
    • milvus_rootcoord_dml_channel_num
    • milvus_rootcoord_id_alloc_count
    • milvus_proxy_req_count
    • milvus_proxy_cache_hit_count
    • milvus_rootcoord_ddl_req_count
    • milvus_proxy_req_in_queue_latency_sum
    • milvus_proxy_req_in_queue_latency_count
    • milvus_rootcoord_ddl_req_latency_in_queue_sum
    • milvus_rootcoord_ddl_req_latency_in_queue_count
  • custom metric family: Custom Metrics|Vector DB|...

Expected AppDymanics Custom Metric Paths

  • Custom Metrics|Vector DB|QueryCoord Collection Count
  • Custom Metrics|Vector DB|RootCoord DML Channel Count
  • Custom Metrics|Vector DB|RootCoord ID Alloc Count
  • Custom Metrics|Vector DB|Proxy|{status}|Request Count
  • Custom Metrics|Vector DB|Proxy Request Count
  • Custom Metrics|Vector DB|Proxy|Cache|{cache_state}|Hit Count
  • Custom Metrics|Vector DB|Proxy|Cache|{cache_name}|Hit Count (agg)
  • Custom Metrics|Vector DB|Proxy|{function_name}|Request Count (agg)
  • Custom Metrics|Vector DB|RootCoord|{function_name}|DDL Request Count (agg)
  • Custom Metrics|Vector DB|Proxy|{function_name}|Avg Latency in Queue (ms) (agg)
  • Custom Metrics|Vector DB|RootCoord|{function_name}|Avg DDL Latency in Queue (ms) (agg)

Dashboard Dependencies

appd_vectordb.json
JSON 
{
    "schemaVersion": null,
    "dashboardFormatVersion": "4.0",
    "name": "dashboard-ai-pod-vector-database",
    "description": null,
    "properties": null,
    "templateEntityType": "APPLICATION_COMPONENT_NODE",
    "associatedEntityTemplates": null,
    "timeRangeSpecifierType": "GLOBAL",
    "minutesBeforeAnchorTime": -1,
    "startDate": null,
    "endDate": null,
    "refreshInterval": 120000,
    "backgroundColor": 15856629,
    "color": 15856629,
    "height": 768,
    "width": 1024,
    "canvasType": "CANVAS_TYPE_GRID",
    "layoutType": "",
    "widgetTemplates": [
        {
            "widgetType": "MetricLabelWidget",
            "title": "Proxy - Collections",
            "height": 2,
            "width": 3,
            "minHeight": 0,
            "minWidth": 0,
            "x": 0,
            "y": 0,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": true,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": true,
            "numDecimals": 2,
            "removeZeros": true,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": null,
            "legendPosition": null,
            "legendColumnCount": null,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 0",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "CURRENT",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|QueryCoord Collection Count",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|QueryCoord Collection Count"
                        },
                        "rollupMetricData": true,
                        "expressionString": null,
                        "useActiveBaseline": true,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": null
                }
            ],
            "text": null,
            "textAlign": "RIGHT",
            "margin": 15,
            "showLabel": false,
            "showBaseline": false,
            "useBaselineColor": false,
            "reverseBaselineColorOrder": false
        },
        {
            "widgetType": "GraphWidget",
            "title": "Proxy - Total Operations",
            "height": 2,
            "width": 9,
            "minHeight": 0,
            "minWidth": 0,
            "x": 3,
            "y": 0,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy Request Count",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy Request Count"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "Milvus Proxy - Operations by Status",
            "height": 2,
            "width": 6,
            "minHeight": 0,
            "minWidth": 0,
            "x": 0,
            "y": 2,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|*|Request Count",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|*|Request Count"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "Milvus Proxy : Cache Hits",
            "height": 2,
            "width": 6,
            "minHeight": 0,
            "minWidth": 0,
            "x": 6,
            "y": 2,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|Cache|*|Hit Count",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|Cache|*|Hit Count"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "Milvus Proxy - Request Latency by Function",
            "height": 2,
            "width": 6,
            "minHeight": 0,
            "minWidth": 0,
            "x": 0,
            "y": 4,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|*|Avg Latency in Queue (ms) (agg)",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|*|Avg Latency in Queue (ms) (agg)"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "Milvus Proxy - Operations by Function",
            "height": 2,
            "width": 6,
            "minHeight": 0,
            "minWidth": 0,
            "x": 6,
            "y": 4,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|*|Request Count (agg)",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|Proxy|*|Request Count (agg)"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "RootCoord - DDL Operations",
            "height": 2,
            "width": 12,
            "minHeight": 0,
            "minWidth": 0,
            "x": 0,
            "y": 6,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord|*|DDL Request Count (agg)",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord|*|DDL Request Count (agg)"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "Milvus Coord - DDL Operations in queue latency",
            "height": 2,
            "width": 9,
            "minHeight": 0,
            "minWidth": 0,
            "x": 3,
            "y": 8,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord|*|Avg DDL Latency in Queue (ms) (agg)",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord|*|Avg DDL Latency in Queue (ms) (agg)"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "GraphWidget",
            "title": "RootCoord - Count of ID allocated",
            "height": 2,
            "width": 12,
            "minHeight": 0,
            "minWidth": 0,
            "x": 0,
            "y": 10,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": false,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": null,
            "numDecimals": 0,
            "removeZeros": null,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": true,
            "legendPosition": "POSITION_BOTTOM",
            "legendColumnCount": 1,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 1",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "VALUE",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord ID Alloc Count",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord ID Alloc Count"
                        },
                        "rollupMetricData": false,
                        "expressionString": null,
                        "useActiveBaseline": false,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": "LEFT"
                }
            ],
            "verticalAxisLabel": null,
            "hideHorizontalAxis": null,
            "horizontalAxisLabel": null,
            "axisType": "LINEAR",
            "stackMode": null,
            "multipleYAxis": null,
            "customVerticalAxisMin": null,
            "customVerticalAxisMax": null,
            "showEvents": null,
            "interpolateDataGaps": false,
            "showAllTooltips": null,
            "staticThresholdList": [],
            "eventFilterTemplate": null
        },
        {
            "widgetType": "MetricLabelWidget",
            "title": "RootCoord - DML Channels",
            "height": 2,
            "width": 3,
            "minHeight": 0,
            "minWidth": 0,
            "x": 0,
            "y": 8,
            "label": null,
            "description": null,
            "drillDownUrl": null,
            "openUrlInCurrentTab": false,
            "useMetricBrowserAsDrillDown": true,
            "drillDownActionType": null,
            "backgroundColor": 16777215,
            "backgroundColors": null,
            "backgroundColorsStr": "16777215,16777215",
            "color": 1646891,
            "fontSize": 12,
            "useAutomaticFontSize": true,
            "borderEnabled": false,
            "borderThickness": 0,
            "borderColor": 14408667,
            "backgroundAlpha": 1,
            "showValues": false,
            "formatNumber": true,
            "numDecimals": 2,
            "removeZeros": true,
            "compactMode": false,
            "showTimeRange": false,
            "renderIn3D": false,
            "showLegend": null,
            "legendPosition": null,
            "legendColumnCount": null,
            "timeRangeSpecifierType": "BEFORE_NOW",
            "startTime": null,
            "endTime": null,
            "minutesBeforeAnchorTime": 15,
            "isGlobal": true,
            "propertiesMap": null,
            "dataSeriesTemplates": [
                {
                    "seriesType": "LINE",
                    "metricType": null,
                    "showRawMetricName": false,
                    "colorPalette": null,
                    "name": "Series 0",
                    "metricMatchCriteriaTemplate": {
                        "entityMatchCriteria": {
                            "matchCriteriaType": "SpecificEntities",
                            "entityType": "APPLICATION_COMPONENT_NODE",
                            "agentTypes": null,
                            "entityNames": [
                                {
                                    "applicationName": "Server & Infrastructure Monitoring",
                                    "entityType": "APPLICATION_COMPONENT_NODE",
                                    "entityName": "worker3.flashstack.local",
                                    "scopingEntityType": "APPLICATION_COMPONENT",
                                    "scopingEntityName": "Root",
                                    "subtype": null,
                                    "uniqueKey": null
                                }
                            ],
                            "summary": false
                        },
                        "metricExpressionTemplate": {
                            "metricExpressionType": "Logical",
                            "functionType": "CURRENT",
                            "displayName": "null",
                            "inputMetricText": true,
                            "inputMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord DML Channel Count",
                            "relativeMetricPath": "Custom Metrics|Temp2|Vector DB2|RootCoord DML Channel Count"
                        },
                        "rollupMetricData": true,
                        "expressionString": null,
                        "useActiveBaseline": true,
                        "sortResultsAscending": false,
                        "maxResults": 20,
                        "evaluationScopeType": null,
                        "baselineName": null,
                        "applicationName": "Server & Infrastructure Monitoring",
                        "metricDisplayNameStyle": "DISPLAY_STYLE_AUTO",
                        "metricDisplayNameCustomFormat": null,
                        "includeHistoricalNodes": false,
                        "includeAbove": true,
                        "includeBelow": false,
                        "includeBoth": false,
                        "includeBand12": false,
                        "includeBand23": false,
                        "includeBand34": false,
                        "includeBand45": false,
                        "includeShade": false
                    },
                    "axisPosition": null
                }
            ],
            "text": null,
            "textAlign": "RIGHT",
            "margin": 15,
            "showLabel": false,
            "showBaseline": false,
            "useBaselineColor": false,
            "reverseBaselineColorOrder": false
        }
    ],
    "warRoom": false,
    "template": false
}

Troubleshooting

  • if queue-latency widgets are empty, verify that both _sum and _count histogram series are present upstream.
  • distinguish the zero values from missing paths before changing the exporter YAML.