Recon API

The Recon API v1 is a set of HTTP endpoints that help you understand the current state of an Ozone cluster and to troubleshoot if needed.

HTTP Endpoints

Containers

  • /containers

    URL Structure

    1. GET /api/v1/containers

    Parameters

    • prevKey (optional)

      Only returns the containers with ID greater than the given prevKey. Example: prevKey=1

    • limit (optional)

      Only returns the limited number of results. The default limit is 1000.

    Returns

    Returns all the ContainerMetadata objects.

    1. {
    2.   "data": {
    3.     "totalCount": 3,
    4.     "containers": [
    5.       {
    6.         "ContainerID": 1,
    7.         "NumberOfKeys": 834
    8.       },
    9.       {
    10.         "ContainerID": 2,
    11.         "NumberOfKeys": 833
    12.       },
    13.       {
    14.         "ContainerID": 3,
    15.         "NumberOfKeys": 833
    16.       }
    17.     ]
    18.   }
    19. }

  • /containers/:id/keys

    URL Structure

    1. GET /api/v1/containers/:id/keys

    Parameters

    • prevKey (optional)

      Only returns the keys that are present after the given prevKey key prefix. Example: prevKey=/vol1/bucket1/key1

    • limit (optional)

      Only returns the limited number of results. The default limit is 1000.

    Returns

    Returns all the KeyMetadata objects for the given ContainerID.

    1. {
    2.   "totalCount":7,
    3.   "keys": [
    4.     {
    5.       "Volume":"vol-1-73141",
    6.       "Bucket":"bucket-3-35816",
    7.       "Key":"key-0-43637",
    8.       "DataSize":1000,
    9.       "Versions":[0],
    10.       "Blocks": {
    11.         "0": [
    12.           {
    13.             "containerID":1,
    14.             "localID":105232659753992201
    15.           }
    16.         ]
    17.       },
    18.       "CreationTime":"2020-11-18T18:09:17.722Z",
    19.       "ModificationTime":"2020-11-18T18:09:30.405Z"
    20.     },
    21.     ...
    22.   ]
    23. }

  • /containers/missing

    URL Structure

    1. GET /api/v1/containers/missing

    Parameters

    No parameters.

    Returns

    Returns the MissingContainerMetadata objects for all the missing containers.

    1. {
    2.     "totalCount": 26,
    3.     "containers": [{
    4.         "containerID": 1,
    5.         "missingSince": 1605731029145,
    6.         "keys": 7,
    7.         "pipelineID": "88646d32-a1aa-4e1a",
    8.         "replicas": [{
    9.             "containerId": 1,
    10.             "datanodeHost": "localhost-1",
    11.             "firstReportTimestamp": 1605724047057,
    12.             "lastReportTimestamp": 1605731201301
    13.         }, 
    14.         ...
    15.         ]
    16.     },
    17.     ...
    18.     ]
    19. }

  • /containers/:id/replicaHistory

    URL Structure

    1. GET /api/v1/containers/:id/replicaHistory

    Parameters

    No parameters.

    Returns

    Returns all the ContainerHistory objects for the given ContainerID.

    1. [
    2.   {
    3.     "containerId": 1,
    4.     "datanodeHost": "localhost-1",
    5.     "firstReportTimestamp": 1605724047057,
    6.     "lastReportTimestamp": 1605730421294
    7.   },
    8.   ...
    9. ]

  • /containers/unhealthy

    URL Structure

    1. GET /api/v1/containers/unhealthy

    Parameters

    • batchNum (optional)

      The batch number (like “page number”) of results to return. Passing 1, will return records 1 to limit. 2 will return limit + 1 to 2 * limit, etc.

    • limit (optional)

      Only returns the limited number of results. The default limit is 1000.

    Returns

    Returns the UnhealthyContainerMetadata objects for all the unhealthy containers.

    1. {
    2.    "missingCount": 2,
    3.    "underReplicatedCount": 0,
    4.    "overReplicatedCount": 0,
    5.    "misReplicatedCount": 0,
    6.    "containers": [{
    7.        "containerID": 1,
    8.        "containerState": "MISSING",
    9.        "unhealthySince": 1605731029145,
    10.        "expectedReplicaCount": 3,
    11.        "actualReplicaCount": 0,
    12.        "replicaDeltaCount": 3,
    13.        "reason": null,
    14.        "keys": 7,
    15.        "pipelineID": "88646d32-a1aa-4e1a",
    16.        "replicas": [{
    17.            "containerId": 1,
    18.            "datanodeHost": "localhost-1",
    19.            "firstReportTimestamp": 1605722960125,
    20.            "lastReportTimestamp": 1605731230509
    21.        }, 
    22.        ...
    23.        ]
    24.    },
    25.    ...
    26.    ]
    27. }

  • /containers/unhealthy/:state

    URL Structure

    1. GET /api/v1/containers/unhealthy/:state

    Parameters

    • batchNum (optional)

      The batch number (like “page number”) of results to return. Passing 1, will return records 1 to limit. 2 will return limit + 1 to 2 * limit, etc.

    • limit (optional)

      Only returns the limited number of results. The default limit is 1000.

    Returns

    Returns the UnhealthyContainerMetadata objects for the containers in the given state. Possible unhealthy container states are MISSING, MIS_REPLICATED, UNDER_REPLICATED, OVER_REPLICATED. The response structure is same as /containers/unhealthy.

ClusterState

  • /clusterState

    URL Structure

    1. GET /api/v1/clusterState

    Parameters

    No parameters.

    Returns

    Returns a summary of the current state of the Ozone cluster.

    1. {
    2.    "pipelines": 5,
    3.    "totalDatanodes": 4,
    4.    "healthyDatanodes": 4,
    5.    "storageReport": {
    6.        "capacity": 1081719668736,
    7.        "used": 1309212672,
    8.        "remaining": 597361258496
    9.    },
    10.    "containers": 26,
    11.    "volumes": 6,
    12.    "buckets": 26,
    13.    "keys": 25
    14. }

Datanodes

  • /datanodes

    URL Structure

    1. GET /api/v1/datanodes

    Parameters

    No parameters.

    Returns

    Returns all the datanodes in the cluster.

    1. {
    2.    "totalCount": 4,
    3.    "datanodes": [{
    4.        "uuid": "f8f8cb45-3ab2-4123",
    5.        "hostname": "localhost-1",
    6.        "state": "HEALTHY",
    7.        "lastHeartbeat": 1605738400544,
    8.        "storageReport": {
    9.            "capacity": 270429917184,
    10.            "used": 358805504,
    11.            "remaining": 119648149504
    12.        },
    13.        "pipelines": [{
    14.            "pipelineID": "b9415b20-b9bd-4225",
    15.            "replicationType": "RATIS",
    16.            "replicationFactor": 3,
    17.            "leaderNode": "localhost-2"
    18.        }, {
    19.            "pipelineID": "3bf4a9e9-69cc-4d20",
    20.            "replicationType": "RATIS",
    21.            "replicationFactor": 1,
    22.            "leaderNode": "localhost-1"
    23.        }],
    24.        "containers": 17,
    25.        "leaderCount": 1
    26.    },
    27.    ...
    28.    ]
    29. }

Pipelines

  • /pipelines

    URL Structure

    1. GET /api/v1/pipelines

    Parameters

    No parameters.

    Returns

    Returns all the pipelines in the cluster.

    1. {
    2.    "totalCount": 5,
    3.    "pipelines": [{
    4.        "pipelineId": "b9415b20-b9bd-4225",
    5.        "status": "OPEN",
    6.        "leaderNode": "localhost-1",
    7.        "datanodes": ["localhost-1", "localhost-2", "localhost-3"],
    8.        "lastLeaderElection": 0,
    9.        "duration": 23166128,
    10.        "leaderElections": 0,
    11.        "replicationType": "RATIS",
    12.        "replicationFactor": 3,
    13.        "containers": 0
    14.    },
    15.    ...
    16.    ]
    17. }

Tasks

  • /task/status

    URL Structure

    1. GET /api/v1/task/status

    Parameters

    No parameters.

    Returns

    Returns the status of all the Recon tasks.

    1. [
    2.   {
    3.    "taskName": "OmDeltaRequest",
    4.    "lastUpdatedTimestamp": 1605724099147,
    5.    "lastUpdatedSeqNumber": 186
    6.   },
    7.   ...
    8. ]

Utilization

  • /utilization/fileCount

    URL Structure

    1. GET /api/v1/utilization/fileCount

    Parameters

    • volume (optional)

      Filters the results based on the given volume name.

    • bucket (optional)

      Filters the results based on the given bucket name.

    • fileSize (optional)

      Filters the results based on the given fileSize.

    Returns

    Returns the file counts within different file ranges with fileSize in the response object being the upper cap for file size range.

    1. [{
    2.    "volume": "vol-2-04168",
    3.    "bucket": "bucket-0-11685",
    4.    "fileSize": 1024,
    5.    "count": 1
    6. }, {
    7.    "volume": "vol-2-04168",
    8.    "bucket": "bucket-1-41795",
    9.    "fileSize": 1024,
    10.    "count": 1
    11. }, {
    12.    "volume": "vol-2-04168",
    13.    "bucket": "bucket-2-93377",
    14.    "fileSize": 1024,
    15.    "count": 1
    16. }, {
    17.    "volume": "vol-2-04168",
    18.    "bucket": "bucket-3-50336",
    19.    "fileSize": 1024,
    20.    "count": 2
    21. }]

Metrics

  • /metrics/:api

    URL Structure

    1. GET /api/v1/metrics/:api

    Parameters

    Refer to Prometheus HTTP API Reference for complete documentation on querying.

    Returns

    This is a proxy endpoint for Prometheus and returns the same response as the prometheus endpoint. Example: /api/v1/metrics/query?query=ratis_leader_election_electionCount

    1. {
    2.   "status": "success",
    3.   "data": {
    4.     "resultType": "vector",
    5.     "result": [
    6.       {
    7.         "metric": {
    8.           "__name__": "ratis_leader_election_electionCount",
    9.           "exported_instance": "33a5ac1d-8c65-4c74-a0b8-9314dfcccb42",
    10.           "group": "group-03CA9397D54B",
    11.           "instance": "ozone_datanode_1:9882",
    12.           "job": "ozone"
    13.         },
    14.         "value": [
    15.           1599159384.455,
    16.           "5"
    17.         ]
    18.       }
    19.     ]
    20.   }
    21. }