Java SDK

Create real-time synchronization task

Create a task on the Dashboard or through the command line, and use Auto Push (autonomous push) as the data source type.

Import Java SDK

The library of the SDK need to be imported into the project before using the SDK. The library can be obtained in the following two ways:

• Get the source code and compile it yourself and deploy the SDK package to the local warehouse, see How to Build.

• Imported through maven dependency like this:

  <dependency>
      <groupId>org.apache.inlong</groupId>
      <artifactId>dataproxy-sdk</artifactId>
      <version>1.11.0-SNAPSHOT</version>
  </dependency>

Data report process

After import the SDK, you can instantiate a MessageSender object, call sync(sendMessage()) or async(asyncSendMessage()) interface to report single or multiple(batch) data. see Send Demo. The overall process includes the following three steps：

Initialize SDK

From the demo code, we can see that the client initialization is mainly done in the getMessageSender() function:

public DefaultMessageSender getMessageSender(String localIP, String inLongManagerAddr, String inLongManagerPort, String inlongGroupId, boolean isLocalVisit, boolean isReadProxyIPFromLocal, String configBasePath, int msgType) {
    ProxyClientConfig dataProxyConfig = null;
    DefaultMessageSender messageSender = null;
    try {
        // Initialize client configuration.  'test', '123456' is username and password, which need to be replaced according to the environment configuration in actual use.
        dataProxyConfig = new ProxyClientConfig(localIP, isLocalVisit, inLongManagerAddr, Integer.valueOf(inLongManagerPort), inlongGroupId, "test", "123456");
        // Set the local save path of the configuration. This setting is optional. By default, the SDK will create a "/.inlong/" directory under the current user's working directory to store the configuration.
        if (StringUtils.isNotEmpty(configBasePath)) {
            dataProxyConfig.setConfStoreBasePath(configBasePath);
        }
        // Set whether to use the local saved configuration or not. This setting is optional. By default, do not use. 
        dataProxyConfig.setReadProxyIPFromLocal(isReadProxyIPFromLocal);
        // Initialize MessageSender object, if there is an exception, an exception will be thrown.
        messageSender = DefaultMessageSender.generateSenderByClusterId(dataProxyConfig);
        // Set message type to send. This setting is optional. By default, send data in binary format.
        messageSender.setMsgtype(msgType);
    } catch (Exception e) {
        logger.error("getMessageSender has exception e = {}", e);
    }
    // Return initialization result.
    return messageSender;
}

ProxyClientConfig configuration

Call the send interface to report data

The SDK data send interface is thread safe, support send single or multiple messages by sync and async two ways. The following demo uses a single sync way to send, and the message does not contain property information:

public void sendTcpMessage(DefaultMessageSender sender, String inlongGroupId, String inlongStreamId, String messageBody, long dt) {
    SendResult result = null;
    try {
        // Sends a single message in sync mode, and does not contain property information    
        result = sender.sendMessage(messageBody.getBytes("utf8"), inlongGroupId, inlongStreamId,
                0, String.valueOf(dt), 20, TimeUnit.SECONDS);
    } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
    }
    logger.info("messageSender {}", result);
}

You can also choose different send interfaces to report data according to your business needs. For the details of the interface, please refer to the definition in the MessageSender interface file, which has a detailed introduction, no additional explanation here.

Close SDK

In Demo, there is no close operation. When in use, we can call the close() function of the MessageSender interface object to stop data reporting.

Warning

• The MessageSender interface object is initialized based on the inlongGroupId, so each MessageSender object can be used differently based on the inlongGroupId, and multiple MessageSender objects can be created in the same process.
• The SDK provides three different network interaction ways: TCP, HTTP, and UDP. Examples of these three ways are given in the example (refer to TcpClientExample.java, HttpClientExample.java, UdpClientExample.java), and the business can be customized according to its own needs to initialize different MessageSender object.
• The SDK contains complex network interactions, MessageSender should be used as a resident object. Avoid frequent initialization and shutdown of MessageSender (frequent initialization and shutdown will have a large resource overhead and will affect the timeliness of data reporting).
• The SDK does not resend the failed message. When using the SDK to report data, if send fails, you need to decide whether to resend according to your own needs.

Error Code Introduction

Common error codes are as follows.

ProxyClientConfig Configuration Introduction