Resource Center

If you want to use the resource upload function, you can select the local file directory for a single machine(this operation does not need to deploy Hadoop). Or you can also upload to a Hadoop or MinIO cluster, at this time, you need to have Hadoop (2.6+) or MinIO and other related environments

Note:

  • If the resource upload function is used, the deployment user in installation and deployment must to have operation authority
  • If you using Hadoop cluster with HA, you need to enable HDFS resource upload, and you need to copy the core-site.xml and hdfs-site.xml under the Hadoop cluster to /opt/dolphinscheduler/conf, otherwise Skip step

hdfs resource configuration

  • Upload resource files and udf functions, all uploaded files and resources will be stored on hdfs, so the following configuration items are required:
  1. conf/common/common.properties
  2.     # Users who have permission to create directories under the HDFS root path
  3.     hdfs.root.user=hdfs
  4.     # data base dir, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions。"/escheduler" is recommended
  5.     data.store2hdfs.basepath=/dolphinscheduler
  6.     # resource upload startup type : HDFS,S3,NONE
  7.     res.upload.startup.type=HDFS
  8.     # whether kerberos starts
  9.     hadoop.security.authentication.startup.state=false
  10.     # java.security.krb5.conf path
  11.     java.security.krb5.conf.path=/opt/krb5.conf
  12.     # loginUserFromKeytab user
  13.     login.user.keytab.username=hdfs-mycluster@ESZ.COM
  14.     # loginUserFromKeytab path
  15.     login.user.keytab.path=/opt/hdfs.headless.keytab
  17. conf/common/hadoop.properties
  18.     # ha or single namenode,If namenode ha needs to copy core-site.xml and hdfs-site.xml
  19.     # to the conf directory，support s3，for example : s3a://dolphinscheduler
  20.     fs.defaultFS=hdfs://mycluster:8020
  21.     #resourcemanager ha note this need ips , this empty if single
  22.     yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx
  23.     # If it is a single resourcemanager, you only need to configure one host name. If it is resourcemanager HA, the default configuration is fine
  24.     yarn.application.status.address=http://xxxx:8088/ws/v1/cluster/apps/%s
  • Only one address needs to be configured for yarn.resourcemanager.ha.rm.ids and yarn.application.status.address, and the other address is empty.
  • You need to copy core-site.xml and hdfs-site.xml from the conf directory of the Hadoop cluster to the conf directory of the dolphinscheduler project, and restart the api-server service.

File management

It is the management of various resource files, including creating basic txt/log/sh/conf/py/java and other files, uploading jar packages and other types of files, and can do edit, rename, download, delete and other operations.

Resource - 图1

  • Create a file

    The file format supports the following types: txt, log, sh, conf, cfg, py, java, sql, xml, hql, properties

Resource - 图2

  • upload files

Upload file: Click the “Upload File” button to upload, drag the file to the upload area, the file name will be automatically completed with the uploaded file name

Resource - 图3

  • File View

For the file types that can be viewed, click the file name to view the file details

Resource - 图4

  • download file

Click the “Download” button in the file list to download the file or click the “Download” button in the upper right corner of the file details to download the file

  • File rename

Resource - 图5

  • delete

    File list -> Click the “Delete” button to delete the specified file

UDF management

Resource management

The resource management and file management functions are similar. The difference is that the resource management is the uploaded UDF function, and the file management uploads the user program, script and configuration file. Operation function: rename, download, delete.

  • Upload udf resources

    Same as uploading files.

Function management

  • Create UDF function

    Click “Create UDF Function”, enter the udf function parameters, select the udf resource, and click “Submit” to create the udf function.

Currently only supports temporary UDF functions of HIVE

  • UDF function name: the name when the UDF function is entered
  • Package name Class name: Enter the full path of the UDF function
  • UDF resource: Set the resource file corresponding to the created UDF

Resource - 图6