Development & Debugging with Kubernetes

This document describes how to use Kubernetes technology to simplify the development and debugging of Linkis projects. Before the introduction of Kubernetes tools, debugging Linkis was a very tedious and complex task, and sometimes it might be necessary to set up a Hadoop cluster for test. To improve this problem, we introduce an alternative approach , using Kubernetes technology, to create a Hadoop cluster on the development machine and pull up all Linkis services, which is a distributed environment and can be pulled up and destroyed at any time, and the developer connects to these services to performs step-by-step debugging through the JVM remote debugger. Here we use the following technologies:

  • Docker: A containerization technology to support the creation and use of Linux containers;
  • Kubernetes: An open source platform that automates the deployment and management of Linux containers, Kubernetes also integrates networking, storage, security, telemetry and other services to provide a comprehensive container-based infrastructure;
  • KinD: A tool that uses Docker containers as “Kubernetes nodes” to run local Kubernetes clusters;

  • Helm: An open source package management tool on Kubernetes that manages user resources on Kubernetes via the Helm command line tool and installation package (Chart);

  • Docker, minimum version v20.10.8+

  • Kubernetes, minimum version v1.21.0+
  • Helm, minimum version v3.0.0+.
  • KinD, minimum version v0.11.0+.

Helm is an open source package management tool on Kubernetes. Helm’s original goal was to provide users with a better way to manage all Kubernetes YAML files created on Kubernetes. When using Charts, the user provides a variable file, Helm uses the variables defined in this variable file to render the corresponding Chart, produce a Kubernetes YAML file, and then invoke the Kubernetes api to create the resources. Each Charts released to Kubernetes is called a Release, and a Chart can typically be installed multiple times into the same cluster, with a new Release being created each time it is installed.

Helm is relatively simple to install, please refer to the official documentation for installation: Installing Helm

Creating a Kubernetes test environment locally is a very common requirement, and the Kubernetes community offers a variety of solutions, such as MiniKube or MicroK8s. KinD, as the name suggests (Kubernetes in Docker), it uses Docker container to host nodes to create a test-oriented Kubernetes cluster.

KinD Architecture

Development & Debugging with Kubernetes - 图1

Deploying KinD is also very easy, please refer to the official deployment documentation: KinD Installation, please install Docker before install KinD.

⚠️ Note: KinD is a tool for testing purposes and cannot be used for production deployments. For example, KinD clusters cannot be used after the development machine is rebooted and need to be recreated (because KinD performs a series of initialization tasks after the Node container is created, which cannot be automatically reverted after the machine is rebooted).

Linkis provides several images, all of which have their Dockerfile and related scripts in the linkis-dist/docker directory. Linkis images include the following.

  • linkis: The Linkis service image, which contains binary packages of all components of Apache Linkis and various scripts.
  • linkis-web: Linkis Web console image, which contains the binary packages and various scripts of the Apache Linkis Web console, using nginx as the web server.
  • linkis-ldh: LDH is a test-oriented image, LDH image provides a complete, pseudo-distributed mode Apache Hadoop runtime environment, including HDFS, YARN, HIVE, Spark, Flink and Zookeeper, can be easily pulled up in the development environment of a fully real Hadoop environment to test the functionality of Linkis.

For details, please refer to: Linkis Docker Image Package.

Linkis Helm Chart is a Helm installation package developed according to the Helm Chart specification and is located in the linkis-dist/helm directory. The module directory structure is as follows:

  1. linkis-dist/helm
  2. ├── charts                                 # Charts directory, currently only contains Linkis Helm Chart
  3. │ └── linkis                               # Linkis Helm Chart directory
  4. │ ├── Chart.yaml                           # - Chart metadata
  5. │ ├── templates                            # - Chart template file containing Kubernetes YAML templates for all linkis components
  6. │ │ ├── NOTES.txt                          #   - Chart notes
  7. │ │ ├── _helpers.tpl                       #   - Chart variable helper templates
  8. │ │ ├── configmap-init-sql.yaml            #   - Database initialization SQL script template
  9. │ │ ├── configmap-linkis-config.yaml       #   - Linkis service configuration file template
  10. │ │ ├── configmap-linkis-web-config.yaml   #   - Linkis Web Console configuration file template
  11. │ │ ├── jobs.yaml                          #   - Kubernetes Job template, currently only includes a database initialization job, the database
  12. | | |                                      #     initialization SQL script will be executed by the job
  13. │ │ ├── linkis-cg-engineconnmanager.yaml   #   - Linkis EngineConnManager deployment template, which is a Kubernetes Deployment type workload
  14. │ │ ├── linkis-cg-engineplugin.yaml        #   - Linkis EngineConn deployment template, a Kubernetes Deployment type workload
  15. │ │ ├── linkis-cg-entrance.yaml            #   - Linkis Entrance deployment template, a Kubernetes Deployment type workload
  16. │ │ ├── linkis-cg-linkismanager.yaml       #   - Linkis Manager deployment template, a Kubernetes Deployment type workload
  17. │ │ ├── linkis-mg-eureka.yaml              #   - Linkis Eureka deployment template, a Kubernetes Statefulset type workload
  18. │ │ ├── linkis-mg-gateway.yaml             #   - Linkis Gateway deployment template, a Kubernetes Deployment type workload
  19. │ │ ├── linkis-ps-publicservice.yaml       #   - Linkis PublicService deployment template, a Kubernetes Deployment type workload
  20. │ │ ├── linkis-web.yaml                    #   - Linkis Web Console deployment template, a Kubernetes Deployment type workload
  21. │ │ └── serviceaccount.yaml                #   - Linkis related Kubernetes Service Account template
  22. │ └── values.yaml                          # - Linkis Helm Chart variable file, which provides Linkis Local schema related variables by default
  23. ├── scripts                                # Some tool scripts to simplify development and debugging
  24. │ ├── common.sh                            #   - public scripts, defining some public methods and variables
  25. │ ├── create-kind-cluster.sh               #   - Creates KinD clusters
  26. │ ├── install-charts-with-ldh.sh           #   - Deploy Linkis service on KinD cluster, using On-LDH deployment method, calling install-linkis.sh
  27. │ ├── install-charts.sh                    #   - Deploy Linkis service on KinD cluster, use Local deployment method, call install-linkis.sh
  28. │ ├── install-ldh.sh                       #   - Deploy LDH deployments on KinD clusters
  29. │ ├── install-linkis.sh                    #   - Deploy the Linkis service on the KinD cluster, either in Local or On-LDH mode
  30. │ ├── install-mysql.sh                     #   - Deploy a MySQL instance on the KinD cluster
  31. │ ├── login-pod.sh                         #   - Login to a Pod and open Bash for interaction
  32. │ ├── remote-debug-proxy.sh                #   - Turn on the JVM remote debug proxy
  33. │ └── resources                            #   - some resource files
  34. │ ├── kind-cluster.yaml                    #   - KinD cluster configuration, default is single Node 
  35. │ ├── ldh                                  #   - LDH related resource files
  36. │ │ │ ├── configmaps                       #     - LDH configuration files for each component
  37. │ │ │ │ ├── configmap-flink.yaml           #       - Flink configuration file
  38. │ │ │ │ ├── configmap-hadoop.yaml          #       - Hdfs & Yarn configuration file
  39. │ │ │ ├── configmap-hive.yaml              #       - Hive configuration file
  40. │ │ │ ├── configmap-spark.yaml             #       - Spark configuration file
  41. │ │ │ └── configmap-zookeeper.yaml         #       - Zookeeper configuration file
  42. │ │ └── ldh.yaml                           #   - LDH Kubernetes YAML, used to deploy LDH instances on KinD
  43. │ └── mysql.yaml                           # - MySQL Kubernetes YAML, for deploying MySQL instances on KinD

This project provides a set of tool scripts for quickly creating a Linkis environment for development testing. For production deployment, you need to modify the values.yaml file according to the actual cluster, and then deploy it using the Helm CLI. There are two common approaches to deploying with the Helm CLI:

  1. deploy directly using the helm install command. This is suitable for non-customized deployments.
  2. use the helm template command to generate Kubernetes YAML files, then manually modify these files, add custom configuration, and then deploy using the kubectl apply command. For advanced users who need to customize Kubernetes features that are not supported by Linkis Helm Charts, such as the need to use specific StorageClass or PVs.

LDH is a Hadoop cluster image for testing purposes, which provides a pseudo-distributed hadoop cluster for quick testing of On Hadoop deployment mode. This image contains the following hadoop components, and the default mode of the engine in LDH is on-yarn.

  • Hadoop 2.7.2 , included HDFS and YARN
  • Hive 2.3.3
  • Spark 2.4.3
  • Flink 1.12.2
  • ZooKeeper 3.5.9

LDH will start some initialization operations, such as format hdfs, create the initialization directory on HDFS, etc., these operations are defined in linkis-dist/docker/scripts/entry-point-ldh.sh file, add, modify, delete some initialization operations need to recreate LDH image to take effect.

In addition, the Hive component in LDH depends on external MySQL instance, you need to deploy MySQL instance first before you can use the Hive component in LDH.

  1. # Create a KinD cluster and deploy Linkis and LDH instances
  2. $> sh ./scripts/create-kind-cluster.sh \
  3.    && sh ./scripts/install-mysql.sh \
  4.    && sh ./scripts/install-ldh.sh
  6. # Quick Experience on LDH
  7. $> kubectl exec -it -n ldh $(kubectl get pod -n ldh -o jsonpath='{.items[0].metadata.name}') -- bash
  9. [root@ldh-96bdc757c-dnkbs /]# hdfs dfs -ls /
  10. Found 4 items
  11. drwxrwxrwx   - root supergroup          0 2022-07-31 02:48 /completed-jobs
  12. drwxrwxrwx   - root supergroup          0 2022-07-31 02:48 /spark2-history
  13. drwxrwxrwx   - root supergroup          0 2022-07-31 02:49 /tmp
  14. drwxrwxrwx   - root supergroup          0 2022-07-31 02:48 /user
  16. [root@ldh-96bdc757c-dnkbs /]# beeline -u jdbc:hive2://ldh.ldh.svc.cluster.local:10000/ -n hadoop
  17. Connecting to jdbc:hive2://ldh.ldh.svc.cluster.local:10000/
  18. Connected to: Apache Hive (version 2.3.3)
  19. Driver: Hive JDBC (version 2.3.3)
  20. Transaction isolation: TRANSACTION_REPEATABLE_READ
  21. Beeline version 2.3.3 by Apache Hive
  22. 0: jdbc:hive2://ldh.ldh.svc.cluster.local:100> create database demo;
  23. No rows affected (1.306 seconds)
  24. 0: jdbc:hive2://ldh.ldh.svc.cluster.local:100> use demo;
  25. No rows affected (0.046 seconds)
  26. 0: jdbc:hive2://ldh.ldh.svc.cluster.local:100> create table t1 (id int, data string);
  27. No rows affected (0.709 seconds)
  28. 0: jdbc:hive2://ldh.ldh.svc.cluster.local:100> insert into t1 values(1, 'linikis demo');
  29. WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
  30. No rows affected (5.491 seconds)
  31. 0: jdbc:hive2://ldh.ldh.svc.cluster.local:100> select * from t1;
  32. +--------+---------------+
  33. | t1.id  |    t1.data    |
  34. +--------+---------------+
  35. | 1      | linikis demo  |
  36. +--------+---------------+
  37. 1 row selected (0.39 seconds)
  38. 0: jdbc:hive2://ldh.ldh.svc.cluster.local:100> !q
  40. [root@ldh-96bdc757c-dnkbs /]# spark-sql
  41. 22/07/31 02:53:18 INFO hive.metastore: Trying to connect to metastore with URI thrift://ldh.ldh.svc.cluster.local:9083
  42. 22/07/31 02:53:18 INFO hive.metastore: Connected to metastore.
  43. ...
  44. 22/07/31 02:53:19 INFO spark.SparkContext: Running Spark version 2.4.3
  45. 22/07/31 02:53:19 INFO spark.SparkContext: Submitted application: SparkSQL::10.244.0.6
  46. ...
  47. 22/07/31 02:53:27 INFO yarn.Client: Submitting application application_1659235712576_0001 to ResourceManager
  48. 22/07/31 02:53:27 INFO impl.YarnClientImpl: Submitted application application_1659235712576_0001
  49. 22/07/31 02:53:27 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1659235712576_0001 and attemptId None
  50. 22/07/31 02:53:28 INFO yarn.Client: Application report for application_1659235712576_0001 (state: ACCEPTED)
  51. ...
  52. 22/07/31 02:53:36 INFO yarn.Client: Application report for application_1659235712576_0001 (state: RUNNING)
  53. ...
  54. Spark master: yarn, Application Id: application_1659235712576_0001
  55. 22/07/31 02:53:46 INFO thriftserver.SparkSQLCLIDriver: Spark master: yarn, Application Id: application_1659235712576_0001
  56. spark-sql> use demo;
  57. Time taken: 0.074 seconds
  58. 22/07/31 02:58:02 INFO thriftserver.SparkSQLCLIDriver: Time taken: 0.074 seconds
  59. spark-sql> select * from t1;
  60. ...
  61. 1       linikis demo
  62. 2       linkis demo spark sql
  63. Time taken: 3.352 seconds, Fetched 2 row(s)
  64. spark-sql> quit;
  66. [root@ldh-96bdc757c-dnkbs /]# zkCli.sh
  67. Connecting to localhost:2181
  68. Welcome to ZooKeeper!
  69. JLine support is enabled
  70. WATCHER::
  72. WatchedEvent state:SyncConnected type:None path:null
  74. [zk: localhost:2181(CONNECTED) 0] get -s /zookeeper/quota
  76. cZxid = 0x0
  77. ctime = Thu Jan 01 00:00:00 UTC 1970
  78. mZxid = 0x0
  79. mtime = Thu Jan 01 00:00:00 UTC 1970
  80. pZxid = 0x0
  81. cversion = 0
  82. dataVersion = 0
  83. aclVersion = 0
  84. ephemeralOwner = 0x0
  85. dataLength = 0
  86. numChildren = 0
  87. [zk: localhost:2181(CONNECTED) 1] quit
  89. # Start a Flink job in per-job cluster mode
  90. [root@ldh-96bdc757c-dnkbs /]# HADOOP_CLASSPATH=`hadoop classpath` flink run -t yarn-per-job /opt/ldh/current/flink/examples/streaming/TopSpeedWindowing.jar
  91. # Start Flink jobs in session mode,
  92. # Flink session is started when LDH Pod starts.
  93. [root@ldh-96bdc757c-dnkbs /]# flink run /opt/ldh/current/flink/examples/streaming/TopSpeedWindowing.jar
  94. Executing TopSpeedWindowing example with default input data set.
  95. Use --input to specify file input.
  96. Printing result to stdout. Use --output to specify output path.
  97. ...

The default KinD cluster description file used by the Linkis project is linkis-dist/helm/scripts/resources/kind-cluster.yaml, which creates a KinD cluster with one node by default. Multiple nodes can be added by remove the comments.

⚠️Note that KinD clusters are for testing purposes only.

  1. # linkis-dist/helm/scripts/resources/kind-cluster.yaml
  2. kind: Cluster
  3. apiVersion: kind.x-k8s.io/v1alpha4
  4. nodes:
  5.   - role: control-plane
  6.     extraMounts:
  7.       - hostPath: ${KIND_CLUSTER_HOST_PATH}     # Points to a directory on the development machine. This directory 
  8.                                                 # is mapped to the `/data` directory in the Kind Node container, which
  9.                                                 # Linkis Helm Charts uses by default as the data directory to mount into 
  10.                                                 # the Pod of each Linkis component. When Linkis is deployed in Local mode, 
  11.                                                 # all components actually use the same directory on the development machine
  12.                                                 # as if they were on the same machine, thus emulating the behavior of Local
  13.                                                 # mode. When deployed in On-Hadoop mode, this directory is not used.
  14.         containerPath: /data
  15. #  - role: worker                               # Remove comments to add 2 KinD nodes. Adding KinD nodes increases the time 
  16.                                                 # it takes to load Docker images to the KinD cluster, so it is not turned on 
  17.                                                 # by default. 
  18. #    extraMounts:
  19. #      - hostPath: ${KIND_CLUSTER_HOST_PATH}
  20. #        containerPath: /data
  21. #  - role: worker
  22. #    extraMounts:
  23. #      - hostPath: ${KIND_CLUSTER_HOST_PATH}
  24. #        containerPath: /data

The following steps describe how to develop and debug using Linkis containerized components (currently only supported for Linux and MacOS). Please confirm the following before proceeding.

  1. whether the Docker engine is already installed on the development machine
  2. whether Helm is installed on the development machine
  3. whether KinD has been installed on the development machine
  4. whether the Linkis image has been created as described in Linkis Docker image packaging

This step will create a KinD cluster and deploy MySQL, Linkis and LDH instances on it.

  1. $> cd linkis-dist/helm
  2. $> sh ./scripts/create-kind-cluster.sh \
  3. >    && sh ./scripts/install-mysql.sh \
  4. >    && sh ./scripts/install-ldh.sh \
  5. >    && sh ./scripts/install-charts-with-ldh.sh
  7. # Creating KinD cluster ...
  8. - kind cluster config: /var/folders/9d/bb6ggdm177j25q40yf5d50dm0000gn/T/kind-XXXXX.Fc2dFJbG/kind-cluster.yaml
  9. ...
  10. kind: Cluster
  11. apiVersion: kind.x-k8s.io/v1alpha4
  12. nodes:
  13.   - role: control-plane
  14.     extraMounts:
  15.       - hostPath: /var/folders/9d/bb6ggdm177j25q40yf5d50dm0000gn/T/kind-XXXXX.Fc2dFJbG/data
  16.         containerPath: /data
  17. ...
  18. Creating cluster "test-helm" ...
  19.  ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
  20.  ✓ Preparing nodes 📦  
  21.  ✓ Writing configuration 📜 
  22.  ✓ Starting control-plane 🕹️ 
  23.  ✓ Installing CNI 🔌 
  24.  ✓ Installing StorageClass 💾 
  25. Set kubectl context to "kind-test-helm"
  26. You can now use your cluster with:
  28. kubectl cluster-info --context kind-test-helm
  30. Have a nice day! 👋
  31. # Loading MySQL image ...
  32. Image: "mysql:5.7" with ID "sha256:3147495b3a5ce957dee2319099a8808c1418e0b0a2c82c9b2396c5fb4b688509" not yet present on node "test-helm-control-plane", loading...
  33. # Deploying MySQL ...
  34. namespace/mysql created
  35. service/mysql created
  36. deployment.apps/mysql created
  37. # LDH version: dev
  38. # Loading LDH image ...
  39. Image: "linkis-ldh:dev" with ID "sha256:aa3bde0a31bf704413fb75673fc2894b03a0840473d8fe15e2d7f7dd22f1f854" not yet present on node "test-helm-control-plane", loading...
  40. # Deploying LDH ...
  41. namespace/ldh created
  42. configmap/flink-conf created
  43. configmap/hadoop-conf created
  44. configmap/hive-conf created
  45. configmap/spark-conf created
  46. configmap/zookeeper-conf created
  47. service/ldh created
  48. deployment.apps/ldh created
  49. # Loading Linkis image ...
  50. Image: "linkis:dev" with ID "sha256:0dfa7882c4216305a80cf57efa8cfb483d006bae5ba931288ffb8025e1db4e58" not yet present on node "test-helm-control-plane", loading...
  51. Image: "linkis-web:dev" with ID "sha256:1dbe0e9319761dbe0e93197665d38077cb2432b8b755dee834928694875c8a22" not yet present on node "test-helm-control-plane", loading...
  52. # Installing linkis, image tag=dev,local mode=false ...
  53. NAME: linkis-demo
  54. NAMESPACE: linkis
  55. STATUS: deployed
  56. REVISION: 1
  57. TEST SUITE: None
  58. NOTES:
  59. ...
  61. ---
  62. Welcome to Apache Linkis (v1.3.0)!
  64. .___    .___ .______  .____/\ .___ .________
  65. |   |   : __|:      \ :   /  \: __||    ___/
  66. |   |   | : ||       ||.  ___/| : ||___    \
  67. |   |/\ |   ||   |   ||     \ |   ||       /
  68. |   /  \|   ||___|   ||      \|   ||__:___/
  69. |______/|___|    |___||___\  /|___|   : v1.3.0
  70.                            \/
  72. Linkis builds a layer of computation middleware between upper applications and underlying engines.
  73. Please visit https://linkis.apache.org/ for details.
  75. Enjoy!
  76. configmap/flink-conf created
  77. configmap/hadoop-conf created
  78. configmap/hive-conf created
  79. configmap/spark-conf created
  80. configmap/zookeeper-conf created
  82. $> kubectl get pods -n ldh -o wide
  83. NAME                   READY   STATUS    RESTARTS   AGE     IP           NODE                      NOMINATED NODE   READINESS GATES
  84. ldh-6648554447-ml2bn   1/1     Running   0          6m25s   10.244.0.6   test-helm-control-plane   <none>           <none>
  86. $> kubectl get pods -n linkis -o wide
  87. NAME                                                READY   STATUS             RESTARTS   AGE     IP            NODE                      NOMINATED NODE   READINESS GATES
  88. init-db-bcp85                                       0/1     Completed          0          4m52s   10.244.0.14   test-helm-control-plane   <none>           <none>
  89. linkis-demo-cg-engineconnmanager-659bf85689-ddvhw   1/1     Running            1          4m52s   10.244.0.7    test-helm-control-plane   <none>           <none>
  90. linkis-demo-cg-engineplugin-98bd6945-tsgjl          1/1     Running            1          4m52s   10.244.0.10   test-helm-control-plane   <none>           <none>
  91. linkis-demo-cg-entrance-858f74c868-xrd82            1/1     Running            0          4m52s   10.244.0.12   test-helm-control-plane   <none>           <none>
  92. linkis-demo-cg-linkismanager-6f96f69b8b-ns6st       1/1     Running            0          4m52s   10.244.0.11   test-helm-control-plane   <none>           <none>
  93. linkis-demo-mg-eureka-0                             1/1     Running            0          4m52s   10.244.0.13   test-helm-control-plane   <none>           <none>
  94. linkis-demo-mg-gateway-68ddb8c547-xgvhn             1/1     Running            0          4m52s   10.244.0.15   test-helm-control-plane   <none>           <none>
  95. linkis-demo-ps-publicservice-6bbf99fcd7-sc922       1/1     Running            0          4m52s   10.244.0.8    test-helm-control-plane   <none>           <none>
  96. linkis-demo-web-554bd7659f-nmdjl                    1/1     Running            0          4m52s   10.244.0.9    test-helm-control-plane   <none>           <none>

Each component has a JVM remote debug port of 5005 within the container, and these ports are mapped to different ports on the host as follows.

  • mg-eureka: 5001
  • mg-gateway: 5002
  • ps-publicservice: 5004
  • cg-linkismanager: 5007
  • cg-entrance: 5008
  • cg-engineconnmanager: 5009
  • cg-engineplugin: 5010

In addition, the Web Console is mapped to port 8087 on the host, which can be accessed by typing http://localhost:8087 in your browser.

  1. $> ./scripts/remote-debug-proxy.sh start      
  2. - starting port-forwad for [web] with mapping [local->8087:8087->pod] ...
  3. - starting port-forwad for [mg-eureka] with mapping [local->5001:5005->pod] ...
  4. - starting port-forwad for [mg-gateway] with mapping [local->5002:5005->pod] ...
  5. - starting port-forwad for [ps-publicservice] with mapping [local->5004:5005->pod] ...
  6. - starting port-forwad for [cg-linkismanager] with mapping [local->5007:5005->pod] ...
  7. - starting port-forwad for [cg-entrance] with mapping [local->5008:5005->pod] ...
  8. - starting port-forwad for [cg-engineconnmanager] with mapping [local->5009:5005->pod] ...
  9. - starting port-forwad for [cg-engineplugin] with mapping [local->5010:5005->pod] ..
  11. $> ./scripts/remote-debug-proxy.sh list 
  12. user            10972   0.0  0.1  5052548  31244 s001  S    12:57AM   0:00.10 kubectl port-forward -n linkis pod/linkis-demo-cg-engineplugin-98bd6945-tsgjl 5010:5005 --address=0.0.0.0
  13. user            10970   0.0  0.1  5053060  30988 s001  S    12:57AM   0:00.12 kubectl port-forward -n linkis pod/linkis-demo-cg-engineconnmanager-659bf85689-ddvhw 5009:5005 --address=0.0.0.0
  14. user            10968   0.0  0.1  5054084  30428 s001  S    12:57AM   0:00.10 kubectl port-forward -n linkis pod/linkis-demo-cg-entrance-858f74c868-xrd82 5008:5005 --address=0.0.0.0
  15. user            10966   0.0  0.1  5053316  30620 s001  S    12:57AM   0:00.11 kubectl port-forward -n linkis pod/linkis-demo-cg-linkismanager-6f96f69b8b-ns6st 5007:5005 --address=0.0.0.0
  16. user            10964   0.0  0.1  5064092  31152 s001  S    12:57AM   0:00.10 kubectl port-forward -n linkis pod/linkis-demo-ps-publicservice-6bbf99fcd7-sc922 5004:5005 --address=0.0.0.0
  17. user            10962   0.0  0.1  5051012  31244 s001  S    12:57AM   0:00.12 kubectl port-forward -n linkis pod/linkis-demo-mg-gateway-68ddb8c547-xgvhn 5002:5005 --address=0.0.0.0
  18. user            10960   0.0  0.1  5053060  31312 s001  S    12:57AM   0:00.13 kubectl port-forward -n linkis pod/linkis-demo-mg-eureka-0 5001:5005 --address=0.0.0.0
  20. ...
  22. # After debugging is complete, you can stop port forwarding with the following command
  23. $> ./scripts/remote-debug-proxy.sh stop 
  24. - stopping port-forward for [web] with mapping [local->8087:8087->pod] ...
  25. - stopping port-forward for [mg-eureka] with mapping [local->5001:5005->pod] ...
  26. - stopping port-forward for [mg-gateway] with mapping [local->5002:5005->pod] ...
  27. - stopping port-forward for [ps-publicservice] with mapping [local->5004:5005->pod] ...
  28. - stopping port-forward for [cg-linkismanager] with mapping [local->5007:5005->pod] ...
  29. - stopping port-forward for [cg-entrance] with mapping [local->5008:5005->pod] ...
  30. - stopping port-forward for [cg-engineconnmanager] with mapping [local->5009:5005->pod] ...
  31. - stopping port-forward for [cg-engineplugin] with mapping [local->5010:5005->pod] ...

Configure the IDE as follows to enable remote debugging:

Development & Debugging with Kubernetes - 图2

Turn on remote debugger Development & Debugging with Kubernetes - 图3

Set a breakpoint and submit a job for debugging

  1. $> ./scripts/login-pod.sh mg-gateway
  3. - login [mg-gateway]'s bash ...
  4. bash-4.2$ ./bin/./linkis-cli -engineType shell-1 -codeType shell -code "echo \"hello\" "  -submitUser hadoop -proxyUser hadoop
  5. =====Java Start Command=====
  6. exec /etc/alternatives/jre/bin/java -server -Xms32m -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/linkis/logs/linkis-cli -XX:ErrorFile=/opt/linkis/logs/linkis-cli/ps_err_pid%p.log -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC    -classpath /opt/linkis/conf/linkis-cli:/opt/linkis/lib/linkis-computation-governance/linkis-client/linkis-cli/*:/opt/linkis/lib/linkis-commons/public-module/*: -Dconf.root=/etc/linkis-conf -Dconf.file=linkis-cli.properties -Dlog.path=/opt/linkis/logs/linkis-cli -Dlog.file=linkis-client..log.20220925171540947077800  org.apache.linkis.cli.application.LinkisClientApplication '-engineType shell-1 -codeType shell -code echo "hello"  -submitUser hadoop -proxyUser hadoop'
  7. ...

Development & Debugging with Kubernetes - 图4

After debugging, you can use the following command to clean up the entire environment:

  1. $> kind delete clusters test-helm                                              
  2. Deleted clusters: ["test-helm"]
  1. $> kubectl logs -n linkis linkis-demo-cg-engineconnmanager-659bf85689-ddvhw -f
  3. + RUN_IN_FOREGROUND=true
  4. + /opt/linkis/sbin/linkis-daemon.sh start cg-engineconnmanager
  5. Start to check whether the cg-engineconnmanager is running
  6. Start server, startup script:  /opt/linkis/sbin/ext/linkis-cg-engineconnmanager
  7. =====Java Start Command=====
  8. java   -DserviceName=linkis-cg-engineconnmanager -Xmx512M -XX:+UseG1GC -Xloggc:/var/logs/linkis/linkis-cg-engineconnmanager-gc.log -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005  -cp /etc/linkis-conf:/opt/linkis/lib/linkis-commons/public-module/*:/opt/linkis/lib/linkis-computation-governance/linkis-cg-engineconnmanager/* org.apache.linkis.ecm.server.LinkisECMApplication  --eureka.instance.prefer-ip-address=true  --eureka.instance.instance-id=${spring.cloud.client.ip-address}:${spring.application.name}:${server.port} 2>&1
  9. OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
  10. Listening for transport dt_socket at address: 5005
  11. 16:32:41.101 [main] INFO  org.apache.linkis.common.conf.BDPConfiguration$ - ******************* Notice: The Linkis configuration file is linkis.properties ! *******************
  12. 16:32:41.130 [main] INFO  org.apache.linkis.common.conf.BDPConfiguration$ - *********************** Notice: The Linkis serverConf file is linkis-cg-engineconnmanager.properties ! ******************
  13. 16:32:41.222 [main] INFO  org.apache.linkis.LinkisBaseServerApp - Ready to start linkis-cg-engineconnmanager with args: --eureka.instance.prefer-ip-address=true
  14. --eureka.instance.instance-id=${spring.cloud.client.ip-address}:${spring.application.name}:${server.port}
  15. ...

Use . /scripts/login-pod.sh <component name> to access the component’s Pod to open a Bash for interactive operation, where the component name can be :

  • cg-engineconnmanager
  • cg-engineplugin
  • cg-entrance
  • cg-linkismanager
  • mg-eureka
  • mg-gateway
  • ps-publicservice
  • web
  1. $> ./scripts/login-pod.sh cg-engineconnmanager
  2. - login [cg-engineconnmanager]'s bash ...
  3. bash-4.2$ pwd
  4. /opt/linkis
  5. bash-4.2$  env |grep LINKIS
  6. LINKIS_DEMO_PS_PUBLICSERVICE_SERVICE_HOST=127.0.0.1
  7. LINKIS_DEMO_CG_LINKISMANAGER_PORT_9101_TCP_PROTO=tcp
  8. LINKIS_DEMO_WEB_PORT_8087_TCP_PORT=8087
  9. ...
  10. LINKIS_CLIENT_CONF_DIR=/etc/linkis-conf
  11. bash-4.2$ ps aux |grep linkis
  12. hadoop         1  0.0  0.0  11708  2664 ?        Ss   16:32   0:00 /bin/bash /opt/linkis/sbin/linkis-daemon.sh start cg-engineconnmanager
  13. hadoop        10  0.0  0.0  11708  2624 ?        S    16:32   0:00 sh /opt/linkis/sbin/ext/linkis-cg-engineconnmanager
  14. hadoop        11  0.0  0.0  11712  2536 ?        S    16:32   0:00 sh /opt/linkis/sbin/ext/linkis-common-start
  15. hadoop        12  4.0  3.2 4146404 400084 ?      Sl   16:32   0:35 java -DserviceName=linkis-cg-engineconnmanager -Xmx512M -XX:+UseG1GC -Xloggc:/var/logs/linkis/linkis-cg-engineconnmanager-gc.log -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -cp /etc/linkis-conf:/opt/linkis/lib/linkis-commons/public-module/*:/opt/linkis/lib/linkis-computation-governance/linkis-cg-engineconnmanager/* org.apache.linkis.ecm.server.LinkisECMApplication --eureka.instance.prefer-ip-address=true --eureka.instance.instance-id=${spring.cloud.client.ip-address}:${spring.application.name}:${server.port}
  16. bash-4.2$ exit
  17. exit