Unverified Commit b6a0d559 authored by rockxsj's avatar rockxsj Committed by GitHub
Browse files

fix and update k8s deploy (#3215) 



* Modify the AWS S3 request encryption method to V4.

* * update Dockerfile to add mysql-client for mysql database service test
* update wechat alert config replace placeholder from  to {xx} because of the  will be treat as env variable
* fix unit test of EnterpriseWeChatUtilsTest
* update values.yaml default config for helm deploy DS cluster
* update values.yaml default config for custom install
* update templates of helm for custom install
* update some name of env varibale, for example change from POSTGRESQL_HOST to DATABASE_HOST
* delele the redundancy requirements.yaml for heml
* update README* in the docker dir and subdir
* update startup scripts for support mysql as backend database

* * fix the ENV name wrong in docker-compose.yml and docker-stack.yml
* remove the didn't need port mapping in postgresql & zookeeper & dolphinscheduler-api

* * keep expose the port of postgresql & zk & api because of unit test needed

* * remove mysql-client package from dockerfile

Co-authored-by: default avatarrockxsj <junjun@mgtv.com>
parent 3aa34bc7

ambari_plugin/common-services/DOLPHIN/1.2.1/package/scripts/params.py

+2 −2
Original line number Diff line number Diff line
@@ -76,8 +76,8 @@ else: 
dolphin_alert_map = {}

wechat_push_url = 'https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=$token'

wechat_token_url = 'https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=$corpId&corpsecret=$secret'

wechat_team_send_msg = '{\"toparty\":\"$toParty\",\"agentid\":\"$agentId\",\"msgtype\":\"text\",\"text\":{\"content\":\"$msg\"},\"safe\":\"0\"}'

wechat_user_send_msg = '{\"touser\":\"$toUser\",\"agentid\":\"$agentId\",\"msgtype\":\"markdown\",\"markdown\":{\"content\":\"$msg\"}}'

wechat_team_send_msg = '{\"toparty\":\"{toParty}\",\"agentid\":\"{agentId}\",\"msgtype\":\"text\",\"text\":{\"content\":\"{msg}\"},\"safe\":\"0\"}'

wechat_user_send_msg = '{\"touser\":\"{toUser}\",\"agentid\":\"{agentId}\",\"msgtype\":\"markdown\",\"markdown\":{\"content\":\"{msg}\"}}'



dolphin_alert_map['enterprise.wechat.push.ur'] = wechat_push_url

dolphin_alert_map['enterprise.wechat.token.url'] = wechat_token_url

ambari_plugin/common-services/DOLPHIN/1.3.0/package/scripts/params.py

+2 −2
Original line number Diff line number Diff line
@@ -77,8 +77,8 @@ else: 
dolphin_alert_map = {}

wechat_push_url = 'https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=$token'

wechat_token_url = 'https://qyapi.weixin.qq.com/cgi-bin/gettoken?corpid=$corpId&corpsecret=$secret'

wechat_team_send_msg = '{\"toparty\":\"$toParty\",\"agentid\":\"$agentId\",\"msgtype\":\"text\",\"text\":{\"content\":\"$msg\"},\"safe\":\"0\"}'

wechat_user_send_msg = '{\"touser\":\"$toUser\",\"agentid\":\"$agentId\",\"msgtype\":\"markdown\",\"markdown\":{\"content\":\"$msg\"}}'

wechat_team_send_msg = '{\"toparty\":\"{toParty}\",\"agentid\":\"{agentId}\",\"msgtype\":\"text\",\"text\":{\"content\":\"{msg}\"},\"safe\":\"0\"}'

wechat_user_send_msg = '{\"touser\":\"{toUser}\",\"agentid\":\"{agentId}\",\"msgtype\":\"markdown\",\"markdown\":{\"content\":\"{msg}\"}}'



dolphin_alert_config_map = config['configurations']['dolphin-alert']

docker/build/Dockerfile

+1 −1
Original line number Diff line number Diff line
@@ -42,7 +42,7 @@ ADD ./apache-dolphinscheduler-incubating-${VERSION}-dolphinscheduler-bin.tar.gz 
RUN mv /opt/apache-dolphinscheduler-incubating-${VERSION}-dolphinscheduler-bin/ /opt/dolphinscheduler/

ENV DOLPHINSCHEDULER_HOME /opt/dolphinscheduler



#4. install pg

#4. install database, if use mysql as your backend database, the `mysql-client` package should be installed

RUN apk add postgresql postgresql-contrib

#5. modify nginx

RUN echo "daemon off;" >> /etc/nginx/nginx.conf && \

docker/build/README.md

+42 −24
Original line number Diff line number Diff line
@@ -16,7 +16,7 @@ Official Website: https://dolphinscheduler.apache.org 
#### You can start a dolphinscheduler instance

```

$ docker run -dit --name dolphinscheduler \ 

-e POSTGRESQL_USERNAME=test -e POSTGRESQL_PASSWORD=test -e POSTGRESQL_DATABASE=dolphinscheduler \

-e DATABASE_USERNAME=test -e DATABASE_PASSWORD=test -e DATABASE_DATABASE=dolphinscheduler \

-p 8888:8888 \

dolphinscheduler all

```
@@ -25,14 +25,14 @@ The default postgres user `root`, postgres password `root` and database `dolphin 


The default zookeeper is created in the `startup.sh`.



#### Or via Environment Variables **`POSTGRESQL_HOST`** **`POSTGRESQL_PORT`** **`POSTGRESQL_DATABASE`** **`ZOOKEEPER_QUORUM`**

#### Or via Environment Variables **`DATABASE_HOST`** **`DATABASE_PORT`** **`DATABASE_DATABASE`** **`ZOOKEEPER_QUORUM`**



You can specify **existing postgres service**. Example:



```

$ docker run -dit --name dolphinscheduler \

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

-p 8888:8888 \

dolphinscheduler all

```
@@ -42,7 +42,7 @@ You can specify **existing zookeeper service**. Example: 
```

$ docker run -dit --name dolphinscheduler \

-e ZOOKEEPER_QUORUM="l92.168.x.x:2181"

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" -e DATABASE_DATABASE="dolphinscheduler" \

-p 8888:8888 \

dolphinscheduler all

```
@@ -56,8 +56,8 @@ You can start a standalone dolphinscheduler server. 
```

$ docker run -dit --name dolphinscheduler \

-e ZOOKEEPER_QUORUM="l92.168.x.x:2181"

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

dolphinscheduler master-server

```
@@ -66,8 +66,8 @@ dolphinscheduler master-server 
```

$ docker run -dit --name dolphinscheduler \

-e ZOOKEEPER_QUORUM="l92.168.x.x:2181"

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

dolphinscheduler worker-server

```
@@ -75,8 +75,8 @@ dolphinscheduler worker-server 


```

$ docker run -dit --name dolphinscheduler \

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

-p 12345:12345 \

dolphinscheduler api-server

```
@@ -85,8 +85,8 @@ dolphinscheduler api-server 


```

$ docker run -dit --name dolphinscheduler \

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

dolphinscheduler alert-server

```
@@ -99,7 +99,7 @@ $ docker run -dit --name dolphinscheduler \ 
dolphinscheduler frontend

```



**Note**: You must be specify `POSTGRESQL_HOST` `POSTGRESQL_PORT` `POSTGRESQL_DATABASE` `POSTGRESQL_USERNAME` `POSTGRESQL_PASSWORD` `ZOOKEEPER_QUORUM` when start a standalone dolphinscheduler server.

**Note**: You must be specify `DATABASE_HOST` `DATABASE_PORT` `DATABASE_DATABASE` `DATABASE_USERNAME` `DATABASE_PASSWORD` `ZOOKEEPER_QUORUM` when start a standalone dolphinscheduler server.



## How to build a docker image
@@ -124,33 +124,51 @@ Please read `./docker/build/hooks/build` `./docker/build/hooks/build.bat` script 


The Dolphin Scheduler image uses several environment variables which are easy to miss. While none of the variables are required, they may significantly aid you in using the image.



**`POSTGRESQL_HOST`**

**`DATABASE_TYPE`**



This environment variable sets the host for PostgreSQL. The default value is `127.0.0.1`.

This environment variable sets the type for database. The default value is `postgresql`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.



**`POSTGRESQL_PORT`**

**`DATABASE_DRIVER`**



This environment variable sets the port for PostgreSQL. The default value is `5432`.

This environment variable sets the type for database. The default value is `org.postgresql.Driver`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.



**`POSTGRESQL_USERNAME`**

**`DATABASE_HOST`**



This environment variable sets the username for PostgreSQL. The default value is `root`.

This environment variable sets the host for database. The default value is `127.0.0.1`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.



**`POSTGRESQL_PASSWORD`**

**`DATABASE_PORT`**



This environment variable sets the password for PostgreSQL. The default value is `root`.

This environment variable sets the port for database. The default value is `5432`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`. 



**`POSTGRESQL_DATABASE`**

**`DATABASE_USERNAME`**



This environment variable sets the database for PostgreSQL. The default value is `dolphinscheduler`.

This environment variable sets the username for database. The default value is `root`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.



**`DATABASE_PASSWORD`**



This environment variable sets the password for database. The default value is `root`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.



**`DATABASE_DATABASE`**



This environment variable sets the database for database. The default value is `dolphinscheduler`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.



**`DATABASE_PARAMS`**



This environment variable sets the database for database. The default value is `characterEncoding=utf8`.



**Note**: You must be specify it when start a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`, `alert-server`.

docker/build/README_zh_CN.md

+42 −24
Original line number Diff line number Diff line
@@ -16,7 +16,7 @@ Official Website: https://dolphinscheduler.apache.org 
#### 你可以运行一个dolphinscheduler实例

```

$ docker run -dit --name dolphinscheduler \ 

-e POSTGRESQL_USERNAME=test -e POSTGRESQL_PASSWORD=test -e POSTGRESQL_DATABASE=dolphinscheduler \

-e DATABASE_USERNAME=test -e DATABASE_PASSWORD=test -e DATABASE_DATABASE=dolphinscheduler \

-p 8888:8888 \

dolphinscheduler all

```
@@ -25,14 +25,14 @@ dolphinscheduler all 


同时，默认的`Zookeeper`也会在`startup.sh`脚本中被创建。



#### 或者通过环境变量 **`POSTGRESQL_HOST`** **`POSTGRESQL_PORT`** **`ZOOKEEPER_QUORUM`** 使用已存在的服务

#### 或者通过环境变量 **`DATABASE_HOST`** **`DATABASE_PORT`** **`ZOOKEEPER_QUORUM`** 使用已存在的服务



你可以指定一个已经存在的 **`Postgres`** 服务. 如下:



```

$ docker run -dit --name dolphinscheduler \

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

-p 8888:8888 \

dolphinscheduler all

```
@@ -42,7 +42,7 @@ dolphinscheduler all 
```

$ docker run -dit --name dolphinscheduler \

-e ZOOKEEPER_QUORUM="l92.168.x.x:2181"

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" -e DATABASE_DATABASE="dolphinscheduler" \

-p 8888:8888 \

dolphinscheduler all

```
@@ -56,8 +56,8 @@ dolphinscheduler all 
```

$ docker run -dit --name dolphinscheduler \

-e ZOOKEEPER_QUORUM="l92.168.x.x:2181"

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

dolphinscheduler master-server

```
@@ -66,8 +66,8 @@ dolphinscheduler master-server 
```

$ docker run -dit --name dolphinscheduler \

-e ZOOKEEPER_QUORUM="l92.168.x.x:2181"

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

dolphinscheduler worker-server

```
@@ -75,8 +75,8 @@ dolphinscheduler worker-server 


```

$ docker run -dit --name dolphinscheduler \

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

-p 12345:12345 \

dolphinscheduler api-server

```
@@ -85,8 +85,8 @@ dolphinscheduler api-server 


```

$ docker run -dit --name dolphinscheduler \

-e POSTGRESQL_HOST="192.168.x.x" -e POSTGRESQL_PORT="5432" -e POSTGRESQL_DATABASE="dolphinscheduler" \

-e POSTGRESQL_USERNAME="test" -e POSTGRESQL_PASSWORD="test" \

-e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \

-e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \

dolphinscheduler alert-server

```
@@ -99,7 +99,7 @@ $ docker run -dit --name dolphinscheduler \ 
dolphinscheduler frontend

```



**注意**: 当你运行dolphinscheduler中的部分服务时，你必须指定这些环境变量 `POSTGRESQL_HOST` `POSTGRESQL_PORT` `POSTGRESQL_DATABASE` `POSTGRESQL_USERNAME` `POSTGRESQL_PASSWORD` `ZOOKEEPER_QUORUM`。

**注意**: 当你运行dolphinscheduler中的部分服务时，你必须指定这些环境变量 `DATABASE_HOST` `DATABASE_PORT` `DATABASE_DATABASE` `DATABASE_USERNAME` `DATABASE_PASSWORD` `ZOOKEEPER_QUORUM`。



## 如何构建一个docker镜像
@@ -124,33 +124,51 @@ c:\incubator-dolphinscheduler>.\docker\build\hooks\build.bat 


Dolphin Scheduler映像使用了几个容易遗漏的环境变量。虽然这些变量不是必须的，但是可以帮助你更容易配置镜像并根据你的需求定义相应的服务配置。



**`POSTGRESQL_HOST`**

**`DATABASE_TYPE`**



配置`PostgreSQL`的`HOST`， 默认值 `127.0.0.1`。

配置`database`的`TYPE`， 默认值 `postgresql`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`POSTGRESQL_PORT`**

**`DATABASE_DRIVER`**



配置`PostgreSQL`的`PORT`， 默认值 `5432`。

配置`database`的`DRIVER`， 默认值 `org.postgresql.Driver`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`POSTGRESQL_USERNAME`**

**`DATABASE_HOST`**



配置`PostgreSQL`的`USERNAME`， 默认值 `root`。

配置`database`的`HOST`， 默认值 `127.0.0.1`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`POSTGRESQL_PASSWORD`**

**`DATABASE_PORT`**



配置`PostgreSQL`的`PASSWORD`， 默认值 `root`。

配置`database`的`PORT`， 默认值 `5432`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`POSTGRESQL_DATABASE`**

**`DATABASE_USERNAME`**



配置`PostgreSQL`的`DATABASE`， 默认值 `dolphinscheduler`。

配置`database`的`USERNAME`， 默认值 `root`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`DATABASE_PASSWORD`**



配置`database`的`PASSWORD`， 默认值 `root`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`DATABASE_DATABASE`**



配置`database`的`DATABASE`， 默认值 `dolphinscheduler`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。



**`DATABASE_PARAMS`**



配置`database`的`PARAMS`， 默认值 `characterEncoding=utf8`。



**注意**: 当运行`dolphinscheduler`中`master-server`、`worker-server`、`api-server`、`alert-server`这些服务时，必须指定这个环境变量，以便于你更好的搭建分布式服务。