Merge pull request #3 from apache/dev

---

name: Bug report

about: Create a report to help us improve

title: "[BUG] bug title "

labels: bug

assignees: ''

---

*For better global communication, please give priority to using English description, thx! *

**Describe the bug**

A clear and concise description of what the bug is.

**To Reproduce**

Steps to reproduce the behavior, for example:

1. Go to '...'

2. Click on '....'

3. Scroll down to '....'

4. See error

**Expected behavior**

A clear and concise description of what you expected to happen.

**Screenshots**

If applicable, add screenshots to help explain your problem.

**Which version of Easy Scheduler:**

 -[1.1.0-preview]

**Additional context**

Add any other context about the problem here.

**Requirement or improvement

- Please describe about your requirements or improvement suggestions.

---

name: Feature request

about: Suggest an idea for this project

title: "[Feature]"

labels: new feature

assignees: ''

---

**Is your feature request related to a problem? Please describe.**

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**

A clear and concise description of what you want to happen.

**Describe alternatives you've considered**

A clear and concise description of any alternative solutions or features you've considered.

**Additional context**

Add any other context or screenshots about the feature request here.

EasyScheduler提交代码流程

=====

* First from the remote repository *https://github.com/analysys/EasyScheduler.git* fork code to your own repository

* there are three branches in the remote repository currently:

     * master normal delivery branch

            After the stable version is released, the code for the stable version branch is merged into the master branch.

    * dev daily development branch

            The daily development branch, the newly submitted code can pull requests to this branch.

    * branch-1.0.0 release version branch

            Release version branch, there will be 2.0 ... and other version branches, the version 

            branch only changes the error, does not add new features.

* Clone your own warehouse to your local

    `git clone https://github.com/analysys/EasyScheduler.git`

* Add remote repository address, named upstream

   `git remote add upstream https://github.com/analysys/EasyScheduler.git`

* View repository:

    `git remote -v`

> There will be two repositories at this time: origin (your own warehouse) and upstream (remote repository)

* Get/update remote repository code (already the latest code, skip it)

   `git fetch upstream`

* Synchronize remote repository code to local repository

```

git checkout origin/dev

git merge --no-ff upstream/dev

```

If remote branch has a new branch `DEV-1.0`, you need to synchronize this branch to the local repository.

```

git checkout -b dev-1.0 upstream/dev-1.0

git push --set-upstream origin dev1.0

```

* After modifying the code locally, submit it to your own repository:

`git commit -m 'test commit'`

`git push`

* Submit changes to the remote repository

* On the github page, click on the new pull request.

<p align = "center">

<img src = "http://geek.analysys.cn/static/upload/221/2019-04-02/90f3abbf-70ef-4334-b8d6-9014c9cf4c7f.png"width ="60%"/>

</ p>

* Select the modified local branch and the branch to merge past to create a pull request.

<p align = "center">

<img src = "http://geek.analysys.cn/static/upload/221/2019-04-02/fe7eecfe-2720-4736-951b-b3387cf1ae41.png"width ="60%"/>

</ p>

*  Next, the administrator is responsible for **merging** to complete the pull request

---

* 首先从远端仓库*https://github.com/analysys/EasyScheduler.git* fork一份代码到自己的仓库中

* 远端仓库中目前有三个分支：

* 把自己仓库clone到本地

    `git clone https://github.com/**/EasyScheduler.git`

    `git clone https://github.com/analysys/EasyScheduler.git`

*  添加远端仓库地址，命名为upstream

> 此时会有两个仓库：origin(自己的仓库)和upstream（远端仓库）

*  获取远端仓库代码（已经是最新代码，就跳过）

*  获取/更新远端仓库代码（已经是最新代码，就跳过）

    `git fetch upstream `

*  更新远端仓库代码

```

git checkout upstream/dev

git pull upstream dev

```

* 同步远端仓库代码到本地仓库

* 在本地修改代码以后，提交到自己仓库：

    `git ca -m 'test commit'`

    `git commit -m 'test commit'`

    `git push`

* 将修改提交到远端仓库

		<p align="center">

	   <img src="http://geek.analysys.cn/static/upload/221/2019-04-02/fe7eecfe-2720-4736-951b-b3387cf1ae41.png" width="60%" />

	 </p>

* 接下来由管理员负责将**Merge**完成此次pull request

#Maintin by jimmy

#Email: zhengge2012@gmail.com

FROM anapsix/alpine-java:8_jdk

WORKDIR /tmp

RUN wget http://archive.apache.org/dist/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz

RUN tar -zxvf apache-maven-3.6.1-bin.tar.gz && rm apache-maven-3.6.1-bin.tar.gz 

RUN mv apache-maven-3.6.1 /usr/lib/mvn

RUN chown -R root:root /usr/lib/mvn

RUN ln -s /usr/lib/mvn/bin/mvn /usr/bin/mvn

RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz

RUN tar -zxvf zookeeper-3.4.6.tar.gz

RUN mv zookeeper-3.4.6 /opt/zookeeper

RUN rm -rf zookeeper-3.4.6.tar.gz

RUN echo "export ZOOKEEPER_HOME=/opt/zookeeper" >>/etc/profile

RUN echo "export PATH=$PATH:$ZOOKEEPER_HOME/bin"  >>/etc/profile

ADD conf/zoo.cfg /opt/zookeeper/conf/zoo.cfg

#RUN source /etc/profile

#RUN zkServer.sh start

RUN apk add --no-cache git npm nginx mariadb mariadb-client mariadb-server-utils pwgen

WORKDIR /opt

RUN git clone https://github.com/analysys/EasyScheduler.git

WORKDIR /opt/EasyScheduler

RUN mvn -U clean package assembly:assembly -Dmaven.test.skip=true

RUN mv /opt/EasyScheduler/target/escheduler-1.0.0-SNAPSHOT /opt/easyscheduler

WORKDIR /opt/EasyScheduler/escheduler-ui

RUN npm install

RUN npm audit fix

RUN npm run build

RUN mkdir -p /opt/escheduler/front/server

RUN cp -rfv dist/* /opt/escheduler/front/server

WORKDIR /

RUN rm -rf /opt/EasyScheduler

#configure mysql server https://github.com/yobasystems/alpine-mariadb/tree/master/alpine-mariadb-amd64

ADD conf/run.sh /scripts/run.sh

RUN mkdir /docker-entrypoint-initdb.d && \

    mkdir /scripts/pre-exec.d && \

    mkdir /scripts/pre-init.d && \

    chmod -R 755 /scripts

RUN rm -rf /var/cache/apk/*

EXPOSE 8888

ENTRYPOINT ["/scripts/run.sh"]

Easy Scheduler

============

[![License](https://img.shields.io/badge/license-Apache%202-4EB1BA.svg)](https://www.apache.org/licenses/LICENSE-2.0.html)

[![Total Lines](https://tokei.rs/b1/github/analysys/EasyScheduler?category=lines)](https://github.com/analysys/EasyScheduler)

> Easy Scheduler for Big Data

**设计特点：** 一个分布式易扩展的可视化DAG工作流任务调度系统。致力于解决数据处理流程中错综复杂的依赖关系，使调度系统在数据处理流程中`开箱即用`。

其主要目标如下：

 - 以DAG图的方式将Task按照任务的依赖关系关联起来，可实时可视化监控任务的运行状态

 - 支持丰富的任务类型：Shell、MR、Spark、SQL(mysql、postgresql、hive、sparksql),Python,Sub_Process、Procedure等

 - 支持工作流定时调度、依赖调度、手动调度、手动暂停/停止/恢复，同时支持失败重试/告警、从指定节点恢复失败、Kill任务等操作

 - 支持工作流优先级、任务优先级及任务的故障转移及任务超时告警/失败

 - 支持工作流全局参数及节点自定义参数设置

 - 支持资源文件的在线上传/下载，管理等，支持在线文件创建、编辑

 - 支持任务日志在线查看及滚动、在线下载日志等

 - 实现集群HA，通过Zookeeper实现Master集群和Worker集群去中心化

 - 支持对`Master/Worker` cpu load，memory，cpu在线查看

 - 支持工作流运行历史树形/甘特图展示、支持任务状态统计、流程状态统计

 - 支持补数

 - 支持多租户

 - 支持国际化

 - 还有更多等待伙伴们探索

[![Stargazers over time](https://starchart.cc/analysys/EasyScheduler.svg)](https://starchart.cc/analysys/EasyScheduler)

### 与同类调度系统的对比

[![EN doc](https://img.shields.io/badge/document-English-blue.svg)](README.md)

[![CN doc](https://img.shields.io/badge/文档-中文版-blue.svg)](README_zh_CN.md)

![调度系统对比](http://geek.analysys.cn/static/upload/47/2019-03-01/9609ca82-cf8b-4d91-8dc0-0e2805194747.jpeg)

### 系统部分截图

### Design features: 

![](http://geek.analysys.cn/static/upload/221/2019-03-29/0a9dea80-fb02-4fa5-a812-633b67035ffc.jpeg)

A distributed and easy-to-expand visual DAG workflow scheduling system. Dedicated to solving the complex dependencies in data processing, making the scheduling system `out of the box` for data processing.

Its main objectives are as follows:

![](http://geek.analysys.cn/static/upload/221/2019-04-01/83686def-a54f-4169-8cae-77b1f8300cc1.png)

 - Associate the Tasks according to the dependencies of the tasks in a DAG graph, which can visualize the running state of task in real time.

 - Support for many task types: Shell, MR, Spark, SQL (mysql, postgresql, hive, sparksql), Python, Sub_Process, Procedure, etc.

 - Support process scheduling, dependency scheduling, manual scheduling, manual pause/stop/recovery, support for failed retry/alarm, recovery from specified nodes, Kill task, etc.

 - Support process priority, task priority and task failover and task timeout alarm/failure

 - Support process global parameters and node custom parameter settings

 - Support online upload/download of resource files, management, etc. Support online file creation and editing

 - Support task log online viewing and scrolling, online download log, etc.

 - Implement cluster HA, decentralize Master cluster and Worker cluster through Zookeeper

 - Support online viewing of `Master/Worker` cpu load, memory

 - Support process running history tree/gantt chart display, support task status statistics, process status statistics

 - Support backfilling data

 - Support multi-tenant

 - Support internationalization

 - There are more waiting partners to explore

![](http://geek.analysys.cn/static/upload/221/2019-03-29/83c937c7-1793-4d7a-aa28-b98460329fe0.jpeg)

### 文档

### What's in Easy Scheduler

- <a href="https://analysys.github.io/easyscheduler_docs_cn/后端部署文档.html" target="_blank">后端部署文档</a>

 Stability | Easy to use | Features | Scalability |

 -- | -- | -- | --

Decentralized multi-master and multi-worker | Visualization process defines key information such as task status, task type, retry times, task running machine, visual variables and so on at a glance.  |  Support pause, recover operation | support custom task types 

HA is supported by itself | All process definition operations are visualized, dragging tasks to draw DAGs, configuring data sources and resources. At the same time, for third-party systems, the api mode operation is provided. | Users on easyscheduler can achieve many-to-one or one-to-one mapping relationship through tenants and Hadoop users, which is very important for scheduling large data jobs. " | The scheduler uses distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster. Master and Worker support dynamic online and offline. 

Overload processing: Task queue mechanism, the number of schedulable tasks on a single machine can be flexibly configured, when too many tasks will be cached in the task queue, will not cause machine jam. | One-click deployment | Supports traditional shell tasks, and also support big data platform task scheduling: MR, Spark, SQL (mysql, postgresql, hive, sparksql), Python, Procedure, Sub_Process |  |

- <a href="https://analysys.github.io/easyscheduler_docs_cn/前端部署文档.html" target="_blank">前端部署文档</a>

- [**使用手册**](https://analysys.github.io/easyscheduler_docs_cn/系统使用手册.html?_blank "系统使用手册") 

- [**升级文档**](https://analysys.github.io/easyscheduler_docs_cn/升级文档.html?_blank "升级文档") 

- <a href="http://52.82.13.76:8888" target="_blank">我要体验</a> 普通用户登录：demo/demo123

### System partial screenshot

更多文档请参考 <a href="https://analysys.github.io/easyscheduler_docs_cn/" target="_blank">easyscheduler中文在线文档</a>

![image](https://user-images.githubusercontent.com/48329107/61368744-1f5f3b00-a8c1-11e9-9cf1-10f8557a6b3b.png)

![image](https://user-images.githubusercontent.com/48329107/61368966-9dbbdd00-a8c1-11e9-8dcc-a9469d33583e.png)

### 近期研发计划

![image](https://user-images.githubusercontent.com/48329107/61372146-f347b800-a8c8-11e9-8882-66e8934ada23.png)

EasyScheduler的工作计划：<a href="https://github.com/analysys/EasyScheduler/projects/1" target="_blank">研发计划</a> ，其中 In Develop卡片下是1.0.2版本的功能，TODO卡片是待做事项(包括 feature ideas)

### 贡献代码

### Document

非常欢迎大家来参与贡献代码，提交代码流程请参考：

https://github.com/analysys/EasyScheduler/blob/master/CONTRIBUTING.md

- <a href="https://analysys.github.io/easyscheduler_docs_cn/后端部署文档.html" target="_blank">Backend deployment documentation</a>

- <a href="https://analysys.github.io/easyscheduler_docs_cn/前端部署文档.html" target="_blank">Front-end deployment documentation</a>

### 感谢

- [**User manual**](https://analysys.github.io/easyscheduler_docs_cn/系统使用手册.html?_blank "User manual") 

Easy Scheduler使用了很多优秀的开源项目，比如google的guava、guice、grpc，netty，ali的bonecp，quartz，以及apache的众多开源项目等等，

正是由于站在这些开源项目的肩膀上，才有Easy Scheduler的诞生的可能。对此我们对使用的所有开源软件表示非常的感谢！我们也希望自己不仅是开源的受益者，也能成为开源的

贡献者，于是我们决定把易调度贡献出来，并承诺长期维护。也希望对开源有同样热情和信念的伙伴加入进来，一起为开源献出一份力！

- [**Upgrade document**](https://analysys.github.io/easyscheduler_docs_cn/升级文档.html?_blank "Upgrade document") 

- <a href="http://106.75.43.194:8888" target="_blank">Online Demo</a> 

### 帮助

More documentation please refer to <a href="https://analysys.github.io/easyscheduler_docs_cn/" target="_blank">[EasyScheduler online documentation]</a>

### Recent R&D plan

Work plan of Easy Scheduler: [R&D plan](https://github.com/analysys/EasyScheduler/projects/1), where `In Develop` card is the features of 1.1.0 version , TODO card is to be done (including feature ideas)

### How to contribute code

Welcome to participate in contributing code, please refer to the process of submitting the code:

[[How to contribute code](https://github.com/analysys/EasyScheduler/issues/310)]

### Thanks

Easy Scheduler uses a lot of excellent open source projects, such as google guava, guice, grpc, netty, ali bonecp, quartz, and many open source projects of apache, etc.

It is because of the shoulders of these open source projects that the birth of the Easy Scheduler is possible. We are very grateful for all the open source software used! We also hope that we will not only be the beneficiaries of open source, but also be open source contributors, so we decided to contribute to easy scheduling and promised long-term updates. We also hope that partners who have the same passion and conviction for open source will join in and contribute to open source!

### Get Help

The fastest way to get response from our developers is to submit issues,  or add our wechat : 510570367

### License

Please refer to [LICENSE](https://github.com/analysys/EasyScheduler/blob/dev/LICENSE) file.