Commit dd5d4a5e authored by Jin Hai's avatar Jin Hai Committed by GitHub
Browse files

Merge branch '0.5.1' into master

Former-commit-id: 4187de60052a91a0b20b28d362cf6a986572c74a
parents 6893619d 5e8fc0dc
Loading
Loading
Loading
Loading
+24 −1
Original line number Diff line number Diff line
@@ -5,10 +5,33 @@ Please mark all change in change log and use the ticket from JIRA.
# Milvus 0.5.1 (TODO)

## Bug
- \#134 - JFrog cache error
- \#161 - Search IVFSQHybrid crash on gpu
- \#169 - IVF_FLAT search out of memory

## Feature
- \#90 - The server start error messages could be improved to enhance user experience
- \#104 - test_scheduler core dump
- \#115 - Using new structure for tasktable
- \#139 - New config option use_gpu_threshold
- \#146 - Add only GPU and only CPU version for IVF_SQ8 and IVF_FLAT
- \#164 - Add CPU version for building index

## Improvement
- \#64 - Improvement dump function in scheduler
- \#80 - Print version information into log during server start
- \#82 - Move easyloggingpp into "external" directory
- \#92 - Speed up CMake build process
- \#96 - Remove .a file in milvus/lib for docker-version
- \#118 - Using shared_ptr instead of weak_ptr to avoid performance loss
- \#122 - Add unique id for Job
- \#130 - Set task state MOVED after resource copy it completed
- \#149 - Improve large query optimizer pass
- \#156 - Not return error when search_resources and index_build_device set cpu
- \#159 - Change the configuration name from 'use_gpu_threshold' to 'gpu_search_threshold'
- \#168 - Improve result reduce
- \#175 - add invalid config unittest

## Feature
## Task

# Milvus 0.5.0 (2019-10-21)
+1 −1
Original line number Diff line number Diff line
@@ -28,7 +28,7 @@ Contributions to Milvus fall into the following categories.

If you have improvements to Milvus, send us your pull requests! For those just getting started, see [GitHub workflow](#github-workflow).

The Milvus team members will review your pull requests, and once it is accepted, it will be given a `ready to merge` label. This means we are working on submitting your pull request to the internal repository. After the change has been submitted internally, your pull request will be merged automatically on GitHub.
The Milvus team members will review your pull requests, and once it is accepted, the status of the projects to which it is associated will be changed to **Reviewer approved**. This means we are working on submitting your pull request to the internal repository. After the change has been submitted internally, your pull request will be merged automatically on GitHub.

### GitHub workflow

+1 −1
Original line number Diff line number Diff line
@@ -25,6 +25,6 @@
| libunwind     | [MIT](https://github.com/libunwind/libunwind/blob/master/LICENSE)                                                          |
| gperftools    | [BSD 3-Clause](https://github.com/gperftools/gperftools/blob/master/COPYING)                                                 |
| grpc          | [Apache 2.0](https://github.com/grpc/grpc/blob/master/LICENSE)                                                   |
| EASYLOGGINGPP | [MIT](https://github.com/zuhd-org/easyloggingpp/blob/master/LICENSEhttps://github.com/zuhd-org/easyloggingpp/blob/master/LICENSE)                                                          |
| EASYLOGGINGPP | [MIT](https://github.com/zuhd-org/easyloggingpp/blob/master/LICENSE)                                                          |
| Json          | [MIT](https://github.com/nlohmann/json/blob/develop/LICENSE.MIT)                                                          |
+29 −150
Original line number Diff line number Diff line

![Milvuslogo](https://github.com/milvus-io/docs/blob/master/assets/milvus_logo.png)


![LICENSE](https://img.shields.io/badge/license-Apache--2.0-brightgreen)
![Language](https://img.shields.io/badge/language-C%2B%2B-blue)
[![codebeat badge](https://codebeat.co/badges/e030a4f6-b126-4475-a938-4723d54ec3a7?style=plastic)](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)
![Release](https://img.shields.io/badge/release-v0.5.0-orange)
![Release_date](https://img.shields.io/badge/release_date-October-yellowgreen)

- [Slack Community](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)
- [Twitter](https://twitter.com/milvusio)
- [Facebook](https://www.facebook.com/io.milvus.5)
- [Blog](https://www.milvus.io/blog/)
- [CSDN](https://zilliz.blog.csdn.net/)
- [中文官网](https://www.milvus.io/zh-CN/)


# Welcome to Milvus
[中文版](README_CN.md) 

## What is Milvus

Milvus is the world's fastest similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.

Milvus provides stable Python, Java and C++ APIs.

Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/releases/v0.5.0/).

- Heterogeneous computing

  Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency. 

- Multiple indexes

  Milvus supports a variety of indexing types that employs quantization, tree-based, and graph indexing techniques. 

- Intelligent resource management

  Milvus automatically adapts search computation and index building processes based on your datasets and available resources.

- Horizontal scalability

  Milvus supports online / offline expansion to scale both storage and computation resources with simple commands.

- High availability

  Milvus is integrated with Kubernetes framework so that all single point of failures could be avoided.

- High compatibility

  Milvus is compatible with almost all deep learning models and major programming languages such as Python, Java and C++, etc.
Milvus is the world's fastest similarity search engine for massive feature vectors. Built with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.

- Ease of use
For more detailed introduction of Milvus and its architecture, see [Milvus overview](https://www.milvus.io/docs/en/aboutmilvus/overview/).

  Milvus can be easily installed in a few steps and enables you to exclusively focus on feature vectors. 
Milvus provides stable [Python](https://pypi.org/project/pymilvus/), [Java](https://milvus-io.github.io/milvus-sdk-java/javadoc/io/milvus/client/package-summary.html) and C++ APIs. 

- Visualized monitor

  You can track system performance on Prometheus-based GUI monitor dashboards.

## Architecture

![Milvus_arch](https://github.com/milvus-io/docs/blob/master/assets/milvus_arch.png)
Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://www.milvus.io/docs/en/release/v0.5.0/).

## Get started

### Hardware Requirements

| Component | Recommended configuration           |
| --------- | ----------------------------------- |
| CPU       | Intel CPU Haswell or higher         |
| GPU       | NVIDIA Pascal series or higher      |
| Memory    | 8 GB or more (depends on data size) |
| Storage   | SATA 3.0 SSD or higher              |

### Install using docker

Use Docker to install Milvus is a breeze. See the [Milvus install guide](https://milvus.io/docs/en/userguide/install_milvus/) for details.

### Build from source

#### Software requirements
See the [Milvus install guide](https://www.milvus.io/docs/en/userguide/install_milvus/) for using Docker containers. To install Milvus from source code, see [build from source](install.md).

- Ubuntu 18.04 or higher
- CMake 3.14 or higher
- CUDA 10.0 or higher
- NVIDIA driver 418 or higher

#### Compilation

##### Step 1 Install dependencies

```shell
$ cd [Milvus sourcecode path]/core
./ubuntu_build_deps.sh
```

##### Step 2 Build

```shell
$ cd [Milvus sourcecode path]/core
$ ./build.sh -t Debug
or 
$ ./build.sh -t Release
```

When the build is completed, all the stuff that you need in order to run Milvus will be installed under `[Milvus root path]/core/milvus`.

#### Launch Milvus server

```shell
$ cd [Milvus root path]/core/milvus
```

Add `lib/` directory to `LD_LIBRARY_PATH`

```
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/milvus/lib
```

Then start Milvus server:

```
$ cd scripts
$ ./start_server.sh
```

To stop Milvus server, run:

```shell
$ ./stop_server.sh
```

To edit Milvus settings in `conf/server_config.yaml` and `conf/log_config.conf`, please read [Milvus Configuration](https://github.com/milvus-io/docs/blob/master/reference/milvus_config.md).
To edit Milvus settings, read [Milvus configuration](https://www.milvus.io/docs/en/reference/milvus_config/).

### Try your first Milvus program

#### Run Python example code

Make sure [Python 3.5](https://www.python.org/downloads/) or higher is already installed and in use.

Install Milvus Python SDK.

```shell
# Install Milvus Python SDK
$ pip install pymilvus==0.2.3
```

Create a new file `example.py`, and add [Python example code](https://github.com/milvus-io/pymilvus/blob/master/examples/AdvancedExample.py) to it.

Run the example code.

```shell
# Run Milvus Python example
$ python3 example.py
```
Try running a program with Milvus using [Python](https://www.milvus.io/docs/en/userguide/example_code/) or [Java example code](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples). 

#### Run C++ example code
To use C++ example code, use below command:

```shell
 # Run Milvus C++ example
@@ -160,41 +38,42 @@ $ python3 example.py
 $ ./sdk_simple
```

#### Run Java example code
Make sure Java 8 or higher is already installed.
## Roadmap

Refer to [this link](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples) for the example code.
Please read our [roadmap](https://milvus.io/docs/en/roadmap/) for upcoming features.

## Contribution guidelines

Contributions are welcomed and greatly appreciated. If you want to contribute to Milvus, please read our [contribution guidelines](CONTRIBUTING.md). This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.
Contributions are welcomed and greatly appreciated. Please read our [contribution guidelines](CONTRIBUTING.md) for detailed contribution workflow. This project adheres to the [code of conduct](CODE_OF_CONDUCT.md) of Milvus. By participating, you are expected to uphold this code.

We use [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) to track issues and bugs. For general questions and public discussions, please join our community.

## Join the Milvus community
## Join our community

To connect with other users and contributors, welcome to join our [slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk). 
To connect with other users and contributors, welcome to join our [Slack channel](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk). 

## Milvus Roadmap
## Thanks

Please read our [roadmap](https://milvus.io/docs/en/roadmap/) to learn about upcoming features.
We greatly appreciate the help of the following people.

## Resources
- [akihoni](https://github.com/akihoni) found a broken link and a small typo in the README file.

[Milvus official website](https://www.milvus.io)
## Resources

[Milvus docs](https://www.milvus.io/docs/en/userguide/install_milvus/)
- [Milvus.io](https://www.milvus.io)

[Milvus bootcamp](https://github.com/milvus-io/bootcamp)
- [Milvus bootcamp](https://github.com/milvus-io/bootcamp)

[Milvus blog](https://www.milvus.io/blog/)
- [Milvus Medium](https://medium.com/@milvusio)

[Milvus CSDN](https://zilliz.blog.csdn.net/)
- [Milvus CSDN](https://zilliz.blog.csdn.net/)

[Milvus roadmap](https://milvus.io/docs/en/roadmap/)
- [Milvus Twitter](https://twitter.com/milvusio)

- [Milvus Facebook](https://www.facebook.com/io.milvus.5)

## License

[Apache 2.0 license](LICENSE)
[Apache License 2.0](LICENSE)

README_CN.md

0 → 100644
+197 −0
Original line number Diff line number Diff line
![Milvuslogo](https://raw.githubusercontent.com/milvus-io/docs/master/assets/milvus_logo.png)

![LICENSE](https://img.shields.io/badge/license-Apache--2.0-brightgreen)
![Language](https://img.shields.io/badge/language-C%2B%2B-blue)
[![codebeat badge](https://codebeat.co/badges/e030a4f6-b126-4475-a938-4723d54ec3a7?style=plastic)](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)

![Release](https://img.shields.io/badge/release-v0.5.0-orange)
![Release_date](https://img.shields.io/badge/release_date-October-yellowgreen)

- [Slack 频道](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)
- [Twitter](https://twitter.com/milvusio)
- [Facebook](https://www.facebook.com/io.milvus.5)
- [博客](https://www.milvus.io/blog/)
- [CSDN](https://zilliz.blog.csdn.net/)
- [中文官网](https://www.milvus.io/zh-CN/)

# 欢迎来到 Milvus

## Milvus 是什么

Milvus 是一款开源的、针对海量特征向量的相似性搜索引擎。基于异构众核计算框架设计,成本更低,性能更好。在有限的计算资源下,十亿向量搜索仅毫秒响应。

Milvus 提供稳定的 Python、Java 以及 C++ 的 API 接口。

通过 [版本发布说明](https://milvus.io/docs/zh-CN/release/v0.5.0/) 获取最新发行版本的 Milvus。

- 异构众核

  Milvus 基于异构众核计算框架设计,成本更低,性能更好。

- 多元化索引

  Milvus 支持多种索引方式,使用量化索引、基于树的索引和图索引等算法。

- 资源智能管理

  Milvus 根据实际数据规模和可利用资源,智能调节优化查询计算和索引构建过程。

- 水平扩容

  Milvus 支持在线 / 离线扩容,仅需执行简单命令,便可弹性伸缩计算节点和存储节点。

- 高可用性

  Milvus 集成了 Kubernetes 框架,能有效避免单点障碍情况的发生。

- 简单易用

  Milvus 安装简单,使用方便,并可使您专注于特征向量。

- 可视化监控

  您可以使用基于 Prometheus 的图形化监控,以便实时跟踪系统性能。

## 整体架构

![Milvus_arch](https://github.com/milvus-io/docs/blob/master/assets/milvus_arch.png)

## 开始使用 Milvus

### 硬件要求

| 硬件设备 | 推荐配置                              |
| -------- | ------------------------------------- |
| CPU      | Intel CPU Haswell 及以上              |
| GPU      | NVIDIA Pascal 系列及以上              |
| 内存     | 8 GB 或以上(取决于具体向量数据规模) |
| 硬盘     | SATA 3.0 SSD 及以上                   |

### 使用 Docker

您可以方便地使用 Docker 安装 Milvus。具体请查看 [Milvus 安装指南](https://milvus.io/docs/zh-CN/userguide/install_milvus/)

### 从源代码编译

#### 软件要求

- Ubuntu 18.04 及以上
- CMake 3.14 及以上
- CUDA 10.0 及以上
- NVIDIA driver 418 及以上

#### 编译

##### 第一步 安装依赖项

```shell
$ cd [Milvus sourcecode path]/core
$ ./ubuntu_build_deps.sh
```

##### 第二步 编译

```shell
$ cd [Milvus sourcecode path]/core
$ ./build.sh -t Debug
or 
$ ./build.sh -t Release
```

当您成功编译后,所有 Milvus 必需组件将安装在`[Milvus root path]/core/milvus`路径下。

##### 启动 Milvus 服务

```shell
$ cd [Milvus root path]/core/milvus
```

`LD_LIBRARY_PATH` 中添加 `lib/` 目录:

```shell
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/milvus/lib
```

启动 Milvus 服务:

```shell
$ cd scripts
$ ./start_server.sh
```

若要停止 Milvus 服务,请使用如下命令:

```shell
$ ./stop_server.sh
```

若需要修改 Milvus 配置文件 `conf/server_config.yaml``conf/log_config.conf`,请查看 [Milvus 配置](https://milvus.io/docs/zh-CN/reference/milvus_config/)

### 开始您的第一个 Milvus 程序

#### 运行 Python 示例代码

请确保系统的 Python 版本为 [Python 3.5](https://www.python.org/downloads/) 或以上。

安装 Milvus Python SDK。

```shell
# Install Milvus Python SDK
$ pip install pymilvus==0.2.3
```

创建 `example.py` 文件,并向文件中加入 [Python 示例代码](https://github.com/milvus-io/pymilvus/blob/master/examples/advanced_example.py)

运行示例代码

```shell
# Run Milvus Python example
$ python3 example.py
```

#### 运行 C++ 示例代码

```shell
 # Run Milvus C++ example
 $ cd [Milvus root path]/core/milvus/bin
 $ ./sdk_simple
```

#### 运行 Java 示例代码

请确保系统的 Java 版本为 Java 8 或以上。

请从[此处](https://github.com/milvus-io/milvus-sdk-java/tree/master/examples)获取 Java 示例代码。

## 贡献者指南

我们由衷欢迎您推送贡献。关于贡献流程的详细信息,请参阅 [贡献者指南](https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md)。本项目遵循 Milvus [行为准则](https://github.com/milvus-io/milvus/blob/master/CODE_OF_CONDUCT.md)。如果您希望参与本项目,请遵守该准则的内容。

我们使用 [GitHub issues](https://github.com/milvus-io/milvus/issues/new/choose) 追踪问题和补丁。若您希望提出问题或进行讨论,请加入我们的社区。

## 加入 Milvus 社区

欢迎加入我们的 [Slack 频道](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk) 以便与其他用户和贡献者进行交流。

## Milvus 路线图

请阅读我们的[路线图](https://milvus.io/docs/zh-CN/roadmap/)以获得更多即将开发的新功能。

## 相关链接

[Milvus 官方网站](https://www.milvus.io/)

[Milvus 文档](https://www.milvus.io/docs/en/userguide/install_milvus/)

[Milvus 在线训练营](https://github.com/milvus-io/bootcamp)

[Milvus 博客](https://www.milvus.io/blog/)

[Milvus CSDN](https://zilliz.blog.csdn.net/)

[Milvus 路线图](https://milvus.io/docs/en/roadmap/)

## 许可协议

[Apache 许可协议2.0版](https://github.com/milvus-io/milvus/blob/master/LICENSE)
Loading