Commit 320e5af8 authored by Jin Hai's avatar Jin Hai Committed by GitHub
Browse files

Merge pull request #87 from milvus-io/master 

Merge from master

Former-commit-id: df85d7d6895b385a9e367609872fc86edc379176
parents 05fda8c7 796fdcf1

CHANGELOG.md

+1 −1
Original line number Diff line number Diff line
@@ -11,7 +11,7 @@ Please mark all change in change log and use the ticket from JIRA. 
## Feature

## Task



# Milvus 0.5.0 (TODO)

# Milvus 0.5.0 (2019-10-21)



## Bug

- MS-568 - Fix gpuresource free error

README.md

+33 −16
Original line number Diff line number Diff line 
![Milvuslogo](https://github.com/milvus-io/docs/blob/branch-0.5.0/assets/milvus_logo.png)

![Milvuslogo](https://github.com/milvus-io/docs/blob/master/assets/milvus_logo.png)





![LICENSE](https://img.shields.io/badge/license-Apache--2.0-brightgreen)

![Language](https://img.shields.io/badge/language-C%2B%2B-blue)

[![codebeat badge](https://codebeat.co/badges/e030a4f6-b126-4475-a938-4723d54ec3a7?style=plastic)](https://codebeat.co/projects/github-com-jinhai-cn-milvus-master)



- [Slack Community](https://join.slack.com/t/milvusio/shared_invite/enQtNzY1OTQ0NDI3NjMzLWNmYmM1NmNjOTQ5MGI5NDhhYmRhMGU5M2NhNzhhMDMzY2MzNDdlYjM5ODQ5MmE3ODFlYzU3YjJkNmVlNDQ2ZTk)

- [Twitter](https://twitter.com/milvusio)
@@ -13,34 +15,49 @@ 


# Welcome to Milvus



Firstly, welcome, and thanks for your interest in [Milvus](https://milvus.io)! ​​No matter who you are, what you do, we greatly appreciate your contribution to help us reinvent data science with Milvus.​ :beers:



## What is Milvus



Milvus is an open source vector search engine which provides state-of-the-art similarity search and analysis for billion-scale feature vectors.

Milvus is an open source similarity search engine for massive feature vectors. Designed with heterogeneous computing architecture for the best cost efficiency. Searches over billion-scale vectors take only milliseconds with minimum computing resources.



Milvus provides stable Python, Java and C++ APIs.



Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/Releases/v0.4.0/).

Keep up-to-date with newest releases and latest updates by reading Milvus [release notes](https://milvus.io/docs/en/Releases/v0.5.0/).



- GPU-accelerated search engine

- Heterogeneous computing



  Milvus uses CPU/GPU heterogeneous computing architecture to process feature vectors, and are orders of magnitudes faster than traditional databases.

  Milvus is designed with heterogeneous computing architecture for the best performance and cost efficiency. 



- Various indexes

- Multiple indexes



  Milvus supports quantization indexing, tree-based indexing, and graph indexing algorithms. 

  Milvus supports a variety of indexing types that employs quantization, tree-based, and graph indexing techniques. 



- Intelligent scheduling

- Intelligent resource management



  Milvus optimizes the search computation and index building according to your data size and available resources. 

  Milvus automatically adapts search computation and index building processes based on your datasets and available resources.



- Horizontal scalability



  Milvus expands computation and storage by adding nodes during runtime, which allows you to scale the data size without redesigning the system.

  Milvus supports online / offline expansion to scale both storage and computation resources with simple commands.



- High availability



  Milvus is integrated with Kubernetes framework so that all single point of failures could be avoided.



- High compatibility



  Milvus is compatible with almost all deep learning models and major programming languages such as Python, Java and C++, etc.



- Ease of use



  Milvus can be easily installed in a few steps and enables you to exclusively focus on feature vectors. 



- Visualized monitor



  You can track system performance on Prometheus-based GUI monitor dashboards.



## Architecture

![Milvus_arch](https://github.com/milvus-io/docs/blob/branch-0.5.0/assets/milvus_arch.jpg)



![Milvus_arch](https://github.com/milvus-io/docs/blob/master/assets/milvus_arch.png)



## Get started
@@ -117,20 +134,20 @@ To edit Milvus settings in `conf/server_config.yaml` and `conf/log_config.conf`, 


#### Run Python example code



Make sure [Python 3.4](https://www.python.org/downloads/) or higher is already installed and in use.

Make sure [Python 3.5](https://www.python.org/downloads/) or higher is already installed and in use.



Install Milvus Python SDK.



```shell

# Install Milvus Python SDK

$ pip install pymilvus==0.2.0

$ pip install pymilvus==0.2.3

```



Create a new file `example.py`, and add [Python example code](https://github.com/milvus-io/pymilvus/blob/master/examples/AdvancedExample.py) to it.



Run the example code.



```python

```shell

# Run Milvus Python example

$ python3 example.py

```

codecov.yaml

0 → 100644
+14 −0
Original line number Diff line number Diff line 
#Configuration File for CodeCov

coverage:

  precision: 2

  round: down

  range: "70...100"



  status:

    project: on

    patch: yes

    changes: no



comment:

  layout: "header, diff, changes, tree"

  behavior: default

core/src/scheduler/task/SearchTask.cpp

+65 −65
Original line number Diff line number Diff line
@@ -307,71 +307,71 @@ XSearchTask::MergeTopkToResultSet(const std::vector<int64_t>& input_ids, const s 
    }

}



void

XSearchTask::MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,

                            const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance,

                            uint64_t src_input_k, uint64_t nq, uint64_t topk, bool ascending) {

    if (src_ids.empty() || src_distance.empty()) {

        return;

    }



    uint64_t output_k = std::min(topk, tar_input_k + src_input_k);

    std::vector<int64_t> id_buf(nq * output_k, -1);

    std::vector<float> dist_buf(nq * output_k, 0.0);



    uint64_t buf_k, src_k, tar_k;

    uint64_t src_idx, tar_idx, buf_idx;

    uint64_t src_input_k_multi_i, tar_input_k_multi_i, buf_k_multi_i;



    for (uint64_t i = 0; i < nq; i++) {

        src_input_k_multi_i = src_input_k * i;

        tar_input_k_multi_i = tar_input_k * i;

        buf_k_multi_i = output_k * i;

        buf_k = src_k = tar_k = 0;

        while (buf_k < output_k && src_k < src_input_k && tar_k < tar_input_k) {

            src_idx = src_input_k_multi_i + src_k;

            tar_idx = tar_input_k_multi_i + tar_k;

            buf_idx = buf_k_multi_i + buf_k;

            if ((ascending && src_distance[src_idx] < tar_distance[tar_idx]) ||

                (!ascending && src_distance[src_idx] > tar_distance[tar_idx])) {

                id_buf[buf_idx] = src_ids[src_idx];

                dist_buf[buf_idx] = src_distance[src_idx];

                src_k++;

            } else {

                id_buf[buf_idx] = tar_ids[tar_idx];

                dist_buf[buf_idx] = tar_distance[tar_idx];

                tar_k++;

            }

            buf_k++;

        }



        if (buf_k < output_k) {

            if (src_k < src_input_k) {

                while (buf_k < output_k && src_k < src_input_k) {

                    src_idx = src_input_k_multi_i + src_k;

                    buf_idx = buf_k_multi_i + buf_k;

                    id_buf[buf_idx] = src_ids[src_idx];

                    dist_buf[buf_idx] = src_distance[src_idx];

                    src_k++;

                    buf_k++;

                }

            } else {

                while (buf_k < output_k && tar_k < tar_input_k) {

                    tar_idx = tar_input_k_multi_i + tar_k;

                    buf_idx = buf_k_multi_i + buf_k;

                    id_buf[buf_idx] = tar_ids[tar_idx];

                    dist_buf[buf_idx] = tar_distance[tar_idx];

                    tar_k++;

                    buf_k++;

                }

            }

        }

    }



    tar_ids.swap(id_buf);

    tar_distance.swap(dist_buf);

    tar_input_k = output_k;

}

// void

// XSearchTask::MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,

//                            const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance,

//                            uint64_t src_input_k, uint64_t nq, uint64_t topk, bool ascending) {

//    if (src_ids.empty() || src_distance.empty()) {

//        return;

//    }

//

//    uint64_t output_k = std::min(topk, tar_input_k + src_input_k);

//    std::vector<int64_t> id_buf(nq * output_k, -1);

//    std::vector<float> dist_buf(nq * output_k, 0.0);

//

//    uint64_t buf_k, src_k, tar_k;

//    uint64_t src_idx, tar_idx, buf_idx;

//    uint64_t src_input_k_multi_i, tar_input_k_multi_i, buf_k_multi_i;

//

//    for (uint64_t i = 0; i < nq; i++) {

//        src_input_k_multi_i = src_input_k * i;

//        tar_input_k_multi_i = tar_input_k * i;

//        buf_k_multi_i = output_k * i;

//        buf_k = src_k = tar_k = 0;

//        while (buf_k < output_k && src_k < src_input_k && tar_k < tar_input_k) {

//            src_idx = src_input_k_multi_i + src_k;

//            tar_idx = tar_input_k_multi_i + tar_k;

//            buf_idx = buf_k_multi_i + buf_k;

//            if ((ascending && src_distance[src_idx] < tar_distance[tar_idx]) ||

//                (!ascending && src_distance[src_idx] > tar_distance[tar_idx])) {

//                id_buf[buf_idx] = src_ids[src_idx];

//                dist_buf[buf_idx] = src_distance[src_idx];

//                src_k++;

//            } else {

//                id_buf[buf_idx] = tar_ids[tar_idx];

//                dist_buf[buf_idx] = tar_distance[tar_idx];

//                tar_k++;

//            }

//            buf_k++;

//        }

//

//        if (buf_k < output_k) {

//            if (src_k < src_input_k) {

//                while (buf_k < output_k && src_k < src_input_k) {

//                    src_idx = src_input_k_multi_i + src_k;

//                    buf_idx = buf_k_multi_i + buf_k;

//                    id_buf[buf_idx] = src_ids[src_idx];

//                    dist_buf[buf_idx] = src_distance[src_idx];

//                    src_k++;

//                    buf_k++;

//                }

//            } else {

//                while (buf_k < output_k && tar_k < tar_input_k) {

//                    tar_idx = tar_input_k_multi_i + tar_k;

//                    buf_idx = buf_k_multi_i + buf_k;

//                    id_buf[buf_idx] = tar_ids[tar_idx];

//                    dist_buf[buf_idx] = tar_distance[tar_idx];

//                    tar_k++;

//                    buf_k++;

//                }

//            }

//        }

//    }

//

//    tar_ids.swap(id_buf);

//    tar_distance.swap(dist_buf);

//    tar_input_k = output_k;

//}



}  // namespace scheduler

}  // namespace milvus

core/src/scheduler/task/SearchTask.h

+4 −4
Original line number Diff line number Diff line
@@ -42,10 +42,10 @@ class XSearchTask : public Task { 
    MergeTopkToResultSet(const std::vector<int64_t>& input_ids, const std::vector<float>& input_distance,

                         uint64_t input_k, uint64_t nq, uint64_t topk, bool ascending, scheduler::ResultSet& result);



    static void

    MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,

                   const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance, uint64_t src_input_k,

                   uint64_t nq, uint64_t topk, bool ascending);

    //    static void

    //    MergeTopkArray(std::vector<int64_t>& tar_ids, std::vector<float>& tar_distance, uint64_t& tar_input_k,

    //                   const std::vector<int64_t>& src_ids, const std::vector<float>& src_distance, uint64_t

    //                   src_input_k, uint64_t nq, uint64_t topk, bool ascending);



 public:

    TableFileSchemaPtr file_;