Skip to content

feat: support milvus #1681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Jul 19, 2023
Merged

feat: support milvus #1681

merged 16 commits into from
Jul 19, 2023

Conversation

jupyterjazz
Copy link
Contributor

@jupyterjazz jupyterjazz commented Jul 2, 2023 •
edited
Loading

Taking over from here #1666

  • Find (vector search)
  • Filter
  • Get/Del
  • Hybrid Search (find+filter)
  • Subindex
  • Tests

Functionality-wise everything's ready, only thing left is to make the execution faster (milvus job takes 13 mins on CI)

Docs will be done in a separate PR

Note:
I had to bump protobuf and tensorflow versions

jupyterjazz and others added 2 commits July 2, 2023 23:16
@jupyterjazz
chore: first commit
b0c675b 
Signed-off-by: jupyterjazz <[email protected]>
@maxwelljin @jupyterjazz
feat: support milvus (#1666)
0e24ce9 
Signed-off-by: maxwelljin2 <[email protected]>
Co-authored-by: Saba Sturua <[email protected]>
@github-actions
Copy link

github-actions bot commented Jul 2, 2023

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz jupyterjazz linked an issue Jul 2, 2023 that may be closed by this pull request
Feature: please add milvus as storage backend to docarray v2 #1549
Closed
@jupyterjazz
refactor: some changes
d1edd96 
Signed-off-by: jupyterjazz <[email protected]>
@jupyterjazz
fix: merge conflicts
2113cdd 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@codecov
Copy link

codecov bot commented Jul 13, 2023 •
edited
Loading

Codecov Report

Patch coverage: 90.70% and project coverage change: +0.17 🎉

Comparison is base (b306c80) 85.51% compared to head (059a8f4) 85.68%.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1681      +/-   ##
==========================================
+ Coverage   85.51%   85.68%   +0.17%     
==========================================
  Files         132      133       +1     
  Lines        8303     8566     +263     
==========================================
+ Hits         7100     7340     +240     
- Misses       1203     1226      +23
Flag Coverage Δ
docarray 85.68% <90.70%> (+0.17%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
docarray/index/backends/redis.py 88.93% <ø> (+0.23%) ⬆️
docarray/utils/_internal/misc.py 65.57% <ø> (ø)
docarray/index/__init__.py 82.50% <80.00%> (+4.72%) ⬆️
docarray/index/backends/milvus.py 90.76% <90.76%> (ø)
docarray/index/backends/hnswlib.py 95.33% <100.00%> (+0.05%) ⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@jupyterjazz
chore: extra name
d8fdb49 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
chore: refactor extra
93f2dc0 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
chore: install milvus
06d0602 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
fix: installing pymilvus
30d133b 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
chore: bump tensorflow
9ddb0ff 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
refactor: collection loading
eff5ab1 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
feat: subindex
da8187e 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
feat: configurations
cf8aa92 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
feat: query builder
87d487d 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz
refactor: minor adjustments
6dce316 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@jupyterjazz jupyterjazz marked this pull request as ready for review July 14, 2023 19:00
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

samsja
samsja reviewed Jul 18, 2023
View reviewed changes
docarray/index/backends/milvus.py
Comment on lines +183 to +192
type_map = {
int: DataType.INT64,
float: DataType.FLOAT,
str: DataType.VARCHAR,
bytes: DataType.VARCHAR,
np.ndarray: DataType.FLOAT_VECTOR,
list: DataType.FLOAT_VECTOR,
AnyTensor: DataType.FLOAT_VECTOR,
AbstractTensor: DataType.FLOAT_VECTOR,
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can move this dict away from the function imo

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm but all other backends have it inside

samsja
samsja reviewed Jul 18, 2023
View reviewed changes
Copy link
Member

@samsja samsja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added one minor comments

@jupyterjazz
chore: merge recent changes
059a8f4 
Signed-off-by: jupyterjazz <[email protected]>
@github-actions
Copy link

This PR exceeds the recommended size of 1000 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size.

@github-actions
Copy link

📝 Docs are deployed on https://ft-feat-support-milvus--jina-docs.netlify.app 🎉

samsja
samsja reviewed Jul 19, 2023
View reviewed changes
.github/workflows/ci.yml
@@ -198,7 +198,7 @@ jobs:
python -m pip install --upgrade pip
python -m pip install poetry
poetry install --all-extras
poetry run pip install protobuf==3.19.0 # we check that we support 3.19
poetry run pip install protobuf==3.20.0 # we check that we support 3.19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
poetry run pip install protobuf==3.20.0 # we check that we support 3.19
poetry run pip install protobuf==3.20.0 # we check that we support 3.20

@JoanFM JoanFM merged commit 007f113 into main Jul 19, 2023
@JoanFM JoanFM deleted the feat-support-milvus branch July 19, 2023 11:22
@jupyterjazz jupyterjazz mentioned this pull request Aug 1, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JoanFM JoanFM JoanFM approved these changes

@samsja samsja samsja approved these changes

@JohannesMessner JohannesMessner Awaiting requested review from JohannesMessner

Assignees
No one assigned
Labels
area/cicd area/core area/housekeeping area/setup area/testing size/xl
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature: please add milvus as storage backend to docarray v2
4 participants
@jupyterjazz @JoanFM @samsja @maxwelljin