Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JavaScript heap out of memory #13

Open
uafrontender opened this issue Aug 14, 2021 · 7 comments
Open

JavaScript heap out of memory #13

uafrontender opened this issue Aug 14, 2021 · 7 comments

Comments

@uafrontender
Copy link

I have >40k records in firestore and I got issue when I try to index more than 3 fields. Please check screenshots

Selection_009
Selection_010
Selection_011
Selection_012
@jasonbosco
Copy link
Member

@mkarkachov This could be related to #10. May I know what version of the extension you're running?

@uafrontender
Copy link
Author

@mkarkachov This could be related to #10. May I know what version of the extension you're running?

typesense/[email protected]

@jasonbosco
Copy link
Member

Hmmm. Could you share a sample firestore document? I'm curious to know the size of each document particularly and the data types.

@matslb
Copy link

matslb commented Dec 6, 2021 •
edited
Loading

I get the same error

@jasonbosco
Copy link
Member

@matslb Could you share a sample firestore document?

@matslb
Copy link

matslb commented Dec 16, 2021 •
edited
Loading

Here is a typical document in my firestore:

{
    "Sugar": "<3",
    "SubType": "Hvitvin",
    "RatingFetchDate": 1639440356127,
    "Id": "7855101",
    "ProductStatusSaleName": "Utsolgt",
    "PriceIsLowered": true,
    "SortingDiscount": 87.87,
    "RatingComment": "En tørr og frisk litt bitter vin som tåler 3-4 år. Godt kjøp.",
    "Stock": {
        "Stores": [
            {
                "pointOfService": {
                    "displayName": "Oslo, Tveita",
                    "id": "145",
                    "name": "145"
                },
                "stockInfo": {
                    "stockLevel": 21
                }
            }
        ]
    },
    "LatestPrice": 99.9,
    "Literprice": 134,
    "PriceChange": 66.64,
    "PriceChanges": 5,
    "LastUpdated": 1638662400000,
    "Stores": [
        "145"
    ],
    "ComparingPrice": 149.9,
    "Volume": 0.75,
    "PriceHistory": {
        "1625090400000": 149.9,
        "1638662400000": 99.9,
        "1598911200000": 169.7,
        "1594072800000": 169.6,
        "1609452000000": 170.6
    },
    "Rating": 84,
    "Country": [
        "Portugal"
    ],
    "Alcohol": 12.5,
    "Types": [
        "Hvitvin"
    ],
    "Type": "Svakvin",
    "Acid": "6,3",
    "LiterPriceAlcohol": 1072,
    "Description": {
        "recommendedFood": [
            {
                "foodDesc": "Skalldyr",
                "foodId": "B"
            },
            {
                "foodDesc": "Fisk",
                "foodId": "C"
            },
            {
                "foodId": "D",
                "foodDesc": "Lyst kjøtt"
            }
        ],
        "sweetness": "01",
        "fullness": "05",
        "bitterness": "",
        "characteristics": {
            "odour": "Lukter av friske sitronaromaer og noe grapefrukt.",
            "colour": "Lys gyllengul.",
            "taste": "Frisk syre med god frukt. God konsentrasjon."
        },
        "freshness": "07",
        "tannins": ""
    },
    "Name": "Quinta de Chocapalha Arinto 2017",
    "ManufacturerName": "Quinta de Chocapalha",
    "Discount": 66.6444296197465
}

These are the fields being indexed by Typesense, set in the extension configuration in firebase:

  • Name, Types, Country, PriceIsLowered, PriceChange, LiterPriceAlcohol ,Rating ,LastUpdated, Stores, Id

The fields match the collection schema

@0x80
Copy link

0x80 commented Oct 22, 2023

Reading the extension code I think heap issues are also to be expected in the backfill function, as it first loads the complete set of documents from the firestore collection before it starts batching documents and sending them to the typesense API.

I work on a project with a users collection that is nearing 1M records, containing user profile info and application settings. You do not want to load all of that upfront to the heap before you start processing chunks.

I wrote utility functions to deal with these kinds of scenarios, but I haven't gotten around to packaging and releasing them to the public.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jasonbosco @0x80 @uafrontender @matslb