DB error when using cortex

Data & Analysis
1

not sure it is a bug or our db has some issues.
We can run other cortex features pretty quickly (such as survey, surveyScore) but for some features such as Screen_Active it take forever and stucked into getting raw data step.

[INFO:feature_types:_wrapper2] Processing raw feature "lamp.screen_state"...
[INFO:feature_types:_wrapper2] No saved raw data found, getting new...

I looked into the couchDB and it return the error below:

[error] 2022-02-17T03:01:09.030902Z nonode@nohost <0.13411.7> -------- Invalid Index Def [{<<"map">>,{[{<<"fields">>,{[{<<"#parent">>,<<"desc">>},{<<"sensor">>,<<"desc">>},{<<"timestamp">>,<<"desc">>}]}},{<<"partial_filter_selector">>,{[]}}]}},{<<"reduce">>,<<"_count">>}]. Error: error, Reason: {badmatch,undefined}
[error] 2022-02-17T03:01:09.031004Z nonode@nohost <0.13411.7> -------- Invalid Index Def [{<<"map">>,{[{<<"fields">>,{[{<<"#parent">>,<<"desc">>},{<<"timestamp">>,<<"desc">>}]}},{<<"partial_filter_selector">>,{[]}}]}},{<<"reduce">>,<<"_count">>}]. Error: error, Reason: {badmatch,undefined}
[error] 2022-02-17T03:01:09.031114Z nonode@nohost <0.13411.7> -------- Invalid Index Def [{<<"map">>,{[{<<"fileds">>,{[]}}]}},{<<"reduce">>,<<"_count">>}]. Error: error, Reason: {badmatch,undefined}

Does anyone know how to fix this issue?

Thanks,
Chunlei

2

Your database node might not be powerful enough (core count or memory), since these CouchDB errors are usually thrown when the index is overwhelmed. You may also not have enough disk space for the index to be computed properly.

(These CouchDB errors could also be making the API Server hang, which is why Cortex appears to take “forever” to run, which we would want to investigate in more detail if so.)

We’d recommend switching from CouchDB to MongoDB (or a managed solution like AWS DocumentDB or Azure CosmosDB) if you can for future studies!

1 Like
3

Hi @avaidyam, Thanks very much! we scale up the node to c5d.2xlarge and deploy a python notebook docker service into the same node as you suggested. All the issue fixed and it runs very fast!

4

Great! Exactly as our team discovered: spending more money :money_with_wings: on AWS usually solves our issues. :sweat_smile: