GridFS implementation should do the job but it is for files and altering the node driver code is not an option for obvious reasons (although we tried some minor alterations of internal objects such as ReadableCursorStream without success). However, this does not seem possible for objects (no config options, web refs or code features available to address it). The solution we seek for this particular issue would be to consume cursor data as a string/buffer stream to avoid loading whole objects in the memory. However, with really big objects (multiples megabytes), the problem still arises because the node-mongodb cursor emits one object at a time ( see the node mongo stream api)) so under high load we would still have a ridiculously high memory footprint. This works great, the memory buildup is perfectly managed as long as we deal with reasonably sized objects. To achieve this, we directly control the MongoDB cursor data emission as documented on the official node driver doc. Contextįor performance reasons (memory accumulation mainly), we switched from a classic data consumption mode (retrieve whole objects in memory and send them to the client) to a streaming consumption mode.
Sorry if this question already exists but I could not find resources on this topic neither on the web nor in the source code of the NodeJS MongoDB driver.
0 kommentar(er)
