MongoDB: Minimum working example

[ssooffiiaannee]

Os : Windows 11
MSVC 2017

Goal:
Make a minimum working example of a MongoDB client writing to a DB (make it compile at least, then make it work)

Code example :

#include <Poco/MongoDB/Connection.h>
#include <Poco/MongoDB/Database.h>
#include <Poco/MongoDB/Document.h>
#include <Poco/MongoDB/Array.h>
#include <Poco/MongoDB/QueryRequest.h>
#include <Poco/MongoDB/ResponseMessage.h>
#include <Poco/Exception.h>
#include <iostream>
#include <string>

int main() {
        Poco::MongoDB::Connection connection("localhost", 27017); 

        Poco::MongoDB::Database db("test_db"); 
        std::string collectionName = "test_collection";

        auto insertRequest = db.createInsertRequest(collectionName);
        Poco::MongoDB::Document *doc = new Poco::MongoDB::Document();
        doc->add("name", "John Doe");
        doc->add("age", 30);
        doc->add("email", "[email protected]");
        insertRequest->documents().push_back(doc);

        connection.sendRequest(*insertRequest);
        std::cout << "Document inserted successfully!" << std::endl;

        Poco::MongoDB::QueryRequest *query = new Poco::MongoDB::QueryRequest(collectionName);
        query->selector().add("name", "John Doe");  // Search for documents where 'name' is 'John Doe'

        Poco::MongoDB::ResponseMessage response;
        connection.sendRequest(*query, response);
    
    return 0;
}

Command line used :
I have cloned the project to : D:\poco
I have built the project in D:\poco\cmake-build folder
I did not install, I'd rather link to the build folder

"C:\path\to\mvs\2017\Community\Common7\Tools\VsDevCmd.bat"
"C:\path\to\mvs\2017\Community\VC\Tools\MSVC\14.16.27023\bin\Hostx86\x86\cl.exe" /std:c++17 /EHsc \mongodb_testing.cpp /I"D:\poco\MongoDB\include" /I"D:\poco\Net\include" /I"D:\poco\Foundation\include" /link "D:\poco\cmake-build\lib\PocoFoundation.lib"

Error:

mongodb_testing.obj
LINK : fatal error LNK1104: cannot open file 'PocoFoundationmt.lib'

I can't find the PocoFoundationmt lib after the build, I don't know why it is trying to link to a non-existant library

[matejk]

Does installing and using CMake module from installation work?

[ssooffiiaannee]

what do you mean by "CMake module from installation " ?
I used to following instructions

$ git clone -b main https://github.com/pocoproject/poco.git
$ cd poco
$ mkdir cmake-build
$ cd cmake-build
$ cmake ..
$ cmake --build . --config Release

I used cmake v3.30.3
The build went fine, no errors in sight, the libraries have been built, cmake-build\lib\PocoFoundation.lib and PocoMongoDB.lib are there. But somehow it is loooking for PocoFoundationmt.lib which cannot be found anywhere.

[matejk]

You wrote above that you did not install.

Install Poco to some directory:

cmake --install . --prefix "SOME DIR"

Then use CMake find_package(Poco) in your example instead of setting include path and linking directly.

That is the supported way and should work.

[matejk]

Another option for simple examples like yours is to define POCO_NO_AUTOMATIC_LIBS on the command line.

[ssooffiiaannee]

Now that I managed to build Poco, and link it to my example code, the DB is not added after running the executable, no error reported, the server is running in localhost:default port, and I launched a nodejs app that writes into the db, and It worked, so the problem is not in the server side.

I am not sure what went wrong, I can't find any doc about this.

#include <Poco/MongoDB/Connection.h>
#include <Poco/MongoDB/Database.h>
#include <Poco/MongoDB/Document.h>
#include <Poco/MongoDB/Array.h>
#include <Poco/MongoDB/QueryRequest.h>
#include <Poco/MongoDB/ResponseMessage.h>
#include <iostream>
#include <string>

int main() {
     
        Poco::MongoDB::Connection connection("127.0.0.1", 27017);

        Poco::MongoDB::Database db("test_db");  
        std::string collectionName = "test_collection";

        auto insertRequest = db.createInsertRequest(collectionName);
        Poco::MongoDB::Document::Ptr doc = new Poco::MongoDB::Document();
        doc->add("name", "John Doe");
        doc->add("age", 30);
        doc->add("email", "[email protected]");
        insertRequest->documents().push_back(doc);

        connection.sendRequest(*insertRequest);
        std::cout << "Document inserted successfully!" << std::endl;
        return 0;
}

TCP traffic capture shows TCP handshakes and insertion request submitted.

[matejk]

Use the function that captures the response and then print it. See unit tests.

[ssooffiiaannee]

this is what I get :

Error: I/O error: Failed to read from socket

int main() {
    try {
        Poco::MongoDB::Connection connection("127.0.0.1", 27017); 

        auto insertReq = Poco::MongoDB::InsertRequest("test_ddb.test_collection");

        Poco::MongoDB::Document::Ptr doc = new Poco::MongoDB::Document();
        doc->add("name", "John Doe");
        doc->add("age", 30);
        doc->add("email", "[email protected]");
        insertReq.documents().push_back(doc);

        Poco::MongoDB::ResponseMessage responseInsert;
        connection.sendRequest(insertReq, responseInsert);
        std::cout << "Document inserted successfully!" << std::endl;

         for (const auto& doc : responseInsert.documents()) {
            std::cout << "Found document:" << std::endl;
            std::cout << std::endl;
        }
    } catch (const Poco::Exception& ex) {
        std::cerr << "Error: " << ex.displayText() << std::endl;
        return 1;
    }

    return 0;
}

[ssooffiiaannee]

Ok, so my previous code was using the deprecated wire protocol OP_INSERT.
But still after using OP_MSG, I get the following response :

Êq¦║okerrmsgyIncorrect BSON length 654311424 should be less or equal to 167 in element with field name '?' in object with unknown _idcodecodeName
InvalidBSON0

With the example code from unit test testOpCmdInsert() in "MongoDBTestOpMsg.cpp" :

#include <Poco/MongoDB/Connection.h>
#include <Poco/MongoDB/InsertRequest.h>
#include <Poco/MongoDB/Document.h>
#include <Poco/MongoDB/Database.h>
#include <Poco/MongoDB/QueryRequest.h>
#include <Poco/Exception.h>
#include <Poco/Nullable.h>
#include <iostream>
#include <string>

int main() {
    Poco::MongoDB::Connection connection("127.0.0.1", 27017);
    Poco::MongoDB::Document::Ptr player = new Poco::MongoDB::Document();
	player->add("lastname", std::string("Braem"));
	player->add("firstname", std::string("Franky"));

	Poco::DateTime birthdate;
	birthdate.assign(1969, 3, 9);
	player->add("birthdate", birthdate.timestamp());

	player->add("start", 1993);
	player->add("active", false);

	Poco::DateTime now;
	player->add("lastupdated", now.timestamp());

	Poco::MongoDB::Database db("team");
	Poco::SharedPtr<Poco::MongoDB::OpMsgMessage> request = db.createOpMsgMessage("players");
	request->setCommandName(Poco::MongoDB::OpMsgMessage::CMD_INSERT);
	request->documents().push_back(player);

	try
	{
		Poco::MongoDB::OpMsgMessage response;
		connection.sendRequest(*request, response);
        response.send(std::cout);
        std::cout << response.responseOk() << std::endl;
	}
	catch(Poco::NotImplementedException& nie)
	{
		std::cout << nie.message() << std::endl;
	}

    return 0;
}

[matejk]

response.send(std::cout) creates BSON and sends it to cout. Use response.toString to print human readable document.

[ssooffiiaannee]

Response is of type OpMsgMessage, OpMsgMessage has no member function toString

[matejk]

You can also dump player before sending insert request.

[ssooffiiaannee]

Even after dumping player :

{"ok":0,"errmsg":"Incorrect BSON length 654311424 should be less or equal to 42 in element with field name '?' in object with unknown _id","code":22,"codeName":"InvalidBSON"}
0
I am investigating the mongoDB source code, it's coming from "file":"src\mongo\bson\bson_validate.cpp","line":439

[matejk]

Which version of MongoDB do you have?

[matejk]

It seems like the request that is sent is not correct.

[ssooffiiaannee]

MongoDB v8.0.3

[matejk]

@ssooffiiaannee, it seems that there is something causing the problem on your end: either build environment, compiler, ...

I just tested the attached project on MacOS 15 and MongoDB 8.0.3 and it works fine.

CMakeLists.txt

#include <Poco/MongoDB/Connection.h>
#include <Poco/MongoDB/InsertRequest.h>
#include <Poco/MongoDB/Document.h>
#include <Poco/MongoDB/Database.h>
#include <Poco/MongoDB/QueryRequest.h>
#include <Poco/Exception.h>
#include <Poco/Nullable.h>
#include <iostream>
#include <string>

int main() {
	Poco::MongoDB::Connection connection("127.0.0.1", 27017);
	Poco::MongoDB::Document::Ptr player = new Poco::MongoDB::Document();
	player->add("lastname", std::string("Braem"));
	player->add("firstname", std::string("Franky"));

	Poco::DateTime birthdate;
	birthdate.assign(1969, 3, 9);
	player->add("birthdate", birthdate.timestamp());

	player->add("start", 1993);
	player->add("active", false);

	Poco::DateTime now;
	player->add("lastupdated", now.timestamp());

	Poco::MongoDB::Database db("team");
	Poco::SharedPtr<Poco::MongoDB::OpMsgMessage> request = db.createOpMsgMessage("players");
	request->setCommandName(Poco::MongoDB::OpMsgMessage::CMD_INSERT);
	request->documents().push_back(player);

	try
	{
		const auto bi = db.queryBuildInfo(connection);
		std::cout << bi->toString(2) << std::endl;

		Poco::MongoDB::OpMsgMessage response;
		connection.sendRequest(*request, response);
		std::cout << response.responseOk() << std::endl;
		std::cout << response.body().toString(2) << std::endl;

		Poco::SharedPtr<Poco::MongoDB::OpMsgMessage> query = db.createOpMsgMessage("players");
		query->setCommandName(Poco::MongoDB::OpMsgMessage::CMD_FIND);

		connection.sendRequest(*query, response);
		std::cout << response.responseOk() << std::endl;
		for (const auto d: response.documents())
		{
			std::cout << d->toString(2) << std::endl;
		}
		std::cout << response.responseOk() << std::endl;

	}
	catch(Poco::Exception& nie)
	{
		std::cout << nie.message() << std::endl;
	}

	return 0;
}

Output:

{
  "version" : "8.0.3",
  "gitVersion" : "89d97f2744a2b9851ddfb51bdf22f687562d9b06",
  "modules" : [

  ],
  "allocator" : "tcmalloc-google",
  "javascriptEngine" : "mozjs",
  "sysInfo" : "deprecated",
  "versionArray" : [
    8,
    0,
    3,
    0
  ],
  "openssl" : {
    "running" : "OpenSSL 3.0.13 30 Jan 2024",
    "compiled" : "OpenSSL 3.0.13 30 Jan 2024"
  },
  "buildEnvironment" : {
    "distmod" : "ubuntu2404",
    "distarch" : "aarch64",
    "cc" : "/opt/mongodbtoolchain/v4/bin/gcc: gcc (GCC) 11.3.0",
    "ccflags" : "-Werror -include mongo/platform/basic.h -ffp-contract=off -fasynchronous-unwind-tables -g2 -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -gdwarf-5 -fno-omit-frame-pointer -fno-strict-aliasing -O2 -march=armv8.2-a -mtune=generic -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-const-variable -Wno-unused-but-set-variable -Wno-missing-braces -Wno-psabi -fstack-protector-strong -gdwarf64 -Wa,--nocompress-debug-sections -Wimplicit-fallthrough=5",
    "cxx" : "/opt/mongodbtoolchain/v4/bin/g++: g++ (GCC) 11.3.0",
    "cxxflags" : "-Woverloaded-virtual -Wpessimizing-move -Wno-maybe-uninitialized -fsized-deallocation -std=c++20",
    "linkflags" : "-Wl,--fatal-warnings -B/opt/mongodbtoolchain/v4/bin -gdwarf-5 -pthread -Wl,-z,now -fuse-ld=lld -fstack-protector-strong -gdwarf64 -Wl,--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro -Wl,--compress-debug-sections=none -Wl,-z,origin -Wl,--enable-new-dtags",
    "target_arch" : "aarch64",
    "target_os" : "linux",
    "cppdefines" : "SAFEINT_USE_INTRINSICS 0 PCRE2_STATIC ABSL_ALLOCATOR_NOTHROW NDEBUG _XOPEN_SOURCE 700 _GNU_SOURCE _FORTIFY_SOURCE 2 ABSL_FORCE_ALIGNED_ACCESS BOOST_ENABLE_ASSERT_DEBUG_HANDLER BOOST_FILESYSTEM_NO_CXX20_ATOMIC_REF BOOST_LOG_NO_SHORTHAND_NAMES BOOST_LOG_USE_NATIVE_SYSLOG BOOST_LOG_WITHOUT_THREAD_ATTR BOOST_MATH_NO_LONG_DOUBLE_MATH_FUNCTIONS BOOST_SYSTEM_NO_DEPRECATED BOOST_THREAD_USES_DATETIME BOOST_THREAD_VERSION 5"
  },
  "bits" : 64,
  "debug" : false,
  "maxBsonObjectSize" : 16777216,
  "storageEngines" : [
    "devnull",
    "wiredTiger"
  ],
  "ok" : 1
}
{
  "n" : 1,
  "ok" : 1
}
1
1
{
  "_id" : 67542747aaaa36dfc8cb5d77,
  "lastname" : "Braem",
  "firstname" : "Franky",
  "birthdate" : "1969-03-09T00:00:00.000000Z",
  "start" : 1993,
  "active" : false,
  "lastupdated" : "2024-12-07T10:45:27.385000Z"
}
1

[ssooffiiaannee]

Well after installing the libs and linking to them, the ldd is different :

linux-vdso.so.1 (0x00007ffd9f3ca000)
libPocoFoundation.so.103 => /home/sofiane/Desktop/poco_ex/poco/cmake-build/install/lib/libPocoFoundation.so.103 (0x00007f7c2e1ff000)
libPocoNet.so.103 => /home/sofiane/Desktop/poco_ex/poco/cmake-build/install/lib/libPocoNet.so.103 (0x00007f7c2e087000)
libPocoMongoDB.so.103 => /home/sofiane/Desktop/poco_ex/poco/cmake-build/install/lib/libPocoMongoDB.so.103 (0x00007f7c2e03b000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7c2de38000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7c2de1d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7c2dc29000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7c2dc06000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7c2dc00000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7c2dbf6000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7c2daa7000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7c2e4ab000)

[matejk]

Well, unfortunately you will have to figure out the cause because you are the only one able to reproduce this problem.

[ssooffiiaannee]

Yes I knew It would come to this point, and building mongodb server to troubleshoot this is a headache on linux, let alone on windows.

[matejk]

The problem is definitely not in MongoDB, but somewhere in client software.

[ssooffiiaannee]

Yes, I don't know how that whole protocol thing works, but with custom built mongodb server, I may find out why the server is reacting this way.

[ssooffiiaannee]

I may have found the root cause, it's a malformation of the OpMsg frame, coming from OpMsgMessage.cpp line 234 :

BinaryWriter socketWriter(ostr, BinaryWriter::LITTLE_ENDIAN_BYTE_ORDER);

// Serialise the body
std::stringstream ss;
BinaryWriter writer(ss, BinaryWriter::LITTLE_ENDIAN_BYTE_ORDER);
writer << _flags;

writer << PAYLOAD_TYPE_0;   // is of type uint8_t, but operator<<(int) is called in some compiler versions
_body.write(writer);

writer << PAYLOAD_TYPE_0; calls BinaryWriter& BinaryWriter::operator << (int value) instead of BinaryWriter& BinaryWriter::operator << (unsigned char value)

And as a consequence, PAYLOAD_TYPE_0 is written as a 4 byte binary instead of 1 in the OpMsg Frame, the other 3 bytes of 0x00s push the document length's least significant bits into higher addresses, leading to a Big Endian representation of the Document length field (654311424 instead of 39) because server sees 0x00 0x00 0x00 0x27 as next field due to MSBs of payload type.

Seems to affect GCC versions < 10.1, and MSVC 15, and maybe other versions.

Next steps : either upgrade GCC version to versions >= 10.1 (Readme shows 8.0)
or change all occurences of payload_type binary serialization to :
writer << (std::uint8_t) PAYLOAD_TYPE_0;

I'll push the change after your confirmation.

[matejk]

@ssooffiiaannee, thank you very much for this finding.

I'll take a look tomorrow.

[matejk]

@ssooffiiaannee, does it work properly if you change enum declaration in OpMsgMessage.h like this and don't use typecast:

	static_assert(std::is_same(Poco::UInt8, unsigned char);

	enum PayloadType : unsigned char
	{
		PAYLOAD_TYPE_0	= 0,
		PAYLOAD_TYPE_1	= 1
	};

I don't have a suitable compiler to test.

[ssooffiiaannee]

it seems to behave the same with gcc versions <= 9.5 , the assertion is true though,
You can test with different gcc versions in https://godbolt.org/z/Yh9Wzv5Kf

[matejk]

PR #4822.