Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use of parameters via python driver seems to break Dozer enhanced images #32

Open
Johnno1011 opened this issue Nov 26, 2024 · 7 comments

Comments

@Johnno1011
Copy link

Johnno1011 commented Nov 26, 2024

Hi guys! Great that I was able to get dozer up and running! Unfortunately I think perhaps the recent feature allowing dropping of databases might still be unstable. I'll share my findings here.

docker-compose.yml

services:
  dozer-neo4j:
    image: graphstack/dozerdb:5.25.1
    container_name: neo4j-test
    restart: always
    ports:
      - 7700:7474
      - 7701:7687
    volumes:
      - ./plugins:/plugins
      - neo4j_data:/data
    environment:
      - NEO4J_AUTH=neo4j/neo4j
      - NEO4J_apoc_export_file_enabled=true
      - NEO4J_apoc_import_file_enabled=true
      - NEO4J_apoc_import_file_use__neo4j__config=true
      - NEO4J_dbms_security_procedures_unrestricted=apoc.*, db.*, gds.*, dozer.*
      - NEO4J_dbms_security_procedures_allowlist=apoc.*, db.*, apoc.meta.data(), gds.*, dozer.*

volumes:
  neo4j_data:

In /plugins I inserted open-gds-2.8.0-alpha01.jar mentioned here and apoc-5.15.1-core.jar from here.

Now, from the web interface of Neo4j, I'm able to create multiple databases with CREATE DATABASE and delete them as I please with DROP DATABASE.

Using the python client (5.26.0 - Neo4j Bolt driver for Python). I can sometimes create a database as shown below, but this does not work robustly, if I keep adding new ones eventually something goes wrong.

from neo4j import GraphDatabase

params = {"databaseName": "pikachu"}
query = "CREATE DATABASE $databaseName"
database = None

with GraphDatabase.driver(
    services["graph_store_url"], auth=("neo4j", "neo4j")
) as driver:
    with driver.session(database=database) as session:
        result = session.run(query, parameters=params)  # type: ignore
        print([r.data() for r in result])

When trying to then do anything using these databases (for example add new data with new properties using MATCH statements), I get the following error from the python client:
image

After this happens, the frontend UI appears to show the container is a bit fragged and the container logs of course complain that it cannot perform the next operation since the DB doesn't exist and got corrupted in some way (I have to destroy and reinitialise the container and DB volume).
image
image

I hope this is enough information to get a discussion going around this topic and reproduction of what I'm experiencing. Please let me know if there's anything else you need. Thanks !

@Johnno1011 Johnno1011 changed the title Creation / Deletion of DBs via python client seems to break dozer Creation / Deletion of DBs via python client seems to break Dozer Nov 26, 2024
@Johnno1011
Copy link
Author

Johnno1011 commented Nov 26, 2024

Think this is also related. If I run the command where the placeholder for the DB name is '$name', it works fine! But then if I change this placeholder name from name to anything else it seems to trigger this break! Odd, but the error message ERROR Client triggered an unexpected error [Neo.DatabaseError.Statement.ExecutionFailed]: No such property, 'name'., is definitely the best signal of the underlying problem.
The other lead is that using placeholders in the cypher command causes this too. IE: CREATE DATABASE test works as expected, but CREATE DATABASE $databaseName and passing the db name in the parameters argument of .run() triggers the problem.
Thanks.

@Johnno1011
Copy link
Author

I've played with this some more. It's basically impossible to use parameters using the python client driver as this causes the container to act up and produce the following logs:

Dump from debug.log:

2024-11-29 11:48:06.000+0000 INFO  [o.n.k.a.DatabaseAvailabilityGuard] [neo4j/dbf26e6f] Requirement `Database unavailable` makes database neo4j unavailable.
2024-11-29 11:48:06.000+0000 INFO  [o.n.k.a.DatabaseAvailabilityGuard] [neo4j/dbf26e6f] DatabaseId{dbf26e6f[neo4j]} is unavailable.
2024-11-29 11:48:06.004+0000 WARN  [o.n.k.i.s.MetaDataStore] [neo4j/dbf26e6f] Missing counts store, rebuilding it.
2024-11-29 11:48:06.004+0000 WARN  [o.n.k.i.s.MetaDataStore] [neo4j/dbf26e6f] Counts store rebuild completed.
2024-11-29 11:48:06.012+0000 INFO  [o.n.k.d.Database] [neo4j/dbf26e6f] Starting transaction log [/data/transactions/neo4j/neostore.transaction.db.0] at version=0
2024-11-29 11:48:06.021+0000 INFO  [o.n.k.d.Database] [neo4j/dbf26e6f] Starting transaction log [/data/transactions/neo4j/checkpoint.0] at version=0
2024-11-29 11:48:06.022+0000 INFO  [o.n.k.i.t.l.f.c.CheckpointLogFile] [neo4j/dbf26e6f] Scanning log file with version 0 for checkpoint entries
2024-11-29 11:48:06.026+0000 INFO  [o.n.k.a.DatabaseAvailabilityGuard] [neo4j/dbf26e6f] Fulfilling of requirement 'Database unavailable' makes database neo4j available.
2024-11-29 11:48:06.026+0000 INFO  [o.n.k.a.DatabaseAvailabilityGuard] [neo4j/dbf26e6f] DatabaseId{dbf26e6f[neo4j]} is ready.
2024-11-29 11:48:06.039+0000 INFO  [o.n.k.i.a.i.IndexPopulationJob] [neo4j/dbf26e6f] Index population started: [Index( id=2, name='index_f7700477', type='LOOKUP', schema=()-[:<any-types>]-(), indexProvider='token-lookup-1.0' )]
2024-11-29 11:48:06.063+0000 INFO  [o.n.k.i.a.i.IndexPopulationJob] [neo4j/dbf26e6f] Index creation finished for index [Index( id=2, name='index_f7700477', type='LOOKUP', schema=()-[:<any-types>]-(), indexProvider='token-lookup-1.0' )].
2024-11-29 11:48:06.067+0000 INFO  [o.n.k.i.a.i.IndexPopulationJob] [neo4j/dbf26e6f] Index population started: [Index( id=1, name='index_343aff4e', type='LOOKUP', schema=(:<any-labels>), indexProvider='token-lookup-1.0' )]
2024-11-29 11:48:06.093+0000 INFO  [o.n.k.i.a.i.IndexPopulationJob] [neo4j/dbf26e6f] Index creation finished for index [Index( id=1, name='index_343aff4e', type='LOOKUP', schema=(:<any-labels>), indexProvider='token-lookup-1.0' )].
2024-11-29 11:48:06.101+0000 INFO  [o.n.k.i.t.l.c.CheckPointerImpl] [neo4j/dbf26e6f] Checkpoint triggered by "Database init completed." @ txId: 3, append index: 3 checkpoint started...
2024-11-29 11:48:06.367+0000 INFO  [o.n.k.i.t.l.c.CheckPointerImpl] [neo4j/dbf26e6f] Checkpoint triggered by "Database init completed." @ txId: 3, append index: 3 checkpoint completed in 265ms. Checkpoint flushed 74 pages (0% of total available pages), in 57 IOs. Checkpoint performed with IO limit: unlimited, paused in total 0 times( 0 millis).
2024-11-29 11:48:06.368+0000 INFO  [o.n.k.i.t.l.p.LogPruningImpl] [neo4j/dbf26e6f] No log version pruned. The strategy used was '2 days 2147483648 size'. 
2024-11-29 11:48:06.412+0000 INFO  [o.n.b.p.c.c.n.SocketNettyConnector] Bolt enabled on 0.0.0.0:7687.
2024-11-29 11:48:06.414+0000 INFO  [o.n.b.p.c.c.n.SocketNettyConnector] Bolt (Routing) enabled on 0.0.0.0:7688.
2024-11-29 11:48:06.415+0000 WARN  [i.n.b.ServerBootstrap] Unknown channel option 'SO_REUSEADDR' for channel '[id: 0x0c02a49c]'
2024-11-29 11:48:06.421+0000 INFO  [o.n.b.BoltServer] Bolt server started
2024-11-29 11:48:06.421+0000 INFO  [o.n.s.A.ServerComponentsLifecycleAdapter] Starting web server
2024-11-29 11:48:07.138+0000 INFO  [o.n.s.CommunityNeoWebServer] HTTP enabled on 0.0.0.0:7474.
2024-11-29 11:48:07.138+0000 INFO  [o.n.s.CommunityNeoWebServer] Remote interface available at http://localhost:7474/
2024-11-29 11:48:07.138+0000 INFO  [o.n.s.A.ServerComponentsLifecycleAdapter] Web server started.
2024-11-29 11:48:07.141+0000 INFO  [o.n.g.f.DatabaseManagementServiceFactory] id: F316F07FB4274E1F48A4174AF3015A16C6E075568A74C7FCC3D272E003127A6B
2024-11-29 11:48:07.141+0000 INFO  [o.n.g.f.DatabaseManagementServiceFactory] name: system
2024-11-29 11:48:07.142+0000 INFO  [o.n.g.f.DatabaseManagementServiceFactory] creationDate: 2024-11-29T11:47:57.734Z
2024-11-29 12:00:32.357+0000 WARN  [o.n.d.d.r.SystemGraphReadOnlyDatabaseLookupFactory] Unable to lookup readonly databases from the system database due to error! Using previous lookup SystemGraphLookup[alwaysReadOnly=false, databaseIds=[]].
Underlying error: No such property, 'name'.
2024-11-29 12:01:01.171+0000 ERROR [o.n.b.f.StateMachineImpl] Client triggered an unexpected error [Neo.DatabaseError.Statement.ExecutionFailed]: No such property, 'name'., reference 50ead15f-4682-4a15-a8a8-b5bb88c61820.
2024-11-29 12:01:01.171+0000 ERROR [o.n.b.f.StateMachineImpl] Client triggered an unexpected error [Neo.DatabaseError.Statement.ExecutionFailed]: No such property, 'name'., reference 50ead15f-4682-4a15-a8a8-b5bb88c61820.
org.neo4j.bolt.protocol.common.fsm.error.TransactionStateTransitionException: No such property, 'name'.
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.CreateStatementStateTransition.process(CreateStatementStateTransition.java:56) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.CreateStatementStateTransition.process(CreateStatementStateTransition.java:32) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.TransactionalStateTransition.process(TransactionalStateTransition.java:42) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.fsm.state.MultiTransitionState.process(MultiTransitionState.java:53) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.fsm.StateMachineImpl.process(StateMachineImpl.java:159) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection$ProcessJob.perform(AtomicSchedulingConnection.java:634) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.executeJob(AtomicSchedulingConnection.java:338) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.doExecuteJobs(AtomicSchedulingConnection.java:317) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.connector.connection.AtomicSchedulingConnection.executeJobs(AtomicSchedulingConnection.java:212) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.neo4j.bolt.tx.error.statement.StatementExecutionException: No such property, 'name'.
	at org.neo4j.bolt.tx.TransactionImpl.run(TransactionImpl.java:143) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.CreateStatementStateTransition.process(CreateStatementStateTransition.java:48) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	... 13 more
Caused by: org.neo4j.kernel.impl.query.QueryExecutionKernelException: No such property, 'name'.
	at org.neo4j.bolt.tx.statement.StatementQuerySubscriber.assertSuccess(StatementQuerySubscriber.java:93) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.tx.TransactionImpl.run(TransactionImpl.java:140) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.CreateStatementStateTransition.process(CreateStatementStateTransition.java:48) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	... 13 more
Caused by: org.neo4j.exceptions.CypherExecutionException: No such property, 'name'.
	at org.neo4j.exceptions.CypherExecutionException.internalError(CypherExecutionException.java:72) ~[neo4j-exceptions-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.tx.statement.StatementQuerySubscriber.assertSuccess(StatementQuerySubscriber.java:93) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.tx.TransactionImpl.run(TransactionImpl.java:140) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.CreateStatementStateTransition.process(CreateStatementStateTransition.java:48) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	... 13 more
Caused by: org.neo4j.graphdb.NotFoundException: No such property, 'name'.
	at org.neo4j.kernel.impl.core.AbstractEntity.getProperty(AbstractEntity.java:68) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.kernel.impl.core.NodeEntity.getProperty(NodeEntity.java:246) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.dbms.systemgraph.CommunityTopologyGraphDbmsModelUtil.getDatabaseId(CommunityTopologyGraphDbmsModelUtil.java:311) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.dbms.systemgraph.CommunityTopologyGraphDbmsModelUtil.lambda$getAllPrimaryStandardDatabaseReferencesInRoot$1(CommunityTopologyGraphDbmsModelUtil.java:59) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.neo4j.dbms.systemgraph.CommunityTopologyGraphDbmsModel.getAllDatabaseReferences(CommunityTopologyGraphDbmsModel.java:76) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.kernel.database.SystemGraphDatabaseReferenceRepository.execute(SystemGraphDatabaseReferenceRepository.java:93) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.kernel.database.SystemGraphDatabaseReferenceRepository.getAllDatabaseReferences(SystemGraphDatabaseReferenceRepository.java:66) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.kernel.database.MapCachingDatabaseReferenceRepository.getAllDatabaseReferences(MapCachingDatabaseReferenceRepository.java:59) ~[neo4j-kernel-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.administration.DatabaseListParameterTransformerFunction.transform(DatabaseListParameterTransformerFunction.scala:80) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.procs.SystemCommandExecutionPlan.$anonfun$runSpecific$1(SystemCommandExecutionPlan.scala:73) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.procs.SystemCommandExecutionPlan.$anonfun$withFullDatabaseAccess$1(SystemCommandExecutionPlan.scala:108) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at scala.util.Using$.resource(Using.scala:262) ~[scala-library-2.13.11.jar:?]
	at org.neo4j.cypher.internal.procs.SystemCommandExecutionPlan.withFullDatabaseAccess(SystemCommandExecutionPlan.scala:107) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.procs.SystemCommandExecutionPlan.runSpecific(SystemCommandExecutionPlan.scala:69) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.procs.SystemCommandExecutionPlan.runSpecific(SystemCommandExecutionPlan.scala:46) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.procs.ChainedExecutionPlan.run(ChainedExecutionPlan.scala:95) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.CypherCurrentCompiler$CypherExecutableQuery.innerExecute(CypherCurrentCompiler.scala:537) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.CypherCurrentCompiler$CypherExecutableQuery.execute(CypherCurrentCompiler.scala:475) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.ExecutionEngine.doExecute(ExecutionEngine.scala:280) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.ExecutionEngine.$anonfun$executeSubquery$1(ExecutionEngine.scala:197) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.ExecutionEngine.closing(ExecutionEngine.scala:203) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.ExecutionEngine.executeSubquery(ExecutionEngine.scala:177) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.ExecutionEngine.execute(ExecutionEngine.scala:110) ~[neo4j-cypher-5.25.1.jar:5.25.1]
	at org.neo4j.cypher.internal.javacompat.ExecutionEngine.executeQuery(ExecutionEngine.java:150) ~[neo4j-cypher-5.25.1.jar:5.25.1-1.1.0-alpha.1]
	at org.neo4j.router.impl.transaction.database.LocalDatabaseTransaction.lambda$executeQuery$1(LocalDatabaseTransaction.java:142) ~[neo4j-query-router-5.25.1.jar:5.25.1]
	at org.neo4j.router.impl.transaction.database.LocalDatabaseTransaction.translateLocalError(LocalDatabaseTransaction.java:163) ~[neo4j-query-router-5.25.1.jar:5.25.1]
	at org.neo4j.router.impl.transaction.database.LocalDatabaseTransaction.executeQuery(LocalDatabaseTransaction.java:134) ~[neo4j-query-router-5.25.1.jar:5.25.1]
	at org.neo4j.router.impl.QueryRouterImpl.executeQuery(QueryRouterImpl.java:278) ~[neo4j-query-router-5.25.1.jar:5.25.1]
	at org.neo4j.router.impl.bolt.QueryRouterBoltSpi$Transaction.executeQuery(QueryRouterBoltSpi.java:161) ~[neo4j-query-router-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.tx.TransactionImpl.run(TransactionImpl.java:136) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	at org.neo4j.bolt.protocol.common.fsm.transition.transaction.CreateStatementStateTransition.process(CreateStatementStateTransition.java:48) ~[neo4j-bolt-5.25.1.jar:5.25.1]
	... 13 more

@Johnno1011 Johnno1011 changed the title Creation / Deletion of DBs via python client seems to break Dozer Use of parameters via python driver seems to break Dozer enhanced images Nov 29, 2024
@jmsuhy
Copy link
Contributor

jmsuhy commented Dec 3, 2024

Thanks for submitting this issue. I believe I know what’s going on. There are additional options that the driver adds which weren’t implemented yet. We will get them implemented so you can use the drivers to also create and drop dbs.

We’re working on other features right now so this will be scheduled during our next team meeting.

@Johnno1011
Copy link
Author

Hi @jmsuhy thanks for letting me know!
Just to clarify, it's not just with CREATE and DROP queries that parameters breaks with the python driver - it's any use of parameters for any type of query.

Many thanks and please do let me know if you need any further information !

@jmsuhy
Copy link
Contributor

jmsuhy commented Dec 13, 2024

Thank you - we identified the issue and the upcoming fix will address all those issues relating to commands through drivers.

I will update this ticket. With the holidays and everyone out for the holidays it may take a litter longer to get this merged.

thank you for identifying this!

@Johnno1011
Copy link
Author

Hey @jmsuhy, thanks for letting me know - happy new year !
Any updates on this?
Cheers.

Thank you - we identified the issue and the upcoming fix will address all those issues relating to commands through drivers.

I will update this ticket. With the holidays and everyone out for the holidays it may take a litter longer to get this merged.

thank you for identifying this!

@jmsuhy
Copy link
Contributor

jmsuhy commented Jan 7, 2025

Yes, now that holidays are past, we are working on finishing this as well as release the 5.26.x LTS version starting this week. I will update the ticket with details once we have verified its fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants