Unique index with neo4j

[etabard]

Hi ! I was looking for unique indexes with bulbs and neo4j and I found this in the source :

# Uncdocumented -- experimental
class UniqueIndex(ExactIndex):
    pass

class Property(object):

     ...

     # These aren't implemented yet.         
     # TODO: unique creates an index
     self.indexed = indexed
     self.unique = unique

Could you give me status on that ? Do you need help ?

[espeed]

Bulbs has put_unique and get_unique on the default Neo4j index
class (ExactIndex) -- I'm not sure if a separate UniqueIndex class is
needed:

https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/index.py#L295

But these methods predate Neo4j's new unique endpoints so it would
be better to use new ones:

POST http://localhost:7474/db/data/index/relationship/people?unique

See http://docs.neo4j.org/chunked/snapshot/rest-api-unique-indexes.html

Add it to the Bulbs issue tracker or even better, send me a pull request :)

It should be simple if you want to update Neo4jClient to add the
put_unique_vertex and put_unique_edge methods -- just copy put_vertex
and put_edge and add "?unique" to the path (not the URI var because
that's the edge URI you are indexing).

See
https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/client.py#L837

And
https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/client.py#L921

Then simply update the ExactIndex class to use the new methods:

https://github.com/espeed/bulbs/blob/master/bulbs/neo4jserver/index.py

Note that if you are using the unique index inside a transaction,
the unique REST endpoint probably isn't useful. For transactions, you
can create a Gremlin script that uses the Neo4j Java API methods for this.

See...

Unique Nodes (this should work for relationships too if you
substitute "relationship" for "node").
http://docs.neo4j.org/chunked/stable/transactions-unique-nodes.html

Put if Absent
http://components.neo4j.org/neo4j/1.7/apidocs/org/neo4j/graphdb/index/Index.html#putIfAbsent%28T,%20java.lang.String,%20java.lang.Object%29

• James

On Tue, Aug 21, 2012 at 10:13 AM, Emmanuel Tabard
notifications@github.comwrote:

Hi ! I was looking for unique indexes with bulbs and neo4j and I found
this in the source :

Uncdocumented -- experimentalclass UniqueIndex(ExactIndex):

pass

class Property(object):

 ...

 # These aren't implemented yet.
 # TODO: unique creates an index
 self.indexed = indexed
 self.unique = unique

Could you give me status on that ? Do you need help ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58.

Bulbflow: A Python framework for graph databases (http://bulbflow.com)

[etabard]

Thanks for the quick answer ! Here is my use case :

• I need unique and not reusable ids across nodes and relations
• So I extended Node model with a property id = String(default=generateUniqueId, unique=True)
• My id generator should generate unique ids but I want to add a unique constraint on the index to be sure.
• However I saw that neo4j has different indexes for nodes and relations so you can never be sure ... and have to lookup vertices index and edges index ...

I don't know the best practices here ... if you have a recommandation, I'm all ears :)

[espeed]

Yeah, the permanent ID thing is an issue that's been brought up several
times on the Neo4j list. They keep saying they're going to make ID resuse
configurable, but I don't think the option is in yet.

This also comes into play with Titan, backed by Cassandra.

But even if Neo4j had the ID reuse option, if you start out using Neo4j for
a Web app and use the ID in the URL, and then later you need to scale up to
something like Titan, you'll break all your links (or at least you'll have
an ID mapping layer to mess with).

All this to say, I think creating your own unique ID makes sense in several
situations, although I might call it a "key" instead of an ID to avoid
confusion.

Note that the new Neo4j unique endpoints behave differently than the Bulbs
methods. In the case where an element already exists, it doesn't update the
element -- instead it returns the existing element.

See http://docs.neo4j.org/chunked/snapshot/rest-api-unique-indexes.html

How are you creating the unique IDs?

• James

On Tue, Aug 21, 2012 at 11:18 AM, Emmanuel Tabard
notifications@github.comwrote:

Thanks for the quick answer ! Here is my use case :

• I need unique and not reusable ids across nodes and relations
• So I extended Node model with a property id =
  String(default=generateUniqueId, unique=True)
• My id generator should generate unique ids but I want to add a
  unique constraint on the index to be sure.
• However I saw that neo4j has different indexes for nodes and
  relations so you can never be sure ... and have to lookup vertices index
  and edges index ...

I don't know the best practices here ... if you have a recommandation, I'm
all ears :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58#issuecomment-7906603.

Bulbflow: A Python framework for graph databases (http://bulbflow.com)

[etabard]

I started using my own random implementation : sha1("%s%s" % (random(), time()) and ended with bson objectid specification :

4-byte timestamp (seconds since epoch), a 3-byte machine id, a 2-byte process id, and a 3-byte counter

And I added an extra byte to type my ids. For example 0 for vertices and 1 for edges.

[espeed]

Did you look at UUIDs?

On Wed, Aug 22, 2012 at 8:50 AM, Emmanuel Tabard
notifications@github.comwrote:

I started using my own random implementation : sha1("%s%s" % (random(),
time()) and ended with bson objectid specification :

4-byte timestamp (seconds since epoch), a 3-byte machine id, a 2-byte
process id, and a 3-byte counter

And I added an extra byte to type my ids. For example 0 for vertices and 1
for edges.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/58#issuecomment-7934637.

Bulbflow: A Python framework for graph databases (http://bulbflow.com)

[etabard]

Sure, first thing I looked at. But I prefer the idea of having "logic" ids instead of the black magic of uuids. I like the idea of being able to sort indexes by the 4-byte timestamp and maybe archive old nodes if I really need to. Both solutions are great, it just seems smarter to me.

[espeed]

Ok, well enforcing global uniqueness via separate vertex and edge indices won't work so you're going to have to rely on the ID algo. And I agree, sortability is a nice feature, which will come into play in Titan/Cassandra.

Note that the "unique" Property attribute is not implemented yet:

https://github.com/espeed/bulbs/blob/master/bulbs/property.py#L61

When I was writing the Bulbs 0.3 release, Neo4j unique indexes were in a state of flux so I postponed it until things were finalized, but I'm not sure they have yet. Neo4j is redoing its entire indexing framework so that indexing is automatic -- this is important for Cypher because Cypher doesn't have any notion of indexing.

Let me know If you have ideas on implementing a stopgap unique-index Property attribute.

[etabard]

ObjectId spec ensure the uniqueness of ids (used by mongodb). However the last byte (0 vertex, 1 edge) ensure the uniqueness by index, it's impossible to have conflicts between vertex index and edges index.

I need to read more about neo4j unique indexes and I'll come back to you as soon as I have an idea !

You talk a lot about Titan, do you prefer it over neo4j ? I know it's scalable but is it as powerful as neo4j ? Does bulbs support it as well ?

Sorry for the lengthy discussion :)

[espeed]

Titan is a distributed OLTP graph DB, and Marko (the creator of Gremlin) is one of the guys behind it. It has pluggable backend storage (HBAse, Cassandra, or BDB so far), and has much higher write performance.

One of Titan's recent benchmarks shows it can do 5000-10000 transactions per second, vs a few hundred for Neo4j (http://thinkaurelius.com/2012/08/06/titan-provides-real-time-big-graph-data/). Neo4j should be faster for traversals since everything is local, but I haven't seen an apples to apples comparison.

Titan can run in Rexster, and Bulbs will support Titan via the Rexster adapter. It does indexing differently than all the other Blueprints DBs so I'll have to rework the Rexster indexing code to make it work with Bulbs.

[etabard]

Wow, awesome !
Next project, I'll give it a try !

Tanks a lot :)

[elventear]

Has there been any progress with declaring uniqueness of elements? I would me happy to contribute the code if I could get some guidance on where to start.

[espeed]

Neo4j is about to release its new indexing framework, which will include autoindexing and unique indexes.

See https://groups.google.com/d/msg/neo4j/d73yLtLqPw4/HM3qxBpn0JcJ

Since Neo4j and Titan are the two primary DBs supported by Neo4j Server and Rexster, anything done prior to seeing Neo4j's new indexing framework would be a stop-gap measure.

Indexing is one of the things that varies significantly from DB to DB so to accommodate for this, there will have to be unique_index methods in each server's Client class, each written to a universally compatible method signature.

Then the DB's index classes would have methods using these Client methods.

And finally we would add support for unique=True in the Model and Property class.

[espeed]

Bulbs 0.4 will include support for Neo4j's new indexing framework.