Tag: cypher

NEO4J: Exploring the Data

Since I seem to frequently acquire orphaned platforms with no documentation, I figure it would be good to figure out how to investigate an unknown NEO4J platform to see what it’s got. “SHOW” is very useful in these cases. The full list of SHOW commands is:

"ALIAS"
"ALIASES"
"ALL"
"BTREE"
"BUILT"
"CONSTRAINT"
"CONSTRAINTS"
"CURRENT"
"DATABASE"
"DATABASES"
"DEFAULT"
"EXIST"
"EXISTENCE"
"EXISTS"
"FULLTEXT"
"FUNCTION"
"FUNCTIONS"
"HOME"
"INDEX"
"INDEXES"
"KEY"
"LOOKUP"
"NODE"
"POINT"
"POPULATED"
"PRIVILEGE"
"PRIVILEGES"
"PROCEDURE"
"PROCEDURES"
"PROPERTY"
"RANGE"
"REL"
"RELATIONSHIP"
"ROLE"
"ROLES"
"SERVER"
"SERVERS"
"SETTING"
"SETTINGS"
"SUPPORTED"
"TEXT"
"TRANSACTION"
"TRANSACTIONS"
"UNIQUE"
"UNIQUENESS"
"USER"
"USERS"

The most useful ones for figuring out what you’ve got … the “show databases” command I know from MySQL/MariaDB does what I expected – you can also include a specific database name, but “show database ljrtest” doesn’t appear to list any more information than the generic show databases command. .

There’s also a “show users” command that outputs the users in the database – although I’m using the community edition without authentication, so there isn’t much interesting information being output here.

And roles, if roles are being used, should be output with “SHOW ALL ROLES” … but mine just says “Unsupported administration command”

Once you know what databases you’ve got and who can log in and do stuff, we’d want to look at the data. There are some built-in functions that will help us out here. The db.labels() function will list the labels in the selected database.

You can also return a distinct list of labels along with the count of nodes with that label:

Since a node can have multiple, comparing that total with a count of nodes would give you an idea if there are many more labels than nodes. In my case, either view shows 156 … so I know there are few (if any) nodes with multiple labels.

To view the types of relationships defined in the selected database, use “CALL db.relationshipTypes()”

Similarly, you can return the relationship types along with counts

There is a function to list the property keys used within the data – interesting to note that keys that were used but the nodes using them were subsequently deleted … they still show up as property keys. Basically, anything that is there is on this list but some things on this list may not be there anymore. In this example, ‘parentm’ and ‘parentf’ were labels I used to build relationships programmatically.

I’ve found db.schema.nodeTypeProperties to be more useful in this regard – it does not appear to list properties that are not in use, and the output includes field types

To see if there are any custom procedures or functions registered, look on the server(s). Use ps -efww to view the running command – there will be some folders listed after “-cp” … you could find procedures or plugins in any of those folders. In my case, the plugins are in /plugins

And the only “custom” things registered are the APOC and APOC-Extended jar’s

 

Neo4J – The Importance of the Data Model

While I am certain table-based SQL databases required planning to establish a reasonable data model – optimizing storage, defining foreign keys, indexing … I have found it more challenging to create a good data model in Neo4j. Maybe that’s because I normally populate SQL tables with custom scripts that can be modified to handle all sorts of edge cases. Maybe I’m still thinking in tables, but there seems to be more trial and error in defining the data model than I’ve ever had in SQL databases.

In the import-from-html-table example, a candidate often is associated with multiple elections. Storing candidates as nodes and elections as other nodes that contain results (electoral college votes for winner & loser and popular votes for winner & loser) then associating candidates with elections allowed me to store data about US elections in the Graph. I know who ran, who won and who lost, and what the results were for each election.

Associating the results with candidates didn’t work because Franklin D Roosevelt only has one property for “EC_VOTES” … which election does that reflect? I could also have added the vote totals to the relationship but that would either separate the data (the loser’s votes are stored on LOST relationships and the winner’s are stored on WON relationships) or data duplication (both WON and LOST relationships contain the same vote numbers).

Query used to populate the data:

CALL apoc.load.html("https://www.iweblists.com/us/government/PresidentialElectionResults.html",
{electionyear: "#listtable tbody tr td:eq(0)"
, winner: "#listtable tbody tr td:eq(1)"
, loser: "#listtable tbody tr td:eq(2)"
, electoral_win: "#listtable tbody tr td:eq(3)"
, electoral_lose: "#listtable tbody tr td:eq(4)"
, popular_win: "#listtable tbody tr td:eq(5)"
, popular_delta: "#listtable tbody tr td:eq(6)" }) yield value
WITH value, size(value.electionyear) as rangeup

UNWIND range(0,rangeup) as i WITH value.electionyear[i].text as ElectionYear
, value.winner[i].text as Winner, value.loser[i].text as Loser
, value.electoral_win[i].text as EC_Winner, value.electoral_lose[i].text as EC_Loser
, value.popular_win[i].text as Pop_Vote_Winner
, value.popular_delta[i].text as Pop_Vote_Margin

MERGE (election:Election {year: coalesce(ElectionYear,"Unknown")})
SET election.EC_Votes_Winner = coalesce(EC_Winner,"Unknown")
SET election.EC_Votes_Loser = coalesce(EC_Loser,"Unknown")
SET election.Pop_Votes_Winner = apoc.text.replace(Pop_Vote_Winner, ",", "")
SET election.Pop_Votes_Loser = apoc.number.exact.sub(apoc.text.replace(Pop_Vote_Winner, ",", ""), apoc.text.replace(Pop_Vote_Margin, ",", ""))

MERGE (ew:CANDIDATE {name: coalesce(Winner,"Unknown")})
MERGE (el:CANDIDATE {name: coalesce(Loser,"Unknown")})

MERGE (ew)-[:WON]->(election) MERGE (el)-[:LOST]->(election);

 

NEO4J: More Cypher Queries

To get a count of returned records, Cypher uses COUNT pretty much the same way as SQL does

Interestingly, there are other aggregation functions that remind me of using the ELK API — I can get averages, min/max, and standard deviation.

 

Chaining MATCH statements functions similarly to a SQL JOIN — get the items with this label, add in some other stuff. And, just like an INNER JOIN, this means no data is returned when one of the conditions has no matches — Bill Clinton never lost an election, so we get a null data set here:

The equivalent of an outer join is an OPTIONAL MATCH — here, the records from the first MATCH will be returned even if there is no corresponding record matching the second MATCH

ORDER BY also works in the same way it does in SQL. Multiple order parameters are separated by a comma and add DESC to do a DESCENDING ORDER

WHERE can be used to create the equivalent of a LIKE query — the where =~ operator uses regular expression syntax, so you don’t just use % or * as a wildcard. Regex wildcards like .* (match any character zero or more times) are used.

NEO4J: Using APOC To Load HTML Table Data

I’ve been playing around with loading neo4j data from random tables on the web using apoc.load.html from the extended APOC library. The first trick to it is knowing how to use jquery to find elements of a webpage — the table named “listtable” then the path down to the data elements (tbody tr td) and column numbers.

Once you have extracted the data, you can then manipulate it, map it into fields, create relationships, etc.

UNWIND is used as a “for each” loop that allows us to iterate through the result set.

MERGE creates or updates records (which, in this case, means I have a poor data model … someone could well have run in multiple elections and I am not really accommodating those cases well. Since I don’t actually want a database of presidential elections but was really just testing some new-to-me functionality … we’re going to ignore these logic problems)

SET adds (or updates) properties of the node.

CALL apoc.load.html("https://www.iweblists.com/us/government/PresidentialElectionResults.html",
{electionyear: "#listtable tbody tr td:eq(0)"
	, winner: "#listtable tbody tr td:eq(1)"
	, loser: "#listtable tbody tr td:eq(2)"
	, electoral_win: "#listtable tbody tr td:eq(3)"
	, electoral_lose: "#listtable tbody tr td:eq(4)"
	, popular_win: "#listtable tbody tr td:eq(5)"
	, popular_delta: "#listtable tbody tr td:eq(6)" }) yield value 

WITH value, size(value.electionyear) as rangeup

UNWIND range(0,rangeup) as i WITH value.electionyear[i].text as ElectionYear
	, value.winner[i].text as Winner
	, value.loser[i].text as Loser
	, value.electoral_win[i].text as EC_Winner
	, value.electoral_lose[i].text as EC_Loser
	, value.popular_win[i].text as Pop_Vote_Winner
	, value.popular_delta[i].text as Pop_Vote_Delta

MERGE (ew:Candidate {name: coalesce(Winner,"Unknown")}) 
MERGE (el:Record {name: coalesce(Loser,"Unknown")}) 
SET ew.EC_Votes = coalesce(EC_Winner,"Unknown") 
SET el.EC_Votes = coalesce(EC_Loser,"Unknown")
SET ew.Year = ElectionYear
SET el.Year = ElectionYear

WITH *, replace(Pop_Vote_Delta,",","") as Pop_Vote_Delta_Int, replace(Pop_Vote_Winner,",","") as Pop_Winner_Int

SET ew.Pop_Votes = Pop_Winner_Int
SET el.Pop_Votes = apoc.number.exact.sub(Pop_Winner_Int, Pop_Vote_Delta_Int)

MERGE (ew)-[:DEFEATED]->(el);

NEO4J: WITH and Scope

I have encountered another manual reading failure error — if you actually read the Cypher documentation for WITH, it clearly states that entering a WITH block creates a new scope into which previous variables are not imported. Unless you specifically include them. You can individually include variables in this new scope (WITH oldvariable1, oldvariable2, newSomething as newvariable1) or just use * to include all previous variables.

Doing neither will produce an error that a variable that you are absolutely positive exists does not, in fact, exist.

NEO4J: Union Sets

There are a lot of places where cypher queries don’t seem to have an “OR” type of functionality … but you can union two returned sets provided they have the same properties list
MATCH z=(:TOMATO {name: ‘Black Krim’})
RETURN z
UNION
MATCH z=(:TOMATO {name: ‘Cherokee Purple’})
RETURN z;

NEO4J: Debugging a Cypher Query

It is somewhat ironic that I continue to use print statements as my debugging tool of choice when programming but spent a decent bit of time trying to find a cypher query debugger. Just use a print statement — or, in this case, return.

When my query returned an error indicating that the variable isn’t defined even though I copy/pasted the variable name from whence I defined it:

I could just omit the component of the query with the error and try returning this variable

And performing operations on null values may not get me anywhere. Adding a replacement command to drop the commas produces integer values:

NEO4J: Adding Indices

Like SQL-based databases, you can create indices in Neo4j to optimize frequently performed searches. In my particular case, I am usually searching by the plant name. Adding an index on the plant’s name, therefore, makes sense:

CREATE INDEX index_tomato_name IF NOT EXISTS
FOR (n:TOMATO)
ON (n.name)

I am also interested in mapping the genetic lineage, so I’ve added indices for the male and female parent plants:

CREATE INDEX index_tomato_parentm IF NOT EXISTS
for ()-[r:PARENT_M]-()
on (r.name)

CREATE INDEX index_tomato_parentf IF NOT EXISTS
for ()-[r:PARENT_F]-()
on (r.name)

 

Hypothetically, you can use the procedure CALL db.indexes(); to view all of the indices in a database, but SHOW PROCEDURES; shows me that procedure isn’t registered in the community edition.

╒═════════════════════════════════════════════════════════╤══════════════════════════════════════════════════════════════════════╤═══════╤═════════════╕
│name                                                     │description                                                           │mode   │worksOnSystem│
╞═════════════════════════════════════════════════════════╪══════════════════════════════════════════════════════════════════════╪═══════╪═════════════╡
│"db.awaitIndex"                                          │"Wait for an index to come online (for example: CALL db.awaitIndex(\"M│"READ" │true         │
│                                                         │yIndex\", 300))."                                                     │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.awaitIndexes"                                        │"Wait for all indexes to come online (for example: CALL db.awaitIndexe│"READ" │true         │
│                                                         │s(300))."                                                             │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.clearQueryCaches"                                    │"Clears all query caches."                                            │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.createLabel"                                         │"Create a label"                                                      │"WRITE"│false        │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.createProperty"                                      │"Create a Property"                                                   │"WRITE"│false        │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.createRelationshipType"                              │"Create a RelationshipType"                                           │"WRITE"│false        │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.index.fulltext.awaitEventuallyConsistentIndexRefresh"│"Wait for the updates from recently committed transactions to be appli│"READ" │true         │
│                                                         │ed to any eventually-consistent full-text indexes."                   │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.index.fulltext.listAvailableAnalyzers"               │"List the available analyzers that the full-text indexes can be config│"READ" │true         │
│                                                         │ured with."                                                           │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.index.fulltext.queryNodes"                           │"Query the given full-text index. Returns the matching nodes, and thei│"READ" │true         │
│                                                         │r Lucene query score, ordered by score. Valid keys for the options map│       │             │
│                                                         │ are: 'skip' to skip the top N results; 'limit' to limit the number of│       │             │
│                                                         │ results returned; 'analyzer' to use the specified analyzer as search │       │             │
│                                                         │analyzer for this query."                                             │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.index.fulltext.queryRelationships"                   │"Query the given full-text index. Returns the matching relationships, │"READ" │true         │
│                                                         │and their Lucene query score, ordered by score. Valid keys for the opt│       │             │
│                                                         │ions map are: 'skip' to skip the top N results; 'limit' to limit the n│       │             │
│                                                         │umber of results returned; 'analyzer' to use the specified analyzer as│       │             │
│                                                         │ search analyzer for this query."                                     │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.info"                                                │"Provides information regarding the database."                        │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.labels"                                              │"List all available labels in the database."                          │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.ping"                                                │"This procedure can be used by client side tooling to test whether the│"READ" │true         │
│                                                         │y are correctly connected to a database. The procedure is available in│       │             │
│                                                         │ all databases and always returns true. A faulty connection can be det│       │             │
│                                                         │ected by not being able to call this procedure."                      │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.prepareForReplanning"                                │"Triggers an index resample and waits for it to complete, and after th│"READ" │true         │
│                                                         │at clears query caches. After this procedure has finished queries will│       │             │
│                                                         │ be planned using the latest database statistics."                    │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.propertyKeys"                                        │"List all property keys in the database."                             │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.relationshipTypes"                                   │"List all available relationship types in the database."              │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.resampleIndex"                                       │"Schedule resampling of an index (for example: CALL db.resampleIndex(\│"READ" │true         │
│                                                         │"MyIndex\"))."                                                        │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.resampleOutdatedIndexes"                             │"Schedule resampling of all outdated indexes."                        │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.schema.nodeTypeProperties"                           │"Show the derived property schema of the nodes in tabular form."      │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.schema.relTypeProperties"                            │"Show the derived property schema of the relationships in tabular form│"READ" │true         │
│                                                         │."                                                                    │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.schema.visualization"                                │"Visualizes the schema of the data based on available statistics. A ne│"READ" │true         │
│                                                         │w node is returned for each label. The properties represented on the n│       │             │
│                                                         │ode include: `name` (label name), `indexes` (list of indexes), and `co│       │             │
│                                                         │nstraints` (list of constraints). A relationship of a given type is re│       │             │
│                                                         │turned for all possible combinations of start and end nodes. The prope│       │             │
│                                                         │rties represented on the relationship include: `name` (type name). Not│       │             │
│                                                         │e that this may include additional relationships that do not exist in │       │             │
│                                                         │the data due to the information available in the count store. "       │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.stats.clear"                                         │"Clear collected data of a given data section. Valid sections are 'QUE│"READ" │true         │
│                                                         │RIES'"                                                                │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.stats.collect"                                       │"Start data collection of a given data section. Valid sections are 'QU│"READ" │true         │
│                                                         │ERIES'"                                                               │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.stats.retrieve"                                      │"Retrieve statistical data about the current database. Valid sections │"READ" │true         │
│                                                         │are 'GRAPH COUNTS', 'TOKENS', 'QUERIES', 'META'"                      │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.stats.retrieveAllAnonymized"                         │"Retrieve all available statistical data about the current database, i│"READ" │true         │
│                                                         │n an anonymized form."                                                │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.stats.status"                                        │"Retrieve the status of all available collector daemons, for this data│"READ" │true         │
│                                                         │base."                                                                │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"db.stats.stop"                                          │"Stop data collection of a given data section. Valid sections are 'QUE│"READ" │true         │
│                                                         │RIES'"                                                                │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.cluster.routing.getRoutingTable"                   │"Returns the advertised bolt capable endpoints for a given database, d│"DBMS" │true         │
│                                                         │ivided by each endpoint's capabilities. For example an endpoint may se│       │             │
│                                                         │rve read queries, write queries and/or future getRoutingTable requests│       │             │
│                                                         │."                                                                    │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.components"                                        │"List DBMS components and their versions."                            │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.info"                                              │"Provides information regarding the DBMS."                            │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.killConnection"                                    │"Kill network connection with the given connection id."               │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.killConnections"                                   │"Kill all network connections with the given connection ids."         │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.listCapabilities"                                  │"List capabilities"                                                   │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.listConfig"                                        │"List the currently active config of Neo4j."                          │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.listConnections"                                   │"List all accepted network connections at this instance that are visib│"DBMS" │true         │
│                                                         │le to the user."                                                      │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.queryJmx"                                          │"Query JMX management data by domain and name. For instance, \"*:*\"" │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.routing.getRoutingTable"                           │"Returns the advertised bolt capable endpoints for a given database, d│"DBMS" │true         │
│                                                         │ivided by each endpoint's capabilities. For example an endpoint may se│       │             │
│                                                         │rve read queries, write queries and/or future getRoutingTable requests│       │             │
│                                                         │."                                                                    │       │             │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.upgrade"                                           │"Upgrade the system database schema if it is not the current schema." │"WRITE"│true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"dbms.upgradeStatus"                                     │"Report the current status of the system database sub-graph schema."  │"READ" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"tx.getMetaData"                                         │"Provides attached transaction metadata."                             │"DBMS" │true         │
├─────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────┼───────┼─────────────┤
│"tx.setMetaData"                                         │"Attaches a map of data to the transaction. The data will be printed w│"DBMS" │false        │
│                                                         │hen listing queries, and inserted into the query log."                │       │             │
└─────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────┴

NEO4J: Searching

Returning data from NEO4J is matching — we can match a node using MATCH p=(a:TOMATO {name: 'Black Krim'}) return p;

where the object “p” becomes a set of nodes labeled with ‘TOMATO’ where the value of ‘name’ is ‘Black Krim’

More advanced matches set the return object to a set of nodes and relationships — here we set p to the set of items starting at the node labeled TOMATO with name ‘PLANT0’ with relationships and nodes until you get to a node labeled TOMATO with name ‘Tomato0000007’

MATCH p=(a:TOMATO {name: 'PLANT0'})-[*]->(b:TOMATO {name: 'Tomato0000007'}) RETURN p