David Allen Profile picture
Sep 14, 2021 9 tweets 4 min read Read on X
Example thread on how to load #JSON into #Neo4j Aura -- working up from simple to more complex. Let's use the .@TheHackersNews public API to load a mini-feed of stories.

First: head endpoint with best stories, and simplest JSON load: Image
the apoc.load.json call always returns "value" with whatever came back. HackerNews is sending results, an array of post IDs.

We can extract out just the post IDs with a bit of extra cypher like this. Nice clean array of long values. Image
One step further; now we will UNWIND the array, turning the nested array into each individual item, and then build the URL we'll ask of HackerNews to get the detail of each story. This is how we build URLs one by one; we just take the story ID and concat it into a string URL Image
Now, let's just add an apoc.load.json call on to each of those URLs we constructed, and see what the resulting JSON of each story is. This gives us a result set of the JSON payload of every story on the site.

#BuildingStepByStep #neo4j Image
Simplest graph format we can do here is just a user submitted a story. Something like:

(:User)-[:SUBMITTED]->(:Story)

So let's do that.
Here's the load code with the MERGE statements that put our Users and Stories into the graph. And it works! Image
resulting snapshot of the graph in #Neo4jAura Image
some slightly fancy things in that merge statement explained:

(1) hackernews reports times in UNIX time (seconds since the epoch) so

datetime({ epochSeconds: value.time })

turns that into a neo4j datetime
it's possible for the URL to be missing from a story! So:

coalesce(value.url, 'REMOVED')

this returns the first non-null value; so if value.url is missing, you get a url of 'removed'

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with David Allen

David Allen Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @mdavidallen

Feb 8, 2023
Having recently started at @grafana I just want to thread a minute about a meta-pattern I'm seeing in software communities that my graph Neo4j people might relate to 🧵
caring about community one of the things I like to look at is most viewed community posts as a broad aggregate view of what people have trouble with / what they wonder about, which I was recently doing with grafana's community site
a pattern that jumps out is "security architecture" type questions. Things like:

- how to embed X web resource into my web page
- how to set up SSL certs in a secure way
- certain error messages relating to security config
Read 13 tweets
Jan 4, 2022
Back in my government consulting days we used to think about policies & laws in terms of this 4-square (with some examples). Who pays, and who benefits?

"Concentrated" means focused on an identifiable group; "diffuse" means spread out among everyone
Concentrated benefits + diffuse costs are typically the easiest things to get done, and the hardest things to kill. An example is all plumbers get a $2k tax break. You have a motivated constituency who will fight HARD for it. And the cost per taxpayer might be $0.05
concentrated costs & benefits feel the most fair: like a toll booth. Don't wanna pay? Don't take the expressway. But they tend not to work except in the context of a specific activity (plumbing, or using the highway

Main opposition is "make the gov't or someone else pay"
Read 18 tweets
Jan 3, 2022
Interesting stuff. Spotify track shuffle was once truly random and people complained. It didn't "feel random". So they made it not random and people were happier

medium.com/immensity/how-…
I generated these random numbers online (1-5)

1 3 4 2 2 2 1 3 2 1

they don't "feel random" though because too many 2s
now we logically know that the chance of a 2 in each slot is 1 in 5 and independent of other draws

but your brain is hardwired to sense pattern even when there is none and pattern "isn't random"
Read 7 tweets
Oct 20, 2021
The cat, as a metaphor for software engineering
Sub-systems -- and "System of Systems"
Abstraction
Read 7 tweets
Oct 18, 2021
For some reason, some mutuals got me thinking about smoking

Memory: working as a teen, I knew a guy who was 100% sure that it was the glue in cigarette papers that caused cancer, not the tobacco. He smoked natural tobacco leaf-wrapped cigars, thought he was all good
at the shop where I worked, there was a 90-yr old guy who walked with a cane. Was the spitting image of William S. Burroughs. Every day, he came in and got his usual, 2 packs of Lucky Strikes unfiltered Image
when you're a teenage smoker yourself, a 90-yr old with a 2-pack a day Lucky Strikes habit is like a hero, but even then I knew he was more lucky than "right"
Read 8 tweets
Oct 1, 2021
#SQL vs. #Cypher for hierarchies

Recursive common-table expressions (SQL)

vs.

built-in path variables in #graph ImageImageImage
This only shows part of the picture; with recursive CTEs in #SQL I believe it is the case that:

- You can constrain max recursion, but not minimum without writing extra code

Contrast to cypher:

MATCH (e:Employee)-[:REPORTS_TO*3..5]->(:Employee)
recursive CTEs are a good example of "implementing a graph abstraction on top of something else". Namely the employee table represents a bunch of node instances; the managerID is a relationship, and the CTE implements a graph traversal.
Read 5 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(