Import form JSON

If you are migrating from MongoDB or any other DBMS that exports data in JSON format, the JSON extractor is what you need. For more information look also at: Import-from-PARSE.

This is the input file stored in /tmp/database.json file:

  1. [
  2.  {
  3.   "name": "Joe",
  4.   "id": 1,
  5.   "friends": [2,4,5],
  6.   "enemies": [6]
  7.  },
  8.  {
  9.   "name": "Suzie",
  10.   "id": 2,
  11.   "friends": [1,4,6],
  12.   "enemies": [5,2]
  13.  }
  14. ]

Note that friends and enemies represent relationships with nodes of the same type. They are in the form of an array of IDs. This is what we need:

  • Use the Vertex class “Account” to store nodes
  • Use the Edge classes “Friend” and “Enemy” to connect vertices
  • Merge and Lookups will be on id property of Account class that will be unique
  • In case the connected friend hasn’t been inserted yet, create it (“unresolvedLinkAction”: “CREATE”)
  • To speed up lookups, a unique index will be created on Account.it

And this pipeline (log is at debug level to show all the messages):

  1. {
  2.   "config": {
  3.     "log": "debug"
  4.   },
  5.   "source" : {
  6.     "file": { "path": "/tmp/database.json" }
  7.   },
  8.   "extractor" : {
  9.     "json": {}
  10.   },
  11.   "transformers" : [
  12.     { "merge": { "joinFieldName": "id", "lookup": "Account.id" } },
  13.     { "vertex": { "class": "Account"} },
  14.     { "edge": {
  15.       "class": "Friend",
  16.       "joinFieldName": "friends",
  17.       "lookup": "Account.id",
  18.       "unresolvedLinkAction": "CREATE"
  19.     } },
  20.     { "edge": {
  21.       "class": "Enemy",
  22.       "joinFieldName": "enemies",
  23.       "lookup": "Account.id",
  24.       "unresolvedLinkAction": "CREATE"
  25.     } }
  26.   ],
  27.   "loader" : {
  28.     "orientdb": {
  29.       "dbURL": "plocal:/tmp/databases/db",
  30.       "dbUser": "admin",
  31.       "dbPassword": "admin",
  32.       "dbAutoDropIfExists": true,
  33.       "dbAutoCreate": true,
  34.       "standardElementConstraints": false,
  35.       "tx": false,
  36.       "wal": false,
  37.       "batchCommit": 1000,
  38.       "dbType": "graph",
  39.       "classes": [{"name": "Account", "extends":"V"}, {"name": "Friend", "extends":"E"}, {"name": 'Enemy', "extends":"E"}],
  40.       "indexes": [{"class":"Account", "fields":["id:integer"], "type":"UNIQUE_HASH_INDEX" }]
  41.     }
  42.   }
  43. }

Note also the setting

  1.       "standardElementConstraints": false,

This is needed, in order to allow importing the property “id” in the OrientDB Loader. Without this option, the Blueprints standard would reject it, because “id” is a reserved name.

By executing the ETL process, this is the output:

  1. OrientDB etl v.2.1-SNAPSHOT www.orientechnologies.com
  2. feb 09, 2015 2:46:42 AM com.orientechnologies.common.log.OLogManager log
  3. INFORMAZIONI: OrientDB auto-config DISKCACHE=10.695MB (heap=3.641MB os=16.384MB disk=42.205MB)
  4. [orientdb] INFO Dropping existent database 'plocal:/tmp/databases/db'...
  5. BEGIN ETL PROCESSOR
  6. [file] DEBUG Reading from file /tmp/database.json
  7. [orientdb] DEBUG - OrientDBLoader: created vertex class 'Account' extends 'V'
  8. [orientdb] DEBUG orientdb: found 0 vertices in class 'null'
  9. [orientdb] DEBUG - OrientDBLoader: created edge class 'Friend' extends 'E'
  10. [orientdb] DEBUG orientdb: found 0 vertices in class 'null'
  11. [orientdb] DEBUG - OrientDBLoader: created edge class 'Enemy' extends 'E'
  12. [orientdb] DEBUG orientdb: found 0 vertices in class 'null'
  13. [orientdb] DEBUG - OrientDBLoader: created property 'Account.id' of type: integer
  14. [orientdb] DEBUG - OrientDocumentLoader: created index 'Account.id' type 'UNIQUE_HASH_INDEX' against Class 'Account', fields [id:integer]
  15. [0:merge] DEBUG Transformer input: {name:Joe,id:1,friends:[3],enemies:[1]}
  16. [0:merge] DEBUG joinValue=1, lookupResult=null
  17. [0:merge] DEBUG Transformer output: {name:Joe,id:1,friends:[3],enemies:[1]}
  18. [0:vertex] DEBUG Transformer input: {name:Joe,id:1,friends:[3],enemies:[1]}
  19. [0:vertex] DEBUG Transformer output: v(Account)[#11:0]
  20. [0:edge] DEBUG Transformer input: v(Account)[#11:0]
  21. [0:edge] DEBUG joinCurrentValue=2, lookupResult=null
  22. [0:edge] DEBUG created new vertex=Account#11:1{id:2} v1
  23. [0:edge] DEBUG created new edge=e[#12:0][#11:0-Friend->#11:1]
  24. [0:edge] DEBUG joinCurrentValue=4, lookupResult=null
  25. [0:edge] DEBUG created new vertex=Account#11:2{id:4} v1
  26. [0:edge] DEBUG created new edge=e[#12:1][#11:0-Friend->#11:2]
  27. [0:edge] DEBUG joinCurrentValue=5, lookupResult=null
  28. [0:edge] DEBUG created new vertex=Account#11:3{id:5} v1
  29. [0:edge] DEBUG created new edge=e[#12:2][#11:0-Friend->#11:3]
  30. [0:edge] DEBUG Transformer output: v(Account)[#11:0]
  31. [0:edge] DEBUG Transformer input: v(Account)[#11:0]
  32. [0:edge] DEBUG joinCurrentValue=6, lookupResult=null
  33. [0:edge] DEBUG created new vertex=Account#11:4{id:6} v1
  34. [0:edge] DEBUG created new edge=e[#13:0][#11:0-Enemy->#11:4]
  35. [0:edge] DEBUG Transformer output: v(Account)[#11:0]
  36. [1:merge] DEBUG Transformer input: {name:Suzie,id:2,friends:[3],enemies:[2]}
  37. [1:merge] DEBUG joinValue=2, lookupResult=Account#11:1{id:2,in_Friend:[#12:0]} v2
  38. [1:merge] DEBUG merged record Account#11:1{id:2,in_Friend:[#12:0],name:Suzie,friends:[3],enemies:[2]} v2 with found record={name:Suzie,id:2,friends:[3],enemies:[2]}
  39. [1:merge] DEBUG Transformer output: Account#11:1{id:2,in_Friend:[#12:0],name:Suzie,friends:[3],enemies:[2]} v2
  40. [1:vertex] DEBUG Transformer input: Account#11:1{id:2,in_Friend:[#12:0],name:Suzie,friends:[3],enemies:[2]} v2
  41. [1:vertex] DEBUG Transformer output: v(Account)[#11:1]
  42. [1:edge] DEBUG Transformer input: v(Account)[#11:1]
  43. [1:edge] DEBUG joinCurrentValue=1, lookupResult=Account#11:0{name:Joe,id:1,friends:[3],enemies:[1],out_Friend:[#12:0, #12:1, #12:2],out_Enemy:[#13:0]} v5
  44. [1:edge] DEBUG created new edge=e[#12:3][#11:1-Friend->#11:0]
  45. [1:edge] DEBUG joinCurrentValue=4, lookupResult=Account#11:2{id:4,in_Friend:[#12:1]} v2
  46. [1:edge] DEBUG created new edge=e[#12:4][#11:1-Friend->#11:2]
  47. [1:edge] DEBUG joinCurrentValue=6, lookupResult=Account#11:4{id:6,in_Enemy:[#13:0]} v2
  48. [1:edge] DEBUG created new edge=e[#12:5][#11:1-Friend->#11:4]
  49. [1:edge] DEBUG Transformer output: v(Account)[#11:1]
  50. [1:edge] DEBUG Transformer input: v(Account)[#11:1]
  51. [1:edge] DEBUG joinCurrentValue=5, lookupResult=Account#11:3{id:5,in_Friend:[#12:2]} v2
  52. [1:edge] DEBUG created new edge=e[#13:1][#11:1-Enemy->#11:3]
  53. [1:edge] DEBUG joinCurrentValue=2, lookupResult=Account#11:1{id:2,in_Friend:[#12:0],name:Suzie,friends:[3],enemies:[2],out_Friend:[#12:3, #12:4, #12:5],out_Enemy:[#13:1]} v6
  54. [1:edge] DEBUG created new edge=e[#13:2][#11:1-Enemy->#11:1]
  55. [1:edge] DEBUG Transformer output: v(Account)[#11:1]
  56. END ETL PROCESSOR
  57. + extracted 2 entries (0 entries/sec) - 2 entries -> loaded 2 vertices (0 vertices/sec) Total time: 228ms [0 warnings, 0 errors]

Once ready, let’s open the database with Studio and this is the result:

Import from JSON - 图1