ideas

[max-mapper]

npm itself is pretty modular and many parts of it can be used here, e.g.:

https://npmjs.org/package/init-package-json
https://github.com/isaacs/promzard
https://npmjs.org/package/read-installed

(the list goes on)

it would be nice to support this workflow:

npm install datapackage -g
datapackage init

then it would prompt you in the same way that npm init works, and you end up with a valid package.json

I also would like to find common ground for a module that I can depend on from dat to manage a package.json for data in every dat repo that gets initialized. maybe a module that uses init-package-json with a schema specific to SDF? then this module and dat could both use it

[max-mapper]

check this out, npm install and npm link: https://github.com/maxogden/datapackage-json

then:

macintosh-3:datapackage-json max$ mkdir foo
macintosh-3:datapackage-json max$ cd foo
macintosh-3:foo max$ datapackage-json
name: (foo) 
version: (0.0.0) 
description: 
git repository: 
keywords: 
license: (BSD) 
About to write to /Users/max/src/js/datapackage-json/foo/package.json:

{
  "name": "foo",
  "version": "0.0.0",
  "description": "",
  "license": "BSD"
}


Is this ok? (yes) 
macintosh-3:foo max$ cat package.json 
{
  "name": "foo",
  "version": "0.0.0",
  "description": "",
  "license": "BSD"
}
macintosh-3:foo max$ 

we can edit prompt.js to port over the validation code etc from this repo

[max-mapper]

one thing i'm unsure of is if theres a programmatic API for init-package-json (e.g. bypass prompts and write a package.json out with defaults)

[rufuspollock]

@maxogden loving this! Do you think we should just direct reuse npm itself perhaps (a fork and rename to dpm with a few tweaks) or incorporate relevant parts of npm into a "dpm"?

I also think we agreed on rename of datapackage.jspon to package.json at the meeting Tuesday. Its a breaking change but worth it if it means we are compatible with npm going forward :-)

To summarize, see 2 components atm:

• dpm - data package manager (may just be npm in disguise). Used for installing, publishing etc (from command line and possibly library use as well but rare)
• datapackage.js - a JS library for doing various things with data packages (this lib at the moment) (reused in data.okfn.org, in the browser etc)

At some point we might imagine the 2 converging but imagine them as separate atm.

wdyt?

[max-mapper]

+1 to using package.json

as far as a fork of npm vs using some of its modules goes -- i'll have to do some research first and talk to isaacs and see what he thinks.

I'm not sure what the exact scope of this repo is then. e.g. I could imagine the CSV stuff in here going in it's own module, and the init stuff going into datapackage-json, leaving this as just a sort of high-level module that bundles together functionality from other modules. also where would this and dpm overlap?

also, just updated datapackage-json, you can do a couple different commands now:

datapackage-json
datapackage-json default
datapackage-json prompt

the first two are the same (default is the default)

default generates a default package.json from the prompt script without using a prompt (just gets default values)

prompt asks the user for values and saves the default value to a script

theres also a non-cli command .write which takes a json obj and writes it as a package.json file (intended to be used with .default

[max-mapper]

also I should mention that I intend to use the datapackage-json commands in dpm, so when you do dpm init it just runs datapackage-json prompt etc.

now that I think about it I should probably rename datapackage-json prompt to datapackage-json init

[max-mapper]

so I think it makes sense for both dat and dpm to use the datapackage-json library

that way if you do dat init, you'll end up with a valid data package that can be read by dpm in case you want to use it to publish your dataset in the future. and vice-versa

there are also deployment tools that can be built on top of dat repos that can use datapackage-json as well.

[rufuspollock]

@maxogden sounds good and awesome to see how this is developing.

From your comments here (esp about using datapackage-json lib in dat) I'm wondering if there is a useful difference between datapackage.js and datapackage-json? I'm very happy to merge and then contribute into datapackage-json :-)

Also how do we hook into the init stuff to add extra datapackage items (primarily resources and for simple data package the schema stuff)

[rufuspollock]

@maxogden I'd like to triple-check a last time before we switch datapackage.json to package.json - should I open an issue in isaacs/npm repo outlining plan to be heavily based on commonjs packages and npm and asking if it is possible to config package.json.

Pros and Cons of a definite switch as I see it:

• (+) compatibility with npm
  • Real question here is whether we absolutely need this to be able to reuse npm or whether it can be config'd or overriden e.g. init-package-json does not explicitly require package.json
• (-) its a breaking change for all existing data packages
• (-) we conflict with node package.json (perhaps unlikely to ever happen)
• (-) nice to have alignment data package => datapackage.json (but relatively minor consideration IMO)

First point would outweigh pretty much all others (by a long way).

[max-mapper]

some thoughts:

• package.json is assumed in not only npm but also the entire ecosystem of tools around it
• the two things absolutely hardcoded in npm are node_modules and package.json (these are cosmetic issues)
• by implement our own superset of npm and using the npm programmatic API we can make our thing nice to use

e.g. https://github.com/maxogden/lunny was an experiment I did last year -- it wraps npm in a command that still installs packages by name from the npmjs.org registry but after installation it does a post-processing step, e.g. it takes java packages and runs an ant build step and/or moves files from node_modules/.... into more convenient locations

what I am unclear on still is exactly how datasets as dependencies should work (at a high level)

if we want to use npm and the ecosystem around it, the following scenario has to be important:

• given an application with dependencies A and B
• if A and B both depend on C, but different versions of C:
• A depends on C@0.0.5
• B depends on C@2.0.1
• given semver rules it means that C@2.0.1 has made breaking changes since C@0.0.5 so they are incomptible
• NPM will install both copies of C, inside both A and B's node_modules folder
• the benefit is that you get the exact working version of each module you need
• the downside is you get multiple copies of the same module in a single dependency tree

now, for small modules of JS code this works great because most are so small that it wouldn't matter if you had thousands of copies of a module in a project because resources (RAM, disk) are relatively cheap and plentiful

it gets less beneficial when you try to support legacy applications in other environments, for instance jQuery for client side development. having 5 copies of jQuery isn't realistic for say a mobile app because of bandwidth limitations.

there are two solutions: don't use npm for client side dev with 'big' libraries like jQuery OR don't use 'big' libraries and instead depend on small components e.g. instead of require('jquery') do require('jquery-toggle')

so, given that explanation, how does it map to data sets?

say I want to make an app for locating voting polling place on the west coast of the US.

first I would dpm install oregon-polling-places, which was published by user A
then I would dpm install california-polling-places, which was published by user B

the oregon data also installed/depended on a package called us-zip-codes at version 1.0.0, which uses a column called ZIP_CODE

but the california data was written later and uses the newer us-zip-codes@2.0.0, which changed the column name to code

(this thought experiment is assuming that semver with data is used to denote only when schema is patched, added to or broken, not the rows of the data itself)

so, on one hand, I know that when I load the oregon-polling-place data into my app, it can reference data in the us-zip-codes@1.0.0 dataset using ZIP_CODE and that the column will definitely exist and my app will work (and not return a bunch of empty zip code fields), and that the same is true for the california data

on the other hand, I have to install two copies of the us-zip-code data at multiple versions. in this sense require('us-zip-codes') is kind of like require('jquery'), whereas require('oregon-zip-codes') would be more like require('jquery-toggle')

so, while this example might be slightly contrived, it demonstrates the fundamental tradeoff when using NPM. NPM works amazingly well for small javascript modules, and I think it would work well for complex dependency trees of dataset metadata, but it's worth discussing in depth the merits of the approach as well

another possible solution to the above zip code example might be that when the us-zip-codes maintainer published version 2.0.0 they also provided a sort of schema migration so that packages that on e.g. version 1.0.0 could just automagically transform the data to a backwards compatible state when using the 2.0.0 dataset

theres a command in npm called npm dedupe which will traverse through the entire dependency tree and analyze semver ranges to remove dependencies at leaf nodes that are semver compatible duplicates of other dependencies at other leaf nodes, resulting in a single package at the common higher level point in the tree that both sub-dependencies can require. i'm sure in the realm of data there are similar optimizations that can be built on top of npm-style dependency trees

[rufuspollock]

So I think on package.json question it definitely seems necessary to go with that - I've therefore opened frictionlessdata/datapackage#73

On the question of data dependencies my sense is that we should not worry too much about inefficiency of storage at the present. My reasoning is:

• Simplicity and ease of use trumps storage considerations, at least at an early stage like this (plus we have options like dedupe if this is a big problem)
• We may not always have local data - that is sometimes dependencies may be on "endpoints" (APIs) that are not local, and hence space is not such an issue (I note this is something I've not totally thought through but its something I've thought about from the start of the work on data packages that we should not confine ourselves to see data as local files on disk but simply as streams (or file-like objects) that can be local or remote. Of course, how this exactly fits with npm etc is to be worked out but I think this is something important to consider).

Given the other plus points (and the well-thought through model of npm) I think it is definitely a win to go with that (and its dependency model) so right now I think we should forge ahead.

so my next question is: to get started on an dpm should we just fork npm and make some mods or do something more systematic?

[sballesteros]

Not sure if I should comment here or in dataprotocols/dataprotocols#73 but +1 for package.json and leveraging npm!

I think it could be nice to use the scripts property and in particular prepublish (see doc here) to fix some of the storage issues and have a good integration with node.

For the storage, the prepublish script could parse the datapackage and publish the data of resources having a path field into a common data registry. The path property of the resource will then be replaced by an url property. Separating the data registry from the common npm registry makes sense as some datapackages describe large data sets frequently appended. In that case, having a new version each time the dataset is appended would be a huge resource waste for the common npm registry, whereas a data registry could handle that efficiently.

For better integration with node, the prepublish script could automatically create an index.js file (and point to it in the main property of package.json) so that the resources of the datapackage are exposed as node.js streams.
For instance, a resource named crime coming from a datapackage named dpkg could be access in JS code with:

var crime = require('dpkg').crime();

require('dpkg').crime() would then return a readable stream.
for resources having a path or url property, a standard stream could be returned whereas for resources having a data property (inline JSON) a stream operating in object mode could be returned (or simply the an object if data is not a list). In case of SDF, we could go further and use the schema property to return a stream operating in object mode and emitting coerced objects (using the type property).

When the datapackage has to be used offline, we could use an install or postinstall script (still in package.json) to cache the data (saving it locally) and re-replacing the url of the resources by path. If the data registry implement SLEEP, we could use dat to ensure the storage locally.

A problem with that, is that in the datapackage spec, resources is defined as a list with no guarantee of unicity for the name property :(

Another use case worth considering is npm modules that have dataset dependencies. I wrote one of these recently: https://github.com/substack/cities1000 It stores the data itself on npm which is not ideal but it's fine for now because the ease of use outweighs the downsides until this dpm command and registry exists.

One way you could have a module like cities1000 on npm with an external data dependency is to have a dataDependencies field in package.json like this:

{
  "dataDependencies": {
    "cities1000": "~0.0.0"
   }
}

dataDependencies names could default to the okfn repository by default but if you want to have dependencies from other repos you could do something like "reponame/pkgname": "~1.2.3" or perhaps pkgname@reponame": "~1.2.3"

and packages published with a dataDependencies field on npm can just add a scripts.postinstall to use dpm to download the datasets

[max-mapper]

I like the dataDependencies as a way to have a multi-registry package.json.

So maybe if you are publishing to NPM you can use dataDependencies, and if you are only using DPM you can just use dependencies like normal (and DPM would be smart enough to try both)