Very interesting project, indeed.
You’ve raised many good questions, which cannot be efficiently discussed in a single forum topic. So, I would suggest to post each question as separate topic. For example, you can start a topic on data format and organization.