Develop dataspectsSystem

From wiki.dataspects.com
Jump to: navigation, search

Run the stack[edit | edit source]

Standard Services dataspects Services

Elasticsearch[edit | edit source]

Kibana[edit | edit source]

ELASTICSEARCH_HOSTS=http://127.0.0.1:9200

Tika[edit | edit source]

UI[edit | edit source]

PORT=3000
REDIS_HOST=<HOSTNAME>
REDIS_PORT=6379
SESSION_SECRET=
MONGODB_URI=mongodb://<HOSTNAME>:27017/<DB_NAME>
SMTP_SERVER=
SMTP_USERNAME=
SMTP_PASSWORD=
FROM_EMAIL=
FROM_NAME=
SITE_URL=https://ui.dataspects.com
OTP_DOMAIN=ui.dataspects.com
API_KEY=
ES_NODE=http://<HOSTNAME>:9200
DOCUMENT_DATA_STORE_API_URL=http://<HOSTNAME>:3003
DOCUMENT_DATA_STORE_API_MASTER_KEY=
APM_SERVER_URL=http://<HOSTNAME>:8200

dataspectsd[edit | edit source]

https://github.com/dataspects/dataspectsd

go/src/github.com/dataspects/dataspectsd$ go run main.go --c config.yml --p 3001 --d dataspectsd.db
# config.yml
apikey: "ping"
elastic-search:
  host: http://localhost
  port: 9200
  username:
  password:
tika:
  host: localhost
  port: 9998

Ideally you run all services except UI, dataspectsd and dataspecter by https://github.com/dataspects/dataspectsSystems.

Feeding data from ResourceSilos to indices[edit | edit source]

MediaWiki[edit | edit source]

PUSH: DataspectsMediaWikiFeeder[edit | edit source]

PULL: dataspecter[edit | edit source]

https://github.com/dataspects/dataspecter

Code Repository (non-git)[edit | edit source]

AAAA-Ingestion "From Repository to dataspectsd by"

Data fed?
Configuration
  1. Configure the datastore
  2. Configure the feeder
Subsequent pipeline
Tools


Git Repository[edit | edit source]

Documents (non-repository)[edit | edit source]

https://github.com/dataspects/dsdocument-cli

AAAA-Ingestion "From File system to Datastore by https://github.com/dataspects/dsdocument-cli"

Data fed?
Configuration

Configure the datastore:

<Datastore ID>        automatic (e.g. 12)
<Datastore Label>     Shown in information sources list on https://ui.dataspects.com/search
<Datastore API Key>   automatic (e.g. c8b89bc3-0139-11wa-8ef3-8c164563716b)
<Datastore Doc Regex> Only file names matching this regex will be fed to the datastore (Regex Tester - Golang)

Configure and run the feeder to index matching files in and below the current folder:

user@workstation:/yourfolder$ ./dsdocument-cli \
                              --id  <Datastore ID>      # From https://ui.dataspects.com/datastores/files \
                              --url https://datastore.dataspects.com \
                              --key <Datastore API Key> # From https://ui.dataspects.com/datastores/files
Subsequent pipeline
Tools


Signals Intelligence (SIGINT)[edit | edit source]

Relevance Engineering (RELENG)[edit | edit source]

ui/relevanceEngineering

Quality Assurance (QA)[edit | edit source]

UI design (results display and interaction design (faceting, drilldown))[edit | edit source]

  • Helper: uncomment link(rel='stylesheet', href='/css/dataspects-meta.css') in dataspects-ui/views/layout.pug
  • RequestTypes: mainRequest, predicateRequest, entityTypeRequest, actionRequest