Develop dataspectsSystem

Run the stack

Standard Services

dataspects Services

Elasticsearch Kibana ELASTICSEARCH_HOSTS=http://127.0.0.1:9200 Tika

UI https://github.com/dataspects/dataspects-ui SystemComponent "ui.dataspectsSystem" PORT=3000 REDIS_HOST=<HOSTNAME> REDIS_PORT=6379 SESSION_SECRET= MONGODB_URI=mongodb://<HOSTNAME>:27017/<DB_NAME> SMTP_SERVER= SMTP_USERNAME= SMTP_PASSWORD= FROM_EMAIL= FROM_NAME= SITE_URL=https://ui.dataspects.com OTP_DOMAIN=ui.dataspects.com API_KEY= ES_NODE=http://<HOSTNAME>:9200 DOCUMENT_DATA_STORE_API_URL=http://<HOSTNAME>:3003 DOCUMENT_DATA_STORE_API_MASTER_KEY= APM_SERVER_URL=http://<HOSTNAME>:8200 dataspectsd https://github.com/dataspects/dataspectsd go/src/github.com/dataspects/dataspectsd$ go run main.go --c config.yml --p 3001 --d dataspectsd.db # config.yml apikey: "ping" elastic-search: host: http://localhost port: 9200 username: password: tika: host: localhost port: 9998

Ideally you run all services except UI, dataspectsd and dataspecter by https://github.com/dataspects/dataspectsSystems.

Feeding data from ResourceSilos to indices

MediaWiki

PUSH: DataspectsMediaWikiFeeder

• https://github.com/dataspects/DataspectsMediaWikiFeeder
• SystemComponent "DataspectsMediaWikiFeeder"

PULL: dataspecter

https://github.com/dataspects/dataspecter

Code Repository (non-git)

AAAA-Ingestion "From Repository to dataspectsd by"

{{#bootstrap_icon: asterisk}} {{#bootstrap_badge:Data fed|color="warning"}}?	Data fed}}
Configuration	Configuration}}
Subsequent pipeline	Subsequent pipeline}}
Tools	Tools}}
{{#lsth:C0334941560|Comment}}

Documents (non-repository)

https://github.com/dataspects/dsdocument-cli

AAAA-Ingestion "From File system to Datastore by https://github.com/dataspects/dsdocument-cli"

{{#bootstrap_icon: asterisk}} {{#bootstrap_badge:Data fed|color="warning"}}?	Data fed}}
Configuration	Configuration}}
Subsequent pipeline	Subsequent pipeline}}
Tools	Tools}}
{{#lsth:C0992221051|Comment}}

Relevance Engineering (RELENG)

ui/relevanceEngineering

UI design (results display and interaction design (faceting, drilldown))

• Helper: uncomment link(rel='stylesheet', href='/css/dataspects-meta.css') in dataspects-ui/views/layout.pug
• RequestTypes: mainRequest, predicateRequest, entityTypeRequest, actionRequest