Monitoring

• e.g. for MediaWiki data
• filtering can occur on the MediaWikiWorkbench and/or Kibana

Get the monitoring data into Elasticsearch

Create a MediaWikiWorkbench filter to Elasticearch:

package main

import (
	"github.com/dataspects/mediawiki-workbench/elasticsearch"
	"github.com/dataspects/mediawiki-workbench/mediawiki"
)

var cofalnet, _ = mediawiki.NewMWClient(mediawiki.MWConfig{
	MWAPIURL: "https://cookbook.findandlearn.net/w/api.php",
})

var localES, _ = elasticsearch.NewESClient(elasticsearch.ESConfig{
	ESURL: "http://localhost:9200",
})

func myFilter(mwpage mediawiki.MediaWikiPage) {
	// Entities having syntaxhighlighting
	if mwpage.Wikitext.Containing_STRING(`< *syntaxhighlight`) {
		mwpage.DNA = "Entities having syntaxhighlighting"
		localES.StoreInINDEX(mwpage, "dataspects1910291146")
	}
	// Entities having mermaid charts
	if mwpage.Wikitext.Containing_STRING(`{{#[m|M]ermaid:`) {
		mwpage.DNA = "Entities having mermaid charts"
		localES.StoreInINDEX(mwpage, "dataspects1910291146")
	}
	// Recipes having no blurb
	if mwpage.Wikitext.Having_exactly_one_TEMPLATE("Recipe") {
		if mwpage.Wikitext.Templates["Recipe"][0].Fields["HasEntityBlurb"].Is_empty() {
			mwpage.DNA = "Recipes having no blurb"
			localES.StoreInINDEX(mwpage, "dataspects1910291146")
		}
	}
}

func main() {
	cofalnet.Ask("Category:Entity", myFilter)
}

Kibana

See https://github.com/dataspects/dataspectsSystemBuilder/blob/master/monitoring/kibana_saved_objects.ndjson

Create saved queries

How:

Why:

Manage: type "query" under /app/kibana#/management/kibana/objects

Create index pattern(s)

How? Under /app/kibana#/management/kibana/index_patterns define index pattern(s) that match(es) your monitoring indices.

Why? Index patterns form the basis of saved searches. They contain scripted fields. They can make use of saved queries.

Manage: type "search" /app/kibana#/management/kibana/objects

Add scripted fields to index patterns

HasEntityType

String mytype = ''; for( int i = 0; i < params._source.annotations.size(); i ++ ) {
    if(params._source.annotations[i]['predicate'] == 'HasEntityType') {
        mytype = params._source.annotations[i]['objectSource'];
        
    }
    
}
return mytype;

HasEntityTitle

String mytitle = ''; for( int i = 0; i < params._source.annotations.size(); i ++ ) {
    if(params._source.annotations[i]['predicate'] == 'HasEntityTitle') {
        mytitle = params._source.annotations[i]['objectSource'];
        
    }
    
}
return mytitle;

HasEntityURL

return params._source.rawUrl;

Create saved searches

E.g. select field "rawUrl" and scripted field "HasEntityType"

{
  "query": {
    "term": {
      "dna.keyword": {
        "value": "Entities having syntaxhighlighting"
      }
    }
  }
}

Create dashboards from panels (saved searches, pies, data tables, etc.)

To add a table listing items (and not aggregations) just add a saved search.