Infrastructure for Typed Knowledge Graphs

In this post we explore infrastructure required for our emerging synthetic field:

Infrastructure for Typed Knowledge Graphs

TGKT = Typed Graph Knowledge Theory
   is a mathematical definition space (purely functional), in which we obtain proofs as knowledge.
   TGKT is an open-world compile-time artifact, used to produce closed-world runtime artifacts
   deployed into TGKB below.
TGKB = Typed Graph Knowledge Base
   is a stateful space of applied effects, in which we reason using compiled artifacts from TGKT above.
   Data received (effect) in TGKB serves as explicit axioms in a queryable form until purged (effect).
   TGKB is closed, predicative, replacable, and equivalent at any moment to a simple set of data files.
   One TGKB is always grounded in a single, permanently fixed immutable TGKT.
   We derive explainable results from such a TGKB.   
   Complex behavior is composable as a sequence of TGKBs, based on a sequence of TGKTs.

General needs 

  • Store and present information from quantitative realms such as finance, physical sciences.
  • Tee up explainable decisions over data
  • Deploy remixable A,B,C technology options in each component and layer.

Computation Activities

  • Type Description
  • Proof and Verification
  • Data Movement and Xform 
Below we rank our options for these core runtime needs, enumerating possible approaches where
required technology is not yet fully standardized and ubiquitous.
  1. exec - executable machine for lab desktop, and symmetrically for server and cloud
  2. nosql - database components which also run on desktop, server, or cloud
  3. cloud - a chosen platform of cloud infrastructure, such as Google Cloud, AWS, MS Azure
  4. xmsg - approach to internal messaging and transactions

Exec machine options

  • exec_main:  ScalaToJvm

NoSQL DB options

Cloud scalable deployment options

Basically ubiquitous in commercial realm, but can be expensive to migrate from one setup to another.
  • cloud_main: Google Cloud 
    • cloud_alt_01: Amazon Web Services
    • cloud_alt_02: Microsoft Azure

XMsg Options

  • xmsg_main:  HTTP + JSON + Akka + NoSQL
    • xmsg_alt_01 : AMQP + Avro + NoSQL
    • Kafka, Flink, Spark, Hadoop
    • Quarkus  Vertex 
 
 
 fin
 

 

 
  

Popular posts from this blog

Overview of AxLam model : Nouns, Types, Morphisms

2023-Feb-03 picking up from tweet thread of 220909  AxLam model: Noun (space), Type (paint), Morphism (arrow) AxLam is for writing models using [concepts, types, and morphisms].  Our concepts may be in the form of some data, or less specifically as things we choose to name or refer to in a model.  Over this concept space, we use types and morphisms to define an adjustable descriptive model that directs and regulates processing of the concept data (or other manifestation).  Alternatively and  more approachably we may write in terms of [nouns,  formal classes, and mappings], and this framing is mostly equivalent for informal purposes. Nouns / Concepts / Terms / Data Nouns (i.e. concepts) describe anything we wish to refer to, and need not be explicit or named.  We think of nouns as the space of things we might try to describe, name or refer to.  "Terms" in a type-theoretic sense are nouns.  Real things in the human world that someone cares abo...
Read more

Streaming analytics in Scala using ZIO and Apache DataSketches (part 1)

Image
First let's look at a trivial example of streaming analytics using ZIO Streams 2.0.9 (on Scala 2.13). https://scastie.scala-lang.org/6AHfPh02S6m0k7aN6bkacA When run, this Scastie just averages some random numbers (gaussian) in a single input stream.  That code is in the trait AvgFun . But we also see a trait called ParAvgFun with the code for rather painless parallel  implementation, using ZIO chunks, via ZStream's  grouped , zipWithIndex , and groupByKey .  Across these two traits, the clean reusability of ZSinks in different pipelines shows a nice design win for the ZIO Streams API.   The code in  ParAvgFun.parAvgUsingChunks  demonstrates a terse and reasonably efficient design for accumulating results using parallel ZStreams (within a single process).  It is applicable to  simple cases where the stream elements are lightweight and similar, and the underlying operations are uniformly easy (like summing numbers).  When inste...
Read more

AxioMagic under Spark on cloud (AWS, Azure, ...)

AxioMagic under Spark on ${ cloud  } AxioMagic relies on functional composition over immutable inputs, thus matching modern evaluation frameworks like Apache Spark, which defines a structured approach to managing data contexts of functional evaluation over data frames.  Spark embodies a refinement of both  ETL and MapReduce methodology.  State machines like Spark require computation and storage resources which may come from any cluster, but today are commonly provided from cloud environments, yielding high volume functional computation at predictable usage-based costs.  We especially value Spark for open scientific dataset analysis, where it is common to work with large data files in interchange formats. Essential features of Spark distributed evaluation Spark assembles batch pipelines to pump datasets of structured records through functional tasks (which may  invoke side-effects, with all necessary stipulations).  This pipeline serves to evaluate...
Read more