Blog

Griffin uses livy as job-submitter of spark, all jobs will be submitted via livy client.

In code base, we wrap livy client into LivyTaskSubmitHelper.java .

SparkSubmitJob.java leverages to quartz scheduler and call livy client to submit jobs

Griffin gives a new sql-like language to define data quality computation, there is an implementation in measure project.

GriffinDslParser.scala is responsible for parsing rules in data quality.

Data sink is a term used to describe a computer or any other medium capable of receiving data.

Sink.scala is an abstract data sink type in griffin which deals with all data processed.