cLoki

like Loki, but for Clickhouse.

cLoki is a clear room design Loki API emulator made with NodeJS, Fastify and Clickhouse
APIs are compatible with Grafana, LogQL and paStash for logs querying, processing and ingestion

Performance is comparable to native Loki, with cLoki outperforming on large range filtered queries.

💡 Get started using the cLoki Wiki

🔥 Beta Stage, Contributors and Testers are Welcome! :octocat:

Project Background

The Loki API and its Grafana native integration are brilliant, simple and appealing - but we just love Clickhouse.

cLoki implements the same API functionality as Loki, buffered by a fast bulking LRU sitting on top of Clickhouse tables and relying on its columnar search and insert performance alongside solid distribution and clustering capabilities for stored data. Just like Loki, cLoki does not parse or index incoming logs, but rather groups log streams using the same label system as Prometheus.

🔥 LogQL: Supported Features

cLoki implements a broad range of LogQL Queries to provide transparent compatibility with the Loki API
The Grafana Loki datasource can be used to natively query logs and display extracted timeseries

🎉 No plugins needed

• Log Stream Selector
• Line Filter Expression
• Label Filter Expression
• Parser Expression
• Log Range Aggregations
• Aggregation operators
• Unwrap Expression.
• Line Format Expression

---

⛽ Log Streams

cLoki supports input via Push API using JSON or Protobuf and it is compatible with Promtail and any other Loki compatible agent such as Telegraf, Fluentbit, Logstash and others.

Our preferred companion for parsing and shipping log streams to cLoki is paStash with extensive interpolation capabilities to create tags and trim any log fat. Sending JSON formatted logs is suggested when dealing with metrics.

---

🔥 CliQL: Experimental 2.0 Features

cLoki implements custom query functions for clickhouse timeseries extraction, allowing direct access to any existing table

Timeseries

Convert columns to tagged timeseries using the emulated loki 2.0 query format

<aggr-op> by (<labels,>) (<function>(<metric>[range_in_seconds])) from <database>.<table> where <optional condition>

Examples

avg by (source_ip) (rate(mos[60])) from my_database.my_table

sum by (ruri_user, from_user) (rate(duration[300])) from my_database.my_table where duration > 10

Clickhouse

Convert columns to tagged timeseries using the experimental clickhouse function

Example

clickhouse({ 
  db="my_database", 
  table="my_table", 
  tag="source_ip", 
  metric="avg(mos)", 
  where="mos > 0", 
  interval="60" 
})

Query Options

parameter	description
db	clickhouse database name
table	clickhouse table name
tag	column(s) for tags, comma separated
metric	function for metric values
where	where condition (optional)
interval	interval in seconds (optional)
timefield	time/date field name (optional)

---

Setup

🚏 GIT (Manual)

Clone this repository, install with npmand run using nodejs 12.x (or higher)

npm install
CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="cloki" node ./cloki.js

🚏 NPM

Install cloki as global package on your system using npm

sudo npm install -g cloki
cd $(dirname $(readlink -f `which cloki`)) \
  && CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="cloki" cloki

🚏 PM2

sudo npm install -g cloki pm2
cd $(dirname $(readlink -f `which cloki`)) \
  && CLICKHOUSE_SERVER="my.clickhouse.server" CLICKHOUSE_AUTH="default:password" CLICKHOUSE_DB="cloki" pm2 start cloki
pm2 save
pm2 startup

🚏 Docker

For a fully working demo, check the docker-compose example

---

Configuration

The following ENV Variables can be used to control cLoki parameters and backend settings.

ENV	Default	Usage
CLICKHOUSE_SERVER	localhost	Clickhouse Server address
CLICKHOUSE_PORT	8123	Clickhouse Server port
CLICKHOUSE_DB	cloki	Clickhouse Database Name
CLICKHOUSE_AUTH	default:	Clickhouse Authentication (user:password)
CLICKHOUSE_PROTO	http	Clickhouse Protocol (http, https)
CLICKHOUSE_TIMEFIELD	record_datetime	Clickhouse DateTime column for native queries
BULK_MAXAGE	2000	Max Age for Bulk Inserts
BULK_MAXSIZE	5000	Max Size for Bulk Inserts
BULK_MAXCACHE	50000	Max Labels in Memory Cache
LABELS_DAYS	7	Max Days before Label rotation
SAMPLES_DAYS	7	Max Days before Timeseries rotation
HOST	0.0.0.0	cLOKi API IP
PORT	3100	cLOKi API PORT
CLOKI_LOGIN	false	Basic HTTP Username
CLOKI_PASSWORD	false	Basic HTTP Password
READONLY	false	Readonly Mode, no DB Init
DEBUG	false	Debug Mode

---

Project Status

API

Loki API Functions are loosely implemented as documented by the Loki API reference.

• /loki/api/v1/push
• /loki/api/v1/query
• /loki/api/v1/query_range
• /loki/api/v1/label
• /loki/api/v1/label/name/values
• /loki/api/v1/tail

Status

Consult the Wiki for a detailed list of supported features

---

Acknowledgements

cLoki is not affiliated or endorsed by Grafana Labs. All rights belong to their respective owners.