KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #codex #ai-agents #claude-code #dsh #dsh-plugin #cli #macos #ai-agent #deepseek-harness #open-source

serviceq

★ 77 Open GitHub ↗

Super fault-tolerant HTTP load balancer & queue. White paper for reference - https://github.com/gptankit/serviceq-paper

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ServiceQ [![Build Status](https://travis-ci.com/gptankit/serviceq.svg?branch=master)](https://travis-ci.com/gptankit/serviceq) [![GoDoc](https://godoc.org/github.com/gptankit/serviceq?status.svg)](https://pkg.go.dev/github.com/gptankit/serviceq?tab=subdirectories)

ServiceQ is a fault-tolerant gateway for HTTP clusters. It employs probabilistic routing to distribute load during partial cluster shutdown (k/n nodes experiencing downtimes, timeouts, connection loss etc) and queues requests during total cluster shutdown (n nodes down). The queued requests are forwarded in FIFO order when the cluster is available next.

Below graph shows the routing probability (P) on a down node (D) in a 8-node cluster with respect to number of requests (r). Notice how quickly the routing probability on D reduces as the requests on D start to fail. Depending on the rate of request, it will only take a few seconds (sometime even milliseconds) to move all requests away from D, thus ensuring more requests are routed to healthier nodes.

Note that, even when requests keep failing on D (however less), ServiceQ retries them on other nodes until they succeed. If they do not succeed on any of the nodes, they are queued and periodically retried on the cluster (using the same approach above), until they succeed.

Noticeable features

* HTTP Load Balancing 
* Probabilistic node selection based on error feedback 
* Failed request queueing and deferred forwarding 
* Upfront request queueing 
* Request retries 
* Concurrent connections limit 
* Complete TLS/SSL support (automatic and manual)

Here are the steps to run ServiceQ -

Download

Clone the project into any directory in your workspace

<pre>
$ git clone https://github.com/gptankit/serviceq
</pre>

Change into directory serviceq

How to Build

<pre>$ make ('make build' will also work)</pre>

Optional: make with debug symbols removed (~25% size reduction)

<pre>$ make build-nodbg</pre>

This will create a Go binary serviceq in the current directory

How to Install

Make sure the current user has root privileges, then -

<pre>$ make install</pre>

This will create a folder serviceq in /usr/local directory and copy the serviceq binary (generated in the build step) to /usr/local/serviceq and sq.properties (serviceq configuration file) to /usr/local/serviceq/config.

How to Run

Before running, make sure the mandatory configurations in /usr/local/serviceq/config/sq.properties are set (LISTENER_PORT, PROTO, ENDPOINTS, CONCURRENCY_PEAK). The configuration file closely resembles a typical INI file so its fairly easy to understand and make changes -

<pre>
#sq.properties

#Port on which serviceq listens on
LISTENER_PORT=5252

#Protocol the endpoints listens on -- 'http' for both http/https
PROTO=http

#Endpoints seperated by comma (,) -- no spaces allowed, can be a combination of http/https
ENDPOINTS=http://my.server1.com:8080,http://my.server2.com:8080,http://my.server3.com:8080

#Concurrency peak defines how many max concurrent connections are allowed to the cluster
CONCURRENCY_PEAK=2048
</pre>

Also, verify timeout value (default is set to 5s). Low value is preferable as it allows retries to be faster -

<pre>
#Timeout (s) is added to each outgoing request to endpoints, the existing timeouts are overriden, value of -1 means no timeout
OUTGOING_REQUEST_TIMEOUT=5
</pre>

By default deferred queue is enabled with all methods and routes allowed. These options can be controlled as -

<pre>
#Enable deferred queue for requests on final failures (cluster down)
ENABLE_DEFERRED_Q=true

#Request format allows given method/route on deferred queue -- picked up if ENABLE_DEFERRED_Q is true
#Q_REQUEST_FORMATS=POST /orders,PUT,PATCH,DELETE
#Q_REQUEST_FORMATS=ALL
Q_REQUEST_FORMATS=POST,PUT,PATCH,DELETE
</pre>

(Note that Q_REQUEST_FORMATS is also considered if ENABLE_UPFRONT_Q is true)

After all is set -

<pre>$ sudo /usr/local/serviceq/serviceq</pre>

Refer wiki for more details: https://github.com/gptankit/serviceq/wiki

Feel free to play around and post feedbacks

serviceq

Was this content helpful?

Recent Feedback

Download README