Summary #

Recently I have been looking into ways to test my Apache Beam pipelines at work. Most common use cases of Beam generally involves either batch reading data from GCS and writing to analytical platforms such as Big Query or stream reading data from Pubsub and writing to perhaps Bigtable.

A pipelines consists of transforms and its generally easy to test them in isolation as a independent unit test per stage, however I am personally a big fan of “end-to-end” testing or “Integration testing” and this is where things can sometimes get tricky.

A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors.

This article will attempt to explain the process or creating and using GHEAT-JAVA, which is a port of famous aspen based gheat and took great inspiration from Gheat.net

Writing a service which would serve heat map tiles is a bit tricky,there are three major components involved

2. The client part i.e. some kind of mapping library which has a concept of layer, I chose Google maps.
3. The data source part, i.e. a spatially aware database or an in memory data structure, I have used postgres ,an in memory quad tree and a flat file as data sources.
4. The renderer part or basically the code which excepts requests , parses tile bounds, fetches data and then renders gradients on the tile and later colorizes them.

The Tiling layer (Client part) #

Google maps allows developers to add a custom layer , the code looks like this