Metric Visualization with Node, InfluxDB, and Grafana

Metric Visualization

There’s a lot of buzz in the software development sphere about metric visualization. Consequently, lots of developer tools have been introduced to assist with implementing these sorts of products. In this article, I’ll walk through setting up a simple visualization client for stock market data using Node.js, InfluxDB, and Grafana panels. But first, I’ll do a brief overview and quick setup of a couple of these technologies. I’ll assume the reader has some familiarity with Node.js, however.

I’ll be running installation commands on an AWS EC2 Ubuntu 16.04 t2.micro instance.

InfluxDB

InfluxDB is a time series database written in Go by InfluxData. It’s perfect for exactly what it’s labeled to do: store time series data — that is, data that is closely related to the time at which it is produced. This could be performance metrics for server hardware, data produced by Internet of Things devices, weather data, and specifically relevant to this post, stock market data. InfluxDB is optimized for storage and retrieval of this type of data.

This data is fundamentally different from something you’d store into a general-purpose relational database such as MySQL. For example, you can gain some value from a single query from MySQL. Imagine a single query used to return all of the compact cars available during a certain time period from a set of rental cars. The return of this query is instantly useful; you can immediately produce a list of cars for the customer to choose from. In contrast, imagine a single query from a time series database: you select the barometric pressure at 10:00 AM on Tuesday, December 11, 2018. There’s nothing immediately interesting about this return; it’s just a pressure reading from a single instance in time.

Time series data becomes interesting when you have massive quantities of it, when you can efficiently grab this data and present it in a form that is easily parsed by the human brain. That’s why you want a specialized database for this category of data.

InfluxDB Installation

First you’ll need to add the InfluxData repository (source):

Once this completes, install InfluxDB and start the service:

If this was successful, you should be able to run influx to access the InfluxDB console. Once you do this, run CREATE DATABASE test followed by SHOW DATABASES and verify you see something similar to:

At this point, you’ve got an InfluxDB configured at the most basic level. As long as your server is accepting inbound traffic to port 8086, you should be able to hook up to it with Node and Grafana, which we’ll do later.

Grafana

Grafana labels itself as “the open platform for beautiful analytics and monitoring.” It offers a variety of different data representation options in the form of “panels.” 

The great thing about Grafana is how well it integrates with different data sources, including InfluxDB, but also ElasticSearch, Graphite, MySQL, and others. You just hook up to a database, configure some queries with a user interface, and the data flows. We’ll get to that shortly.

Grafana Installation

Use these commands for a stable install:

Then:

Then start the Grafana service:

At this point, if all went well, you should be able to visit your server at port 3000 (provided you’ve opened it to inbound access) and arrive at the Grafana login screen. The default login is U/N: admin, P/W: admin (you should definitely change this).

Login to your Grafana and click Add Data Source. Copy this config; the default U/N and P/W is also admin admin. Notice the database name, test. We created that earlier in the console. When you click Save and Test you should get a green box indicating that you’ve successfully connected Grafana to InfluxDB.

Now before we can view any data, we need to set up a simple Node application and start sending to our database.

Node App with Node-Influx

Create a new directory for your project, navigate to it, and intialize your Node project with npm init. Then install Express, Request, and Node-Influx with

For this application, I’ll be making GET requests to the IEX Trading API, which is a free API service. For simplicity in the example, I’ll just set up recurring GET requests with setInterval and have another recurring database-update function.

First, create a basic index.html file with an empty body. We’ll add a Grafana panel iframe to it later. Then create an app.js where we’ll have the route, database setup, and API requests.

Import Influx, Request, and Express into the project:

Then set up the Express app with the single route:

For API documentation, go here. It’s pretty straightforward, and here I’ll set up an object to store updating values as well as the function responsible for the GET requests:

This will pull the latest Microsoft stock data and set the data object’s price parameter to Microsoft’s latest stock price (obviously this won’t update for you if it’s after trading hours). So far, we aren’t putting anything into InfluxDB, but we’ve prepared the updating values for the database insertions.

Next we need to configure Node-Influx. You need to provide it with your hose, database name (test), username, and password. Additionally, it requires the schema for the data you plan on sending to Influx.

If you don’t know anything about InfluxDB, the schema’s object requires a bit of explanation. InfluxDB entries are composed of measurements, fields, tags, and timestamps (timestamps are omitted here – the default is the timestamp at the time of DB insertion). Measurement is analogous to a relational databases table name. Fields are the values for the metric you’re inserting, and tags are meta information for whatever you’re inputting. Here, I’m just tagging the source of the data. Under fields, I’m telling Node-Influx to expect float values for value.

At this point, our Node application is almost done! We just need to write a function that inserts data into the database.

I’m telling Node-Influx to take my database object (we set this up above with all the configuration details) and write Points to it. A point in InfluxDB consists of a single collection of fields in a series. Each point must have a unique timestamp. If you write multiple points in a series with the same timestamp, it will not write multiple points. Again, in my example, I omit the timestamp field, resulting in a default value of the current time.

Here I’m basically saying: “write a point to price_msft, the source is iextrading, and the value is the current value stored in data.price, and by the way, this is in the test database. I want precision by seconds.” 

Finally, we want the requests and the writes to keep occurring while our Node application is running:

Here I have a five second offset to allow the request to update the value; this is just for the sake of the example.

That’s it! When we run app.js, this will begin requesting from the API, updating the value, and writing that value to our Influx database. If you’re still consoled into your InfluxDB, you can run

to see the entries as they come in. Now we’re ready to configure a Grafana panel to view the data.

Grafana Panel Configuration

Head back over to your Grafana UI and on the left, go to Manage Dashboards. Click New Dashboard. For this example, I’ll choose the graph panel variety.

You’ll be presented with this. Click where it says panel title and click Edit. Select Influx next to Data Source (this will only be an option if you configured your data source properly earlier). Then, next to FROM, ensure it matches my settings below. This is essentially setting up a query in the proper format; Grafana will perform the query for you once you configure it here.

Once you do this, you should see your panel above populating with data. In this case, I’m running my app after trading hours, so the value isn’t changing. I might update this image later during trading hours to have a better graph. Here’s what I’ve got at the moment:

Great! Now all we need to do is import the panel into our index.html file. At the top of your Grafana UI, there’s a little square icon with an arrow coming out of it – share dashboard – click that and deselect Template variables. This will give you an updated link to your panel; copy this and place it in an iframe tag inside your index.html:

You can play around with the settings I’ve got here depending on your graph size, but this will output your Grafana panel in the browser.

Conclusion

At this point, we’re all done. You can set up multiple panels in a single Grafana dashboard and dynamically output the panels as desired in your application. Grafana also has more advanced features, like variable configuration: with this, a viewer can select via some selection medium different options that you configure. In this example, you could add queries to other stocks and display them in a single panel simultaneously or separately. You can also experiment with the other data representation options.

Thanks for reading! Here’s the complete app.js in one place: