Satori Docs

Your definitive guide to subscribing and publishing live data

Composer Direct link

 Composer project is experimental

Overview Direct link

Composer is a ready-to-go (configure and run) project that can digest different data sources and republish them into RTM, among other things.

The project comes with a collection of ready-to-use modules (mods) and a runtime for them (including orchestration). Available mods include http poller,  dedup filter, json converters from several formats, publisher to rtm and more; see Mods Suite for a full list of mods with their specifications.

The project is written in Java but no Java skills required; typical use cases could be handled with available mods without coding at: by composing a json config file. It does require building and running it on your machine (or deploying somewhere). It is gradle based Java project, IntelliJ ready (or use another IDE of your choice).

For customizations, if one needs additional implementations, the project is designed extremely modular, in a plug-in fashion, so that extending functionality is a matter of implementing a well defined mod interface.  Any language programmer should feel pretty comfortable with it, as it’s a matter of implementing an interface scoped to a particular user task and following a few contracts for functional orchestration of the mods. All the expert Java plumbing is handled and encapsulated by the composer internally.


Life of a mod

Every mod implements IMod interface. Composer invokes startup and shutdown related functions providing a place for a mod to set up and clean up; onPulse for any continual updates (every second by default); onInput delivers data from other mods, according to the configured connectors. A mod outputs the data (to be passed to other connected mods) by invoking a yield(..) function inside any mod code. JsonNode object used as an input for a yield can encapsulate arbitrary binary data, not just json.

public interface IMod {

 default void init(IModContext context) throws Exception {
 }

 default void dispose() throws Exception {
 }

 default void onStart() throws Exception {
 }

 default void onStop() throws Exception {
 }

 default void onPulse() {
 }

 default void onStats(StatsCycle cycle, IStatsCollector collector) {
 }

 void onInput(String inputName, JsonNode data, IAsyncHandler cont) throws Exception;
}

Examples Direct link

Prerequisite

Recommended:


Verifying examples end-to-end

After running the examples, to verify functionality end-to-end, subscribe to the same channel and observe incoming ticks, published by our example. An easy way to check subscription data or publish test messages is from the Dev Portal Console.

Clock example

Illustrates user generated data (abstracted by a clock, custom mod) being published to RTM

 

In this example, we are adding our own custom mod ClockMod, which delivers a timestamp to its output, on every tick. In addition, we are including a prepackaged rtm publishing mod (RtmPublishMod) from the suite, connecting publisher to the clock. As a result, the clock’s ticks are being published to our rtm channel.  

Mods pipeline for the clock example

Zq59T5fddCRhkCBEW_b4QSP1AcgFpKmhd8p60sjHQliaf_yjGWPe7yqwUlK8LTcWOJKAUS7kLnKiJTsR_a0l_6H0fJkvpxr12n5hxwKDUeIDbTzARSKgGSOo8J3ZKp6FTLveXxn3


Implementation outline
  • Implement ClockMod class extending Mod class
  • Add main entry ClockApp class

  • Add config.json with ClockMod and RtmPublisherMod

  • Build and run; verify ticks are streaming in the channel


ClockMod class

The only thing that ClockMod need to do is to output timestamp and correctly schedule the next tick  (repeating this over and over). ClockMod reports its output (current  msec value)  through the  yield(..) function.


ClockMod.java 

package com.satori.mods.examples;

import com.satori.mods.api.*;
import com.satori.mods.core.async.*;
import com.satori.mods.core.config.*;
import com.satori.mods.suite.*;

import com.fasterxml.jackson.databind.*;
import org.slf4j.*;

public class ClockMod extends Mod {
 public final static Logger log = LoggerFactory.getLogger(ClockMod.class);
 public final static long defaultTick = 3_000;

 private long tick = defaultTick;

 public ClockMod() {
   this(defaultTick);
 }

 public ClockMod(JsonNode settings) throws Exception {
   this(settings != null ? Config.parseAndValidate(settings, ClockModSettings.class) : null);
 }

 public ClockMod(ClockModSettings settings) {
   if (settings != null) {
     tick = settings.tick;
   }
   log.info("created with tick {}", tick);
 }

 public ClockMod(long tick) {
   this.tick = tick;
 }


 @Override
 public void init(IModContext context) throws Exception {
   super.init(context);
   // Post loop to the queue instead of making a direct call.
   // This ensures that loop() will be executed when all startup is done
   // and all dependent mods are ready
   exec(this::loop);
 }

 private void loop() {
   try {
     long msec = System.currentTimeMillis();
     log.info("yield tick {} (next in {} msec)", msec, tick);
     //report output data
     yield(msec, AsyncPromise.from(
       () -> {
         // successful completion
         timer(tick, this::loop);
       },
       cause -> {
         // error completion
         log.error("failure", cause);
       }
     ));
   } catch (Exception e) {
     log.error("failure", e);
   }
 }
}

yield and promise

Our timer(tick, this::loop) call is wrapped inside the promise parameter to the  yield function. This continuation travels down to the connected mods (publisher mod in our case), together with the output data. RtmPublishMod will signal it when it is done processing the data, either with success or failure. Hence, the next tick is scheduled only after the publisher node is done processing data from our previous tick, and only if publish of it has succeeded (in case of a failure, we log it and do not set off a timer). In other works, actual tick period includes timer delay setting and the publishing time.

Setting a timer for next tick (loop) could have been scheduled right inside the loop (outside of the yield), but this implementation is preferred. In real word, ticks are actual heavy operations (consider http polling retrieving bulky data in place of a tick), and continuing in this manner allows to abort the needless effort since no one is processing our output anyways (is failing).

Designing mods to progress in accord with promise completion,  enables a “back-pressure” mechanism within the mod pipeline: so that output generating mod does not overwhelm a mod ahead, in case the latter is not catching up processing all the incoming to it data.


Main class

Add a main java class as shown below: main entry just needs to launch composer runtime passing it a config. Composer will parse this config and orchestrate the mods accordingly.

ClockApp.java (Main entry)

package com.satori.mods.examples;

import com.satori.composer.runtime.*;
import com.satori.mods.resources.*;

import java.io.*;
import java.util.logging.*;

public class ClockApp {

 // entry point

 public static void main(String... args) throws Exception {
   ComposerRuntime.prepare();
   try (final InputStream is = ModResourceLoader.loadAsStream("log.properties")) {
     LogManager.getLogManager().readConfiguration(is);
   }
   ComposerRuntimeConfig config = ComposerRuntime.loadConfig("config.json");
   config.validate();
   ComposerRuntime.start(config);
 }
}


config.json

Add config.json file into application’s res folder. Config specifies two mods: our ClockMod (without any settings, just the class to instantiate) and rtm publisher mod from the prepackaged suite.

Replace RTM connection credentials for your satori project from Dev Portal.  Ensure that the connectors field of the publisher designates the node holding ClockMod: “clock” in our config.

res > config.json

{
  "mods": {
    "clock": {
      "type":"com.satori.mods.examples.ClockMod",
    },
    "publisher": {
      "type":"com.satori.mods.suite.RtmPublishMod",
      "connectors": "clock",
      "settings": {
        "channel" : "YOUR_CHANNEL",
        "host": "YOUR_HOST", //just host, no wss scheme
        "ssl":true,
        "args": {"appkey": "YOUR_APPKEY"},
        //"auth": {"role": "YOUR_ROLE","secret": "YOUR_SECRET"}
      }
    }
  }
}


Run and verify

This completes the example: you should be able to build and run it. You should see log indicating clock ticks:

...
com.satori.mods.examples.ClockMod loop - clock sends tick 1493600820301 (next in 2000 msec) 
com.satori.mods.examples.ClockMod loop - clock sends tick 1493600822305 (next in 2000 msec) 
com.satori.mods.examples.ClockMod loop - clock sends tick 1493600824309 (next in 2000 msec)
...


ClockModSettings (optional)

The full clock example on the github implements ClockModSettings, in addition to the code shown in this example walkthrough. This allows to parametrize our clock mod in config.json rather than hardcoding tick delay in code.

ClockModSettings.java

package com.satori.mods.examples;

import com.satori.mods.core.config.*;

import com.fasterxml.jackson.annotation.*;


@JsonInclude(JsonInclude.Include.NON_DEFAULT)
public class ClockModSettings extends Config {

 @JsonProperty("tick")
 public long tick = 3_000; // in msec

 @Override
 public void validate() throws InvalidConfigException {
   if (tick <= 0) {
     throw new InvalidConfigException("invalid delay");
   }
 }

 public ClockModSettings() {
 }
}

config.json with clock settings

"clock": {
      "type":"com.satori.mods.examples.ClockMod",
      "settings": {
        "tick":2000 //2 sec, in msec
      }
    },





NWS USA Alerts

This example illustrates how to compose republishing a feed provided over http in xml format, such as National Weather Service (NWS) alerts. Check out the data provided in the original source and the resulting channel on Satori to get a feel for it.

Since this feed is provided over http the first thing is to set up a module polling over http, i.e. continually downloading alert updates from the feed. The next part is to have polled data converted from xml to json format. We also want to filter our duplicate alerts, potentially re-polled in a previous request, before publishing them to rtm.

Republishing pipeline for HTTP XML feed looks like this:

feed.png#asset:5016:url


Implementations of the four modules in the above pipeline are already available in the Composer: all there is to it, is to get these modules instantiated and connected; this is done in the json configuration file shown below.

Below config.json lists the four mods to be ran. Each json node in the config specifies a class to instantiate a mod instance from (type field): for instance, http poller is implemented by HttpPollMod class, so there is a json node with 

"type":"com.satori.mods.library.HttpPollMod".


config.json (in Composer modes-examples > nws-usa-alerts > res > com.satori.mods.resources). 

 {
 "stats": {
   "period": 1000, // in ms., 1 sec.
   "console": {
     "period": 10000
   }
 },
 "mods": {
   "poller": {
     "type": "com.satori.mods.suite.HttpPollMod",
     "settings": {
       "delay": 1000, // in ms., 1 sec.
       "format": "binary",
       "host": "alerts.weather.gov",
       "path": "/cap/us.atom",
       "ssl": true
     }
   },
   "converter": {
     "type": "com.satori.mods.suite.XsltMod",
     "connectors": "poller",
     "settings": {
       "xslt": "transform.xslt"
     }
   },
   "filter": {
     "type": "com.satori.mods.suite.DedupMod",
     "connectors": "converter",
     "settings": {
       "expiration-interval": 60000,
       "key-selector": "/id"
     }
   },
   "publisher": {
     "type": "com.satori.mods.suite.RtmPublishMod",
     "connectors": "filter",
     "settings": {
       "channel": "nws-usa-alerts",
       "host": "he3rv0pp.api.satori.com", //just host, no wss scheme
       "ssl": true,
       "args": {"appkey": "dFDa4104a3913EE4DbdDedd257ea54F1"},
       "auth": {"role": "demo", "secret": "dBdaE8021926Ac602EdD4c184e25d09E"}
     }
   }
 }

Node connectors field is what connects the output from one mod to the input of another one. The “poller” mod doesn’t have any connectors because it doesn’t take any input from other mods: it retrieves data from http polling, which is internal to its functionality. It does generate output used by the “converter” mod. Hence the “converter” mod has

"connectors": "poller".

Node names (such as “poller” in our config) are user-defined strings but should match precisely as references in the connectors. This is how composer knows to pass output from which mod to which.

Nested settings object is specific to each mod type.  For instance, http polling mod needs to know where to poll the data from, so it has host and path fields, -- and how often to poll, so it has delay fields. format: “binary” instructs http polling mod to pass retrieved data as a sequence of bytes, without any parsing. Converter from xml to json, XsltMod, takes xml input as binary data.

Other nodes have settings fields relevant to their particular function, as another example, RTM publishing mod has fields for specifying where to connect to , authorization credentials, and the channel to publish the data to. In case there is doubts, settings for each mod are documented in Mods Suite.

The config.json file is parsed on launch (in main); runtime instantiates specified mod instances and runs them, passing output from a mod to the connected mods according to the config.

To try this example in action, check out composer from the github,  modify  mods-examples > nws-usa-alerts config.json with your RTM credentials for publishing, - obtained in your (free) Dev Portal project, -, and run it (with command line or from IntelliJ IDE).

More details about composer, including additional examples, building & running instructions are in the Composer section.


Big-blue-bus example (GTFS)

Illustrates processing two feeds in protobuf format (bus positions and trip updates in GTFS-realtime) and republishing them as json to a single RTM channel.


GTFS feeds

GTFS (General Transit Feed Specification) standard was developed by Google and is by now widely adopted by public transportation agencies. Realtime GTFS data exchange is in Protocol Buffer (protobuf) format. Realtime GTFS is used for communicating transport vehicle locations and their trip updates: cancelled trips, running ahead or behind the schedule, etc.

This example processes transit updates from two protobuf feeds: vehicle positions and trip, -- those are often provided in separate feeds.

incTUJdrJOn5DSE-fxy-_dcd84BiJ-Xa0JodhLA-0uIq2mNF2qNf4uul9JBKVt4g16di6Ff_4vQ_7jvMzLSnNH4Ie2KSHqmpbQghhnm0bydutyOUv0wEHXApDJUqUjNtaySfUXvQ

Big-blue-bus > res > config.json

{
 "stats": {
   "period": 1000, // in ms., 1 sec.
   "console": {
     "period": 10000
   }
 },
 "mods": {
   "poller-vehicle-positions": {
     "type": "com.satori.mods.suite.HttpPollMod",
     "settings": {
       "delay": 1000, // in ms., 1 sec.
       "format": "binary",
       "host": "gtfs.bigbluebus.com",
       "ssl": true,
       "verify-host": false,
       "path": "/vehiclepositions.bin"
     }
   },
   "poller-trip-updates": {
     "type": "com.satori.mods.suite.HttpPollMod",
     "settings": {
       "delay": 1000, // in ms., 1 sec.
       "format": "binary",
       "host": "gtfs.bigbluebus.com",
       "ssl": true,
       "verify-host": false,
       "path": "/tripupdates.bin"
     }
   },
   "converter": {
     "type": "com.satori.mods.suite.GtfsProtoBufToJsonMod",
     "connectors": ["poller-vehicle-positions", "poller-trip-updates"],
     "settings": {
       "user-data": "auckland"
     }
   },
   "dedup-filter": {
     "type": "com.satori.mods.suite.DedupMod",
     "connectors": "converter",
     "settings": {
       "expiration-interval": 600000,
       "override": false,
       "key-selector": "/entity"
     }
   },
   "publisher": {
     "type": "com.satori.mods.suite.RtmPublishMod",
     "connectors": "dedup-filter",
     "settings": {
       "channel": "YOUR_CHANNEL",
       "host": "YOU_HOST", //no wss scheme, just host: x.satori.com
       "ssl": true,
       "args": {"appkey": "YOU_APPKEY"} //,
       // enable if your publish permission needs auth
       //"auth": {"role": "YOUR_ROLE","secret": "YOUR_SECRET"}
     }
   }
 }
}

Converter mod in this example is connected to (taking input from) two mods: poller-vehicle-positions and poller-trip-updates and is processing binary protobuf format. The rest of the pipeline is the same as in nws-usa-alerts example.


Run and verify

Working example log:

com.satori.mods.suite.HttpPollMod processResult - idle poll 
com.satori.mods.suite.HttpPollMod doPolling - polling... 
com.satori.mods.suite.HttpPollMod sendRequest - GET /tripupdates.bin 
com.satori.mods.suite.HttpPollMod processResponseBody - 304, Not Modified (/tripupdates.bin) 
com.satori.mods.suite.HttpPollMod processResult - idle poll 
com.satori.mods.suite.HttpPollMod doPolling - polling... 
com.satori.mods.suite.HttpPollMod sendRequest - GET /vehiclepositions.bin 
com.satori.mods.suite.HttpPollMod processResponseBody - 200, OK





Mods Suite Direct link

  • com.satori.mods.suite package


Composer comes prepackaged with a suite of ready-to-use modules (mods). This document covers available mods and their specifications.

Mods implementations can be found in com.satori.mods.suite package. Each mod has its principal implementation class implementing IMod interface, and accompanying Settings and Stats classes.

Settings class maps to the settings field specified in the json config files and is passed to a mod constructor. Settings classes may reuse common base implementations.

To have a functional connectivity between the mods (passing output from one as input to another mod), the input and output format of the connected mods must be compatible.


Mods

  • HttpPollMod
  • HttpPostMod
  • DedupMod
  • XsltMod
  • GtfsProtoBufToJsonMod
  • RtmPublishMod
  • RtmSubscribeMod
  • WsSubscribeMod


HttpPollMod
  • Continuously polls data over HTTP

Description

Http polling mod schedules and processes responses to GET requests. It uses conditional requests to avoid duplicate responses with unchanged data (such as using If-Modified-Since http header). Properly set up data provider servers would support at least some of the conditional headers, allowing to prevent data transfer and processing of meaningless redundant responses. Note that modified content does not mean there will be no duplicate updates in the the response, just that that there was some change in the content.

Spec

Input -

Output binary | json | text

HttpPollModSettings

field

type

description

required

default

host

string

host

yes

path

string

request-uri
https://tools.ietf.org/html/rfc2616#section-5.1.2

no

"/"

port

int

port

no

ssl?443:80

ssl

bool

whether SSL/TLS is enabled

no

false

verify-host

boolean

whether the hostname verification (for SSL/TLS) is enabled

no

true

trust-all

boolean

set whether all server certificates should be trusted

no

true

args

map of strings

set of query arguments to append to request uri

no

{}

headers

map of strings

set of headers to add to http UPGRADE request

no

{}

delay

long

delay between subsequent requests in ms.

no

1000

format

string

content and output format

"binary"

expects binary content and sends it as binary node to output

"json"  

expects text utf-8 encoded content with json payload parse it and sends parsed json tree to output

"text"

expects text in utf-8 encoded content and sends it as text node to output

no

"json"

connect-timeout

long

the connect timeout in ms.

no

60,000

idle-timeout

the idle timeout in sec.

no

600

compression

boolean

is compression enabled on client side

no

false

error-delay

long

delay between subsequent requests in ms. if first one was unsuccessful

no

delay

disable-etag

boolean

ignore etag header in response

no

false

disable-last-modified

boolean

ignore last-modified header in response

no

false



HttpPollModSettings example

   "poller":{
      "type":"com.satori.mods.suite.HttpPollMod",
      "settings":{
        "delay": 1000, // in ms., 1 sec.
        "format":"binary",
        "host": "alerts.weather.gov",
        "path":"/cap/us.atom",
        "ssl":true
      }



HttpPostMod
  • Performs http post

HttpPostModSettings

Extends HttpRequestConfig

field

type

description

required

default

connect-timeout

int

the connect timeout in ms.

no

60_000

compression

boolean

is compression enabled on client side

no

false

max-pool-size

int

maximum pool size for connections

no

100

max-wait-queue-size

int

maximum requests allowed in the wait queue, any requests beyond the max size will fail.

no

-1 (unbound)

method

string

the http method for the request.

no

"POST"

template

any

no

null




DedupMod
  • Filters out duplicate messages providing pure output stream, without repeating the same message.

Description

Pull based feeds rarely implement versioning mechanisms allowing a requester to get only incremental updates (only new data after the last request). As a result, continual pulling of such feed would retrieve the same messages over and over.

Consider a feed providing location updates where an entity’s location is changed after 1 min. If ones polls this feed every 10 sec, the same location message is re-retrieved 6 times. Streaming data should provide only new updates: this is important both for efficiency and business logic processing on the receiving end.

On receiving a message to its input, dedup mod checks if this message has been already seen (present in the cache) and if so, the mod “swallows” it, meaning the mod does not pass repetitive message to the output.  

Specs

Input: json

Output: json

DedupModSettings

field

type

description

required

default

key-selector

string

Relative json path, which specifies which field inside the message determines the identity of the message.

If omitted, the whole message is compared.

Deep json node comparison is used to deduce if the messages are identical.

no

"/"

expiration-interval

Time interval in ms. after which entry will be forgotten.

no

60_000

override

If true, expiration timer is restarted on every hit.

If false, the entry is forgotten once expiration interval elapsed after it was first observed.

Use false if you need to repeat duplicated messages over some period of time.



Config example

   "filter": {
      "type":"com.satori.mods.suite.DedupMod",
      "connectors": "converter",
      "settings": {
        "expiration-interval": 60000,
        "key-selector":"/id"
      }
   }



XsltMod
  • Converts xml formatted data to json using XSLT transformations

input: binary xml

output: json

XsltModSettings

field

type

description

required

default

xslt

string

Path to xslt transformation

yes



GtfsProtoBufToJsonMod
  • Coverts protobuf formatted data to json

Specs

Input binary protobuf

Output json

GtfsProtoBufToJsonModSettings

field

type

description

required

default

user-data

string

Custom node to inject into output gtfs header

no

null



RtmPublishMod
  • Publishes to RTM

input json

output -

RtmPublishModSettings

Extends RtmBaseConfig

field

type

description

required

default

channel

string

Channel name to publish to

yes

RtmBaseConfig

Extends HttpRequestConfig

field

type

description

required

default

reconnect-delay

long

Relaxation time interval in ms. between unsuccessful attempts to connect  

no

1000

max-frame-size

int

Maximum websocket frame size in bytes

no

10 * 1024 * 1024`

credentials

RtmCredentialsConfig

If present, authorization is performed, with specified  role and secret.

If omitted, no authorization is performed.

no

connect-timeout

long

The connect timeout in ms.

no

10_000

RtmCredentialsConfig

field

type

description

required

default

role

string

Role in Satori  Dev Portal

yes

secret

string

Secret in Satori  Dev Portal

yes


RtmSubscribeMod

RtmSubscribeModSettings

Extends RtmBaseConfig

field

type

description

Required (default)

channel

string

Rtm channel name

yes


WsSubscribeMod

WsSubscribeModSettings

Extends HttpRequestConfig

field

type

description

required

default

connect-timeout

string

Rtm channel name

no

60_000

compression

boolean

Enables websocket compression

no

false

idle-timeout

int

In sec., automatically reconnect if during this interval no data was received

no

600

error-delay

long

Relaxation time interval in ms. between unsuccessful attempts to connect  in msec

no

1000

format

string

"binary"

expects binary frames and sends them as binary nodes to output

"json"  

expects text utf-8 encoded frames with json payload; parses them and sends the parsed json tree to output

"text"

expects text in utf-8 encoded frames and sends them as text nodes to output

no

"json"

HttpConnectionConfig

Extends HttpConnectionConfig

field

type

description

required

default

path

string

request-uri
https://tools.ietf.org/html/rfc2616#section-5.1.2

no

"/"

args

map of strings

Set of query arguments to append to request uri

no

{}

headers

map of strings

Set of headers to add to http UPGRADE request

no

{}



HttpConnectionConfig

field

type

description

Required

default

host

string

host

yes

port

int

port

no

ssl?443:80

ssl

boolean

whether SSL/TLS is enabled

no

false

verify-host

boolean

Whether the hostname verification (for SSL/TLS) is enabled

no

true

trust-all

boolean

Whether all server certificates should be trusted

no

true

Composition

CompositionSettings

field

type

description

required

default

outputs

string[]

List of output connectors

no

[]

mods

string -> CompositionNodeConfig

Children mod nodes

no

{}

CompositionNodeConfig

field

type

description

required

default

connectors

string -> string[]

Inputs connectors

no

{}

type

string

Mod

yes

setting

any

Mod settings

no

null

Building and Running Direct link

Common Setup

Download or check out composer source code from github

git clone https://github.com/satori-com/satori-composer.git composer

We will use nsw-usa-alerts example included in the composer project to show building and running. This example processes US weather alerts feed by polling http and publishes new updates to an RTM channel. The only source modification you will need is to update RTM credentials in the config file.


Common build & run outline

  1. Locate/open the project
  2. If using IDE, may need to set matching Java version
  3. Update config.json with your RTM credentials
  4. Locate the main class
  5. Build and run


Update config.json with RTM credentials

Locate config.json file under resources:

mods-examples > nsw-usa-alerts > res > config.json
cat mods-examples/nws-usa-alerts/res/com/satori/mods/resources/config.json


Edit config.json  “publisher” node to have RTM credentials for your project (obtained in Dev Portal). If you don't need to authenticate for publishing, just comment out “auth” setting. Publish channel permissions depend on your project setup in the Dev Portal. By default, projects are created with a default role allowed to publish, -- no authentication.

config.json

...
​"publisher": {
    "type":"com.satori.mods.suite.RtmPublishMod",
     "connectors": "filter",
     "settings": {
       "channel" : "YOU_CHANNEL",
       "host": "YOUR_HOST", "ssl":true,
       "args": {"appkey": "YOUR_APPKEY"} //,
       //"auth": {"role": "YOUR_ROLE","secret": "YOUR_SECRET"}
}


Locate the main class

The main entry class (the class with the main function) of the application is named after the example, NwsUsaAleters.java.

nsw-usa-alerts > src > com.satori.mods.examples > NwsUsaAleters.java 
cat nws-usa-alerts/src/com/satori/mods/examples/NwsUsaAlerts.java


You don’t need need to do anything with the main class, besides knowing where it is to know what to run. The main function has a handful of code lines for bootstrapping config and launching composer, -- you can copy & paste it to your later projects.

This is the only java file in this application because the example uses all prepackaged mods in the composer.


Build & run

Use can use command line or if using IntelliJ: right click the main class in project panel and click run (see Command line or IntelliJ section if you need more details)


Verify running example

The project outputs to standard output; meaning, you will see log output in IntelliJ Console or terminal (whichever way you are running it)

On successful run, you should see log output indicating http is polling and, when there is new data (200), publishing to rtm.

com.satori.mods.suite.HttpPollMod processResponseBody - 304, Not Modified (/cap/us.atom)
com.satori.mods.suite.HttpPollMod processResult - idle poll
com.satori.mods.suite.HttpPollMod doPolling - polling...
com.satori.mods.suite.HttpPollMod sendRequest - GET /cap/us.atom
com.satori.mods.suite.HttpPollMod processResponseBody - 200, OK (/cap/us.atom)
com.satori.mods.suite.RtmPublishMod publish - paused...
com.satori.mods.suite.RtmPublishMod lambda$publish$57 - resumed...
com.satori.mods.suite.RtmPublishMod publish - paused...

Subscribe to your channel with another client, such as the Web Console widget in the Dev Portal or RTM CLI, to see incoming data, republished by this running instance.


Command line instructions

There is several options. See readme on github for the example for different command line instructions.

With gradle:

$ gradle installDist
$ gradle run



IntelliJ instructions

Prerequisites

  • Installed IntelliJ 2017 , including Java 8 setup

Open the project

From IntelliJ, Open composer source code folder. Choose Java version 1.8 for Gradle JVM on a pop-up window.

After/during opening,  IntelliJ should ask to “Import Gradle project”, confirm to import it.

On successful import, the left pane of the IDE lists project structure and is ready to run.  If you do not see the project structure shown below, clean any previously generated .idea files (reset and `git clean -fxd`) and try Import project as gradle (instead of Open), -- Import as gradle should always work but it removes checked-in run configs (which are optional and you can git reset and refix them, or just add new ones, see run/debug configs later)

Imported project

If you experience build or runtime errors, check Troubleshooting section.

Run

Right click on the main class from the Project panel and choose run or debug. (Another way to run is to set up IntelliJ Run/Debug configs)



Run/Debug configs (optional)

You can set up an IntelliJ run/debug config as alternative way to running. These configs allow you to save run parametrization, if you need any (vm options, environment).

IntelliJ has a dropdown with Run/Debug configs at the top toolbar. Example configs are checked in, but you should verify and fix it, or create one from scratch.

Click config dropdown arrow > Edit Configurations... (Or from menu Run > Edit Configurations)

  • Working run/debug config is show below.
  • Set the main class, classpath module matching the main and JRE version 1.8.
  • Recommended single instance for running.

Prepare a run/debug config to launch NwsUsaAleters class and click run or debug (alternatively, from Run menu > Run or Debug). IntelliJ builds and launches the application.


Troubleshooting

Build errors due to Java versions

If you are getting any errors regarding JDK, java, “-source” versions or support, it’s a matter of ensuring you have matching Java installed on the machine and providing correct versions to the build and IDE. This is likely to occur when multiple Java versions are installed on a machine and IDE (IntelliJ) is used.

Error:(6, 64) java: diamond operator is not supported in -source 1.6
 (use -source 7 or higher to enable diamond operator)

In IntelliJ menu File > Project Structure > Project Settings

  •      Project SDK should point to Java 1.8
  •      Project language level should be 8.


Runtime: connection errors

If you can build and run but observe RTM connection failures in the log at runtime, this is most likely due to wrong publishing credentials in the config.json.

com.satori.composer.rtm.core.RtmBase enterConnectingState - rtm connecting...
com.satori.composer.rtm.core.RtmBase enterConnectedState - rtm connected
com.satori.composer.rtm.core.RtmAuthenticator enterHandshakingState - rtm authenticating...
com.satori.composer.rtm.core.RtmBase leaveConnectedState - rtm disconnected

Fix host, appkey, auth for RTM publisher in the config.json


Runtime: Warning Thread blocked

[WARNING] io.vertx.core.impl.BlockedThreadChecker$1 run - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 6839 ms, time limit is 2000 <br>io.vertx.core.VertxException: Thread blocked

Known issue: it may occur on a normal startup in the first few seconds. It recovers and continues normally, is safe to ignore.


Runtime: other problems

If you are using nsw-usa-alerts example or similar setup with the whole republishing pipeline, such as polling large batches from the network, -- occasional exceptions are expected and recoverable (the project logs exception and continues normally). Also note that some source feeds (is the case with nsw-usa-alerts) get updated in batches so it’s normal to see delays in published results (no modifications for a while and then a big batch).

If you are having persistent failures, try to identify the mod and cause from the logs.  Try the clock example, which is simple (two mods).