Skip to content

Commit

Permalink
Link to latest mantis-example and update commands (#161)
Browse files Browse the repository at this point in the history
  • Loading branch information
liuml07 authored Apr 5, 2022
1 parent 3a3b5dd commit 45d804d
Show file tree
Hide file tree
Showing 10 changed files with 41 additions and 34 deletions.
2 changes: 1 addition & 1 deletion docs/docs/develop/writing-jobs/group-by.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

NOTE: This tutorial is a work in progress.

Until now we've run single stage Mantis jobs which a can run in a single process / container. Much of the power provided by Mantis is that we can design and implement a distributed job. Let's take a look at the [groupby-sample](https://github.com/Netflix/mantis-examples/tree/master/groupby-sample) job definition and then break it down stage by stage.
Until now we've run single stage Mantis jobs which a can run in a single process / container. Much of the power provided by Mantis is that we can design and implement a distributed job. Let's take a look at the [groupby-sample](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-groupby-sample) job definition and then break it down stage by stage.

```java

Expand Down
9 changes: 7 additions & 2 deletions docs/docs/develop/writing-jobs/twitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Our first tutorial primed us for writing and executing a job end-to-end but it w
To proceed you'll need to head over to Twitter and grab yourself a pair of API keys.

## The Source
The source is responsible for ingesting data to be processed within the job. Many Mantis jobs will subscribe to other jobs and can simply use a templatized source such as [`io.mantisrx.connectors.job.source.JobSource`](https://github.com/Netflix/mantis-connectors/blob/master/mantis-connector-job/src/main/java/io/mantisrx/connector/job/source/JobSource.java) which handles all the minutiae of connecting to other jobs for us. If however your job exists on the edge of Mantis it will need to pull data in via a custom source. Since we're reading from the Twitter API we'll need to do this ourselves.
The source is responsible for ingesting data to be processed within the job. Many Mantis jobs will subscribe to other jobs and can simply use a templatized source such as [`io.mantisrx.connectors.job.source.JobSource`](https://github.com/Netflix/mantis/blob/master/mantis-connectors/mantis-connector-job/src/main/java/io/mantisrx/connector/job/source/JobSource.java) which handles all the minutiae of connecting to other jobs for us. If however your job exists on the edge of Mantis it will need to pull data in via a custom source. Since we're reading from the Twitter API we'll need to do this ourselves.

Our [`TwitterSource`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/source/Source.java) must implement [`io.mantisrx.runtime.source.Source`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/source/Source.java) which requires us to implement `call` and optionally `init`. Mantis provides some guarantees here in that `init` will be invoked exactly once and before `call` which will be invoked at least once. This makes `init` the ideal location to perform one time setup and configuration for the source and `call` the ideal location for performing work on the incoming stream. The objective of this entire class is to have `call` return an `Observable<Observable<T>>` which will be passed as a parameter to the first stage of our job.

Expand Down Expand Up @@ -153,4 +153,9 @@ The stage is nearly equivalent to the previous tutorial. We need to add a few li
# Conclusion
We've learned how to create a parameterized source which reads from Twitter and pulls data into the ecosystem. With some slight modifications our previous example's stage deserializes the messages and extracts the data to perform the same word count.

If you've checked out the [`mantis-examples`](https://github.com/Netflix/mantis-examples) repository then running `./gradlew :mantis-examples-twitter-sample:execute --args='consumerKey consumerSecret token tokensecret'` at the root of the repository should begin running the job and expose a local port for SSE streaming. As an exercise consider how you might begin to scale this work out over multiple machines if the workload were too large to perform on a single host. This will be the topic of the next tutorial.
If you've checked out the [`mantis`](https://github.com/Netflix/mantis) repository, then running following commands should begin running the job and expose a local port for SSE streaming.
```bash
$ cd mantis-examples/mantis-examples-twitter-sample
$ ../../gradlew execute --args='consumerKey consumerSecret token tokensecret'
```
As an exercise consider how you might begin to scale this work out over multiple machines if the workload were too large to perform on a single host. This will be the topic of the next tutorial.
15 changes: 8 additions & 7 deletions docs/docs/develop/writing-jobs/word-count.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Writing Your First Mantis Job
We'll be doing the classic word count example for streaming data for the tutorial section. For this example we'll be keeping it simple and focusing on the processing logic and job provider. The tutorials are structured progressively to allow us to incrementally build some experience writing jobs without getting overwhelmed with details. We'll stream text from a Project Gutenberg book, perform some application logic on the stream, and then write the data to a sink for consumption by other Mantis jobs. If you want to follow along check out the [Word Count](https://github.com/Netflix/mantis-examples/tree/master/wordcount) project in the [mantis-examples](https://github.com/Netflix/mantis-examples/) repository.
We'll be doing the classic word count example for streaming data for the tutorial section. For this example we'll be keeping it simple and focusing on the processing logic and job provider. The tutorials are structured progressively to allow us to incrementally build some experience writing jobs without getting overwhelmed with details. We'll stream text from a Project Gutenberg book, perform some application logic on the stream, and then write the data to a sink for consumption by other Mantis jobs. If you want to follow along, please check the [Word Count](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-wordcount) module in Mantis repository.

There are a few things to keep in mind when implementing a Mantis Job;

Expand All @@ -10,7 +10,7 @@ There are a few things to keep in mind when implementing a Mantis Job;

## WordCountJob

The full source of the [WordCountJob](https://github.com/Netflix/mantis-examples/wordcount/...) class is included below with imports elided. This class implements the [`io.mantisrx.runtime.MantisJobProvider`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/MantisJobProvider.java) interface which the Mantis runtime loads. `MantisJobProvider#getJobInstance()` provides the runtime with an entry point to your job's code.
The full source of the [WordCountJob](https://github.com/Netflix/mantis/blob/master/mantis-examples/mantis-examples-wordcount/src/main/java/com/netflix/mantis/examples/wordcount/WordCountJob.java) class is included below with imports elided. This class implements the [`io.mantisrx.runtime.MantisJobProvider`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/MantisJobProvider.java) interface which the Mantis runtime loads. `MantisJobProvider#getJobInstance()` provides the runtime with an entry point to your job's code.

```java
/**
Expand All @@ -20,9 +20,6 @@ The full source of the [WordCountJob](https://github.com/Netflix/mantis-examples
* E.g
* <code> Serving modern HTTP SSE server sink on port: 8650 </code>
* You can curl this port <code> curl localhost:8650</code> to view the output of the job.
*
* To run via gradle
* /gradlew :mantis-examples-wordcount:execute
*/
@Slf4j
public class WordCountJob extends MantisJobProvider<String> {
Expand Down Expand Up @@ -115,11 +112,15 @@ public static ScalarToScalar.Config<String, String> scalarToScalarConfig() {

### The Job Provider

The [`MantisJobProvider`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/MantisJobProvider.java) interface is what the Mantis runtime expects to load. The runtime reads [`resources/META-INF/services/io.mantisrx.runtime.MantisJobProvider`](https://github.com/Netflix/mantis-examples/blob/master/wordcount/src/main/resources/META-INF/services/io.mantisrx.runtime.MantisJobProvider) to discover the fully qualified classname of the MantisJobProvider to be used as an entry point for the application.
The [`MantisJobProvider`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/MantisJobProvider.java) interface is what the Mantis runtime expects to load. The runtime reads [`resources/META-INF/services/io.mantisrx.runtime.MantisJobProvider`](https://github.com/Netflix/mantis/blob/master/mantis-examples/mantis-examples-wordcount/src/main/resources/META-INF/services/io.mantisrx.runtime.MantisJobProvider) to discover the fully qualified classname of the MantisJobProvider to be used as an entry point for the application.

### Main Method

The main method invokes the [`LocalJobExecutorNetworked`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/executor/LocalJobExecutorNetworked.java) `execute` method to run our job locally. The first three tutorials will take advantage of the ability to run jobs locally. In the fourth tutorial we will explore uploading and submitting our job on a Mantis cloud deployment for greater scalability. We can and should run this main method by invoking `./gradlew :mantis-examples-wordcount:execute` at the root of the `mantis-examples` directory.
The main method invokes the [`LocalJobExecutorNetworked`](https://github.com/Netflix/mantis/blob/master/mantis-runtime/src/main/java/io/mantisrx/runtime/executor/LocalJobExecutorNetworked.java) `execute` method to run our job locally. The first three tutorials will take advantage of the ability to run jobs locally. In the fourth tutorial we will explore uploading and submitting our job on a Mantis cloud deployment for greater scalability. We can and should run this main method by following commands from the Mantis repository:
```bash
$ cd mantis-examples/mantis-examples-wordcount
$ ../../gradlew execute
```

```java
public static void main(String[] args) {
Expand Down
10 changes: 5 additions & 5 deletions docs/docs/getting-started/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,20 +125,20 @@ Works the the mantis-publish library to fetch data on-demand from external appli
available to downstream consumers. See the [On-Demand Sample](samples/on-demand.md) to see
this in action.

Users can also build their own source jobs see the [Synthetic Source Job](https://github.com/Netflix/mantis-examples/tree/master/synthetic-sourcejob) example.
Users can also build their own source jobs see the [Synthetic Source Job](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-synthetic-sourcejob) example.

## Job Chaining

One of the unique capabilities of Mantis is the ability for Jobs to communicate with each other to form a kind of
streaming microservices architecture. The Source Job -> Downstream job flow is an example of this Job chaining.
In this case the `source` of the downstream job is the output of the upstream Data source job.
All a job needs to do to connect to the sink of another job is to include the in-built [Job Connnector](https://github.com/Netflix/mantis-connectors/blob/master/mantis-connector-job/src/main/java/io/mantisrx/connector/job/source/JobSource.java)
All a job needs to do to connect to the sink of another job is to include the in-built [Job Connnector](https://github.com/Netflix/mantis/blob/master/mantis-connectors/mantis-connector-job/src/main/java/io/mantisrx/connector/job/source/JobSource.java).

See the [Job Connector Sample](https://github.com/Netflix/mantis-examples/tree/master/jobconnector-sample) to see this in action.
See the [Job Connector Sample](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-jobconnector-sample) to see this in action.

These job to job communications happen directly via in memory socket connections with no intermediate disk persistence.
If buffering/persistence of results is desired then it is recommended to sink the data into persistence queue like Kafka using
the [Kafka Connector](https://github.com/Netflix/mantis-connectors/blob/master/mantis-connector-kafka/src/main/java/io/mantisrx/connector/kafka/sink/KafkaSink.java)
the [Kafka Connector](https://github.com/Netflix/mantis/blob/master/mantis-connectors/mantis-connector-kafka/src/main/java/io/mantisrx/connector/kafka/sink/KafkaSink.java)

Job chaining has proven to be extremely useful while operating at scale. It is widely used in the Netflix deployment of Mantis.

Expand All @@ -156,7 +156,7 @@ as a library within a Job that can benefit from the query and aggregation featur
## Mantis Master
The [Mantis Master](https://github.com/netflix/mantis-control-plane) is a leader elected control plane for the Mantis platform.
The [Mantis Master](https://github.com/Netflix/mantis/tree/master/mantis-control-plane) is a leader elected control plane for the Mantis platform.
It is responsible for managing the life cycle of Job Clusters, Jobs and workers. It also acts as a Resource scheduler
to optimally allocate and schedule resources required by the Jobs. The master stores its meta-data into an external source.
The OSS version ships with a sample file based store. For production deployments a highly available store is recommended.
Expand Down
7 changes: 4 additions & 3 deletions docs/docs/getting-started/samples/on-demand.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,17 @@ This end-to-end example highlights two powerful Mantis concepts

* `SharedMrePublishEventSource` Job Cluster exists.
* `JobConnectorSample` Job cluster exists
* [Java Sample](https://github.com/Netflix/mantis-examples/tree/master/mantis-publish-sample) is setup and running.
* [Java Sample](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-mantis-publish-sample) is setup and running.

> **Note** If you are following the [Mantis Cluster using Docker](../tutorials/docker.md) instructions all of this
>will be already set up.

## Publishing events into Mantis

> **Note**: The local docker setup has already preconfigured a simple [Java Sample](https://github.com/Netflix/mantis-examples/tree/master/mantis-publish-sample) application to publish events
to Mantis.
> **Note**: The local docker setup has already preconfigured a simple
[Java Sample](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-mantis-publish-sample)
application to publish events to Mantis..

## Setting up a Publish Data Source Job

Expand Down
2 changes: 1 addition & 1 deletion docs/docs/getting-started/samples/sine-function.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,6 @@ To stop the job click on the red `Kill Job` button on the top right corner.

## Next Steps

* Explore the [code](https://github.com/Netflix/mantis-examples/tree/master/sine-function)
* Explore the [code](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-sine-function)

* Checkout out the other samples
2 changes: 1 addition & 1 deletion docs/docs/getting-started/samples/twitter.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,6 @@ You should see output of the Twitter job being streamed below
## Terminate the job
To stop the job click on the red `Kill Job` button on the top right corner.

* Explore the [code](https://github.com/Netflix/mantis-examples/tree/master/twitter-sample)
* Explore the [code](https://github.com/Netflix/mantis/tree/master/mantis-examples/mantis-examples-twitter-sample)

* Checkout out the other samples
23 changes: 10 additions & 13 deletions docs/docs/getting-started/tutorials/local.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,25 @@ JDK 8 or higher

## Build and run the synthetic-sourcejob sample

Clone the mantis-examples repo:

```bash
$ git clone https://github.com/Netflix/mantis-examples.git
```

Run the synthetic-sourcejob sample via gradle.

This job outputs request events sourced from an imaginary service. The RequestEvent data
has information such as uri, status, userId, country etc.
Data Source Jobs are mantis jobs that allow consumers to filter the raw stream down to just the events
they are interested in.
This filtering is done by specifying an [MQL query](../../develop/querying/mql.md) while connecting to the sink.

To run the sample execute the following command.
Clone the mantis repo:
```bash
$ cd mantis-examples/synthetic-sourcejob
$ ../gradlew execute
$ git clone https://github.com/Netflix/mantis.git
```
This will launch the job and you would see output like

Run the synthetic-sourcejob sample via gradle.
```bash
$ cd mantis/mantis-examples/mantis-examples-synthetic-sourcejob
$ ../../gradlew execute
```

This will launch the job and you would see output like
```bash
2019-10-06 14:14:07 INFO StageExecutors:254 main - initializing io.mantisrx.sourcejob.synthetic.stage.TaggingStage
2019-10-06 14:14:07 INFO SinkPublisher:82 main - Got sink subscription, onSubscribe=null
2019-10-06 14:14:07 INFO ServerSentEventsSink:141 main - Serving modern HTTP SSE server sink on port: 8436
Expand Down Expand Up @@ -66,7 +63,7 @@ data: {"country":"Liberia","mantis.meta.sourceName":"SyntheticRequestSource","ma
## Next Steps
* Import the project into your IDE to explore the code.
* Try out other samples from the [Mantis examples repository](https://github.com/netflix/mantis-examples).
* Try out other samples from the [Mantis examples module](https://github.com/Netflix/mantis/tree/master/mantis-examples).
* [Set up Mantis locally using Docker](docker.md) and run the samples.
* [Set up Mantis in AWS](cloud.md) and run the samples.
* Learn to write your [first Mantis Job](../../develop/writing-jobs/word-count.md).
3 changes: 3 additions & 0 deletions mantis-examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,6 @@ into your job.

## mantis-publish-sample
An example of using the mantis-publish library to send events to Mantis.

## mantis-examples-wordcount
The classic word count example for streaming data for the tutorial section.
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ configurations.all {


task execute(type: JavaExec) {
main = "com.netflix.mantis.examples.twittersample.TwitterJob"
main = "com.netflix.mantis.examples.wordcount.TwitterJob"
classpath = sourceSets.main.runtimeClasspath
}

Expand Down

0 comments on commit 45d804d

Please sign in to comment.