OpenDMP 0.5.0

• Python and Clojure script processors now run as separate services.
  • Removed . The python script runner is a pure Python Flask service. Much less clunky.
  • This simplifies and greatly reduces the size of the OpenDMP processor service image and reduces CPU load.
• Improved error handling on service calls
• Bumped Keycloak version to 11
• Elasticsearch now supported as a data destination!

OpenDMP 0.4.0

Significant changes here

• Added Plugin support!
• Added Consul to the stack for plugin service discovery
• Started on a Plugin development library.
• Added a plugins folder to the repo.
• Some bug fixes
• Some code reorganization - created a parent POM.

Known issues:

• When ingesting from S3, if a plugin throws a 500 error, the flow will start over trying to re-ingest the same file. Ad infinitum. This can be caused either by a problem with the plugin itself, or corrupt data. I haven't observed this behavior when ingesting from a file drop.

OpenDMP 0.3.0

• S3 Ingest now supported. You can set up a S3 bucket and key prefix to "watch" for new data.
• Raw Data is now tagged when it enters a flow and history of individual data sets (i.e., what processors handle the data as it traverses the flow) are tracked.
• Updated to Apache Camel 3.5. This Camel bug has been fixed, which has allowed me to remove an ugly hack-around in the OpenDMP code.
• External Processor implemented - can execute external scripts and tools as part of a dataflow.
• Collection interface is more fleshed out
  • Can delete collections
  • Can examine properties and history of individual datasets (and delete dataset records as well)

OpenDMP 0.2.0

This is a good point to cut a (very alpha) release:

• S3 Destination for collect processors works (still can't ingest from S3 yet, working on it)
• Can download datasets through the UI now
• Python script processors work
• Datasets are now sortable and paginated
• Errors during runs are now displayed in the Dataflow view (requires refresh - need to implement push via SSE, or perhaps be lazy and do polling).

Still to do:
Lots