diff --git a/README.md b/README.md index 25ded5d..b2375c8 100644 --- a/README.md +++ b/README.md @@ -8,30 +8,29 @@ # Section Overview -As Consensus Flow Entry Point, "Proposing" means taking the responsibility of -**receptioning** requests from customers and consortium and **packaging** them into local proposals. -Each of these local proposals are proposed for transactions into the consortium block stream. +`Proposing` means +- **Receptioning** requests from customers and consortium members +- **Staging** these requests packaged into local proposals for being consumed by the consensus. + +Each of these local proposals once accepted by the consortium will be uniformly transacted on each consortium node. +The transactions will then be appended into each node ledgers. A **Section** is a meaningful Set of Pipelines and Junctions put together. ![proposing-overview](documentation/media/proposing-overview.png) -> Worth Mentioning : [Packaging](/packages/packaging/) is the core of this section. -> It's a good example of Pipeline Implementation in Haskell. - -- [Receptioning](/packages/receptioning) : Client/Server for collecting requests presenting them for being packaged into a proposal. -- [Detecting Starvation](/packages/detecting-starvation) : Notify the `Packaging` pipeline when all the local proposals produced have been consumed. -- [Packaging](/packages/packaging/) : Produce Local Proposal Files with the following properties +- [Receptioning](/packages/receptioning) : Client/Server for collecting requests. +- [Detecting Flow Tension](/packages/detecting-tension) : Detect if the local proposal flow is tensed, meaning if the consensus has consumed more local proposals than being staged. +- [Staging](/packages/staging) : Stage Local Proposal Files for being consumed by the consensus. Local Proposals have the following properties - `Proposals are never empty` - `Proposals file size < configurable size limit` - Filename - `x.proposal` with `x` the offset of proposal produced - [Simulating](/packages/simulating) : Simulate the proposing input streams - - Send dummy requests to `Receptioning` and simulate the downstream consumption of local proposal. - - 2 modes are available : + - Send dummy requests to `Receptioning` and simulate the downstream local proposal consumption. + - 2 available modes - **Overflowing** : Sending more requests than the consortium can consume - **UnderSupplying** : Sending less request than the consortium can consume - # DevOps [Zeus](/packages/zeus) : Local Deployment Tool diff --git a/documentation/development-task.md b/documentation/development-task.md index b303dd9..4fbf2ca 100644 --- a/documentation/development-task.md +++ b/documentation/development-task.md @@ -8,8 +8,8 @@ # Environment Setup - Install [Docker For Mac](https://docs.docker.com/docker-for-mac/install/) +- Install [Multitail](http://macappstore.org/multitail/) - Install the Haskell Platform For Mac (GHC, Cabal build system, [Stack tool](https://docs.haskellstack.org/en/stable/README/), ...) - ```shell curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh ``` @@ -20,28 +20,28 @@ curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh # Run the tests - Manual ```shell -➜ consensus-proposing git:(master) : stack test +stack test ``` - Automatic (will rebuild and test every time a file changes) ```shell -➜ consensus-proposing git:(master) : stack test --fast --file-watch +stack test --fast --file-watch ``` # Install the executables - Manual ```shell -➜ consensus-proposing git:(master) : stack install +stack install ``` - Automatic (will rebuild and install every time a file changes) ```shell -➜ consensus-proposing git:(master) : stack install --fast --file-watch +stack install --fast --file-watch ``` You should see something similar at the end : ```shell Copied executables to /Users/nhenin/.local/bin: -- dolla-consensus-proposing-detecting-starvation -- dolla-consensus-proposing-packaging +- dolla-consensus-proposing-detecting-tension +- dolla-consensus-proposing-staging - dolla-consensus-proposing-receptioning - dolla-consensus-proposing-simulating - dolla-consensus-proposing-zeus @@ -53,6 +53,20 @@ Copied executables to /Users/nhenin/.local/bin: - Make sure you are at the root of the project - Execute Proposing Zeus +```shell +dolla-consensus-proposing-zeus +``` +N.B : If you run it for the first time , it will install the eventstore docker image (eventstore/eventstore:release-5.0.6) +```shell +WARNING: Some next steps will your ADMIN PASSWORD +``` +Next Steps : +- Interactivity with the CLI for getting the desired parameters +- Start the Event Store docker container +- Configure the EventStore for running the consensus properly (**REQUIRE ADMIN PASSWORD**) +- Run the proposing executables in another tab + + Below, an example about how to run Zeus ```shell ➜ consensus-proposing git:(master) : dolla-consensus-proposing-zeus @@ -104,4 +118,4 @@ You can now see the local proposal being produced in `output/nodes/achilles/prop ![proposal-produced](media/proposal-produced.png) -go back to Zeus if you want to stop the Proposing section. \ No newline at end of file +go back to Zeus if you want to stop the Proposing section. (choice 2) \ No newline at end of file diff --git a/documentation/media/proposing-overview.png b/documentation/media/proposing-overview.png index 624d4d8..f2bb386 100644 Binary files a/documentation/media/proposing-overview.png and b/documentation/media/proposing-overview.png differ diff --git a/packages/detecting-starvation/README.md b/packages/detecting-starvation/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/packages/detecting-starvation/app/Executables.hs b/packages/detecting-starvation/app/Executables.hs deleted file mode 100644 index f457fcf..0000000 --- a/packages/detecting-starvation/app/Executables.hs +++ /dev/null @@ -1,11 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -module Executables - ( detectingStarvation - ) where - - -import qualified Dolla.Consensus.Proposing.Starving.Detecting.OverEventStore as Notifying - -detectingStarvation :: IO () -detectingStarvation = Notifying.execute - diff --git a/packages/detecting-starvation/dolla-consensus-proposing-detecting-starvation.cabal b/packages/detecting-starvation/dolla-consensus-proposing-detecting-starvation.cabal deleted file mode 100644 index 773548e..0000000 --- a/packages/detecting-starvation/dolla-consensus-proposing-detecting-starvation.cabal +++ /dev/null @@ -1,87 +0,0 @@ -cabal-version: 1.12 - --- This file has been generated from package.yaml by hpack version 0.33.0. --- --- see: https://github.com/sol/hpack --- --- hash: adccecf237589ae238790098b2f10eb2d947346183933f1a3855432ca855ccfe - -name: dolla-consensus-proposing-detecting-starvation -version: 0.0.0.1 -build-type: Simple -extra-source-files: - README.md - -library - exposed-modules: - Dolla.Consensus.Proposing.Starving.Detecting.Dependencies - Dolla.Consensus.Proposing.Starving.Detecting.ESMerger - Dolla.Consensus.Proposing.Starving.Detecting.Input - Dolla.Consensus.Proposing.Starving.Detecting.Output - Dolla.Consensus.Proposing.Starving.Detecting.OverEventStore - Dolla.Consensus.Proposing.Starving.Detecting.OverMemoryStream - Dolla.Consensus.Proposing.Starving.Detecting.OverPersistedLog - Dolla.Consensus.Proposing.Starving.Detecting.State - other-modules: - Paths_dolla_consensus_proposing_detecting_starvation - hs-source-dirs: - lib - ghc-options: -Wall - build-depends: - QuickCheck ==2.13.2 - , aeson ==1.4.7.1 - , base ==4.13.0.0 - , bytestring ==0.10.10.0 - , containers ==0.6.2.1 - , directory ==1.3.6.0 - , dolla-base ==1.0.0 - , dolla-consensus-base ==1.0.0 - , dolla-consensus-proposing-detecting-starvation-settings - , exceptions ==0.10.4 - , interpolatedstring-perl6 ==1.0.2 - , lens ==4.18.1 - , mtl ==2.2.2 - , streamly ==0.7.2 - , uuid ==1.3.13 - default-language: Haskell2010 - -executable dolla-consensus-proposing-detecting-starvation - main-is: Executables.hs - other-modules: - Paths_dolla_consensus_proposing_detecting_starvation - hs-source-dirs: - app - ghc-options: -Wall -main-is Executables.detectingStarvation -threaded -rtsopts -with-rtsopts=-N - build-depends: - base ==4.13.0.0 - , dolla-consensus-proposing-detecting-starvation - default-language: Haskell2010 - -test-suite dolla-consensus-proposing-detecting-starvation-test - type: exitcode-stdio-1.0 - main-is: Spec.hs - other-modules: - Dolla.Consensus.Proposing.Starving.Detecting.GenInput - Dolla.Consensus.Proposing.Starving.Detecting.GenOutput - Dolla.Consensus.Proposing.Starving.Detecting.IOSpec - Dolla.Consensus.Proposing.Starving.Detecting.OverMemoryStreamSpec - Dolla.Consensus.Proposing.Starving.Detecting.StateSpec - Paths_dolla_consensus_proposing_detecting_starvation - hs-source-dirs: - test - ghc-options: -Wall -threaded -rtsopts -with-rtsopts=-N - build-depends: - QuickCheck - , aeson ==1.4.7.1 - , base - , dolla-consensus-base ==1.0.0 - , dolla-consensus-proposing-detecting-starvation - , generic-arbitrary - , hspec - , interpolatedstring-perl6 ==1.0.2 - , mtl ==2.2.2 - , quickcheck-instances - , random ==1.1 - , streamly ==0.7.2 - , uuid ==1.3.13 - default-language: Haskell2010 diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverEventStore.hs b/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverEventStore.hs deleted file mode 100644 index db6773b..0000000 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverEventStore.hs +++ /dev/null @@ -1,41 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Starving.Detecting.OverEventStore (execute) where - -import Prelude hiding (log) -import Control.Monad.Reader - -import Dolla.Common.Logging.Core -import Dolla.Common.Executable.Executable - -import Dolla.Consensus.Log.EventStoreLog -import Dolla.Libraries.LogEngine.Instances.EventStore.EventStoreLog (EventStoreLog) - -import Dolla.Consensus.Proposing.Starving.Detecting.Settings -import Dolla.Consensus.Proposing.Starving.Detecting.Dependencies -import Dolla.Consensus.Proposing.Starving.Detecting.ESMerger (loadInputProjection) - -import Dolla.Consensus.Proposing.Starving.Detecting.Output - -import qualified Dolla.Consensus.Proposing.Starving.Detecting.State as State -import qualified Dolla.Consensus.Proposing.Starving.Detecting.OverPersistedLog as OverPersistedLog -import qualified Dolla.Consensus.Proposing.Starving.Detecting.OverMemoryStream as OverMemoryStream - -execute :: IO () -execute = executeMicroservice (\Settings {logger} -> logger) start - -start :: ReaderT Dependencies IO () -start = do - Dependencies {eventStoreClient,nodeId,logger} <- ask - _ <- withReaderT (const (nodeId, eventStoreClient)) loadInputProjection - log logger INFO "Starting Service" - let inputLog = getEventStoreLog eventStoreClient ProposingStarvingDetectionInputLog - outputLog = getEventStoreLog eventStoreClient LocalRequestLog :: EventStoreLog Output - OverPersistedLog.detectingPipelineStarving - inputLog - outputLog - (OverMemoryStream.detectingPipelineStarving - State.projection - State.starvingInvariantPredicate) - log logger INFO "Service Down" - diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverMemoryStream.hs b/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverMemoryStream.hs deleted file mode 100644 index d66a954..0000000 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverMemoryStream.hs +++ /dev/null @@ -1,42 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE RecordWildCards #-} -{-# LANGUAGE MonoLocalBinds #-} - -module Dolla.Consensus.Proposing.Starving.Detecting.OverMemoryStream - (detectingPipelineStarving) where - -import Prelude hiding (log,writeFile) - -import Data.Function ((&)) - -import qualified Streamly as S -import qualified Streamly.Prelude as S -import qualified Streamly.Internal.Data.Fold as SF - -import Dolla.Common.Offset -import Dolla.Common.UUID.Deterministic - -import Dolla.Consensus.Proposing.Starving.Detecting.Input -import Dolla.Consensus.Proposing.Starving.Detecting.Output - -detectingPipelineStarving - :: S.MonadAsync m - => SF.Fold m Input state - -> (state -> Bool) - -> S.SerialT m Input - -> S.SerialT m Output -detectingPipelineStarving - projection - starvingInvariantPredicate - inputStream - = inputStream - & S.postscan projection - & S.filter starvingInvariantPredicate - & S.indexed - & S.map (fromIntegralToOffset . fst ) - & S.mapM - (\starvingOffset -> do - let itemId = getDeterministicUUID starvingOffset - return LocalProposalStarvationDetected {..}) - diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverPersistedLog.hs b/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverPersistedLog.hs deleted file mode 100644 index ecef7fe..0000000 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/OverPersistedLog.hs +++ /dev/null @@ -1,38 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE MonoLocalBinds #-} - -module Dolla.Consensus.Proposing.Starving.Detecting.OverPersistedLog - (detectingPipelineStarving) where - -import Prelude hiding (log,writeFile) - -import Data.Function ((&)) - -import Control.Monad.Reader -import Control.Monad.Catch (MonadCatch) - -import qualified Streamly as S -import qualified Streamly.Prelude as S - -import Dolla.Common.Range - -import Dolla.Libraries.LogEngine.LogEngine - -import Dolla.Consensus.Proposing.Starving.Detecting.Input -import Dolla.Consensus.Proposing.Starving.Detecting.Output - -type OverMemoryStreamProcessing m = (S.SerialT m Input -> S.SerialT m Output) - -detectingPipelineStarving - :: ( MemoryStreamLoggable m log - , S.MonadAsync m - , MonadCatch m) - => log Input - -> log Output - -> OverMemoryStreamProcessing m - -> m () -detectingPipelineStarving inputLog outputLog overMemoryStreamProcessing - = stream infinitely inputLog - & overMemoryStreamProcessing - & S.mapM_ (void . nonIdempotentAppend outputLog) diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/State.hs b/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/State.hs deleted file mode 100644 index a9d7901..0000000 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/State.hs +++ /dev/null @@ -1,52 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE MonoLocalBinds #-} - -module Dolla.Consensus.Proposing.Starving.Detecting.State - ( starvingInvariantPredicate - , projection - , State (..) - ) where - -import Prelude hiding (log,writeFile) - -import Data.Monoid -import qualified Streamly.Internal.Data.Fold as SF -import Dolla.Consensus.Proposing.Starving.Detecting.Input - - -type RemainingProposalToConsume = Integer - -data State - = State - { remainingProposalToConsume :: Integer - , isLocalProposalAsked :: Any} deriving Eq - -starvingInvariantPredicate :: State -> Bool -starvingInvariantPredicate = (== State 0 (Any True)) - -projection - :: (Monad m) - => SF.Fold m Input State -projection - = State <$> foldRemainingProposalToConsume - <*> foldIsLocalProposalAsked - -foldIsLocalProposalAsked :: (Monad m , Monoid Any) => SF.Fold m Input Any -foldIsLocalProposalAsked - = SF.foldMap - (\case - LocalProposalAsked -> Any True - _ -> Any False ) - -foldRemainingProposalToConsume :: (Monad m ) => SF.Fold m Input RemainingProposalToConsume -foldRemainingProposalToConsume - = SF.lmapM - (\inputItem -> do - return $ case inputItem of - LocalProposalProduced -> 1 - LocalProposalConsumed -> -1 - LocalProposalAsked -> 0) - SF.sum - diff --git a/packages/detecting-starvation/settings/README.md b/packages/detecting-starvation/settings/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/GenOutput.hs b/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/GenOutput.hs deleted file mode 100644 index f860c66..0000000 --- a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/GenOutput.hs +++ /dev/null @@ -1,14 +0,0 @@ -module Dolla.Consensus.Proposing.Starving.Detecting.GenOutput - ( OutputUnderTests (..)) where - -import Test.QuickCheck.Arbitrary - -import Test.QuickCheck.Instances.UUID () - -import Dolla.Consensus.Proposing.Starving.Detecting.Output - -instance Arbitrary OutputUnderTests where - arbitrary = OutputUnderTests . LocalProposalStarvationDetected <$> arbitrary - -newtype OutputUnderTests = OutputUnderTests { unOutputUnderTests :: Output} deriving Show - diff --git a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/IOSpec.hs b/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/IOSpec.hs deleted file mode 100644 index 569105b..0000000 --- a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/IOSpec.hs +++ /dev/null @@ -1,37 +0,0 @@ -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE ScopedTypeVariables #-} -{-# LANGUAGE DerivingVia #-} -{-# LANGUAGE QuasiQuotes, ExtendedDefaultRules #-} -{-# LANGUAGE NamedFieldPuns #-} - -module Dolla.Consensus.Proposing.Starving.Detecting.IOSpec (spec) where - - -import Data.Aeson -import Test.QuickCheck - -import Test.QuickCheck.Instances () -import Test.Hspec -import Text.InterpolatedString.Perl6 (qc) - -import Dolla.Consensus.Proposing.Starving.Detecting.Input - -import Dolla.Consensus.Proposing.Starving.Detecting.Output -import Dolla.Consensus.Proposing.Starving.Detecting.GenOutput - -spec :: Spec -spec = parallel $ - describe "Proposing.Starving-Detection IOs" $ do - describe "Input" $ do - it "follow the json protocol" $ do - decode [qc| "LocalProposalAsked" |] `shouldBe` Just LocalProposalAsked - decode [qc| "LocalProposalConsumed" |] `shouldBe` Just LocalProposalConsumed - decode [qc| "LocalProposalProduced" |] `shouldBe` Just LocalProposalProduced - describe "Output" $ do - it "follow Proposing Packaging Input Json Protocol" - $ property $ \OutputUnderTests {unOutputUnderTests} - -> do - let encodedItemId = encode $ itemId unOutputUnderTests - encode unOutputUnderTests `shouldBe` [qc|\{"tag":"LocalProposalStarvationDetected","itemId":{encodedItemId}}|] - - diff --git a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/OverMemoryStreamSpec.hs b/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/OverMemoryStreamSpec.hs deleted file mode 100644 index dc8312b..0000000 --- a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/OverMemoryStreamSpec.hs +++ /dev/null @@ -1,82 +0,0 @@ -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE ScopedTypeVariables #-} -{-# LANGUAGE DerivingVia #-} -module Dolla.Consensus.Proposing.Starving.Detecting.OverMemoryStreamSpec (spec) where - -import Data.Coerce (coerce) -import Data.Function ((&)) -import Data.Monoid -import Data.Bifunctor - -import Control.Monad.Cont (liftIO) -import System.Random -import Test.QuickCheck -import Test.QuickCheck.Monadic hiding (assert) -import Test.QuickCheck.Instances () -import Test.Hspec - -import qualified Streamly.Internal.Data.Fold as SF -import qualified Streamly.Prelude as S - - -import Dolla.Consensus.Proposing.Starving.Detecting.Input -import Dolla.Consensus.Proposing.Starving.Detecting.GenInput - -import Dolla.Consensus.Proposing.Starving.Detecting.OverMemoryStream - -spec :: Spec -spec = parallel $ - describe "detecting a pipeline starving over a memory stream" $ do - it "emits a Starving Pipeline output event when the predicate validate the projected state" - $ property shouldEmitAnOutputWhenPredicateIsTrue - - -shouldEmitAnOutputWhenPredicateIsTrue :: ConsistentInputs -> Property -shouldEmitAnOutputWhenPredicateIsTrue consistentInputs - = monadicIO $ do - let indexedStreamOfTrueEvent - = S.fromList (coerce consistentInputs) - & S.postscan projectDeterministicallyAndRandomlyABool - & S.postscan countTruesAndIndexIt - & S.filter snd -- filter by true - indexedStreamOfPipelineStarvingEvent - = S.fromList (coerce consistentInputs) - & detectingPipelineStarving - projectDeterministicallyAndRandomlyABool - withStarvingInvariantWhenTrue - & S.indexed & S.map (first (+1)) -- index starting by 1 and not 0 - liftIO - $ S.drain - $ S.zipAsyncWithM - (\(pipelineStarvingEventOffset,_) (trueOffset,_) -> liftIO $ trueOffset `shouldBe` pipelineStarvingEventOffset) - indexedStreamOfPipelineStarvingEvent - indexedStreamOfTrueEvent - - -projectDeterministicallyAndRandomlyABool :: (Monad m) => SF.Fold m Input Bool -projectDeterministicallyAndRandomlyABool - = SF.Fold - (\(_,stdGen) _ -> do - let (newInt,newGen) = next stdGen - case newInt `mod` 2 of - value | value == 0 -> return (True,newGen) - _ -> return (False,newGen)) - (return (False,mkStdGen 0)) - (return . fst) - - -countTruesAndIndexIt :: (Monad m) => SF.Fold m Bool (Int,Bool) -countTruesAndIndexIt - = (,) - <$> (getSum <$> SF.foldMap - (\case - True -> Sum 1 - False -> Sum 0)) - <*> SF.Fold - (\_ a -> return a) - (return False) - return - - -withStarvingInvariantWhenTrue :: Bool -> Bool -withStarvingInvariantWhenTrue = id \ No newline at end of file diff --git a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/StateSpec.hs b/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/StateSpec.hs deleted file mode 100644 index e03b15c..0000000 --- a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/StateSpec.hs +++ /dev/null @@ -1,60 +0,0 @@ -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE ScopedTypeVariables #-} -{-# LANGUAGE DerivingVia #-} -module Dolla.Consensus.Proposing.Starving.Detecting.StateSpec (spec) where - -import Data.Coerce (coerce) -import Data.Monoid -import Control.Monad.Cont (liftIO) - -import Test.QuickCheck -import Test.QuickCheck.Monadic hiding (assert) -import Test.QuickCheck.Instances () -import Test.Hspec - - -import qualified Streamly.Prelude as S - - -import Dolla.Consensus.Proposing.Starving.Detecting.Input -import Dolla.Consensus.Proposing.Starving.Detecting.GenInput - -import Dolla.Consensus.Proposing.Starving.Detecting.State - - -spec :: Spec -spec = parallel $ - describe "State used for detecting a starving pipeline" $ do - it "provides the number of remaining local proposal not yet consumed" $ property shouldProjectNBRemainingProposalToConsume - it "indicates when a local proposal is asked by the pipeline" $ property shouldProjectIsLocalProposalAsked - - -shouldProjectNBRemainingProposalToConsume :: ConsistentInputs -> Property -shouldProjectNBRemainingProposalToConsume consistentInputs - = monadicIO $ do - let expectedRemainingProposalToConsume = getSum $ delta (coerce consistentInputs) - actualRemainingProposalToConsume <- remainingProposalToConsume <$> S.fold projection (S.fromList (coerce consistentInputs)) - liftIO $ actualRemainingProposalToConsume `shouldBe` expectedRemainingProposalToConsume - -shouldProjectIsLocalProposalAsked :: ConsistentInputs -> Property -shouldProjectIsLocalProposalAsked consistentInputs - = monadicIO $ do - let expectedIsLocalProposalAsked = foldIsLocalProposalAsked (coerce consistentInputs) - actualIsLocalProposalAsked <- isLocalProposalAsked <$> S.fold projection (S.fromList (coerce consistentInputs)) - liftIO $ actualIsLocalProposalAsked `shouldBe` expectedIsLocalProposalAsked - - -delta :: [Input] -> Sum Integer -delta - = foldMap - (\case - LocalProposalConsumed -> - 1 - LocalProposalProduced -> 1 - LocalProposalAsked -> 0) - -foldIsLocalProposalAsked :: [Input] -> Any -foldIsLocalProposalAsked - = foldMap - (\case - LocalProposalAsked -> Any True - _ -> Any False ) \ No newline at end of file diff --git a/packages/detecting-tension/README.md b/packages/detecting-tension/README.md new file mode 100644 index 0000000..078c066 --- /dev/null +++ b/packages/detecting-tension/README.md @@ -0,0 +1,127 @@ +/ [Consensus](https://github.com/dolla-consortium/consensus) / [Proposing](https://github.com/dolla-consortium/consensus-proposing) / Detecting Tension +# Detecting Tension + +- [Overview](#overview) +- [Project Tree](#project-tree) +- [Pipeline](#pipeline) + - [Junction](#junction) + - [IO](#ios) + - [Input/Events](#inputevents) + - [Output/Events](#outputevents) +- [Pipes](#pipes) + - [Detecting Tensed Flow](#detecting-tensed-flow) + +# Overview +`Detecting Tension` is responsible for detecting when the local proposal flow is tensed, meaning if the consensus has consumed more local proposals than being staged. + + ![overview](documentation/media/overview.png) + +# Project Tree +
+### 1. Pipeline + +`DetectingTension` is a simple ***Pipeline*** +- a persisted input stream : [Input.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Input.hs) +- a line of Pipes Welded together in [Pipeline.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Pipeline.hs) + - Sourcing initial inputs : `stream infinitely inputLog` + - `DetectingTensedFlow` : A pipe that detects a tensed flow in [Pipe.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Pipe.hs) using [StateMachine.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachine.hs) + - A Welding : Adapting IOs between pipes + - Sinking final outputs : `sinking outputLog` +- a persisted output stream : [Output.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Output.hs) + +From A CQRS point of view, `DetectingTension` is a projection. It projects outputs from events previously recorded in the system. + +### 2. Execution Environment + +`DetectingTension` is Polymorphic by the Log Engine used. + +You'll find in this folder a concrete pipeline version in [Pipeline.hs](library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Pipeline.hs) over the event store + +### 3. Executable + +`DetectingTension` has some DevOps features as well + +- [Settings.hs](settings/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Settings.hs) always into a separated project `xxxx-detecting-tension-settings` for deployment purposes in Zeus +- [Dependencies.hs](library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Dependencies.hs) are derived from Settings if sub-dependencies are all Healthy + +[Executable.hs](executables/Executables.hs) +- Perform the HealhtChecks to obtain the pipeline dependencies +- Execute the pipeline + load the [junctions](#junction) in the EventStore Microservice +- Put the Microservice back in HealthCheck mode if any Exception bubbles up in the pipeline during execution. + +**N.B** : Microservice configuration and Deployment (Locally/Simulated/Production etc...) are defined in the package [Zeus](../zeus/) + +
+ +# Pipeline +## Junction + +A Junction (Merger) is + - a set of persisted input streams + - a nondeterministic logic for merging these input streams + - a persisted output stream (input of a pipeline) + +The persisted input stream is the junction of 2 upstreams pipelines +- Consening : + - `Local Proposal Accepted` + - `Consensus Reached` +- [Staging](../staging/README.md) with `Local Proposal Staged` + +We are using the "User Defined Projections" EventStore feature to implement this junction + - javaScript snippets + - loaded in the event store microservice directly + - more details : https://eventstore.org/docs/projections/api/index.html + +> Defined in [Junction.hs](library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Junction.hs) + +> Executed in [Executable.hs](executables/Executables.hs) +## IOs +### Input/Events + +The pipeline does not execute commands but just `fold` previous events recorded into the system to provide insights to the downstream flow. +In this situation, it will provide information for optimising the flow +management (purging requests accumulated in the [Staging](../staging/README.md) pipeline) + +```haskell +data Input + = LocalProposalAccepted -- ^ local proposal has been accepted by the consortium + | LocalProposalStaged -- ^ local proposal has been staged by the local staging pipeline + | ConsensusReached -- ^ A global consensus has been reached for the proposals of the current block, + -- All the proposal accepted will be consumed. +``` +> Defined in [Input.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Input.hs) + +### Output/Events + +The pipeline will output +```haskell +data Output = Tensed deriving (Eq, Show +``` + +This output will be sinked and used in the [Staging](../staging/README.md) junction pipeline. + +# Pipes +## Detecting Tensed Flow +### Problem + +Detect if the local proposal flow is tensed, meaning if the consensus has consumed more local proposals than being staged. + +### Approach + +We'll transpose our terminology from the specific consensus domain to a more generic one +```haskell +instance Weldable DetectingTension.Input DetectingTensedFlow.Input where + weld + = \case + DetectingTension.LocalProposalAccepted -> DetectingTensedFlow.Released + DetectingTension.LocalProposalStaged -> DetectingTensedFlow.Staged + DetectingTension.ConsensusReached -> DetectingTensedFlow.Pulled +``` + +A flow is tensed when +- all the items staged have been released and pulled +- we are pulling on `items staged - items release == 0` + +> Implemented in [Pipe.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Pipe.hs) and [StateMachine.hs](library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachine.hs) + +> Tested in [StateMachineSpec.hs](test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachineSpec.hs) diff --git a/packages/detecting-tension/documentation/media/overview.png b/packages/detecting-tension/documentation/media/overview.png new file mode 100644 index 0000000..6a76c34 Binary files /dev/null and b/packages/detecting-tension/documentation/media/overview.png differ diff --git a/packages/detecting-tension/documentation/media/project-tree.png b/packages/detecting-tension/documentation/media/project-tree.png new file mode 100644 index 0000000..5246199 Binary files /dev/null and b/packages/detecting-tension/documentation/media/project-tree.png differ diff --git a/packages/detecting-tension/dolla-consensus-proposing-detecting-tension.cabal b/packages/detecting-tension/dolla-consensus-proposing-detecting-tension.cabal new file mode 100644 index 0000000..668843b --- /dev/null +++ b/packages/detecting-tension/dolla-consensus-proposing-detecting-tension.cabal @@ -0,0 +1,93 @@ +cabal-version: 1.12 + +-- This file has been generated from package.yaml by hpack version 0.33.0. +-- +-- see: https://github.com/sol/hpack +-- +-- hash: 609a42d16391083e1afdec880fd0db8b5ac2ef588d25d18d265cdeb56bc92ea7 + +name: dolla-consensus-proposing-detecting-tension +version: 0.0.0.1 +build-type: Simple +extra-source-files: + README.md + +library + exposed-modules: + Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Dependencies + Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Junction + Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Pipeline + Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Input + Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Output + Dolla.Consensus.Proposing.DetectingTension.Pipeline.Pipeline + Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Input + Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Pipe + Dolla.Consensus.Proposing.DetectingTension.Pipeline.Welding.BluePrint + Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input + Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Output + Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Pipe + Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.StateMachine + other-modules: + Paths_dolla_consensus_proposing_detecting_tension + hs-source-dirs: + library + ghc-options: -Wall + build-depends: + QuickCheck ==2.13.2 + , aeson ==1.4.7.1 + , base ==4.13.0.0 + , bytestring ==0.10.10.0 + , containers ==0.6.2.1 + , directory ==1.3.6.0 + , dolla-base ==1.0.0 + , dolla-consensus-base ==1.0.0 + , dolla-consensus-proposing-detecting-tension-settings + , exceptions ==0.10.4 + , interpolatedstring-perl6 ==1.0.2 + , lens ==4.18.1 + , mtl ==2.2.2 + , streamly ==0.7.2 + , uuid ==1.3.13 + default-language: Haskell2010 + +executable dolla-consensus-proposing-detecting-tension + main-is: Executables.hs + other-modules: + Paths_dolla_consensus_proposing_detecting_tension + hs-source-dirs: + executables + ghc-options: -Wall -main-is Executables.execute -threaded -rtsopts -with-rtsopts=-N + build-depends: + base ==4.13.0.0 + , dolla-base ==1.0.0 + , dolla-consensus-proposing-detecting-tension + , dolla-consensus-proposing-detecting-tension-settings + , mtl ==2.2.2 + , streamly ==0.7.2 + default-language: Haskell2010 + +test-suite dolla-consensus-proposing-detecting-tension-test + type: exitcode-stdio-1.0 + main-is: Spec.hs + other-modules: + Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.GenInput + Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.StateMachineSpec + Paths_dolla_consensus_proposing_detecting_tension + hs-source-dirs: + test + ghc-options: -Wall + build-depends: + QuickCheck + , aeson ==1.4.7.1 + , base + , dolla-consensus-base ==1.0.0 + , dolla-consensus-proposing-detecting-tension + , generic-arbitrary + , hspec + , interpolatedstring-perl6 ==1.0.2 + , mtl ==2.2.2 + , quickcheck-instances + , random ==1.1 + , streamly ==0.7.2 + , uuid ==1.3.13 + default-language: Haskell2010 diff --git a/packages/detecting-tension/executables/Executables.hs b/packages/detecting-tension/executables/Executables.hs new file mode 100644 index 0000000..d034e02 --- /dev/null +++ b/packages/detecting-tension/executables/Executables.hs @@ -0,0 +1,32 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE NamedFieldPuns #-} +module Executables + ( execute + ) where + +import Prelude hiding (log) +import Control.Monad.Reader + +import Dolla.Common.Logging.Core +import Dolla.Common.Executable.Executable + +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Settings +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Dependencies +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Junction (loadJunctionInEventStore) + +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Pipeline (measuringTension) +import qualified Streamly.Prelude as S + +execute :: IO () +execute = executeMicroservice + (\Settings {logger} -> logger) + executePipeline + where + executePipeline :: ReaderT Dependencies IO () + executePipeline + = do + Dependencies {eventStoreClient,nodeId,logger} <- ask + withReaderT (const (nodeId, eventStoreClient)) loadJunctionInEventStore + log logger INFO "Starting Service" + S.drain measuringTension + log logger INFO "Service Down" diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Dependencies.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Dependencies.hs similarity index 82% rename from packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Dependencies.hs rename to packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Dependencies.hs index 99fc18e..57ef7ff 100644 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Dependencies.hs +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Dependencies.hs @@ -1,14 +1,15 @@ {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE MultiParamTypeClasses #-} -module Dolla.Consensus.Proposing.Starving.Detecting.Dependencies (Dependencies (..)) where +module Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Dependencies + (Dependencies (..)) where import Dolla.Common.NodeId import Dolla.Common.Dependencies.Core import Dolla.Common.Logging.Core import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStore -import Dolla.Consensus.Proposing.Starving.Detecting.Settings +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Settings data Dependencies = Dependencies diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/ESMerger.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Junction.hs similarity index 66% rename from packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/ESMerger.hs rename to packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Junction.hs index 5d7c2fa..6a5126a 100644 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/ESMerger.hs +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Junction.hs @@ -1,7 +1,8 @@ {-# LANGUAGE QuasiQuotes, ExtendedDefaultRules #-} {-# LANGUAGE RecordWildCards #-} -module Dolla.Consensus.Proposing.Starving.Detecting.ESMerger (loadInputProjection) where +module Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Junction + (loadJunctionInEventStore) where import Prelude hiding (log) import Data.Coerce (coerce) @@ -14,23 +15,29 @@ import Dolla.Common.Logging.Core import Dolla.Libraries.LogEngine.Instances.EventStore.Projection.Definition import Dolla.Consensus.Log.LogNameIndex +-- | A Junction (Merger) is + --a set of persisted input streams + --a nondeterministic logic for merging these input streams + --a persisted output stream (input of a pipeline) --- | User Defined Projection loaded into the event store for dispatching RB Broadcast Merged output and Result of Voting --- events into the proper input stream for the maestro --- * for more details about event store projection : https://eventstore.org/docs/projections/api/index.html -loadInputProjection :: ReaderT (NodeId, EventStore.Dependencies) IO () -loadInputProjection = do +-- | Junction for Starving Detection Pipeline : Logic to generate the input stream of the pipeline. +-- We are using the "User Defined Projections" feature from the EventStore to implement this junction : +-- - javaScript snippets +-- - loaded in the event store microservice directly +-- - more details : https://eventstore.org/docs/projections/api/index.html +loadJunctionInEventStore :: ReaderT (NodeId, EventStore.Dependencies) IO () +loadJunctionInEventStore = do (nodeId , EventStore.Dependencies {..}) <- ask - let projectionName = coerce nodeId ++ "_proposing_starving_detecting_input" + let projectionName = coerce nodeId ++ "_proposing_starving_detection_input" maestroOutputMergedLogStreamName = getStreamNameFromIndex MaestroOutputMergedLog - proposingPackagingOutputLogStreamName = getStreamNameFromIndex ProposingPackagingOutputLog + proposingStagingOutputLogStreamName = getStreamNameFromIndex ProposingStagingOutputLog proposingStarvingDetectionInputLogStreamName = getStreamNameFromIndex ProposingStarvingDetectionInputLog body = [qc| options(\{ reorderEvents: false, processingLag: 0 }) fromStreams ([ '{maestroOutputMergedLogStreamName}' - , '{proposingPackagingOutputLogStreamName}']) + , '{proposingStagingOutputLogStreamName}']) .when(\{ $any : function(s,e)\{ function getOutputStream() \{ @@ -40,21 +47,21 @@ loadInputProjection = do var messageJson = JSON.parse(e.bodyRaw) if (messageJson.byProposer.proposerId == {nodeId}) \{ emit ( getOutputStream () - , "LocalProposalConsumed" - , "LocalProposalConsumed" + , "LocalProposalAccepted" + , "LocalProposalAccepted" , \{}); } } if (e.eventType == "ConsensusReached" ) \{ emit ( getOutputStream () - , "LocalProposalAsked" - , "LocalProposalAsked" + , "ConsensusReached" + , "ConsensusReached" , \{}); } - if (e.eventType == "LocalProposalProduced" ) \{ + if (e.eventType == "LocalProposalStaged" ) \{ emit ( getOutputStream () - , "LocalProposalProduced" - , "LocalProposalProduced" + , "LocalProposalStaged" + , "LocalProposalStaged" , \{}); } // if } // $any diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Pipeline.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Pipeline.hs new file mode 100644 index 0000000..0aeb414 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Pipeline.hs @@ -0,0 +1,32 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE NamedFieldPuns #-} +{-# LANGUAGE FlexibleContexts #-} +module Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Pipeline + (measuringTension) where + +import Prelude hiding (log) +import Control.Monad.Reader +import Control.Monad.Catch (MonadCatch) + +import qualified Streamly as S + +import Dolla.Consensus.Log.EventStoreLog +import Dolla.Consensus.Log.LogNameIndex + +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Dependencies + +import qualified Dolla.Consensus.Proposing.DetectingTension.Pipeline.Pipeline as Generics + +measuringTension + :: ( S.MonadAsync m + , MonadReader Dependencies m + , MonadCatch m) + => S.SerialT m () +measuringTension = + ask + >>= \Dependencies {eventStoreClient} -> + Generics.measuringTension + (getEventStoreLog eventStoreClient ProposingStarvingDetectionInputLog) + (getEventStoreLog eventStoreClient ProposingStarvingDetectionOutputLog) + + diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Input.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Input.hs new file mode 100644 index 0000000..8a71ae7 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Input.hs @@ -0,0 +1,22 @@ +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DerivingVia #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE UndecidableInstances #-} +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE DeriveGeneric #-} +module Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Input + ( Input (..)) where + +import Data.Aeson +import GHC.Generics +import Dolla.Adapter.Aeson.AesonVia + + +data Input + = LocalProposalAccepted + | LocalProposalStaged + | ConsensusReached + deriving (Eq,Show, Generic) + deriving (ToJSON,FromJSON) via DefaultJSON Input + + diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Output.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Output.hs similarity index 54% rename from packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Output.hs rename to packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Output.hs index 492e1f7..b277c08 100644 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Output.hs +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/IO/Output.hs @@ -4,26 +4,22 @@ {-# LANGUAGE UndecidableInstances #-} {-# LANGUAGE DuplicateRecordFields #-} {-# LANGUAGE DeriveGeneric #-} -{-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Starving.Detecting.Output +module Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Output (Output (..)) where import Data.Aeson -import Data.UUID (UUID) import GHC.Generics import Dolla.Adapter.Aeson.AesonVia import Dolla.Libraries.LogEngine.Appendable -import Dolla.Common.UUID.Provider -newtype Output - = LocalProposalStarvationDetected {itemId :: UUID} +data Output + = LocalProposalFlowTensed deriving (Eq, Show, Generic) deriving (ToJSON, FromJSON) via DefaultJSON Output instance Appendable Output where - getItemName (LocalProposalStarvationDetected _ ) = "LocalProposalStarvationDetected" + getItemName LocalProposalFlowTensed = "LocalProposalFlowTensed" + -instance UUIDProvider Output where - getUUID LocalProposalStarvationDetected {itemId} = itemId diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Pipeline.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Pipeline.hs new file mode 100644 index 0000000..a029f4b --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Pipeline.hs @@ -0,0 +1,36 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE MonoLocalBinds #-} + +module Dolla.Consensus.Proposing.DetectingTension.Pipeline.Pipeline + (measuringTension) where + +import Prelude hiding (log,writeFile) + +import Control.Monad.Catch (MonadCatch) + +import qualified Streamly as S + +import Dolla.Common.Range + +import Dolla.Libraries.LogEngine.LogEngine + + +import Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Input +import Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Output +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Pipe (detectTensedFlow) +import Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Pipe (sinking) +import Dolla.Consensus.Proposing.DetectingTension.Pipeline.Welding.BluePrint () +import Dolla.Common.Pipeline.Weldable ((~>),(.~>)) + +measuringTension + :: ( MemoryStreamLoggable m log + , S.MonadAsync m + , MonadCatch m) + => log Input + -> log Output + -> S.SerialT m () +measuringTension inputLog outputLog + = stream infinitely inputLog -- sourcing + ~> detectTensedFlow + .~> sinking outputLog diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Sinking/Input.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Sinking/Input.hs new file mode 100644 index 0000000..21e8db9 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Sinking/Input.hs @@ -0,0 +1,13 @@ +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DerivingVia #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE UndecidableInstances #-} +{-# LANGUAGE DuplicateRecordFields #-} +module Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Input + ( Input (..)) where + +data Input = SinkLocalProposalFlowTensed + + + + diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Sinking/Pipe.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Sinking/Pipe.hs new file mode 100644 index 0000000..bc658bb --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Sinking/Pipe.hs @@ -0,0 +1,33 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE MonoLocalBinds #-} + +module Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Pipe + (sinking) where + +import Prelude hiding (log,writeFile) + +import Data.Function ((&)) + +import Control.Monad.Reader + +import qualified Streamly as S +import qualified Streamly.Prelude as S + +import Dolla.Libraries.LogEngine.LogEngine +import Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Input +import Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Output + +import Dolla.Common.Offset (fromIntegralToOffset) + +sinking + :: ( MemoryStreamLoggable m log + , S.MonadAsync m) + => log Output + -> S.SerialT m Input + -> S.SerialT m () +sinking outputLog inputStream + = inputStream + & S.indexed + & S.map (fromIntegralToOffset . fst ) + & S.mapM (\offset -> void $ append outputLog offset LocalProposalFlowTensed) \ No newline at end of file diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Welding/BluePrint.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Welding/BluePrint.hs new file mode 100644 index 0000000..dc7d7d8 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipeline/Welding/BluePrint.hs @@ -0,0 +1,31 @@ +{-# LANGUAGE LambdaCase #-} +{-# LANGUAGE MultiParamTypeClasses #-} +{-# LANGUAGE FlexibleInstances #-} + +{-# OPTIONS_GHC -fno-warn-orphans #-} + +module Dolla.Consensus.Proposing.DetectingTension.Pipeline.Welding.BluePrint + () where + +import Dolla.Common.Pipeline.Weldable + + +import qualified Dolla.Consensus.Proposing.DetectingTension.Pipeline.IO.Input as DetectingTension + +import qualified Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input as DetectingTensedFlow +import qualified Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Output as DetectingTensedFlow + +import qualified Dolla.Consensus.Proposing.DetectingTension.Pipeline.Sinking.Input as Sinking + +instance Weldable DetectingTension.Input DetectingTensedFlow.Input where + weld + = \case + DetectingTension.LocalProposalAccepted -> DetectingTensedFlow.Released + DetectingTension.LocalProposalStaged -> DetectingTensedFlow.Staged + DetectingTension.ConsensusReached -> DetectingTensedFlow.Pulled + +instance Weldable DetectingTensedFlow.Output Sinking.Input where + weld + = \case + DetectingTensedFlow.Tensed -> Sinking.SinkLocalProposalFlowTensed + diff --git a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Input.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Input.hs similarity index 75% rename from packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Input.hs rename to packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Input.hs index a7d4927..cbd98b9 100644 --- a/packages/detecting-starvation/lib/Dolla/Consensus/Proposing/Starving/Detecting/Input.hs +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Input.hs @@ -4,18 +4,17 @@ {-# LANGUAGE UndecidableInstances #-} {-# LANGUAGE DuplicateRecordFields #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Starving.Detecting.Input +module Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input ( Input (..)) where import Data.Aeson import GHC.Generics import Dolla.Adapter.Aeson.AesonVia - data Input - = LocalProposalConsumed - | LocalProposalProduced - | LocalProposalAsked + = Staged + | Released + | Pulled deriving (Eq,Show, Generic) deriving (ToJSON,FromJSON) via DefaultJSON Input diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Output.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Output.hs new file mode 100644 index 0000000..b916169 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Output.hs @@ -0,0 +1,12 @@ +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE DerivingVia #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE UndecidableInstances #-} +{-# LANGUAGE DuplicateRecordFields #-} + +module Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Output + (Output (..)) where + +data Output = Tensed deriving (Eq, Show) + + diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Pipe.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Pipe.hs new file mode 100644 index 0000000..a466741 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/Pipe.hs @@ -0,0 +1,41 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE FlexibleContexts #-} + +module Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Pipe + (detectTensedFlow ) where + +import Prelude hiding (log,writeFile) + +import Data.Function ((&)) +import qualified Streamly as S +import qualified Streamly.Prelude as S +import qualified Streamly.Internal.Data.Fold as SF + +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Output + +import qualified Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.StateMachine as State + +detectTensedFlow + :: S.MonadAsync m + => S.SerialT m Input + -> S.SerialT m Output +detectTensedFlow + = emitWhenPredicateValid + State.projection + State.isFlowTensed + +emitWhenPredicateValid + :: S.MonadAsync m + => SF.Fold m Input state + -> (state -> Bool) + -> S.SerialT m Input + -> S.SerialT m Output +emitWhenPredicateValid + projection + predicate + inputStream + = inputStream + & S.postscan projection + & S.filter predicate + & S.map (const Tensed) \ No newline at end of file diff --git a/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachine.hs b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachine.hs new file mode 100644 index 0000000..4cc1f10 --- /dev/null +++ b/packages/detecting-tension/library/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachine.hs @@ -0,0 +1,53 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE LambdaCase #-} +{-# LANGUAGE MonoLocalBinds #-} + +module Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.StateMachine + ( isFlowTensed + , projection + , State (..) + ) where + +import Prelude hiding (log,writeFile) + +import Data.Monoid +import qualified Streamly.Internal.Data.Fold as SF +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input + + +type RemainingProposalToConsume = Integer + +data State + = State + { itemsInTransit :: Integer + , isPulled :: Any} deriving Eq + +isFlowTensed :: State -> Bool +isFlowTensed = (== State 0 (Any True)) + +projection + :: (Monad m) + => SF.Fold m Input State +projection + = State <$> foldItemsInTransit + <*> foldIsPulled + + +foldIsPulled :: (Monad m , Monoid Any) => SF.Fold m Input Any +foldIsPulled + = SF.foldMap + (\case + Pulled -> Any True + _ -> Any False ) + +foldItemsInTransit :: (Monad m ) => SF.Fold m Input RemainingProposalToConsume +foldItemsInTransit + = SF.lmapM + (\inputItem -> do + return $ case inputItem of + Staged -> 1 + Released -> -1 + Pulled -> 0) + SF.sum + diff --git a/packages/detecting-starvation/package.yaml b/packages/detecting-tension/package.yaml similarity index 65% rename from packages/detecting-starvation/package.yaml rename to packages/detecting-tension/package.yaml index 1595a18..ae87581 100644 --- a/packages/detecting-starvation/package.yaml +++ b/packages/detecting-tension/package.yaml @@ -1,21 +1,19 @@ -name: dolla-consensus-proposing-detecting-starvation +name: dolla-consensus-proposing-detecting-tension version: 0.0.0.1 extra-source-files: - README.md ghc-options: - -Wall -## - -j library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 - dolla-consensus-base == 1.0.0 - - dolla-consensus-proposing-detecting-starvation-settings - + - dolla-consensus-proposing-detecting-tension-settings - base == 4.13.0.0 - containers == 0.6.2.1 - mtl == 2.2.2 @@ -30,28 +28,28 @@ library: - uuid == 1.3.13 executables: - dolla-consensus-proposing-detecting-starvation : + dolla-consensus-proposing-detecting-tension : main: Executables.hs - source-dirs: app + source-dirs: executables ghc-options: - - -main-is Executables.detectingStarvation + - -main-is Executables.execute - -threaded - -rtsopts - -with-rtsopts=-N dependencies: - - dolla-consensus-proposing-detecting-starvation - base == 4.13.0.0 + - mtl == 2.2.2 + - streamly == 0.7.2 + - dolla-base == 1.0.0 + - dolla-consensus-proposing-detecting-tension + - dolla-consensus-proposing-detecting-tension-settings tests: - dolla-consensus-proposing-detecting-starvation-test: + dolla-consensus-proposing-detecting-tension-test: main: Spec.hs source-dirs: test - ghc-options: - - -threaded - - -rtsopts - - -with-rtsopts=-N dependencies: - - dolla-consensus-proposing-detecting-starvation + - dolla-consensus-proposing-detecting-tension - base - streamly == 0.7.2 - mtl == 2.2.2 diff --git a/packages/detecting-starvation/settings/dolla-consensus-proposing-detecting-starvation-settings.cabal b/packages/detecting-tension/settings/dolla-consensus-proposing-detecting-tension-settings.cabal similarity index 54% rename from packages/detecting-starvation/settings/dolla-consensus-proposing-detecting-starvation-settings.cabal rename to packages/detecting-tension/settings/dolla-consensus-proposing-detecting-tension-settings.cabal index 0910d03..2a0210b 100644 --- a/packages/detecting-starvation/settings/dolla-consensus-proposing-detecting-starvation-settings.cabal +++ b/packages/detecting-tension/settings/dolla-consensus-proposing-detecting-tension-settings.cabal @@ -4,21 +4,19 @@ cabal-version: 1.12 -- -- see: https://github.com/sol/hpack -- --- hash: 2308c0fda4c0467a24caeb206edfc0505aa66a224cb54e0eb1170950c951aef4 +-- hash: bc0ae701896d104ed9c0876f472d6f16ea087264c1844b60f8e2d000002cb6f3 -name: dolla-consensus-proposing-detecting-starvation-settings +name: dolla-consensus-proposing-detecting-tension-settings version: 0.0.0 build-type: Simple -extra-source-files: - README.md library exposed-modules: - Dolla.Consensus.Proposing.Starving.Detecting.Settings + Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Settings other-modules: - Paths_dolla_consensus_proposing_detecting_starvation_settings + Paths_dolla_consensus_proposing_detecting_tension_settings hs-source-dirs: - lib + library ghc-options: -Wall build-depends: aeson ==1.4.7.1 diff --git a/packages/detecting-starvation/settings/lib/Dolla/Consensus/Proposing/Starving/Detecting/Settings.hs b/packages/detecting-tension/settings/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Settings.hs similarity index 82% rename from packages/detecting-starvation/settings/lib/Dolla/Consensus/Proposing/Starving/Detecting/Settings.hs rename to packages/detecting-tension/settings/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Settings.hs index 53ac194..d6fceee 100644 --- a/packages/detecting-starvation/settings/lib/Dolla/Consensus/Proposing/Starving/Detecting/Settings.hs +++ b/packages/detecting-tension/settings/library/Dolla/Consensus/Proposing/DetectingTension/Execution/Environment/EventStore/Settings.hs @@ -1,6 +1,6 @@ {-# LANGUAGE DerivingVia #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Starving.Detecting.Settings (Settings (..)) where +module Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Settings (Settings (..)) where import Data.Aeson import GHC.Generics diff --git a/packages/detecting-starvation/settings/package.yaml b/packages/detecting-tension/settings/package.yaml similarity index 50% rename from packages/detecting-starvation/settings/package.yaml rename to packages/detecting-tension/settings/package.yaml index afcbdbf..a536dd2 100644 --- a/packages/detecting-starvation/settings/package.yaml +++ b/packages/detecting-tension/settings/package.yaml @@ -1,13 +1,10 @@ -name: dolla-consensus-proposing-detecting-starvation-settings -extra-source-files: - - README.md +name: dolla-consensus-proposing-detecting-tension-settings ghc-options: - -Wall -## - -j library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 diff --git a/packages/detecting-starvation/settings/test/Spec.hs b/packages/detecting-tension/settings/test/Spec.hs similarity index 100% rename from packages/detecting-starvation/settings/test/Spec.hs rename to packages/detecting-tension/settings/test/Spec.hs diff --git a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/GenInput.hs b/packages/detecting-tension/test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/GenInput.hs similarity index 60% rename from packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/GenInput.hs rename to packages/detecting-tension/test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/GenInput.hs index 8827d13..f436009 100644 --- a/packages/detecting-starvation/test/Dolla/Consensus/Proposing/Starving/Detecting/GenInput.hs +++ b/packages/detecting-tension/test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/GenInput.hs @@ -1,6 +1,6 @@ {-# LANGUAGE LambdaCase #-} -module Dolla.Consensus.Proposing.Starving.Detecting.GenInput +module Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.GenInput ( ConsistentInputs (..) , InputUnderTests (..)) where @@ -11,15 +11,15 @@ import Test.QuickCheck.Arbitrary import Test.QuickCheck.Gen import Test.QuickCheck.Instances.UUID () -import Dolla.Consensus.Proposing.Starving.Detecting.Input +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input instance Arbitrary InputUnderTests where arbitrary = InputUnderTests <$> oneof - [ return LocalProposalConsumed - , return LocalProposalProduced - , return LocalProposalAsked] + [ return Staged + , return Released + , return Pulled] newtype InputUnderTests = InputUnderTests { unInputUnderTests :: Input} deriving Show @@ -29,13 +29,13 @@ instance Arbitrary ConsistentInputs where arbitrary = suchThat (ConsistentInputs <$> listOf arbitrary) - (\inputs -> delta (coerce inputs) > 0) + (\inputs -> itemInTransit (coerce inputs) > 0) -delta :: [Input] -> Sum Integer -delta +itemInTransit :: [Input] -> Sum Integer +itemInTransit = foldMap (\case - LocalProposalConsumed -> - 1 - LocalProposalProduced -> 1 - LocalProposalAsked -> 0) + Released -> - 1 + Staged -> 1 + Pulled -> 0) diff --git a/packages/detecting-tension/test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachineSpec.hs b/packages/detecting-tension/test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachineSpec.hs new file mode 100644 index 0000000..8197009 --- /dev/null +++ b/packages/detecting-tension/test/Dolla/Consensus/Proposing/DetectingTension/Pipes/DetectingTensedFlow/StateMachineSpec.hs @@ -0,0 +1,67 @@ +{-# LANGUAGE LambdaCase #-} +{-# LANGUAGE ScopedTypeVariables #-} +{-# LANGUAGE DerivingVia #-} +{-# LANGUAGE NamedFieldPuns #-} +module Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.StateMachineSpec (spec) where + +import Data.Coerce (coerce) +import Data.Monoid + +import Test.QuickCheck +import Test.QuickCheck.Instances () +import Test.Hspec + + +import qualified Streamly.Prelude as S +import qualified Streamly as S + +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.Input +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.GenInput + +import Dolla.Consensus.Proposing.DetectingTension.Pipes.DetectingTensedFlow.StateMachine + + +spec :: Spec +spec = parallel $ + describe "State Machine used for detecting when a flow is tensed" $ do + it "provides the number of items in transit withing the flow" + $ property + $ \inputs -> do + let expectedItemsInTransit = getExpectedItemsInTransit inputs + State {itemsInTransit} <- S.fold projection (getGeneratedInputStream inputs) + itemsInTransit `shouldBe` expectedItemsInTransit + it "indicates when an item is pulled" + $ property + $ \inputs -> do + let expectedIsPulled = getExpectedIsPulled inputs + State{isPulled} <- S.fold projection (getGeneratedInputStream inputs) + isPulled `shouldBe` expectedIsPulled + +getGeneratedInputStream + :: Monad m + => ConsistentInputs + -> S.SerialT m Input +getGeneratedInputStream consistentInputs = S.fromList (coerce consistentInputs) + +getExpectedItemsInTransit :: ConsistentInputs -> Integer +getExpectedItemsInTransit inputs + = getSum $ foldItemsInTransit (coerce inputs) + where + foldItemsInTransit :: [Input] -> Sum Integer + foldItemsInTransit + = foldMap + (\case + Released -> - 1 + Staged -> 1 + Pulled -> 0) + +getExpectedIsPulled :: ConsistentInputs -> Any +getExpectedIsPulled inputs + = foldIsPulled (coerce inputs) + where + foldIsPulled :: [Input] -> Any + foldIsPulled + = foldMap + (\case + Pulled -> Any True + _ -> Any False ) \ No newline at end of file diff --git a/packages/detecting-starvation/test/Spec.hs b/packages/detecting-tension/test/Spec.hs similarity index 100% rename from packages/detecting-starvation/test/Spec.hs rename to packages/detecting-tension/test/Spec.hs diff --git a/packages/packaging/README.md b/packages/packaging/README.md deleted file mode 100644 index c20bf2a..0000000 --- a/packages/packaging/README.md +++ /dev/null @@ -1,283 +0,0 @@ -/ [Consensus](https://github.com/dolla-consortium/consensus) / [Proposing](https://github.com/dolla-consortium/consensus-proposing) / [Packaging](#packaging-pipeline) -# Packaging Pipeline -- [Overview](#overview) -- [Pipeline](#pipeline) - - [IOs](#ios) - - [Input/Commands](#inputcommands) - - [Output/Events](#outputevents) - - [Pipe recipe](#pipe-recipe) - - [Polymorphism](#polymorphism) -- [Pipes](#pipes) - - [Serializing](#serializing) - - [NonEmptying](#nonemptying) - - [Capping](#capping) - - [Persisting](#persisting) - - -# Overview - -
-`Packaging` is a ***Pipeline*** -- a persisted input stream : [Input.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Input.hs) -- a line of Pipes Welded together - - composition of **deterministic** *Pipes* - - Welding : Adapting IOs between pipes - - ``` - serializing .~> nonEmptying .~> capping .~> persisting - ``` -- a persisted output stream : [Output.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Output.hs) - -`Serializing, nonEmptying, capping, persisting` are **Pipes**, meaning each of them has -- An Input Stream -- A Stream Processing -- An Output Stream - -`Packaging` is also a **Microservice** -- [Settings.hs](settings/lib/Dolla/Consensus/Proposing/Packaging/Settings.hs) always into a separated project `xxxx-packaging-settings` for deployment purposes in Zeus -- [Dependencies.hs](lib/Dolla/Consensus/Proposing/Packaging/Dependencies.hs) are derived from Settings if sub-dependencies are all Healthy -- [Microservice.hs](lib/Dolla/Consensus/Proposing/Packaging/Microservice.hs) - - Perform the HealhtChecks to obtain the pipeline dependencies - - Execute the pipeline - - Put the Microservice back in HealthCheck mode if any Exception bubbles up in the pipeline during execution. -- Deployment Logic in Zeus : [Pipeline.hs](../zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs) -
- -# Pipeline -## IOs -### Input/Commands - -> Defined in [Input.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Input.hs) - -The persisted input stream is the junction of 2 upstreams pipelines -- [Receptioning](../receptioning/README.md) providing collected requests -- [Detecting-Starvation](../detecting-starvation/README.md) transmitting Local-Proposal-Starvation Notifications - -#### 1. Package request - -Executing that command means -- Accumulate each request into a temporary file. - -``` -FileName : x.tmp with x the offset of local proposal produced -``` - -- Detect when the temporary file is full - -``` -Current Size + new request size < configurable size limit -``` -- Convert this temporary file into a proposal file -``` -x.tmp -> x.proposal -``` -- Redirect the stream into a new Temporary File : `(x+1).tmp` -- Notify the downstream Broadcasting Section that a new local proposal is available. - -#### 2. ForceProposalProduction - -Executing that command means : -- Converting the temporary file into a proposal file if requests are accumulated. - -### Output/Events - -> Defined [Output.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Output.hs) - -Packaging produces -- A Notification : `LocalProposalProduced {localOffset :: Offset}` -- a File `{localOffset}.proposal` containing requests with -``` -0 < Size <= Size Limit -``` -## Pipe recipe - -To produce the expected pipeline output , we are combining different pipes all together by -- A simple function composition (.) -- A welding : `map` to adapt `Output Pipe(x)` with `Input Pipe(x+1)` - -The `Packaging` pipe recipe is -```haskell - stream infinitely inputLog - ~> serializing - .~> nonEmptying - .~> capping proposalSizeLimit - .~> persisting proposalRootFolder - .~> notifying outputLog -``` -> Defined in [Generic.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Generic.hs) - -> The welding between each pipe is defined in [/Welding/BluePrint.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Welding/BluePrint.hs) - -The pipe recipe goals are -### 1. Size properly the proposals -``` -0 < Size <= Size Limit -``` -Under the responsibility of -- [`NonEmptying`](#nonemptying) Pipe -- [`Capping`](#capping) Pipe - -### 2. Convert the temporary file to a proposal file. -Under the responsibility of -- [`Persisting`](#persisting) Pipe - -N.B : [`Serializing`](#serializing) will be removed eventually. We'll evaluate this when addressing data compression. - -## Polymorphism - -The Generic Pipeline is agnostic from -- The log Engine used to retrieve Input and persist Output -- The structure of the request manipulated - -To execute `Packaging`, we need to turn [Generic.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Generic.hs) into a concrete Pipeline -- [OverEventStore.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStore.hs) : EventStore is the log Engine used (Polymorphism reduced) -- [OverEventStoreAndDolla.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStoreAndDolla.hs) : We are using Dummy Dolla Requests with the EventStore (Concrete Pipeline) - -# Pipes -## Serializing - -> Defined in [Pipe.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Pipe.hs) - -Just Transform `request` in `SerializedRequest` - -```haskell -newtype SerializedRequest = SerializedRequest [Word8] deriving (Eq,Show) -``` - -> N.B : `serializing` will be removed eventually. We'll address it when we'll add the request compression features. - -## NonEmptying - -### Problem - -Remembering the initial input of the section -```haskell -data Input request - = ForceProposalProduction -- ^ ask to "Packaging Pipeline" to flush all the requests currently collected - | Package request -- ^ ask to to "Packaging Pipeline" to package the request into a proposal according - -- some properties (see README.md) - deriving (Eq,Show) - - -``` -It's totally natural to receive multiple `ForceProposalProduction` commands consecutively, E.g -- No Requests while many blocks are appended consecutively -- Etc... - -Forcing the production of empty proposal adds no value in our domain and provokes an accidental complexity downstream if not managed. -Therefore, we want to -- Remove the consecutive `ForceProposalProduction` commands from our input stream. -- Never start downstream processing with a `ForceProposalProduction` - -Said differently, we want to get the following property -``` -0 < Proposal File Size -``` - -### Approach - -With the following Natural Transformation -``` haskell -Packaging.Input request ~> Maybe request -``` -We want the following stream property -- Never start by `Nothing` -- Never 2 consecutive `Nothing` - -E.g - We want the following transformation -``` haskell -[Nothing, Just r1, Just r2, Nothing, Nothing, Nothing] -> [Just r1, Just r2, Nothing] -``` -> Implemented in [Pipe.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Pipe.hs) - -> Tested in [PipeSpec.hs](test/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/PipeSpec.hs) - -## Capping - -### Problem - -We can't broadcast proposal files with unlimited memory size. We want the following property - -``` -Proposal File Size <= Size Limit -``` - -`Capping` is welded after `NonEmptying`, combining the 2 pipes will give the following property -``` -0 < Proposal File Size <= Size Limit -``` - - -### Approach - -To increase the expressivity of our previous domain we are doing the following natural transformation -```haskell -instance Weldable (NonEmptying.Output request) (Capping.Input request) where - weld - = \case - Nothing -> Capping.AskForACut - Just request -> Capping.Add request -``` -With the Following Output in [Output.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Output.hs) -```haskell -data Output request - = Cut - | Added request - deriving (Show,Eq) -``` -the polymorphism on `request` is reduced with the following type Class - -```haskell -class Sizable a where - getMemorySize :: a -> Byte - -capping - :: ( S.MonadAsync m - , Sizable item) - => Byte -- ^ proposalSizeLimit - -> S.SerialT m (Input item) - -> S.SerialT m (Output item) -```` -We are cutting the consecutiveness of `Added request` events if the cumulative memory size of requests is > a size limit given. -By "cutting", we mean adding a `Cut` in between 2 `Added request` events. - -Using a `Fold executed with a postscan` with the following State Machine - -![state-machine](documentation/media/state-machine.png) - -> Implemented in [Pipe.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Pipe.hs) - -> Tested in [PipeSpec.hs](test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/PipeSpec.hs) - -## Persisting - -### Prerequesites -This pipe prerequesites the following properties on its stream input -``` -0 < Proposal File Size <= Size Limit -``` -### Input/Output -```haskell -data Input request - = CommitProposal - | Persist request - deriving (Show,Eq,Functor) - -newtype Output - = LocalProposalPersisted {proposalId :: Offset} - deriving (Show,Eq) -``` -### Processing - -Using Streamly FileSystem primitives - -- Create a temporary file : `x.tmp` -- Persist request -- When receiving `CommitProposal` - - Convert `x.tmp` into `x.proposal` - - produce `LocalProposalPersisted x` - - `x = x + 1` - -> Implemented in [Pipe.hs](lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Pipe.hs) - -> Integration Test in [PipeSpec.hs](test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/PipeSpec.hs) \ No newline at end of file diff --git a/packages/packaging/app/Executables.hs b/packages/packaging/app/Executables.hs deleted file mode 100644 index 952dd6c..0000000 --- a/packages/packaging/app/Executables.hs +++ /dev/null @@ -1,10 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -module Executables - ( packaging - ) where - -import qualified Dolla.Consensus.Proposing.Packaging.Microservice as Packaging - -packaging :: IO () -packaging = Packaging.execute - diff --git a/packages/packaging/documentation/media/package-tree.png b/packages/packaging/documentation/media/package-tree.png deleted file mode 100644 index 7a90331..0000000 Binary files a/packages/packaging/documentation/media/package-tree.png and /dev/null differ diff --git a/packages/packaging/documentation/media/project-tree.png b/packages/packaging/documentation/media/project-tree.png deleted file mode 100644 index f54cf0b..0000000 Binary files a/packages/packaging/documentation/media/project-tree.png and /dev/null differ diff --git a/packages/packaging/dolla-consensus-proposing-packaging.cabal b/packages/packaging/dolla-consensus-proposing-packaging.cabal deleted file mode 100644 index 251c3af..0000000 --- a/packages/packaging/dolla-consensus-proposing-packaging.cabal +++ /dev/null @@ -1,118 +0,0 @@ -cabal-version: 1.12 - --- This file has been generated from package.yaml by hpack version 0.33.0. --- --- see: https://github.com/sol/hpack --- --- hash: 68a075e23f5b1ed553239bc60047735e308c888fb432c3f233c4b556d924d700 - -name: dolla-consensus-proposing-packaging -version: 0.0.0.1 -build-type: Simple -extra-source-files: - README.md - -library - exposed-modules: - Dolla.Consensus.Proposing.Packaging.Dependencies - Dolla.Consensus.Proposing.Packaging.Microservice - Dolla.Consensus.Proposing.Packaging.Pipeline.Generic - Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input - Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output - Dolla.Consensus.Proposing.Packaging.Pipeline.OverEventStore - Dolla.Consensus.Proposing.Packaging.Pipeline.OverEventStoreAndDolla - Dolla.Consensus.Proposing.Packaging.Pipeline.Welding.BluePrint - Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Input - Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Output - Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Pipe - Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Sizable - Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Input - Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Output - Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Pipe - Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Input - Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Pipe - Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Input - Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Output - Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Pipe - Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input - Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Output - Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Pipe - Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.SerializedRequest - other-modules: - Paths_dolla_consensus_proposing_packaging - hs-source-dirs: - lib - ghc-options: -Wall - build-depends: - aeson ==1.4.7.1 - , base ==4.13.0.0 - , bytestring ==0.10.10.0 - , containers ==0.6.2.1 - , directory ==1.3.6.0 - , dolla-base ==1.0.0 - , dolla-consensus-base ==1.0.0 - , dolla-consensus-proposing-packaging-settings - , exceptions ==0.10.4 - , interpolatedstring-perl6 ==1.0.2 - , lens ==4.18.1 - , mtl ==2.2.2 - , natural-transformation - , random ==1.1 - , streamly ==0.7.2 - , text ==1.2.4.0 - , uuid ==1.3.13 - default-language: Haskell2010 - -executable dolla-consensus-proposing-packaging - main-is: Executables.hs - other-modules: - Paths_dolla_consensus_proposing_packaging - hs-source-dirs: - app - ghc-options: -Wall -main-is Executables.packaging -threaded -rtsopts -with-rtsopts=-N - build-depends: - base ==4.13.0.0 - , dolla-consensus-proposing-packaging - default-language: Haskell2010 - -test-suite dolla-consensus-proposing-packaging-test - type: exitcode-stdio-1.0 - main-is: Spec.hs - other-modules: - Dolla.Consensus.Proposing.Packaging.DummyRequest - Dolla.Consensus.Proposing.Packaging.Pipeline.IO.GenInput - Dolla.Consensus.Proposing.Packaging.Pipeline.IO.GenOutput - Dolla.Consensus.Proposing.Packaging.Pipeline.IO.InputSpec - Dolla.Consensus.Proposing.Packaging.Pipeline.IO.OutputSpec - Dolla.Consensus.Proposing.Packaging.Pipes.Capping.GenInput - Dolla.Consensus.Proposing.Packaging.Pipes.Capping.PipeSpec - Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.PipeSpec - Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.GenInput - Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.PipeSpec - Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.GenInput - Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.PipeSpec - Paths_dolla_consensus_proposing_packaging - hs-source-dirs: - test - ghc-options: -Wall -threaded -rtsopts -with-rtsopts=-N - build-depends: - MissingH - , QuickCheck - , aeson ==1.4.7.1 - , base - , bytestring ==0.10.10.0 - , directory ==1.3.6.0 - , dolla-base - , dolla-consensus-base ==1.0.0 - , dolla-consensus-proposing-packaging - , generic-arbitrary - , hspec - , hspec-core ==2.7.1 - , interpolatedstring-perl6 ==1.0.2 - , mtl ==2.2.2 - , quickcheck-instances - , random ==1.1 - , streamly ==0.7.2 - , turtle ==1.5.19 - , uuid ==1.3.13 - default-language: Haskell2010 diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Microservice.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Microservice.hs deleted file mode 100644 index dec7a3d..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Microservice.hs +++ /dev/null @@ -1,30 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Packaging.Microservice (execute) where - -import Prelude hiding (log) -import Control.Monad.Reader - -import Streamly.Prelude as S -import qualified Streamly.Internal.Prelude as SIP - -import Dolla.Common.Logging.Core -import Dolla.Common.Executable.Executable - -import Dolla.Consensus.Proposing.Packaging.Pipeline.OverEventStoreAndDolla (packaging) -import Dolla.Consensus.Proposing.Packaging.Settings -import Dolla.Consensus.Proposing.Packaging.Dependencies - -execute :: IO () -execute - = executeMicroservice - (\Settings {logger} -> logger) - executePipeline - where - executePipeline :: ReaderT Dependencies IO () - executePipeline = do - dependencies@Dependencies {logger} <- ask - log logger INFO "Pipeline Starting" - lift $ drain $ SIP.runReaderT dependencies packaging - log logger INFO "End Of Pipeline" - diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Generic.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Generic.hs deleted file mode 100644 index 46fe3cc..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Generic.hs +++ /dev/null @@ -1,63 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE NamedFieldPuns #-} - -module Dolla.Consensus.Proposing.Packaging.Pipeline.Generic - (packaging) - where - -import Prelude -import Data.Aeson.Types (ToJSON,FromJSON) - -import Control.Monad.Reader -import Control.Monad.Catch (MonadCatch) - - -import qualified Streamly as S - -import Dolla.Libraries.LogEngine.LogEngine (MemoryStreamLoggable (..)) -import Dolla.Common.Range (infinitely) - -import Dolla.Consensus.Proposing.Packaging.Dependencies - -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output - -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Pipe (serializing) -import Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Pipe (nonEmptying) -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Pipe (capping) -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Pipe (persisting) -import Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Pipe (notifying) - -import Dolla.Common.Pipeline.Weldable ((~>),(.~>)) -import Dolla.Consensus.Proposing.Packaging.Pipeline.Welding.BluePrint () - --- | Packaging Pipeline expressed in its most polymorphic way : --- - Log Engine Agnostic --- - Request Agnostic -packaging - :: ( ToJSON request - , FromJSON request - , Show request - , MonadCatch m - , MonadReader Dependencies m - , S.MonadAsync m - , MemoryStreamLoggable m log - ) - => log (Input request) - -> log Output - -> S.SerialT m () -packaging inputLog outputLog - = do - Dependencies {proposalSizeLimit,proposalRootFolder} <- ask - stream infinitely inputLog - ~> serializing - .~> nonEmptying - .~> capping proposalSizeLimit - .~> persisting proposalRootFolder - .~> notifying outputLog - - - - - diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Input.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Input.hs deleted file mode 100644 index 0997dac..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Input.hs +++ /dev/null @@ -1,33 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE OverloadedStrings #-} -module Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input - ( Input (..)) where - --- Data Origins --- The Packaging input stream is the junction of 2 upstream pipelines : --- - Receptioning : providing collected requests --- - Detecting-Starvation : transmitting Local-Proposal-Starvation Notifications - -import Data.Aeson -import qualified Data.Text as Text - - -data Input request - = ForceProposalProduction -- ^ ask to "Packaging Pipeline" to flush all the requests currently collected - | Package request -- ^ ask to to "Packaging Pipeline" to package the request into a proposal according - -- some properties (see README.md) - deriving (Eq,Show) - - --- | Integration with upstream junction --- We are reinterpreting events from upstream pipelines into commands for the new pipeline. --- (See tests for a more details) -instance FromJSON request => FromJSON (Input request) where - parseJSON (Object jsonObject) = do - tagMaybe <- jsonObject .: "tag" - case Text.unpack <$> tagMaybe of - Just "LocalProposalStarvationDetected" -> return ForceProposalProduction - Just "Receptioned" -> Package <$> jsonObject .: "contents" - anythingElse -> error $ "tag " ++ show anythingElse ++ "unknown, please verify integration with junction" - parseJSON _ = error $ "Json format not expected" - diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Welding/BluePrint.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Welding/BluePrint.hs deleted file mode 100644 index 71ddca3..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/Welding/BluePrint.hs +++ /dev/null @@ -1,58 +0,0 @@ -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE MultiParamTypeClasses #-} -{-# LANGUAGE FlexibleInstances #-} -{-# LANGUAGE RecordWildCards #-} - -{-# OPTIONS_GHC -fno-warn-orphans #-} - -module Dolla.Consensus.Proposing.Packaging.Pipeline.Welding.BluePrint - () where - -import Data.Word (Word8) -import Dolla.Common.Pipeline.Weldable -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.SerializedRequest - -import qualified Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input as Packaging - -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input as Serializing -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Output as Serializing - -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Input as NonEmptying -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Output as NonEmptying - -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Input as Capping -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Output as Capping - -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Input as Persisting -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Output as Persisting - -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Input as Notifying - -instance Weldable (Packaging.Input request) (Serializing.Input request) where - weld - = \case - Packaging.ForceProposalProduction {} -> Serializing.ForceProposalProduction - Packaging.Package request -> Serializing.Serialize request - -instance Weldable Persisting.Output Notifying.Input where - weld - = \case - Persisting.LocalProposalPersisted {..} -> Notifying.LocalProposalProduced {..} - -instance Weldable (Serializing.Output a) (NonEmptying.Input a) where - weld - = \case - Serializing.ProposalProductionNotForced -> Nothing - Serializing.Serialized a -> Just a - -instance Weldable (NonEmptying.Output a) (Capping.Input a) where - weld - = \case - Nothing -> Capping.AskForACut - Just a -> Capping.Add a - -instance Weldable (Capping.Output SerializedRequest) (Persisting.Input [Word8]) where - weld - = \case - Capping.Cut -> Persisting.CommitProposal - Capping.Added (SerializedRequest word8s) -> Persisting.Persist word8s diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Input.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Input.hs deleted file mode 100644 index c4d7a8a..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Input.hs +++ /dev/null @@ -1,5 +0,0 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Input - (Input ) - where - -type Input request = Maybe request \ No newline at end of file diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Output.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Output.hs deleted file mode 100644 index e0fac73..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Output.hs +++ /dev/null @@ -1,5 +0,0 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Output - (Output ) - where - -type Output request = Maybe request \ No newline at end of file diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Notifying/Input.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Notifying/Input.hs deleted file mode 100644 index 5a15210..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Notifying/Input.hs +++ /dev/null @@ -1,10 +0,0 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Input - ( Input (..)) where - -import Dolla.Common.Offset (Offset) - - -newtype Input = LocalProposalProduced {proposalId :: Offset} - - - diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Input.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Input.hs deleted file mode 100644 index c0b4d8f..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Input.hs +++ /dev/null @@ -1,10 +0,0 @@ -{-# LANGUAGE DeriveFunctor #-} - -module Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input - ( Input (..)) where - -data Input request - = ForceProposalProduction - | Serialize request - deriving (Functor,Show) - diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Output.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Output.hs deleted file mode 100644 index e60243a..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Output.hs +++ /dev/null @@ -1,11 +0,0 @@ -{-# LANGUAGE DeriveFunctor #-} - -module Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Output - (Output (..)) - where - -data Output a - = ProposalProductionNotForced - | Serialized a - deriving (Eq,Show,Functor) - diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Pipe.hs b/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Pipe.hs deleted file mode 100644 index 8148d60..0000000 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/Pipe.hs +++ /dev/null @@ -1,29 +0,0 @@ -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE LambdaCase #-} - -module Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Pipe - (serializing) - where - -import Prelude -import Data.ByteString hiding (length,map,append) -import Data.Coerce (coerce) -import Data.Aeson.Types (ToJSON) - -import qualified Streamly as S -import qualified Streamly.Prelude as S - -import Dolla.Common.UUID.Deterministic -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input as Input -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Output as Output -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.SerializedRequest - -serializing - :: ( Monad m, ToJSON a) - => S.SerialT m (Input a) - -> S.SerialT m (Output SerializedRequest) -serializing - = S.map (\case - ForceProposalProduction -> ProposalProductionNotForced -- Isomorphism with Maybe - Serialize request -> Serialized $ (coerce . unpack . getEncodedItem) request) \ No newline at end of file diff --git a/packages/packaging/settings/README.md b/packages/packaging/settings/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/GenInput.hs b/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/GenInput.hs deleted file mode 100644 index efb73cc..0000000 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/GenInput.hs +++ /dev/null @@ -1,21 +0,0 @@ -module Dolla.Consensus.Proposing.Packaging.Pipeline.IO.GenInput - ( InputUnderTests (..)) where - -import Test.QuickCheck.Arbitrary -import Test.QuickCheck.Gen -import Test.QuickCheck.Instances.UUID () -import Test.QuickCheck.Instances () - -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input - - -instance Arbitrary a => Arbitrary (InputUnderTests a) where - arbitrary - = InputUnderTests - <$> oneof - [ return ForceProposalProduction - , Package <$> arbitrary] - - -newtype InputUnderTests a = InputUnderTests { unInputUnderTests :: Input a} deriving Show - diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/GenOutput.hs b/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/GenOutput.hs deleted file mode 100644 index 428b4f7..0000000 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/GenOutput.hs +++ /dev/null @@ -1,15 +0,0 @@ -module Dolla.Consensus.Proposing.Packaging.Pipeline.IO.GenOutput - ( OutputUnderTests (..) - ) where - -import Test.QuickCheck.Arbitrary -import Test.QuickCheck.Instances.UUID () -import Test.QuickCheck.Instances () -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output -import Dolla.Common.Offset - -newtype OutputUnderTests = OutputUnderTests { unOutputUnderTests :: Output} deriving Show - -instance Arbitrary OutputUnderTests where - arbitrary = OutputUnderTests . LocalProposalProduced . Offset <$> arbitrary - diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/InputSpec.hs b/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/InputSpec.hs deleted file mode 100644 index dc12d2e..0000000 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/InputSpec.hs +++ /dev/null @@ -1,33 +0,0 @@ -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE ScopedTypeVariables #-} -{-# LANGUAGE DerivingVia #-} -{-# LANGUAGE QuasiQuotes, ExtendedDefaultRules #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE UndecidableInstances #-} -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE DeriveGeneric #-} -{-# LANGUAGE DeriveFunctor #-} -{-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Packaging.Pipeline.IO.InputSpec (spec) where - -import Data.Aeson - -import Test.QuickCheck.Instances () -import Test.Hspec -import Text.InterpolatedString.Perl6 (qc) - -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input - -import Dolla.Consensus.Proposing.Packaging.DummyRequest - -spec :: Spec -spec = parallel $ - describe "Input follow the integration protocol" $ do - it "ForceProposalProduction is an interpretation of LocalProposalStarvationDetected from Detecting-Starvation Pipeline" $ do - decode [qc|\{"tag":"LocalProposalStarvationDetected","itemId":"bf428e1d-f221-55de-a77f-a61755a4d727"}|] - `shouldBe` (Just ForceProposalProduction :: (Maybe (Input DummyRequest))) - it "Package request is an interpretation of Receptioned request from Receptioning Pipeline" $ do - decode [qc|\{"tag":"Receptioned","contents":\{"tag":"DummyRequest","fieldA":true,"fieldB":false}}|] - `shouldBe` (Just $ Package (DummyRequest {fieldA = True, fieldB = False}) :: (Maybe (Input DummyRequest))) - - diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/OutputSpec.hs b/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/OutputSpec.hs deleted file mode 100644 index d3405e5..0000000 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/OutputSpec.hs +++ /dev/null @@ -1,31 +0,0 @@ -{-# LANGUAGE LambdaCase #-} -{-# LANGUAGE ScopedTypeVariables #-} -{-# LANGUAGE DerivingVia #-} -{-# LANGUAGE QuasiQuotes, ExtendedDefaultRules #-} -{-# LANGUAGE FlexibleContexts #-} -{-# LANGUAGE UndecidableInstances #-} -{-# LANGUAGE DuplicateRecordFields #-} -{-# LANGUAGE DeriveGeneric #-} -{-# LANGUAGE DeriveFunctor #-} -{-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Packaging.Pipeline.IO.OutputSpec (spec) where - -import Data.Aeson - -import Test.QuickCheck.Instances () -import Test.QuickCheck -import Test.Hspec -import Text.InterpolatedString.Perl6 (qc) - -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.GenOutput - - -spec :: Spec -spec = parallel $ - describe "Output" $ do - it "follow Broadcasting Input json protocol " - $ property $ \OutputUnderTests {unOutputUnderTests} - -> encode unOutputUnderTests - `shouldBe` [qc|\{"tag":"LocalProposalProduced","localOffset":{show $ localOffset unOutputUnderTests}}|] - diff --git a/packages/receptioning/README.md b/packages/receptioning/README.md index e69de29..f8e8fbf 100644 --- a/packages/receptioning/README.md +++ b/packages/receptioning/README.md @@ -0,0 +1,50 @@ +/ [Consensus](https://github.com/dolla-consortium/consensus) / [Proposing](https://github.com/dolla-consortium/consensus-proposing) / Receptioning +# Receptioning + +- [Overview](#overview) +- [Project Tree](#project-tree) + +# Overview + +`Receptioning` collects requests from customers and consortium members + +![overview](documentation/media/overview.png) + +# Project Tree + +
+### 1. Service +This is a non-deterministic service, A merge of clients requests +- Send Requests over the network via [Client.hs](library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Client.hs) +- Receive Request via [Server.hs](library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Server.hs) +- Append Non Idempotently Request directly into the Packaging Input Stream + +### 2. Execution Environment +`Receptioning` is Polymorphic by +- The Log Engine used +- The Business Logic used on top of the consensus layer (requests in that context) +- The type of server used + +You'll find in this folder different version of Service.hs "polymorphically reduced" and concrete +- Over the event store + - [Service.hs](library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Service.hs) + - Over Dolla Dummy Requests + - [Service.hs](library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Service.hs) + - Over Warp using Servant library (concrete version) + - [Client.hs](library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Client.hs) + - [Server.hs](library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Server.hs) + +### 3. Executable + +`Receptioning` has some DevOps features as well + +- Settings.hs always into a separated project `xxxx-receptioning-settings` for deployment purposes in Zeus +- Dependencies.hs are derived from Settings if sub-dependencies are all Healthy + +[Executable.hs](executables/Executables.hs) +- Perform the HealhtChecks to obtain the pipeline dependencies +- Execute the server +- Put the Microservice back in HealthCheck mode if any Exception bubbles up in the server during execution. + +**N.B** : Microservice configuration and Deployment (Locally/Simulated/Production etc...) are defined in the package [Zeus](../zeus/) +
logger) + runServerOnWarp diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/Generic.hs b/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/Generic.hs deleted file mode 100644 index 84edefe..0000000 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/Generic.hs +++ /dev/null @@ -1,37 +0,0 @@ -{-# LANGUAGE FlexibleContexts #-} -module Dolla.Consensus.Proposing.Receptioning.Service.Generic - ( transmitRequestToProposingPackagingPipeline - , transmitRequestsToProposingPackagingPipeline) where - -import Data.List.NonEmpty -import Dolla.Libraries.LogEngine.LogEngine -import Dolla.Consensus.Request -import Dolla.Libraries.LogEngine.Appendable -import Dolla.Common.UUID.Provider -import Dolla.Consensus.Proposing.Receptioning.Output - -transmitRequestToProposingPackagingPipeline - :: ( Appendable clientRequest - , Appendable consortiumRequest - , UUIDProvider clientRequest - , UUIDProvider consortiumRequest - , MemoryStreamLoggable m log) - => log (Output (Request clientRequest consortiumRequest)) - -> Request clientRequest consortiumRequest - -> m () -transmitRequestToProposingPackagingPipeline eventStoreLog request - = nonIdempotentAppend eventStoreLog $ Receptioned request - -transmitRequestsToProposingPackagingPipeline - :: ( Appendable clientRequest - , Appendable consortiumRequest - , UUIDProvider clientRequest - , UUIDProvider consortiumRequest - , Show clientRequest - , Show consortiumRequest - , MemoryStreamLoggable m log) - => log (Output (Request clientRequest consortiumRequest)) - -> NonEmpty (Request clientRequest consortiumRequest) - -> m () -transmitRequestsToProposingPackagingPipeline eventStoreLog requests - = nonIdempotentAppendList eventStoreLog $ Receptioned <$> requests \ No newline at end of file diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/OverDolla.hs b/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/OverDolla.hs deleted file mode 100644 index 428a283..0000000 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/OverDolla.hs +++ /dev/null @@ -1,34 +0,0 @@ - -module Dolla.Consensus.Proposing.Receptioning.Service.OverDolla - ( persistClientRequest - , persistClientRequests) where - -import Data.List.NonEmpty -import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStoreClient - -import Dolla.Consensus.Dummy.Client.Request -import Dolla.Consensus.Consortium.Request -import qualified Dolla.Consensus.Proposing.Receptioning.Service.OverEventStore as OverEventStore -import Dolla.Consensus.Request -import Control.Monad.IO.Class - - -persistClientRequest - :: MonadIO m - => EventStoreClient.Dependencies - -> DollaClientRequest - -> m () -persistClientRequest eventStoreClientDependencies clientRequest = - OverEventStore.transmitRequestToProposingPackagingPipeline - eventStoreClientDependencies - (ClientReq clientRequest :: Request DollaClientRequest ConsortiumRequest) - -persistClientRequests - :: MonadIO m - => EventStoreClient.Dependencies - -> NonEmpty DollaClientRequest - -> m () -persistClientRequests eventStoreClientDependencies clientRequests = - OverEventStore.transmitRequestsToProposingPackagingPipeline - eventStoreClientDependencies - ((ClientReq <$> clientRequests) :: NonEmpty (Request DollaClientRequest ConsortiumRequest)) diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/OverEventStore.hs b/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/OverEventStore.hs deleted file mode 100644 index 9d78cb9..0000000 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Service/OverEventStore.hs +++ /dev/null @@ -1,47 +0,0 @@ - -module Dolla.Consensus.Proposing.Receptioning.Service.OverEventStore - ( transmitRequestToProposingPackagingPipeline - , transmitRequestsToProposingPackagingPipeline) where - -import Data.List.NonEmpty -import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStoreClient - -import Dolla.Consensus.Request - -import qualified Dolla.Consensus.Proposing.Receptioning.Service.Generic as Service.Generic -import Dolla.Consensus.Log.EventStoreLog -import Dolla.Libraries.LogEngine.Appendable -import Dolla.Common.UUID.Provider -import Control.Monad.IO.Class - -transmitRequestToProposingPackagingPipeline - :: ( MonadIO m - , Appendable clientRequest - , Appendable consortiumRequest - , UUIDProvider clientRequest - , UUIDProvider consortiumRequest) - => EventStoreClient.Dependencies - -> Request clientRequest consortiumRequest - -> m () -transmitRequestToProposingPackagingPipeline eventStoreClientDependencies = - Service.Generic.transmitRequestToProposingPackagingPipeline - (getEventStoreLog - eventStoreClientDependencies - LocalRequestLog) - -transmitRequestsToProposingPackagingPipeline - :: ( MonadIO m - , Show clientRequest - , Show consortiumRequest - , Appendable clientRequest - , Appendable consortiumRequest - , UUIDProvider clientRequest - , UUIDProvider consortiumRequest) - => EventStoreClient.Dependencies - -> NonEmpty (Request clientRequest consortiumRequest) - -> m () -transmitRequestsToProposingPackagingPipeline eventStoreClientDependencies = - Service.Generic.transmitRequestsToProposingPackagingPipeline - (getEventStoreLog - eventStoreClientDependencies - LocalRequestLog) \ No newline at end of file diff --git a/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Service.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Service.hs new file mode 100644 index 0000000..a5d1939 --- /dev/null +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Service.hs @@ -0,0 +1,23 @@ + +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Service + ( receptioning) where + +import Data.List.NonEmpty +import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStoreClient + +import Dolla.Consensus.Dummy.Client.Request +import Dolla.Consensus.Consortium.Request +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Service as OverEventStore +import Dolla.Consensus.Request +import Control.Monad.IO.Class + + +receptioning + :: MonadIO m + => EventStoreClient.Dependencies + -> NonEmpty DollaClientRequest + -> m () +receptioning eventStoreClient clientRequests = + OverEventStore.receptioning + eventStoreClient + ((ClientReq <$> clientRequests) :: NonEmpty (Request DollaClientRequest ConsortiumRequest)) diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Client.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Client.hs similarity index 72% rename from packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Client.hs rename to packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Client.hs index 95e5ea4..b6ad9eb 100644 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Client.hs +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Client.hs @@ -1,38 +1,24 @@ {-# LANGUAGE DataKinds #-} {-# LANGUAGE DuplicateRecordFields #-} -module Dolla.Consensus.Proposing.Receptioning.API.Client.Client - ( sendProposalRequest - , sendProposalRequests +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Client + ( sendProposalRequests , sendHealthCheckRequest) where +import Control.Monad.IO.Class +import Data.Validation +import Data.List.NonEmpty + +import Network.HTTP.Client (Manager) import qualified Servant.Client.Streaming as S + import Servant -import Dolla.Consensus.Proposing.Receptioning.API.Definition -import Dolla.Consensus.Dummy.Client.Request -import Data.Validation -import Dolla.Common.Dependencies.Core -import Data.List.NonEmpty import Servant.Client -import Network.HTTP.Client (Manager) -import Control.Monad.IO.Class -sendProposalRequest - :: Manager - -> BaseUrl - -> DollaClientRequest - -> IO (Either String ()) -sendProposalRequest httpClientManager url proposalRequest = - S.withClientM - (sendProposalRequestCall proposalRequest) - (S.mkClientEnv httpClientManager url) - (either - (return . Left. show) - (return . Right)) - where - sendProposalRequestCall :: DollaClientRequest -> S.ClientM () - sendProposalRequestCall = S.client (Proxy :: Proxy SendClientRequest ) +import Dolla.Consensus.Dummy.Client.Request +import Dolla.Common.Dependencies.Core +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Definition sendProposalRequests :: MonadIO m diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Dependencies.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Dependencies.hs similarity index 74% rename from packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Dependencies.hs rename to packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Dependencies.hs index 2842f8b..0745b79 100644 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Dependencies.hs +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Dependencies.hs @@ -1,17 +1,23 @@ {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE MultiParamTypeClasses #-} -module Dolla.Consensus.Proposing.Receptioning.API.Client.Dependencies (Dependencies (..)) where +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Dependencies + (Dependencies (..)) where -import Dolla.Common.Dependencies.Core import Servant.Client + import Network.HTTP.Client (Manager, newManager, defaultManagerSettings) + +import Dolla.Adapter.Servant.Adapter + import Dolla.Common.NodeId -import Dolla.Consensus.Proposing.Receptioning.API.Client.Client +import Dolla.Common.Dependencies.Core import Dolla.Common.Logging.Core -import Dolla.Adapter.Servant.Adapter -import Dolla.Consensus.Proposing.Receptioning.API.Client.Settings + + +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Settings +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Client data Dependencies = Dependencies diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Definition.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Definition.hs similarity index 72% rename from packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Definition.hs rename to packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Definition.hs index 22dc705..e12f35c 100644 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Definition.hs +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Definition.hs @@ -4,29 +4,24 @@ {-# LANGUAGE RankNTypes #-} {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE TypeOperators #-} -module Dolla.Consensus.Proposing.Receptioning.API.Definition +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Definition ( ReceptionistApi , HealthCheckRequest - , SendClientRequest , SendClientRequests ) where import Servant -import Dolla.Consensus.Dummy.Client.Request import Data.List.NonEmpty + +import Dolla.Consensus.Dummy.Client.Request + import Dolla.Common.Dependencies.Core type ReceptionistApi = HealthCheckRequest - :<|> SendClientRequest :<|> SendClientRequests - type HealthCheckRequest = "health" :> Get '[JSON] (Either (NonEmpty UnhealthyDependency) ()) -type SendClientRequest = "consortium" :> "team" :> "sendClientRequest" - :> ReqBody '[JSON] DollaClientRequest - :> PostAccepted '[JSON] () - type SendClientRequests = "consortium" :> "team" :> "sendClientRequests" :> ReqBody '[JSON] (NonEmpty DollaClientRequest) :> PostAccepted '[JSON] () \ No newline at end of file diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Dependencies.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Dependencies.hs similarity index 82% rename from packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Dependencies.hs rename to packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Dependencies.hs index 00e70f2..42944c9 100644 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Dependencies.hs +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Dependencies.hs @@ -1,7 +1,7 @@ {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE MultiParamTypeClasses #-} -module Dolla.Consensus.Proposing.Receptioning.API.Server.Dependencies (Dependencies (..)) where +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Dependencies (Dependencies (..)) where import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStore @@ -9,7 +9,7 @@ import Dolla.Common.Logging.Core import Dolla.Common.Network.Core import Dolla.Common.Dependencies.Core import Dolla.Common.NodeId -import Dolla.Consensus.Proposing.Receptioning.API.Server.Settings +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Settings data Dependencies = Dependencies{ nodeId :: NodeId, diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Server.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Server.hs similarity index 55% rename from packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Server.hs rename to packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Server.hs index 860eda1..3ef63ea 100644 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Server.hs +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Server.hs @@ -1,13 +1,19 @@ {-# LANGUAGE DuplicateRecordFields #-} {-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Receptioning.API.Server.Server (execute) where +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Server + (runServerOnWarp) where import Prelude hiding (log) +import Control.Monad +import Control.Monad.Reader + +import Data.Coerce (coerce) import Data.List.NonEmpty import Data.Validation import Data.Aeson () -import Servant + import Network.Wai.Handler.Warp hiding (Settings) +import Servant import Dolla.Adapter.Servant.Wrapper import Dolla.Common.Logging.Core @@ -15,18 +21,10 @@ import Dolla.Common.Network.Core import Dolla.Common.Dependencies.Core import Dolla.Consensus.Dummy.Client.Request -import qualified Dolla.Consensus.Proposing.Receptioning.API.Server.Settings as Server -import qualified Dolla.Consensus.Proposing.Receptioning.API.Server.Dependencies as Server -import qualified Dolla.Consensus.Proposing.Receptioning.Service.OverDolla as DollaService -import Dolla.Consensus.Proposing.Receptioning.API.Definition -import Control.Monad -import Control.Monad.Reader - -import Dolla.Common.Executable.Executable -import Data.Coerce (coerce) +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Dependencies as Server +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Service +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Definition -execute :: IO () -execute = executeMicroservice (\Server.Settings {logger} -> logger) runServerOnWarp runServerOnWarp :: ReaderT Server.Dependencies IO() runServerOnWarp = do @@ -41,7 +39,6 @@ receptioningServer :: Server.Dependencies -> Server ReceptionistApi receptioningServer dependencies = checkHealthRequest dependencies - :<|> sendClientRequest dependencies :<|> sendClientRequests dependencies checkHealthRequest @@ -49,22 +46,10 @@ checkHealthRequest -> Handler (Either (NonEmpty UnhealthyDependency) ()) checkHealthRequest dependencies = liftIO $ fmap (toEither . void) (checkHealth dependencies) -sendClientRequest - :: Server.Dependencies - -> DollaClientRequest - -> Handler () -sendClientRequest Server.Dependencies {logger,eventStoreClientDependencies} proposalRequest = do - liftIO $ log logger DEBUG $ "client request received :" ++ show proposalRequest - liftIO $ DollaService.persistClientRequest - eventStoreClientDependencies - proposalRequest - sendClientRequests :: Server.Dependencies -> NonEmpty DollaClientRequest -> Handler () -sendClientRequests Server.Dependencies {logger,eventStoreClientDependencies} proposalRequests = do - liftIO $ log logger DEBUG $ "client requests received :" ++ show proposalRequests - liftIO $ DollaService.persistClientRequests - eventStoreClientDependencies - proposalRequests +sendClientRequests Server.Dependencies {logger,eventStoreClientDependencies} requests = do + liftIO $ log logger DEBUG $ "client requests received :" ++ show requests + liftIO $ receptioning eventStoreClientDependencies requests diff --git a/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Service.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Service.hs new file mode 100644 index 0000000..363f8cc --- /dev/null +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Service.hs @@ -0,0 +1,30 @@ + +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Service + (receptioning) where + +import Data.List.NonEmpty +import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStoreClient + +import Dolla.Consensus.Request + +import qualified Dolla.Consensus.Proposing.Receptioning.Service.Service as Generic +import Dolla.Consensus.Log.EventStoreLog +import Dolla.Libraries.LogEngine.Appendable +import Dolla.Common.UUID.Provider +import Control.Monad.IO.Class +import Dolla.Consensus.Log.LogNameIndex + +receptioning + :: ( MonadIO m + , Show clientRequest + , Show consortiumRequest + , Appendable clientRequest + , Appendable consortiumRequest + , UUIDProvider clientRequest + , UUIDProvider consortiumRequest) + => EventStoreClient.Dependencies + -> NonEmpty (Request clientRequest consortiumRequest) + -> m () +receptioning eventStoreClient = + Generic.receptioning + (getEventStoreLog eventStoreClient ProposingReceptioningOutputLog) \ No newline at end of file diff --git a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Output.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Service/Output.hs similarity index 85% rename from packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Output.hs rename to packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Service/Output.hs index ccdf68b..ee7ee61 100644 --- a/packages/receptioning/lib/Dolla/Consensus/Proposing/Receptioning/Output.hs +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Service/Output.hs @@ -5,7 +5,7 @@ {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE DeriveFunctor #-} -module Dolla.Consensus.Proposing.Receptioning.Output +module Dolla.Consensus.Proposing.Receptioning.Service.Output ( Output (..)) where import Data.Aeson @@ -22,7 +22,7 @@ newtype Output request instance Appendable a => Appendable (Output a) where - getItemName (Receptioned request) = "Receptioned." ++ getItemName request + getItemName _ = "Receptioned" instance UUIDProvider a => UUIDProvider (Output a) where getUUID (Receptioned request) = getUUID request diff --git a/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Service/Service.hs b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Service/Service.hs new file mode 100644 index 0000000..390c3db --- /dev/null +++ b/packages/receptioning/library/Dolla/Consensus/Proposing/Receptioning/Service/Service.hs @@ -0,0 +1,26 @@ +{-# LANGUAGE FlexibleContexts #-} +module Dolla.Consensus.Proposing.Receptioning.Service.Service + (receptioning) where + +import Data.List.NonEmpty +import Dolla.Libraries.LogEngine.LogEngine +import Dolla.Consensus.Request +import Dolla.Libraries.LogEngine.Appendable +import Dolla.Common.UUID.Provider +import Dolla.Consensus.Proposing.Receptioning.Service.Output + +receptioning + :: ( Appendable clientRequest + , Appendable consortiumRequest + , UUIDProvider clientRequest + , UUIDProvider consortiumRequest + , Show clientRequest + , Show consortiumRequest + , MemoryStreamLoggable m log) + => log (Output (Request clientRequest consortiumRequest)) + -> NonEmpty (Request clientRequest consortiumRequest) + -> m () +receptioning eventStoreLog requests + = nonIdempotentAppendList + eventStoreLog + (Receptioned <$> requests) \ No newline at end of file diff --git a/packages/receptioning/package.yaml b/packages/receptioning/package.yaml index 4286a19..0ba907c 100644 --- a/packages/receptioning/package.yaml +++ b/packages/receptioning/package.yaml @@ -5,9 +5,9 @@ extra-source-files: ghc-options: - -Wall -## - -j + library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 @@ -24,17 +24,19 @@ library: - aeson == 1.4.7.1 executables: - dolla-consensus-proposing-receptioning : + dolla-consensus-proposing-receptioning-server : main: Executables.hs - source-dirs: app + source-dirs: executables ghc-options: - - -main-is Executables.receptioning + - -main-is Executables.executeServer - -threaded - -rtsopts - -with-rtsopts=-N dependencies: - - dolla-consensus-proposing-receptioning - base == 4.13.0.0 + - dolla-base == 1.0.0 + - dolla-consensus-proposing-receptioning + - dolla-consensus-proposing-receptioning-settings tests: dolla-consensus-proposing-receptioning-test: diff --git a/packages/receptioning/settings/README.md b/packages/receptioning/settings/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/packages/receptioning/settings/dolla-consensus-proposing-receptioning-settings.cabal b/packages/receptioning/settings/dolla-consensus-proposing-receptioning-settings.cabal index ad536ee..881beff 100644 --- a/packages/receptioning/settings/dolla-consensus-proposing-receptioning-settings.cabal +++ b/packages/receptioning/settings/dolla-consensus-proposing-receptioning-settings.cabal @@ -4,22 +4,20 @@ cabal-version: 1.12 -- -- see: https://github.com/sol/hpack -- --- hash: bc4850ba3771a6fa8c32dd27f670db4bf28e3c0133550646434a3f232554ee77 +-- hash: 976655d54f366a5e241d42765414218293f04ffc2e9756e1560146e5474dad15 name: dolla-consensus-proposing-receptioning-settings version: 0.0.0.1 build-type: Simple -extra-source-files: - README.md library exposed-modules: - Dolla.Consensus.Proposing.Receptioning.API.Client.Settings - Dolla.Consensus.Proposing.Receptioning.API.Server.Settings + Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Settings + Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Settings other-modules: Paths_dolla_consensus_proposing_receptioning_settings hs-source-dirs: - lib + library ghc-options: -Wall build-depends: aeson ==1.4.7.1 diff --git a/packages/receptioning/settings/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Settings.hs b/packages/receptioning/settings/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Settings.hs similarity index 78% rename from packages/receptioning/settings/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Settings.hs rename to packages/receptioning/settings/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Settings.hs index 734265f..6e6ff12 100644 --- a/packages/receptioning/settings/lib/Dolla/Consensus/Proposing/Receptioning/API/Client/Settings.hs +++ b/packages/receptioning/settings/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Client/Settings.hs @@ -1,6 +1,7 @@ {-# LANGUAGE DerivingVia #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Receptioning.API.Client.Settings (Settings (..)) where +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Settings + (Settings (..)) where import Data.Aeson import GHC.Generics diff --git a/packages/receptioning/settings/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Settings.hs b/packages/receptioning/settings/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Settings.hs similarity index 84% rename from packages/receptioning/settings/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Settings.hs rename to packages/receptioning/settings/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Settings.hs index fa4659f..c833eb0 100644 --- a/packages/receptioning/settings/lib/Dolla/Consensus/Proposing/Receptioning/API/Server/Settings.hs +++ b/packages/receptioning/settings/library/Dolla/Consensus/Proposing/Receptioning/Execution/Environment/EventStore/Dolla/Warp/Server/Settings.hs @@ -1,6 +1,7 @@ {-# LANGUAGE DerivingVia #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Receptioning.API.Server.Settings (Settings (..)) where +module Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Settings + (Settings (..)) where import Data.Aeson import GHC.Generics diff --git a/packages/receptioning/settings/package.yaml b/packages/receptioning/settings/package.yaml index 05824c9..2f19369 100644 --- a/packages/receptioning/settings/package.yaml +++ b/packages/receptioning/settings/package.yaml @@ -1,13 +1,11 @@ name: dolla-consensus-proposing-receptioning-settings version: 0.0.0.1 -extra-source-files: - - README.md ghc-options: - -Wall -## - -j + library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 diff --git a/packages/receptioning/test/Dolla/Consensus/Proposing/Receptioning/OutputSpec.hs b/packages/receptioning/test/Dolla/Consensus/Proposing/Receptioning/Service/OutputSpec.hs similarity index 83% rename from packages/receptioning/test/Dolla/Consensus/Proposing/Receptioning/OutputSpec.hs rename to packages/receptioning/test/Dolla/Consensus/Proposing/Receptioning/Service/OutputSpec.hs index 1dea134..b7d266b 100644 --- a/packages/receptioning/test/Dolla/Consensus/Proposing/Receptioning/OutputSpec.hs +++ b/packages/receptioning/test/Dolla/Consensus/Proposing/Receptioning/Service/OutputSpec.hs @@ -8,7 +8,7 @@ {-# LANGUAGE DeriveGeneric #-} {-# LANGUAGE DeriveFunctor #-} {-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Receptioning.OutputSpec (spec) where +module Dolla.Consensus.Proposing.Receptioning.Service.OutputSpec (spec) where import Data.Aeson @@ -19,7 +19,7 @@ import Test.QuickCheck.Instances () import Test.Hspec import Text.InterpolatedString.Perl6 (qc) -import Dolla.Consensus.Proposing.Receptioning.Output +import Dolla.Consensus.Proposing.Receptioning.Service.Output data DummyRequest = DummyRequest {fieldA :: Bool, fieldB :: Bool} @@ -29,7 +29,7 @@ data DummyRequest spec :: Spec spec = parallel $ describe "Proposing Receptioning Output" $ do - it "follow Proposing Packaging Input json protocol " $ do + it "follow Proposing Staging Input json protocol " $ do encode (Receptioned (DummyRequest {fieldA = True, fieldB = False}) :: (Output DummyRequest)) `shouldBe` [qc|\{"tag":"Receptioned","contents":\{"tag":"DummyRequest","fieldA":true,"fieldB":false}}|] diff --git a/packages/simulating/README.md b/packages/simulating/README.md index 9f5f739..9d5a3a7 100644 --- a/packages/simulating/README.md +++ b/packages/simulating/README.md @@ -1,16 +1,4 @@ -# dolla-cli-client : Command Line Interface Client +/ [Consensus](https://github.com/dolla-consortium/consensus) / [Proposing](https://github.com/dolla-consortium/consensus-proposing) / Simulating +# Simulating -This project contains 2 executables : - - - `dolla-cli-client` : CLI Client to communicate with the consortium - - `dolla-cli-client-simulator` : A simulator that will heavily send requests to each nodes of the consortium - -## Run Executables - - ```bash - stack run dolla-cli-client - ``` - - ```bash - stack run dolla-cli-client-simulator - ``` \ No newline at end of file +Documentation to be done. \ No newline at end of file diff --git a/packages/simulating/dolla-consensus-proposing-simulating.cabal b/packages/simulating/dolla-consensus-proposing-simulating.cabal index bb114c8..fbc31e6 100644 --- a/packages/simulating/dolla-consensus-proposing-simulating.cabal +++ b/packages/simulating/dolla-consensus-proposing-simulating.cabal @@ -4,7 +4,7 @@ cabal-version: 1.12 -- -- see: https://github.com/sol/hpack -- --- hash: e7cdcbc8ebf839e2eecf3625b7400ac0eb4a5a0653e05287db68138411fa9c34 +-- hash: 69eec4043771efbd175ee4f37c2f4d38bcc31c3ada0fa31381ddccb79da832f6 name: dolla-consensus-proposing-simulating version: 0.0.0.1 @@ -22,7 +22,7 @@ library other-modules: Paths_dolla_consensus_proposing_simulating hs-source-dirs: - lib + library ghc-options: -Wall build-depends: aeson ==1.4.7.1 @@ -31,10 +31,10 @@ library , bytestring ==0.10.10.0 , dolla-base ==1.0.0 , dolla-consensus-base ==1.0.0 - , dolla-consensus-proposing-packaging , dolla-consensus-proposing-receptioning , dolla-consensus-proposing-receptioning-settings , dolla-consensus-proposing-simulating-settings + , dolla-consensus-proposing-staging , exceptions ==0.10.4 , mtl ==2.2.2 , random ==1.1 @@ -48,7 +48,7 @@ executable dolla-consensus-proposing-simulating other-modules: Paths_dolla_consensus_proposing_simulating hs-source-dirs: - app + executables ghc-options: -Wall -main-is Executables.simulating -threaded -rtsopts -with-rtsopts=-N build-depends: base ==4.13.0.0 diff --git a/packages/simulating/app/Executables.hs b/packages/simulating/executables/Executables.hs similarity index 100% rename from packages/simulating/app/Executables.hs rename to packages/simulating/executables/Executables.hs diff --git a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/Dependencies.hs b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/Dependencies.hs similarity index 92% rename from packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/Dependencies.hs rename to packages/simulating/library/Dolla/Consensus/Proposing/Simulating/Dependencies.hs index 578e8dc..71fb483 100644 --- a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/Dependencies.hs +++ b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/Dependencies.hs @@ -10,7 +10,7 @@ module Dolla.Consensus.Proposing.Simulating.Dependencies (Dependencies (..)) whe import Dolla.Common.Logging.Core import Dolla.Common.Dependencies.Core -import qualified Dolla.Consensus.Proposing.Receptioning.API.Client.Dependencies as Receptionist.Client +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Dependencies as Receptionist.Client import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStore import Dolla.Consensus.Proposing.Simulating.Settings import Dolla.Consensus.Proposing.Simulating.StressLoad diff --git a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/GenRequest.hs b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/GenRequest.hs similarity index 100% rename from packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/GenRequest.hs rename to packages/simulating/library/Dolla/Consensus/Proposing/Simulating/GenRequest.hs diff --git a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/OverFlowing.hs b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/OverFlowing.hs similarity index 84% rename from packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/OverFlowing.hs rename to packages/simulating/library/Dolla/Consensus/Proposing/Simulating/OverFlowing.hs index fd868cf..a20d6e9 100644 --- a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/OverFlowing.hs +++ b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/OverFlowing.hs @@ -16,8 +16,8 @@ import Data.Function ((&)) import Dolla.Common.Logging.Core -import qualified Dolla.Consensus.Proposing.Receptioning.API.Client.Dependencies as Receptionist.Client -import Dolla.Consensus.Proposing.Receptioning.API.Client.Client +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Dependencies as Receptionist.Client +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Client import Dolla.Consensus.Proposing.Simulating.Dependencies import Dolla.Common.Memory.Byte (Byte) import Dolla.Consensus.Proposing.Simulating.GenRequest diff --git a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/Simulating.hs b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/Simulating.hs similarity index 71% rename from packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/Simulating.hs rename to packages/simulating/library/Dolla/Consensus/Proposing/Simulating/Simulating.hs index 3f6fb13..3520a4d 100644 --- a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/Simulating.hs +++ b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/Simulating.hs @@ -16,10 +16,8 @@ import qualified Streamly.Internal.Prelude as S import Dolla.Common.Logging.Core import Dolla.Common.Executable.Executable import Dolla.Consensus.Log.EventStoreLog +import Dolla.Consensus.Log.LogNameIndex -import Dolla.Consensus.Maestro.ESMerger (loadMaestroInputProjection) - -import qualified Dolla.Consensus.Proposing.Receptioning.API.Client.Dependencies as Receptionist.Client import Dolla.Consensus.Proposing.Simulating.Dependencies import Dolla.Consensus.Proposing.Simulating.Settings @@ -41,17 +39,15 @@ start { logger , stressLoad , eventStoreClient - , receptioningClient = receptioningClient@Receptionist.Client.Dependencies {nodeId} } <- ask + , receptioningClient} <- ask log logger INFO "Starting Simulating" case stressLoad of OverFlowing proposalSizeLimit -> overflowing proposalSizeLimit - UnderSupplying proposalSizeLimit -> do - -- Need to run the merge of the maestro to simulate the merge of the detecting starvation merge - _ <- withReaderT (const (nodeId, eventStoreClient)) loadMaestroInputProjection + UnderSupplying proposalSizeLimit -> S.drain $ S.runReaderT UnderSupplying.Context - { packagingOutputLog = getEventStoreLog eventStoreClient ProposingPackagingOutputLog + { stagingOutputLog = getEventStoreLog eventStoreClient ProposingStagingOutputLog , getMaestroOutputLog = getEventStoreLog eventStoreClient . MaestroOutputLog , ..} (underSupplying proposalSizeLimit) diff --git a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/UnderSupplying.hs b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/UnderSupplying.hs similarity index 65% rename from packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/UnderSupplying.hs rename to packages/simulating/library/Dolla/Consensus/Proposing/Simulating/UnderSupplying.hs index 601ff63..db88fe1 100644 --- a/packages/simulating/lib/Dolla/Consensus/Proposing/Simulating/UnderSupplying.hs +++ b/packages/simulating/library/Dolla/Consensus/Proposing/Simulating/UnderSupplying.hs @@ -31,16 +31,16 @@ import Dolla.Libraries.LogEngine.LogEngine import Dolla.Consensus.Log.Aggregation import Dolla.Consensus.Dummy.Client.Request import qualified Dolla.Consensus.Maestro.Output as Maestro -import qualified Dolla.Consensus.Proposing.Receptioning.API.Client.Dependencies as Receptionist.Client -import Dolla.Consensus.Proposing.Receptioning.API.Client.Client -import qualified Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output as Packaging +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Dependencies as Receptionist.Client +import Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Client +import qualified Dolla.Consensus.Proposing.Staging.Pipeline.IO.Output as Staging import Dolla.Consensus.Proposing.Simulating.GenRequest data Context log = Context { logger :: Logger , receptioningClient :: Receptionist.Client.Dependencies - , packagingOutputLog :: log Packaging.Output + , stagingOutputLog :: log Staging.Output , getMaestroOutputLog :: ByBlockOffset -> log Maestro.Output} underSupplying @@ -52,18 +52,21 @@ underSupplying -> S.SerialT m () underSupplying proposalSizeLimit = do - Context {packagingOutputLog,logger} <- ask - lift $ - generateRequests (proposalSizeLimit `div` 2) - >>= sendSimulatedRequest - >> simulateFirstConsensusReached - stream infinitely packagingOutputLog - & S.mapM (\Packaging.LocalProposalProduced {localOffset} -> do - log logger INFO $ "Local Proposal " ++ show localOffset ++ " Produced." - generateRequests (proposalSizeLimit `div` 2) - >>= sendSimulatedRequest - >> simulateLocalProposalConsumptionForBlock (localOffset + 2) - >> simulateConsensusReachedForBlock (localOffset + 2)) + Context {stagingOutputLog,logger} <- ask + lift $ do + generateRequests (proposalSizeLimit `div` 2) >>= sendSimulatedRequest + log logger INFO "Requests sent." + _ <- simulateFirstConsensusReached + log logger INFO "First Consensus Reached Simulated." + stream infinitely stagingOutputLog + & S.mapM (\Staging.LocalProposalStaged {localOffset} -> do + log logger INFO $ "Local Proposal " ++ show localOffset ++ " Staged." + generateRequests (proposalSizeLimit `div` 2) >>= sendSimulatedRequest + log logger INFO "Requests sent." + simulateLocalProposalConsumptionForBlock (localOffset + 2) + log logger INFO "Local Proposal Consumption Simulated." + simulateConsensusReachedForBlock (localOffset + 2) + log logger INFO "Consensus Reached Simulated.") simulateFirstConsensusReached :: ( MemoryStreamLoggable m log @@ -73,8 +76,8 @@ simulateFirstConsensusReached = do Context {getMaestroOutputLog} <- ask let byBlockOffset = ByBlockOffset 1 - maestroStream = getMaestroOutputLog byBlockOffset - void $ append maestroStream 0 (Maestro.ConsensusReached byBlockOffset) + maestroLog = getMaestroOutputLog byBlockOffset + void $ append maestroLog 0 (Maestro.ConsensusReached byBlockOffset) simulateLocalProposalConsumptionForBlock :: ( MemoryStreamLoggable m log @@ -86,8 +89,8 @@ simulateLocalProposalConsumptionForBlock blockOffset Context{getMaestroOutputLog, receptioningClient = Receptionist.Client.Dependencies {nodeId}} <- ask let byBlockOffset = ByBlockOffset {blockOffset} byProposer = ByProposer { blockOffset = blockOffset , proposerId = coerce nodeId } - maestroStream = getMaestroOutputLog byBlockOffset - void $ append maestroStream 0 (Maestro.ProposalAccepted {byProposer}) + maestroLog = getMaestroOutputLog byBlockOffset + void $ append maestroLog 0 (Maestro.ProposalAccepted {byProposer}) simulateConsensusReachedForBlock :: ( MemoryStreamLoggable m log @@ -98,8 +101,8 @@ simulateConsensusReachedForBlock blockOffset = do Context{getMaestroOutputLog} <- ask let byBlockOffset = ByBlockOffset {blockOffset} - maestroStream = getMaestroOutputLog byBlockOffset - void $ append maestroStream 1 (Maestro.ConsensusReached byBlockOffset) + maestroLog = getMaestroOutputLog byBlockOffset + void $ append maestroLog 1 (Maestro.ConsensusReached byBlockOffset) sendSimulatedRequest :: ( MonadIO m diff --git a/packages/simulating/package.yaml b/packages/simulating/package.yaml index ed9670b..310ccb5 100644 --- a/packages/simulating/package.yaml +++ b/packages/simulating/package.yaml @@ -5,7 +5,6 @@ extra-source-files: ghc-options: - -Wall -## - -j dependencies: - dolla-base == 1.0.0 @@ -14,12 +13,12 @@ dependencies: - dolla-consensus-proposing-receptioning-settings library: - source-dirs: lib + source-dirs: library dependencies: - dolla-consensus-base == 1.0.0 - dolla-consensus-proposing-receptioning - - dolla-consensus-proposing-packaging + - dolla-consensus-proposing-staging - mtl == 2.2.2 - text == 1.2.4.0 - byline == 0.3.2.1 @@ -33,7 +32,7 @@ library: executables: dolla-consensus-proposing-simulating : main: Executables.hs - source-dirs: app + source-dirs: executables ghc-options: - -main-is Executables.simulating - -threaded diff --git a/packages/simulating/settings/README.md b/packages/simulating/settings/README.md deleted file mode 100644 index e69de29..0000000 diff --git a/packages/simulating/settings/dolla-consensus-proposing-simulating-settings.cabal b/packages/simulating/settings/dolla-consensus-proposing-simulating-settings.cabal index 85d81f4..00c258e 100644 --- a/packages/simulating/settings/dolla-consensus-proposing-simulating-settings.cabal +++ b/packages/simulating/settings/dolla-consensus-proposing-simulating-settings.cabal @@ -4,13 +4,11 @@ cabal-version: 1.12 -- -- see: https://github.com/sol/hpack -- --- hash: 100fa44ef37d6042b674c42851d1b9ae47aa247f9a53ace731655fd5282dbb2c +-- hash: 9e3865a812e383c6e6d982be4755c64e2768275d7bc0ef31d4ec72b4ed1e4b38 name: dolla-consensus-proposing-simulating-settings version: 0.0.0.1 build-type: Simple -extra-source-files: - README.md library exposed-modules: @@ -19,7 +17,7 @@ library other-modules: Paths_dolla_consensus_proposing_simulating_settings hs-source-dirs: - lib + library ghc-options: -Wall build-depends: aeson ==1.4.7.1 diff --git a/packages/simulating/settings/lib/Dolla/Consensus/Proposing/Simulating/Settings.hs b/packages/simulating/settings/library/Dolla/Consensus/Proposing/Simulating/Settings.hs similarity index 84% rename from packages/simulating/settings/lib/Dolla/Consensus/Proposing/Simulating/Settings.hs rename to packages/simulating/settings/library/Dolla/Consensus/Proposing/Simulating/Settings.hs index e6cf9e7..00127b0 100644 --- a/packages/simulating/settings/lib/Dolla/Consensus/Proposing/Simulating/Settings.hs +++ b/packages/simulating/settings/library/Dolla/Consensus/Proposing/Simulating/Settings.hs @@ -9,7 +9,7 @@ import Dolla.Adapter.Aeson.AesonVia import Dolla.Common.Logging.Core import Dolla.Consensus.Proposing.Simulating.StressLoad -import qualified Dolla.Consensus.Proposing.Receptioning.API.Client.Settings as Receptionist.Client +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Settings as Receptionist.Client import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStore data Settings diff --git a/packages/simulating/settings/lib/Dolla/Consensus/Proposing/Simulating/StressLoad.hs b/packages/simulating/settings/library/Dolla/Consensus/Proposing/Simulating/StressLoad.hs similarity index 100% rename from packages/simulating/settings/lib/Dolla/Consensus/Proposing/Simulating/StressLoad.hs rename to packages/simulating/settings/library/Dolla/Consensus/Proposing/Simulating/StressLoad.hs diff --git a/packages/simulating/settings/package.yaml b/packages/simulating/settings/package.yaml index f0d7050..d5fe09b 100644 --- a/packages/simulating/settings/package.yaml +++ b/packages/simulating/settings/package.yaml @@ -1,14 +1,11 @@ name: dolla-consensus-proposing-simulating-settings version: 0.0.0.1 -extra-source-files: - - README.md ghc-options: - -Wall -## - -j library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 diff --git a/packages/staging/README.md b/packages/staging/README.md new file mode 100644 index 0000000..fe91588 --- /dev/null +++ b/packages/staging/README.md @@ -0,0 +1,318 @@ +/ [Consensus](https://github.com/dolla-consortium/consensus) / [Proposing](https://github.com/dolla-consortium/consensus-proposing) / Staging +# Staging Pipeline +- [Overview](#overview) +- [Project Tree](#project-tree) +- [Pipeline](#pipeline) + - [Junction](#junction) + - [IOs](#ios) + - [Input/Commands](#inputcommands) + - [Output/Events](#outputevents) + - [Pipe recipe](#pipe-recipe) +- [Pipes](#pipes) + - [Serializing](#serializing) + - [NonEmptying](#nonemptying) + - [Capping](#capping) + - [Persisting](#persisting) + + +# Overview + +`Staging` pipeline is responsible for packaging and staging requests from `receptioning` into proposal files. +Each of these local proposals once accepted by the consortium will be uniformly transacted on each consortium node. +The transactions will then be appended into each node ledgers. + +![visual](documentation/media/overview.png) +# Project Tree + +
+ +### 1. Pipeline + +`Staging` is a ***Pipeline*** +- a persisted input stream : [Input.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Input.hs) +- a line of Pipes Welded together in [Pipeline.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/Pipeline.hs) + - Sourcing initial inputs : `stream infinitely inputLog` + - Composition of **deterministic** *Pipes* + - Welding : Adapting IOs between pipes + + ``` + serializing .~> nonEmptying .~> capping .~> persisting + ``` + - Sinking final outputs : `sinking outputLog` +- a persisted output stream : [Output.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Output.hs) + +### 2. Pipes +The pipeline is using **Pipes** `serializing, nonEmptying, capping, persisting`, meaning each of them has +- An Input Stream +- A Stream Processing +- An Output Stream + +### 3. Execution Environment + +The `Staging` model (pipeline) is executed on +- The Log Engine used +- The Business Logic used on top of the consensus layer (requests in that context) + +You'll find in this folder different version of Pipeline.hs "polymorphically reduced" or concrete +- [Pipeline.hs](library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Pipeline.hs) over the event store +- [Pipeline.hs](library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Pipeline.hs) over the event store + Dolla Dummy Requests (Concrete Version) + +### 4. Executable + +`Staging` has some DevOps features as well + +- [Settings.hs](settings/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Settings.hs) always into a separated project `xxxx-packaging-settings` for deployment purposes in Zeus +- [Dependencies.hs](library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dependencies.hs) are derived from Settings if sub-dependencies are all Healthy + +[Executable.hs](executables/Executables.hs) +- Perform the HealhtChecks to obtain the pipeline dependencies +- Execute the pipeline + load the [junctions](#junction) in the EventStore Microservice +- Put the Microservice back in HealthCheck mode if any Exception bubbles up in the pipeline during execution. + +**N.B** : Microservice configuration and Deployment (Locally/Simulated/Production etc...) are defined in the package [Zeus](../zeus/) +
+ +# Pipeline +## Junction + +A Junction (Merger) is + - a set of persisted input streams + - a nondeterministic logic for merging these input streams + - a persisted output stream (input of a pipeline) + +The persisted input stream is the junction of 2 upstreams pipelines +- [Receptioning](../receptioning/README.md) providing collected requests +- [Detecting Tension](../detecting-tension/README.md) transmitting Local-Proposal-Starvation Notifications + +We are using the "User Defined Projections" EventStore feature to implement this junction + - javaScript snippets + - loaded in the event store microservice directly + - more details : https://eventstore.org/docs/projections/api/index.html + +> Defined in [Junction.hs](library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Junction.hs) + +> Executed in [Executable.hs](executables/Executables.hs) + +## IOs +### Input/Commands + +> Defined in [Input.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Input.hs) + +#### 1. Package request + +Executing that command means +- Accumulate each request into a temporary file. + +``` +FileName : x.tmp with x the offset of local proposal produced +``` + +- Detect when the temporary file is full + +``` +Current Size + new request size < configurable size limit +``` +- Convert this temporary file into a proposal file +``` +x.tmp -> x.proposal +``` +- Redirect the stream into a new Temporary File : `(x+1).tmp` +- Notify the downstream Broadcasting Section that a new local proposal is available. + +#### 2. Stage + +This command appears when the Local Proposal flow is **tensed**. The flow is tensed when the consensus is consuming more local proposal than produced. + +Executing it means : +- Converting the temporary file into a proposal file if requests are already accumulated. + +### Output/Events + +> Defined [Output.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Output.hs) + +Staging produces +- A Notification : `LocalProposalStaged {localOffset :: Offset}` +- a File `{localOffset}.proposal` containing requests with +``` +0 < Size <= Size Limit +``` +## Pipe recipe + +To produce the expected pipeline output , we are combining different pipes all together by +- A simple function composition (.) +- A welding : `map` to adapt Output Pipe `x` with Input Pipe `x+1` + +The `Staging` pipe recipe is +```haskell + stream infinitely inputLog -- sourcing + ~> serializing + .~> nonEmptying + .~> capping proposalSizeLimit + .~> persisting proposalRootFolder + .~> sinking outputLog +``` +> Defined in [Pipeline.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/Pipeline.hs) + +> The welding between each pipe is defined in [/Welding/BluePrint.hs](library/Dolla/Consensus/Proposing/Staging/Pipeline/Welding/BluePrint.hs) + +The pipe recipe goals are +### 1. Size properly the proposals +``` +0 < Size <= Size Limit +``` +Under the responsibility of +- [`NonEmptying`](#nonemptying) Pipe +- [`Capping`](#capping) Pipe + +### 2. Convert the temporary file into a proposal file. +Under the responsibility of +- [`Persisting`](#persisting) Pipe + +N.B : [`Serializing`](#serializing) will be removed eventually. We'll evaluate this when addressing data compression. + +# Pipes +## Serializing + +> Defined in [Pipe.hs](library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Pipe.hs) + +Just Transform `request` in `SerializedRequest` + +```haskell +newtype SerializedRequest = SerializedRequest [Word8] deriving (Eq,Show) +``` + +> N.B : `serializing` will be removed eventually. We'll address it when we'll add the request compression features. + +## NonEmptying + +### Problem + +Remembering the initial input of the section +```haskell +data Input request + = Stage -- ^ ask to "Staging Pipeline" to force the stage of a new local proposal with all the requests currently collected + | Package request -- ^ ask to to "Staging Pipeline" to package the request into a proposal according + -- some properties (see README.md) + +``` +It's totally natural to receive multiple `Stage` commands consecutively, E.g +- No Requests while many blocks are appended consecutively +- Etc... + +Forcing the production of empty proposal adds no value in our domain and provokes an accidental complexity downstream if not managed. +Therefore, we want to +- Remove the consecutive `Stage` commands from our input stream. +- Never start downstream processing with a `Stage` + +Said differently, we want to get the following property +``` +0 < Proposal File Size +``` + +### Approach + +With the following Natural Transformation +``` haskell +Staging.Input request ~> Maybe request +``` +We want the following stream property +- Never start by `Nothing` +- Never 2 consecutive `Nothing` + +E.g - We want the following transformation +``` haskell +[Nothing, Just r1, Just r2, Nothing, Nothing, Nothing] -> [Just r1, Just r2, Nothing] +``` +> Implemented in [Pipe.hs](library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Pipe.hs) + +> Tested in [PipeSpec.hs](test/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/PipeSpec.hs) + +## Capping + +### Problem + +We can't broadcast proposal files with unlimited memory size. We want the following property + +``` +Proposal File Size <= Size Limit +``` + +`Capping` is welded after `NonEmptying`, combining the 2 pipes will give the following property +``` +0 < Proposal File Size <= Size Limit +``` + + +### Approach + +To increase the expressivity of our previous domain we are doing the following natural transformation +```haskell +instance Weldable (NonEmptying.Output request) (Capping.Input request) where + weld + = \case + Nothing -> Capping.AskForACut + Just request -> Capping.Add request +``` +With the Following Output in [Output.hs](library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Output.hs) +```haskell +data Output request + = Cut + | Added request + deriving (Show,Eq) +``` +the polymorphism on `request` is reduced with the following type Class + +```haskell +class Sizable a where + getMemorySize :: a -> Byte + +capping + :: ( S.MonadAsync m + , Sizable item) + => Byte -- ^ proposalSizeLimit + -> S.SerialT m (Input item) + -> S.SerialT m (Output item) +```` +We are cutting the consecutiveness of `Added request` events if the cumulative memory size of requests is > a size limit given. +By "cutting", we mean adding a `Cut` in between 2 `Added request` events. + +Using a `Fold executed with a postscan` with the following State Machine + +![state-machine](documentation/media/state-machine.png) + +> Implemented in [Pipe.hs](library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Pipe.hs) + +> Tested in [PipeSpec.hs](test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/PipeSpec.hs) + +## Persisting + +### Prerequesites +This pipe prerequesites the following properties on its stream input +``` +0 < Proposal File Size <= Size Limit +``` +### Input/Output +```haskell +data Input request + = CommitProposal + | Persist request + deriving (Show,Eq,Functor) + +newtype Output + = LocalProposalPersisted {proposalId :: Offset} + deriving (Show,Eq) +``` +### Processing + +Using Streamly FileSystem primitives + +- Create a temporary file : `x.tmp` +- Persist request +- When receiving `CommitProposal` + - Convert `x.tmp` into `x.proposal` + - produce `LocalProposalPersisted x` + - `x = x + 1` + +> Implemented in [Pipe.hs](library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Pipe.hs) + +> Integration Test in [PipeSpec.hs](test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/PipeSpec.hs) diff --git a/packages/staging/documentation/media/overview.png b/packages/staging/documentation/media/overview.png new file mode 100644 index 0000000..8ea8774 Binary files /dev/null and b/packages/staging/documentation/media/overview.png differ diff --git a/packages/staging/documentation/media/package-file-tree.png b/packages/staging/documentation/media/package-file-tree.png new file mode 100644 index 0000000..5a35050 Binary files /dev/null and b/packages/staging/documentation/media/package-file-tree.png differ diff --git a/packages/packaging/documentation/media/state-machine.png b/packages/staging/documentation/media/state-machine.png similarity index 100% rename from packages/packaging/documentation/media/state-machine.png rename to packages/staging/documentation/media/state-machine.png diff --git a/packages/staging/dolla-consensus-proposing-staging.cabal b/packages/staging/dolla-consensus-proposing-staging.cabal new file mode 100644 index 0000000..bd2e53f --- /dev/null +++ b/packages/staging/dolla-consensus-proposing-staging.cabal @@ -0,0 +1,120 @@ +cabal-version: 1.12 + +-- This file has been generated from package.yaml by hpack version 0.33.0. +-- +-- see: https://github.com/sol/hpack +-- +-- hash: 854e5eef0fdfba2dee6f84a75a738313eec78a35bca9eb8efcfab9cd5de2df75 + +name: dolla-consensus-proposing-staging +version: 0.0.0.1 +build-type: Simple +extra-source-files: + README.md + +library + exposed-modules: + Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dependencies + Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dolla.Junction + Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dolla.Pipeline + Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Pipeline + Dolla.Consensus.Proposing.Staging.Pipeline.IO.Input + Dolla.Consensus.Proposing.Staging.Pipeline.IO.Output + Dolla.Consensus.Proposing.Staging.Pipeline.Pipeline + Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Input + Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Pipe + Dolla.Consensus.Proposing.Staging.Pipeline.Welding.BluePrint + Dolla.Consensus.Proposing.Staging.Pipes.Capping.Input + Dolla.Consensus.Proposing.Staging.Pipes.Capping.Output + Dolla.Consensus.Proposing.Staging.Pipes.Capping.Pipe + Dolla.Consensus.Proposing.Staging.Pipes.Capping.Sizable + Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Input + Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Output + Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Pipe + Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Input + Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Output + Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Pipe + Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Input + Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Output + Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Pipe + Dolla.Consensus.Proposing.Staging.Pipes.Serializing.SerializedRequest + other-modules: + Paths_dolla_consensus_proposing_staging + hs-source-dirs: + library + ghc-options: -Wall + build-depends: + aeson ==1.4.7.1 + , base ==4.13.0.0 + , bytestring ==0.10.10.0 + , containers ==0.6.2.1 + , directory ==1.3.6.0 + , dolla-base ==1.0.0 + , dolla-consensus-base ==1.0.0 + , dolla-consensus-proposing-staging-settings + , exceptions ==0.10.4 + , interpolatedstring-perl6 ==1.0.2 + , lens ==4.18.1 + , mtl ==2.2.2 + , natural-transformation + , random ==1.1 + , streamly ==0.7.2 + , text ==1.2.4.0 + , uuid ==1.3.13 + default-language: Haskell2010 + +executable dolla-consensus-proposing-staging + main-is: Executables.hs + other-modules: + Paths_dolla_consensus_proposing_staging + hs-source-dirs: + executables + ghc-options: -Wall -main-is Executables.execute -threaded -rtsopts -with-rtsopts=-N + build-depends: + base ==4.13.0.0 + , dolla-base ==1.0.0 + , dolla-consensus-base ==1.0.0 + , dolla-consensus-proposing-staging + , dolla-consensus-proposing-staging-settings + , exceptions ==0.10.4 + , mtl ==2.2.2 + , streamly ==0.7.2 + default-language: Haskell2010 + +test-suite dolla-consensus-proposing-staging-test + type: exitcode-stdio-1.0 + main-is: Spec.hs + other-modules: + Dolla.Consensus.Proposing.Staging.DummyRequest + Dolla.Consensus.Proposing.Staging.Pipes.Capping.GenInput + Dolla.Consensus.Proposing.Staging.Pipes.Capping.PipeSpec + Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.PipeSpec + Dolla.Consensus.Proposing.Staging.Pipes.Persisting.GenInput + Dolla.Consensus.Proposing.Staging.Pipes.Persisting.PipeSpec + Dolla.Consensus.Proposing.Staging.Pipes.Serializing.GenInput + Dolla.Consensus.Proposing.Staging.Pipes.Serializing.PipeSpec + Paths_dolla_consensus_proposing_staging + hs-source-dirs: + test + ghc-options: -Wall -threaded -rtsopts -with-rtsopts=-N + build-depends: + MissingH + , QuickCheck + , aeson ==1.4.7.1 + , base + , bytestring ==0.10.10.0 + , directory ==1.3.6.0 + , dolla-base + , dolla-consensus-base ==1.0.0 + , dolla-consensus-proposing-staging + , generic-arbitrary + , hspec + , hspec-core ==2.7.1 + , interpolatedstring-perl6 ==1.0.2 + , mtl ==2.2.2 + , quickcheck-instances + , random ==1.1 + , streamly ==0.7.2 + , turtle ==1.5.19 + , uuid ==1.3.13 + default-language: Haskell2010 diff --git a/packages/staging/executables/Executables.hs b/packages/staging/executables/Executables.hs new file mode 100644 index 0000000..6692335 --- /dev/null +++ b/packages/staging/executables/Executables.hs @@ -0,0 +1,34 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE NamedFieldPuns #-} +module Executables + ( execute + ) where + +import Prelude hiding (log) +import Control.Monad.Reader + +import Streamly.Prelude as S +import qualified Streamly.Internal.Prelude as SIP + +import Dolla.Common.Logging.Core +import Dolla.Common.Executable.Executable + +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dolla.Pipeline (staging) +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Settings +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dependencies +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dolla.Junction + +execute :: IO () +execute + = executeMicroservice + (\Settings {logger} -> logger) + executePipeline + where + executePipeline :: ReaderT Dependencies IO () + executePipeline = do + dependencies @ Dependencies {eventStoreClient,nodeId,logger} <- ask + withReaderT (const (nodeId, eventStoreClient)) loadJunctionInEventStore + log logger INFO "Pipeline Starting" + drain $ SIP.runReaderT dependencies staging + log logger INFO "End Of Pipeline" + diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Dependencies.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dependencies.hs similarity index 86% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Dependencies.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dependencies.hs index 8129030..845706e 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Dependencies.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dependencies.hs @@ -1,7 +1,7 @@ {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE MultiParamTypeClasses #-} -module Dolla.Consensus.Proposing.Packaging.Dependencies (Dependencies (..)) where +module Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dependencies (Dependencies (..)) where import Dolla.Common.NodeId import Dolla.Common.Dependencies.Core @@ -10,7 +10,7 @@ import Dolla.Common.Memory.Byte (Byte) import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStore import Dolla.Consensus.Proposal.Persistence -import Dolla.Consensus.Proposing.Packaging.Settings +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Settings data Dependencies = Dependencies diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Junction.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Junction.hs new file mode 100644 index 0000000..3d4b634 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Junction.hs @@ -0,0 +1,73 @@ +{-# LANGUAGE QuasiQuotes, ExtendedDefaultRules #-} +{-# LANGUAGE RecordWildCards #-} + +module Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dolla.Junction + (loadJunctionInEventStore) where + +import Prelude hiding (log) +import Data.Coerce (coerce) +import qualified Dolla.Libraries.LogEngine.Instances.EventStore.Settings as EventStore +import Dolla.Libraries.LogEngine.Instances.EventStore.Projection.Client +import Text.InterpolatedString.Perl6 (qc) +import Dolla.Common.NodeId +import Control.Monad.Reader (ReaderT,ask, withReaderT) +import Dolla.Common.Logging.Core +import Dolla.Libraries.LogEngine.Instances.EventStore.Projection.Definition +import Dolla.Consensus.Log.LogNameIndex + +-- | A Junction (Merger) is + --a set of persisted input streams + --a nondeterministic logic for merging these input streams + --a persisted output stream (input of a pipeline) + +-- | Junction for Staging Pipeline : Logic to generate the input stream of the pipeline. +-- We are using the "User Defined Projections" feature from the EventStore to implement this junction : +-- - javaScript snippets +-- - loaded in the event store microservice directly +-- - more details : https://eventstore.org/docs/projections/api/index.html +loadJunctionInEventStore :: ReaderT (NodeId, EventStore.Dependencies) IO () +loadJunctionInEventStore = do + (nodeId , EventStore.Dependencies {..}) <- ask + let projectionName = coerce nodeId ++ "_proposing_staging_input" + starvingDetectionOutputLogStreamName = getStreamNameFromIndex ProposingStarvingDetectionOutputLog + stagingOutputLogStreamName = getStreamNameFromIndex ProposingReceptioningOutputLog + stagingInputLogLogStreamName = getStreamNameFromIndex ProposingStagingInputLog + body = [qc| options(\{ + reorderEvents: false, + processingLag: 0 + }) + fromStreams ([ '{starvingDetectionOutputLogStreamName}' + , '{stagingOutputLogStreamName}']) + .when(\{ + $any : function(s,e)\{ + function getOutputStream() \{ + return '{stagingInputLogLogStreamName}'} + if (e.eventType == "LocalProposalFlowTensed" ) \{ + emit ( getOutputStream () + , "Stage" + , \{"tag": "Stage"} + , \{}); + } + if (e.eventType == "Receptioned" ) \{ + var messageJson = JSON.parse(e.bodyRaw) + emit ( getOutputStream () + , "Package" + , \{"tag": "Package", + "contents" : messageJson.contents} + , \{}); + } + } // $any + }) + .outputState()|] + + log logger DEBUG "Loading Projection for Maestro Input " + -- TODO : Verified if the right projection version is loaded + result <- withReaderT snd $ createContinuousProjection + (ProjectionName projectionName) + (ProjectionBody body) + (ProjectionEnabled True) + (ProjectionEmitting True) + (ProjectionTrackEmittedStreams True) + log logger DEBUG $ "Projection loaded : " ++ show result + return () + diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStoreAndDolla.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Pipeline.hs similarity index 54% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStoreAndDolla.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Pipeline.hs index f849596..911c738 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStoreAndDolla.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Dolla/Pipeline.hs @@ -1,6 +1,6 @@ {-# LANGUAGE DuplicateRecordFields #-} {-# LANGUAGE FlexibleContexts #-} -module Dolla.Consensus.Proposing.Packaging.Pipeline.OverEventStoreAndDolla (packaging) where +module Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dolla.Pipeline (staging) where import Prelude hiding (log) import Control.Monad.Reader @@ -10,19 +10,19 @@ import Data.Data import qualified Streamly as S -import qualified Dolla.Consensus.Proposing.Packaging.Pipeline.OverEventStore as OverEventStore -import Dolla.Consensus.Proposing.Packaging.Dependencies +import qualified Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Pipeline as OverEventStore +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dependencies import Dolla.Consensus.Request import Dolla.Consensus.Dummy.Client.Request import Dolla.Consensus.Consortium.Request -packaging +staging :: ( MonadReader Dependencies m , S.MonadAsync m , MonadCatch m) => S.SerialT m () -packaging - = OverEventStore.packaging (Proxy :: Proxy (Request DollaClientRequest ConsortiumRequest)) +staging + = OverEventStore.staging (Proxy :: Proxy (Request DollaClientRequest ConsortiumRequest)) diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStore.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Pipeline.hs similarity index 51% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStore.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Pipeline.hs index 4a939c8..0643c19 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/OverEventStore.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Pipeline.hs @@ -1,7 +1,7 @@ {-# LANGUAGE DuplicateRecordFields #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE FlexibleContexts #-} -module Dolla.Consensus.Proposing.Packaging.Pipeline.OverEventStore (packaging) where +module Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Pipeline (staging) where import Prelude hiding (log) import Data.Data @@ -12,13 +12,15 @@ import Control.Monad.Catch (MonadCatch) import qualified Streamly as S -import qualified Dolla.Consensus.Proposing.Packaging.Pipeline.Generic as Generic -import Dolla.Consensus.Proposing.Packaging.Dependencies +import qualified Dolla.Consensus.Proposing.Staging.Pipeline.Pipeline as Generic +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Dependencies import Dolla.Consensus.Log.EventStoreLog -import Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Input +import Dolla.Consensus.Log.LogNameIndex + +import Dolla.Consensus.Proposing.Staging.Pipeline.IO.Input import Dolla.Libraries.LogEngine.Instances.EventStore.EventStoreLog -packaging +staging :: ( ToJSON request , FromJSON request , Show request @@ -27,11 +29,13 @@ packaging , MonadCatch m) => Proxy request -> S.SerialT m () -packaging proxy = do - Dependencies {eventStoreClient} <- ask - Generic.packaging - (asProxyTypeOf (getEventStoreLog eventStoreClient LocalRequestLog) (getProxyLogInput proxy)) - (getEventStoreLog eventStoreClient ProposingPackagingOutputLog) +staging proxy = do + Dependencies {eventStoreClient,proposalRootFolder,proposalSizeLimit} <- ask + Generic.staging + proposalRootFolder + proposalSizeLimit + (asProxyTypeOf (getEventStoreLog eventStoreClient ProposingStagingInputLog) (getProxyLogInput proxy)) + (getEventStoreLog eventStoreClient ProposingStagingOutputLog) getProxyLogInput :: Proxy request -> Proxy (EventStoreLog (Input request)) getProxyLogInput _ = Proxy diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Input.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Input.hs new file mode 100644 index 0000000..aff6c9e --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Input.hs @@ -0,0 +1,25 @@ +{-# LANGUAGE DerivingVia #-} +{-# LANGUAGE FlexibleContexts #-} +{-# LANGUAGE UndecidableInstances #-} +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE DeriveGeneric #-} +module Dolla.Consensus.Proposing.Staging.Pipeline.IO.Input + ( Input (..)) where + +import Data.Aeson +import GHC.Generics +import Dolla.Adapter.Aeson.AesonVia + +-- Data Origins +-- The Staging input stream is the junction of 2 upstream pipelines : +-- - Receptioning : providing collected requests +-- - Detecting-Tension : notify when the local proposal flow is tensed. + +data Input request + = Stage -- ^ ask to "Staging Pipeline" to force the stage of a new local proposal with all the requests currently collected + | Package request -- ^ ask to to "Staging Pipeline" to package the request into a proposal according + -- some properties (see README.md) + deriving (Eq,Show,Generic) + deriving (FromJSON) via DefaultJSON (Input request) + + diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Output.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Output.hs similarity index 75% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Output.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Output.hs index 2c0fb81..e16b85d 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipeline/IO/Output.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/IO/Output.hs @@ -4,7 +4,7 @@ {-# LANGUAGE DuplicateRecordFields #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output +module Dolla.Consensus.Proposing.Staging.Pipeline.IO.Output ( Output (..)) where import Data.Aeson @@ -14,9 +14,9 @@ import Dolla.Common.Offset (Offset) import Dolla.Libraries.LogEngine.Appendable newtype Output - = LocalProposalProduced {localOffset :: Offset} + = LocalProposalStaged {localOffset :: Offset} deriving (Eq, Show, Generic) deriving (ToJSON, FromJSON) via DefaultJSON Output instance Appendable Output where - getItemName LocalProposalProduced {} = "LocalProposalProduced" \ No newline at end of file + getItemName LocalProposalStaged {} = "LocalProposalStaged" \ No newline at end of file diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Pipeline.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Pipeline.hs new file mode 100644 index 0000000..120fef0 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Pipeline.hs @@ -0,0 +1,64 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE FlexibleContexts #-} + +module Dolla.Consensus.Proposing.Staging.Pipeline.Pipeline + (staging) + where + +import Prelude +import Data.Aeson.Types (ToJSON,FromJSON) + +import Control.Monad.Catch (MonadCatch) + + +import qualified Streamly as S + +import Dolla.Libraries.LogEngine.LogEngine (MemoryStreamLoggable (..)) +import Dolla.Common.Range (infinitely) + +import Dolla.Consensus.Proposing.Staging.Pipeline.IO.Input +import Dolla.Consensus.Proposing.Staging.Pipeline.IO.Output + +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Pipe (serializing) +import Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Pipe (nonEmptying) +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Pipe (capping) +import Dolla.Common.Memory.Byte (Byte) +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Pipe (persisting) +import Dolla.Consensus.Proposal.Persistence (ProposalRootFolder) +import Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Pipe (sinking) + +import Dolla.Common.Pipeline.Weldable ((~>),(.~>)) +import Dolla.Consensus.Proposing.Staging.Pipeline.Welding.BluePrint () + +-- | Staging Pipeline expressed in its most polymorphic way : +-- - Log Engine Agnostic +-- - Request Agnostic +staging + :: ( ToJSON request + , FromJSON request + , Show request + , MonadCatch m + , S.MonadAsync m + , MemoryStreamLoggable m log + ) + => ProposalRootFolder + -> Byte + -> log (Input request) + -> log Output + -> S.SerialT m () +staging + proposalRootFolder + proposalSizeLimit + inputLog + outputLog + = stream infinitely inputLog -- sourcing + ~> serializing + .~> nonEmptying + .~> capping proposalSizeLimit + .~> persisting proposalRootFolder + .~> sinking outputLog + + + + + diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Sinking/Input.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Sinking/Input.hs new file mode 100644 index 0000000..cb9d061 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Sinking/Input.hs @@ -0,0 +1,10 @@ +module Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Input + ( Input (..)) where + +import Dolla.Common.Offset (Offset) + + +newtype Input = SinkNewLocalProposal {proposalId :: Offset} + + + diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Notifying/Pipe.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Sinking/Pipe.hs similarity index 57% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Notifying/Pipe.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Sinking/Pipe.hs index 9ad4e73..afc15b2 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Notifying/Pipe.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Sinking/Pipe.hs @@ -2,7 +2,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Pipe (notifying) where +module Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Pipe (sinking) where import Prelude hiding (log,writeFile) import Control.Monad.Reader @@ -11,21 +11,21 @@ import qualified Streamly.Prelude as S hiding (length,bracket) import qualified Streamly as S import Dolla.Libraries.LogEngine.LogEngine -import qualified Dolla.Consensus.Proposing.Packaging.Pipeline.IO.Output as Packaging -import Dolla.Consensus.Proposing.Packaging.Pipes.Notifying.Input +import qualified Dolla.Consensus.Proposing.Staging.Pipeline.IO.Output as Staging +import Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Input -notifying +sinking :: ( MemoryStreamLoggable m log , S.MonadAsync m ) - => log Packaging.Output + => log Staging.Output -> S.SerialT m Input -> S.SerialT m () -notifying outputLog +sinking outputLog = S.mapM - (\LocalProposalProduced {proposalId} -> + (\SinkNewLocalProposal {proposalId} -> void $ append outputLog proposalId - $ Packaging.LocalProposalProduced proposalId) + $ Staging.LocalProposalStaged proposalId) diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Welding/BluePrint.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Welding/BluePrint.hs new file mode 100644 index 0000000..e34ffb5 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipeline/Welding/BluePrint.hs @@ -0,0 +1,56 @@ +{-# LANGUAGE LambdaCase #-} +{-# LANGUAGE MultiParamTypeClasses #-} +{-# LANGUAGE FlexibleInstances #-} +{-# LANGUAGE RecordWildCards #-} + +{-# OPTIONS_GHC -fno-warn-orphans #-} + +module Dolla.Consensus.Proposing.Staging.Pipeline.Welding.BluePrint + () where + +import Data.Word (Word8) +import Data.Coerce (coerce) +import Dolla.Common.Pipeline.Weldable +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.SerializedRequest + +import qualified Dolla.Consensus.Proposing.Staging.Pipeline.IO.Input as Staging + +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Input as Serializing +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Output as Serializing + +import qualified Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Input as NonEmptying +import qualified Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Output as NonEmptying + +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Capping.Input as Capping +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Capping.Output as Capping + +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Input as Persisting +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Output as Persisting + +import qualified Dolla.Consensus.Proposing.Staging.Pipeline.Sinking.Input as Sinking + +instance Weldable (Staging.Input request) (Serializing.Input request) where + weld + = \case + Staging.Stage {} -> Nothing + Staging.Package request -> Just request + +instance Weldable (Serializing.Output a) (NonEmptying.Input a) where + weld = coerce + +instance Weldable (NonEmptying.Output a) (Capping.Input a) where + weld + = \case + Nothing -> Capping.AskForACut + Just a -> Capping.Add a + +instance Weldable (Capping.Output SerializedRequest) (Persisting.Input [Word8]) where + weld + = \case + Capping.Cut -> Persisting.CommitProposal + Capping.Added (SerializedRequest word8s) -> Persisting.Persist word8s + +instance Weldable Persisting.Output Sinking.Input where + weld + = \case + Persisting.LocalProposalPersisted {..} -> Sinking.SinkNewLocalProposal {..} diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Input.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Input.hs similarity index 68% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Input.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Input.hs index 6e2ca86..5ab73dc 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Input.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Input.hs @@ -1,6 +1,6 @@ {-# LANGUAGE DeriveFunctor #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Input +module Dolla.Consensus.Proposing.Staging.Pipes.Capping.Input ( Input (..)) where data Input request diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Output.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Output.hs similarity index 58% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Output.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Output.hs index a4f3c05..e75da01 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Output.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Output.hs @@ -1,4 +1,4 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Output +module Dolla.Consensus.Proposing.Staging.Pipes.Capping.Output (Output (..)) where diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Pipe.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Pipe.hs similarity index 88% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Pipe.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Pipe.hs index adcc983..ce30163 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Pipe.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Pipe.hs @@ -4,7 +4,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE LambdaCase #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Pipe +module Dolla.Consensus.Proposing.Staging.Pipes.Capping.Pipe ( capping) where @@ -19,9 +19,9 @@ import Streamly.Internal.Data.Fold.Types import Dolla.Common.Memory.Byte (Byte) -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Input -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Sizable -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Output +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Sizable +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Output data State item diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Sizable.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Sizable.hs similarity index 66% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Sizable.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Sizable.hs index ab54ad0..bbab77a 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/Sizable.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Capping/Sizable.hs @@ -1,4 +1,4 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Sizable +module Dolla.Consensus.Proposing.Staging.Pipes.Capping.Sizable ( Sizable (..)) where diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Input.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Input.hs new file mode 100644 index 0000000..923cff7 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Input.hs @@ -0,0 +1,5 @@ +module Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Input + (Input ) + where + +type Input request = Maybe request \ No newline at end of file diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Output.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Output.hs new file mode 100644 index 0000000..8227372 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Output.hs @@ -0,0 +1,5 @@ +module Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Output + (Output ) + where + +type Output request = Maybe request \ No newline at end of file diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Pipe.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Pipe.hs similarity index 94% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Pipe.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Pipe.hs index f3a3916..4e7989b 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/Pipe.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/Pipe.hs @@ -2,7 +2,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE LambdaCase #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Pipe +module Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Pipe ( nonEmptying) where diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Input.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Input.hs similarity index 68% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Input.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Input.hs index 5eb10f8..a4817c4 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Input.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Input.hs @@ -1,6 +1,6 @@ {-# LANGUAGE DeriveFunctor #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Input +module Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Input (Input (..)) where diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Output.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Output.hs similarity index 68% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Output.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Output.hs index ffe1468..24a33b2 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Output.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Output.hs @@ -1,4 +1,4 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Output +module Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Output (Output (..)) where diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Pipe.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Pipe.hs similarity index 93% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Pipe.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Pipe.hs index 3a19d8b..d57d73e 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/Pipe.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/Pipe.hs @@ -4,7 +4,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE LambdaCase #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Pipe +module Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Pipe ( persisting ) where @@ -27,8 +27,8 @@ import qualified Streamly.Internal.Prelude as S import qualified Dolla.Common.Streamly as S (groupsBy2,lmapM2, writeChunks2) import Dolla.Common.Offset import Dolla.Consensus.Proposal.Persistence -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Input -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Output +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Output persisting :: (S.MonadAsync m ,Storable a) diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Input.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Input.hs new file mode 100644 index 0000000..42ee351 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Input.hs @@ -0,0 +1,7 @@ +{-# LANGUAGE DeriveFunctor #-} + +module Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Input + ( Input ) where + +type Input request = Maybe request + diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Output.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Output.hs new file mode 100644 index 0000000..8bf69aa --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Output.hs @@ -0,0 +1,6 @@ +module Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Output + (Output (..)) + where + +newtype Output request = Output (Maybe request) deriving (Eq,Show) + diff --git a/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Pipe.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Pipe.hs new file mode 100644 index 0000000..d95d751 --- /dev/null +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/Pipe.hs @@ -0,0 +1,26 @@ +{-# LANGUAGE DuplicateRecordFields #-} +{-# LANGUAGE FlexibleContexts #-} + +module Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Pipe + (serializing) + where + +import Prelude +import Data.ByteString hiding (length,map,append) +import Data.Coerce (coerce) +import Data.Aeson.Types (ToJSON) + +import qualified Streamly as S +import qualified Streamly.Prelude as S + +import Dolla.Common.UUID.Deterministic +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Input as Input +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Output as Output +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.SerializedRequest + +serializing + :: ( Monad m, ToJSON a) + => S.SerialT m (Input a) + -> S.SerialT m (Output SerializedRequest) +serializing = S.map (\input -> Output (coerce . unpack . getEncodedItem <$> input)) + \ No newline at end of file diff --git a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/SerializedRequest.hs b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/SerializedRequest.hs similarity index 63% rename from packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/SerializedRequest.hs rename to packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/SerializedRequest.hs index 486f82e..2a1e43e 100644 --- a/packages/packaging/lib/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/SerializedRequest.hs +++ b/packages/staging/library/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/SerializedRequest.hs @@ -1,9 +1,9 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.SerializedRequest +module Dolla.Consensus.Proposing.Staging.Pipes.Serializing.SerializedRequest ( SerializedRequest (..)) where import Data.Word (Word8) -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Sizable +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Sizable newtype SerializedRequest = SerializedRequest [Word8] deriving (Eq,Show) diff --git a/packages/packaging/package.yaml b/packages/staging/package.yaml similarity index 61% rename from packages/packaging/package.yaml rename to packages/staging/package.yaml index ca76900..12d14e2 100644 --- a/packages/packaging/package.yaml +++ b/packages/staging/package.yaml @@ -1,21 +1,20 @@ -name: dolla-consensus-proposing-packaging +name: dolla-consensus-proposing-staging version: 0.0.0.1 extra-source-files: - README.md ghc-options: - -Wall -## - -j + library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 - dolla-consensus-base == 1.0.0 - - dolla-consensus-proposing-packaging-settings - + - dolla-consensus-proposing-staging-settings - base == 4.13.0.0 - containers == 0.6.2.1 - mtl == 2.2.2 @@ -32,20 +31,26 @@ library: - natural-transformation executables: - dolla-consensus-proposing-packaging : - main: Executables.hs - source-dirs: app - ghc-options: - - -main-is Executables.packaging - - -threaded - - -rtsopts - - -with-rtsopts=-N - dependencies: - - dolla-consensus-proposing-packaging - - base == 4.13.0.0 + dolla-consensus-proposing-staging : + main: Executables.hs + source-dirs: executables + ghc-options: + - -main-is Executables.execute + - -threaded + - -rtsopts + - -with-rtsopts=-N + dependencies: + - base == 4.13.0.0 + - mtl == 2.2.2 + - streamly == 0.7.2 + - exceptions == 0.10.4 + - dolla-base == 1.0.0 + - dolla-consensus-base == 1.0.0 + - dolla-consensus-proposing-staging + - dolla-consensus-proposing-staging-settings tests: - dolla-consensus-proposing-packaging-test: + dolla-consensus-proposing-staging-test: main: Spec.hs source-dirs: test ghc-options: @@ -53,7 +58,7 @@ tests: - -rtsopts - -with-rtsopts=-N dependencies: - - dolla-consensus-proposing-packaging + - dolla-consensus-proposing-staging - dolla-base - base - streamly == 0.7.2 diff --git a/packages/packaging/settings/dolla-consensus-proposing-packaging-settings.cabal b/packages/staging/settings/dolla-consensus-proposing-staging-settings.cabal similarity index 57% rename from packages/packaging/settings/dolla-consensus-proposing-packaging-settings.cabal rename to packages/staging/settings/dolla-consensus-proposing-staging-settings.cabal index 4c273d4..c3920eb 100644 --- a/packages/packaging/settings/dolla-consensus-proposing-packaging-settings.cabal +++ b/packages/staging/settings/dolla-consensus-proposing-staging-settings.cabal @@ -4,21 +4,19 @@ cabal-version: 1.12 -- -- see: https://github.com/sol/hpack -- --- hash: cd442c399b12c857f9e356f24012ad6c3588c608ef09f2a59820f16648a39c11 +-- hash: 99ccbf92ab6ea0490964fb920f828fb0bf190fc021aaa9a0a8101baeb0990665 -name: dolla-consensus-proposing-packaging-settings +name: dolla-consensus-proposing-staging-settings version: 0.0.0.1 build-type: Simple -extra-source-files: - README.md library exposed-modules: - Dolla.Consensus.Proposing.Packaging.Settings + Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Settings other-modules: - Paths_dolla_consensus_proposing_packaging_settings + Paths_dolla_consensus_proposing_staging_settings hs-source-dirs: - lib + library ghc-options: -Wall build-depends: aeson ==1.4.7.1 diff --git a/packages/packaging/settings/lib/Dolla/Consensus/Proposing/Packaging/Settings.hs b/packages/staging/settings/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Settings.hs similarity index 86% rename from packages/packaging/settings/lib/Dolla/Consensus/Proposing/Packaging/Settings.hs rename to packages/staging/settings/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Settings.hs index 774c909..4d85de1 100644 --- a/packages/packaging/settings/lib/Dolla/Consensus/Proposing/Packaging/Settings.hs +++ b/packages/staging/settings/library/Dolla/Consensus/Proposing/Staging/Execution/Environment/EventStore/Settings.hs @@ -1,6 +1,6 @@ {-# LANGUAGE DerivingVia #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Packaging.Settings (Settings (..)) where +module Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Settings (Settings (..)) where import Data.Aeson import GHC.Generics diff --git a/packages/packaging/settings/package.yaml b/packages/staging/settings/package.yaml similarity index 55% rename from packages/packaging/settings/package.yaml rename to packages/staging/settings/package.yaml index abccf0f..1db3fa9 100644 --- a/packages/packaging/settings/package.yaml +++ b/packages/staging/settings/package.yaml @@ -1,14 +1,11 @@ -name: dolla-consensus-proposing-packaging-settings +name: dolla-consensus-proposing-staging-settings version: 0.0.0.1 -extra-source-files: - - README.md ghc-options: - -Wall -## - -j library: - source-dirs: lib + source-dirs: library dependencies: - dolla-base == 1.0.0 diff --git a/packages/packaging/settings/test/Spec.hs b/packages/staging/settings/test/Spec.hs similarity index 100% rename from packages/packaging/settings/test/Spec.hs rename to packages/staging/settings/test/Spec.hs diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/DummyRequest.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/DummyRequest.hs similarity index 90% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/DummyRequest.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/DummyRequest.hs index 1843e1d..1ea538c 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/DummyRequest.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/DummyRequest.hs @@ -1,6 +1,6 @@ {-# LANGUAGE DerivingVia #-} {-# LANGUAGE DeriveGeneric #-} -module Dolla.Consensus.Proposing.Packaging.DummyRequest +module Dolla.Consensus.Proposing.Staging.DummyRequest (DummyRequest (..)) where diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/GenInput.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/GenInput.hs similarity index 77% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/GenInput.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/GenInput.hs index a61cfeb..1a79cf1 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/GenInput.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/GenInput.hs @@ -1,11 +1,11 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Capping.GenInput +module Dolla.Consensus.Proposing.Staging.Pipes.Capping.GenInput ( InputUnderTests (..)) where import Test.QuickCheck.Arbitrary import Test.QuickCheck.Gen import Test.QuickCheck.Instances.UUID () import Test.QuickCheck.Instances () -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Input instance Arbitrary a => Arbitrary (InputUnderTests a) where diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/PipeSpec.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/PipeSpec.hs similarity index 67% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/PipeSpec.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/PipeSpec.hs index cdbd76b..a3280a4 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Capping/PipeSpec.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Capping/PipeSpec.hs @@ -3,7 +3,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE ScopedTypeVariables #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Capping.PipeSpec +module Dolla.Consensus.Proposing.Staging.Pipes.Capping.PipeSpec (spec) where @@ -18,27 +18,27 @@ import qualified Streamly as S import qualified Streamly.Prelude as S import qualified Streamly.Internal.Data.Fold.Types as S -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Input -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.GenInput -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Output +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.GenInput +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Output -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Pipe (capping) +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Pipe (capping) import Data.Function ((&)) import Data.Coerce (coerce) import Dolla.Common.Memory.Byte hiding (getMemorySize) -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Sizable +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Sizable import Data.Monoid spec :: Spec spec = do describe "About Capping item " $ do it "outputs a series of consecutives items having always a cumulated memory size <= limit given" - $ property - $ \inputs - -> getGeneratedInputStream inputs - & capping (10 * kb) - & S.scan consecutivesMemorySizeAdded - & S.mapM_ (`shouldSatisfy` (<= (10 * kb))) + $ property + $ \inputs + -> getGeneratedInputStream inputs + & capping (10 * kb) + & S.scan consecutivesMemorySizeAdded + & S.mapM_ (`shouldSatisfy` (<= (10 * kb))) getGeneratedInputStream :: Monad m diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/PipeSpec.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/PipeSpec.hs similarity index 85% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/PipeSpec.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/PipeSpec.hs index 2fd1761..0b986ae 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/NonEmptying/PipeSpec.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/NonEmptying/PipeSpec.hs @@ -2,7 +2,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE ScopedTypeVariables #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.PipeSpec +module Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.PipeSpec (spec) where import Test.QuickCheck.Instances () @@ -13,7 +13,7 @@ import qualified Streamly.Prelude as S import qualified Streamly.Internal.Prelude as S -import Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Pipe (nonEmptying) +import Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Pipe (nonEmptying) import Data.Function ((&)) spec :: Spec diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/GenInput.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/GenInput.hs similarity index 76% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/GenInput.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/GenInput.hs index 6c54180..f554edc 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/GenInput.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/GenInput.hs @@ -1,11 +1,11 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.GenInput +module Dolla.Consensus.Proposing.Staging.Pipes.Persisting.GenInput ( InputUnderTests (..)) where import Test.QuickCheck.Arbitrary import Test.QuickCheck.Gen import Test.QuickCheck.Instances.UUID () import Test.QuickCheck.Instances () -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Input instance Arbitrary a => Arbitrary (InputUnderTests a) where diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/PipeSpec.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/PipeSpec.hs similarity index 80% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/PipeSpec.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/PipeSpec.hs index 9a06f63..b05d90a 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Persisting/PipeSpec.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Persisting/PipeSpec.hs @@ -4,7 +4,7 @@ {-# LANGUAGE ScopedTypeVariables #-} {-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE LambdaCase #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.PipeSpec +module Dolla.Consensus.Proposing.Staging.Pipes.Persisting.PipeSpec (spec) where import Data.ByteString.Lazy.Char8 (pack) @@ -37,19 +37,18 @@ import Dolla.Common.Data.IsString import Dolla.Common.Offset import Dolla.Common.Memory.Byte hiding (getMemorySize) -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Input -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Output +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Output -import Dolla.Consensus.Proposing.Packaging.Pipes.Persisting.Pipe (persisting) -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Pipe (serializing) -import Dolla.Consensus.Proposing.Packaging.Pipes.NonEmptying.Pipe (nonEmptying) +import Dolla.Consensus.Proposing.Staging.Pipes.Persisting.Pipe (persisting) +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Pipe (serializing) +import Dolla.Consensus.Proposing.Staging.Pipes.NonEmptying.Pipe (nonEmptying) -import Dolla.Consensus.Proposing.Packaging.Pipes.Capping.Sizable +import Dolla.Consensus.Proposing.Staging.Pipes.Capping.Sizable import Dolla.Consensus.Proposal.Persistence (ProposalRootFolder, getLocalProposalFolder) -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.SerializedRequest +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.SerializedRequest -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Output as Serializing -import qualified Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input as Serializing +import qualified Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Output as Serializing spec :: Spec spec = do @@ -117,13 +116,13 @@ getGeneratedEncodedInputStream inputs = getGeneratedInputStream inputs & S.map (\case - CommitProposal -> Serializing.ForceProposalProduction - Persist a -> Serializing.Serialize a ) + CommitProposal -> Nothing + Persist a -> Just a ) & serializing & S.map (\case - Serializing.ProposalProductionNotForced -> CommitProposal - Serializing.Serialized (SerializedRequest word8s) -> Persist word8s) + Serializing.Output Nothing -> CommitProposal + Serializing.Output (Just (SerializedRequest word8s)) -> Persist word8s) getGeneratedInputStream :: S.MonadAsync m diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/GenInput.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/GenInput.hs similarity index 50% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/GenInput.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/GenInput.hs index 65bc6da..7d9b9f7 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/GenInput.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/GenInput.hs @@ -1,19 +1,14 @@ -module Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.GenInput +module Dolla.Consensus.Proposing.Staging.Pipes.Serializing.GenInput ( InputUnderTests (..)) where import Test.QuickCheck.Arbitrary -import Test.QuickCheck.Gen import Test.QuickCheck.Instances.UUID () import Test.QuickCheck.Instances () -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Input instance Arbitrary a => Arbitrary (InputUnderTests a) where - arbitrary - = InputUnderTests - <$> oneof - [ return ForceProposalProduction - , Serialize <$> arbitrary] + arbitrary = InputUnderTests <$> arbitrary newtype InputUnderTests a = InputUnderTests { unInputUnderTests :: Input a} diff --git a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/PipeSpec.hs b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/PipeSpec.hs similarity index 59% rename from packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/PipeSpec.hs rename to packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/PipeSpec.hs index 09de778..6cb81fb 100644 --- a/packages/packaging/test/Dolla/Consensus/Proposing/Packaging/Pipes/Serializing/PipeSpec.hs +++ b/packages/staging/test/Dolla/Consensus/Proposing/Staging/Pipes/Serializing/PipeSpec.hs @@ -2,7 +2,7 @@ {-# LANGUAGE LambdaCase #-} {-# LANGUAGE FlexibleContexts #-} -module Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.PipeSpec +module Dolla.Consensus.Proposing.Staging.Pipes.Serializing.PipeSpec (spec) where import Data.Aeson @@ -17,17 +17,17 @@ import Test.QuickCheck import qualified Streamly.Prelude as S import qualified Streamly as S -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Input -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Output -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.GenInput -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.Pipe (serializing) -import Dolla.Consensus.Proposing.Packaging.Pipes.Serializing.SerializedRequest -import Dolla.Consensus.Proposing.Packaging.DummyRequest +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Input +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Output +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.GenInput +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.Pipe (serializing) +import Dolla.Consensus.Proposing.Staging.Pipes.Serializing.SerializedRequest +import Dolla.Consensus.Proposing.Staging.DummyRequest spec :: Spec spec = parallel $ - describe "Proposing.Packaging.Serializing" $ do - it "Serialize requests and Transmit downstream ForceProposalProduction Command" + describe "Proposing.Staging.Serializing" $ do + it "Serialize requests when available" $ property $ \inputs -> aggregateInputToResult @@ -35,9 +35,9 @@ spec = parallel $ serializing & S.mapM_ (\case - (ForceProposalProduction, result) - -> result `shouldBe` ProposalProductionNotForced - (Serialize request@DummyRequest {}, Serialized result) + (Nothing, Output result) + -> result `shouldBe` Nothing + (Just request@DummyRequest {}, Output (Just result) ) -> result `shouldBe` (coerce.unpackBytes.encode) request (a, b) -> expectationFailure $ "unexpected result = " ++ show (a,b) ) diff --git a/packages/packaging/test/Spec.hs b/packages/staging/test/Spec.hs similarity index 100% rename from packages/packaging/test/Spec.hs rename to packages/staging/test/Spec.hs diff --git a/packages/zeus/README.md b/packages/zeus/README.md index 2fabd23..707c291 100644 --- a/packages/zeus/README.md +++ b/packages/zeus/README.md @@ -1 +1,4 @@ -Zeus Readme \ No newline at end of file +/ [Consensus](https://github.com/dolla-consortium/consensus) / [Proposing](https://github.com/dolla-consortium/consensus-proposing) / Zeus +# Zeus + +Documentation to be done. \ No newline at end of file diff --git a/packages/zeus/dolla-consensus-proposing-zeus.cabal b/packages/zeus/dolla-consensus-proposing-zeus.cabal index fddebd6..c589a2a 100644 --- a/packages/zeus/dolla-consensus-proposing-zeus.cabal +++ b/packages/zeus/dolla-consensus-proposing-zeus.cabal @@ -4,7 +4,7 @@ cabal-version: 1.12 -- -- see: https://github.com/sol/hpack -- --- hash: 2c0ecd7b21144551873bd809ee3c5b53fe4bee7315b145093bcd4e75edb4e91a +-- hash: f7d001e9e2d484cc2f4fbd9d6563308e64ef59a434b0f9219fafcd71e02cfce7 name: dolla-consensus-proposing-zeus version: 0.0.0.1 @@ -25,15 +25,15 @@ library Dolla.Consensus.Proposing.Zeus.CLI Dolla.Consensus.Proposing.Zeus.Local.CLI Dolla.Consensus.Proposing.Zeus.Local.Context - Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.DetectingStarvation - Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Packaging + Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.DetectingTension Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Receptionist Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Simulating + Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Staging Dolla.Consensus.Proposing.Zeus.Local.Pipeline other-modules: Paths_dolla_consensus_proposing_zeus hs-source-dirs: - lib + library ghc-options: -Wall build-depends: adjunctions ==4.4 @@ -46,10 +46,10 @@ library , directory ==1.3.6.0 , dolla-base ==1.0.0 , dolla-consensus-base ==1.0.0 - , dolla-consensus-proposing-detecting-starvation-settings - , dolla-consensus-proposing-packaging-settings + , dolla-consensus-proposing-detecting-tension-settings , dolla-consensus-proposing-receptioning-settings , dolla-consensus-proposing-simulating-settings + , dolla-consensus-proposing-staging-settings , either ==5.0.1 , exceptions ==0.10.4 , hashable @@ -79,7 +79,7 @@ executable dolla-consensus-proposing-zeus other-modules: Paths_dolla_consensus_proposing_zeus hs-source-dirs: - app + executables ghc-options: -Wall -main-is Executables.executeZeus -threaded -rtsopts -with-rtsopts=-N build-depends: base ==4.13.0.0 diff --git a/packages/zeus/app/Executables.hs b/packages/zeus/executables/Executables.hs similarity index 100% rename from packages/zeus/app/Executables.hs rename to packages/zeus/executables/Executables.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Common/Zeus/Haskell/ExecutableSettings.hs b/packages/zeus/library/Dolla/Consensus/Common/Zeus/Haskell/ExecutableSettings.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Common/Zeus/Haskell/ExecutableSettings.hs rename to packages/zeus/library/Dolla/Consensus/Common/Zeus/Haskell/ExecutableSettings.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/Network.hs b/packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/Network.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/Network.hs rename to packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/Network.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/Node.hs b/packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/Node.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/Node.hs rename to packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/Node.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/NodeEntryPoints.hs b/packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/NodeEntryPoints.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/NodeEntryPoints.hs rename to packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/NodeEntryPoints.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/NodeIndex.hs b/packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/NodeIndex.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Common/Zeus/Local/NodeIndex.hs rename to packages/zeus/library/Dolla/Consensus/Common/Zeus/Local/NodeIndex.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Common/Zeus/Logging.hs b/packages/zeus/library/Dolla/Consensus/Common/Zeus/Logging.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Common/Zeus/Logging.hs rename to packages/zeus/library/Dolla/Consensus/Common/Zeus/Logging.hs diff --git a/packages/zeus/lib/Dolla/Consensus/EventStore/Zeus/Local/NodeContainer.hs b/packages/zeus/library/Dolla/Consensus/EventStore/Zeus/Local/NodeContainer.hs similarity index 85% rename from packages/zeus/lib/Dolla/Consensus/EventStore/Zeus/Local/NodeContainer.hs rename to packages/zeus/library/Dolla/Consensus/EventStore/Zeus/Local/NodeContainer.hs index 3fc03cd..67ae44f 100644 --- a/packages/zeus/lib/Dolla/Consensus/EventStore/Zeus/Local/NodeContainer.hs +++ b/packages/zeus/library/Dolla/Consensus/EventStore/Zeus/Local/NodeContainer.hs @@ -30,7 +30,7 @@ delete = liftIO $ catch (do echo $ ">| reset the Event Store data for " <++> coerce nodeId - stdout $ echoCommandAndInShell $ "docker rm -f " <++> getEvenStoreDockerName nodeId <++> " &>/dev/null") + stdout $ echoCommandAndInShell $ "docker rm -f " <++> getEventStoreDockerName nodeId) (\SomeException {} -> return ()) start @@ -42,13 +42,13 @@ start eventStoreSettings@EventStoreSettings {..} echo $ ">| starting containers for " <++> coerce nodeId let command = "docker" <++> " run" - <++> " --name " <++> getEvenStoreDockerName nodeId + <++> " --name " <++> getEventStoreDockerName nodeId <++> " -dit" <++> " -p " <++> show (port projectionUrl) <++> ":2113" <++> " -p " <++> show (port eventStoreUrl) <++> ":1113" - <++> " eventstore/eventstore " <++> " 1>/dev/null" + <++> " eventstore/eventstore:release-5.0.6 " stdout $ echoCommandAndInShell command - pauseThread $ 5 * s -- time for machine to start... + pauseThread $ 10 * s -- time for container to start, otherwise configuration fails :-(... Find a way to know when the container is up and running... configure eventStoreSettings configure @@ -70,7 +70,7 @@ enableByCategoryProjection EventStoreSettings {..} = do <++> fromString (unpack username ) <++> ":" <++> fromString (unpack password) - <++> " &>/dev/null" +-- <++> " &>/dev/null" stdout $ echoCommandAndInShell command where @@ -79,9 +79,9 @@ enableByCategoryProjection EventStoreSettings {..} = do "/projection/%24by_category/command/enable'" -getEvenStoreDockerName +getEventStoreDockerName :: NodeId -> String -getEvenStoreDockerName +getEventStoreDockerName NodeId {..} = "consensus-" ++ unNodeId \ No newline at end of file diff --git a/packages/zeus/lib/Dolla/Consensus/EventStore/Zeus/Local/Settings.hs b/packages/zeus/library/Dolla/Consensus/EventStore/Zeus/Local/Settings.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/EventStore/Zeus/Local/Settings.hs rename to packages/zeus/library/Dolla/Consensus/EventStore/Zeus/Local/Settings.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/CLI.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/CLI.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/CLI.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/CLI.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/CLI.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/CLI.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/CLI.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/CLI.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/Context.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/Context.hs similarity index 100% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/Context.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/Context.hs diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/DetectingStarvation.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/DetectingTension.hs similarity index 92% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/DetectingStarvation.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/DetectingTension.hs index 1137f84..266e4e0 100644 --- a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/DetectingStarvation.hs +++ b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/DetectingTension.hs @@ -1,6 +1,6 @@ {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.DetectingStarvation +module Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.DetectingTension ( MicroServiceSettings (..)) where @@ -13,7 +13,7 @@ import Dolla.Common.Logging.Core import Dolla.Consensus.EventStore.Zeus.Local.Settings import Dolla.Consensus.Common.Zeus.Haskell.ExecutableSettings -import Dolla.Consensus.Proposing.Starving.Detecting.Settings +import Dolla.Consensus.Proposing.DetectingTension.Execution.Environment.EventStore.Settings import Dolla.Consensus.Common.Zeus.Logging data MicroServiceSettings diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Receptionist.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Receptionist.hs similarity index 92% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Receptionist.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Receptionist.hs index 71f02f2..826922f 100644 --- a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Receptionist.hs +++ b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Receptionist.hs @@ -14,7 +14,7 @@ import Dolla.Common.Network.Core import Dolla.Consensus.Common.Zeus.Haskell.ExecutableSettings import Dolla.Consensus.Common.Zeus.Logging import Dolla.Consensus.EventStore.Zeus.Local.Settings -import qualified Dolla.Consensus.Proposing.Receptioning.API.Server.Settings as Receptionist +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Server.Settings as Receptionist data MicroServiceSettings = MicroServiceSettings diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Simulating.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Simulating.hs similarity index 93% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Simulating.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Simulating.hs index 4c918f7..d7e197a 100644 --- a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Simulating.hs +++ b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Simulating.hs @@ -18,7 +18,7 @@ import Dolla.Consensus.EventStore.Zeus.Local.Settings import Dolla.Consensus.Proposing.Simulating.Settings import Dolla.Consensus.Proposing.Simulating.StressLoad -import qualified Dolla.Consensus.Proposing.Receptioning.API.Client.Settings as Receptionist.Client +import qualified Dolla.Consensus.Proposing.Receptioning.Execution.Environment.EventStore.Dolla.Warp.Client.Settings as Receptionist.Client data MicroserviceSettings diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Packaging.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Staging.hs similarity index 93% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Packaging.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Staging.hs index 0205296..04a645c 100644 --- a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Packaging.hs +++ b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/MicroserviceSettings/Staging.hs @@ -1,6 +1,6 @@ {-# LANGUAGE RecordWildCards #-} {-# LANGUAGE NamedFieldPuns #-} -module Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Packaging +module Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Staging ( MicroServiceSettings (..)) where @@ -15,7 +15,7 @@ import Dolla.Common.Logging.Core import Dolla.Consensus.EventStore.Zeus.Local.Settings import Dolla.Consensus.Common.Zeus.Haskell.ExecutableSettings -import Dolla.Consensus.Proposing.Packaging.Settings +import Dolla.Consensus.Proposing.Staging.Execution.Environment.EventStore.Settings import Dolla.Consensus.Common.Zeus.Logging data MicroServiceSettings diff --git a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs similarity index 94% rename from packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs rename to packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs index 685f80a..f38b3b4 100644 --- a/packages/zeus/lib/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs +++ b/packages/zeus/library/Dolla/Consensus/Proposing/Zeus/Local/Pipeline.hs @@ -28,8 +28,8 @@ import Dolla.Consensus.Proposing.Zeus.Local.Context import qualified Dolla.Consensus.EventStore.Zeus.Local.Settings as EventStore import qualified Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Simulating as Simulating import qualified Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Receptionist as Receptioning -import qualified Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Packaging as Packaging -import qualified Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.DetectingStarvation as Flushing +import qualified Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.Staging as Staging +import qualified Dolla.Consensus.Proposing.Zeus.Local.MicroserviceSettings.DetectingTension as Flushing start @@ -80,18 +80,18 @@ getProposerMicroservicesSettings configFolder = nodeFolder ++ "configuration/Proposer/" logFolder = nodeFolder ++ "log/" return [ getExecutableSettings - Packaging.MicroServiceSettings + Staging.MicroServiceSettings { nodeId - , executableName = "dolla-consensus-proposing-packaging" - , logFileLocation = FileSystemLocation {rootFolder = logFolder, fileName = "packaging.log"} - , configurationLocation = FileSystemLocation {rootFolder = configFolder, fileName = "packaging.config"} + , executableName = "dolla-consensus-proposing-staging" + , logFileLocation = FileSystemLocation {rootFolder = logFolder, fileName = "staging.log"} + , configurationLocation = FileSystemLocation {rootFolder = configFolder, fileName = "staging.config"} , proposalSizeLimit , eventStore , proposalRootFolder} , getExecutableSettings Receptioning.MicroServiceSettings { nodeId - , executableName = "dolla-consensus-proposing-receptioning" + , executableName = "dolla-consensus-proposing-receptioning-server" , logFileLocation = FileSystemLocation {rootFolder = logFolder, fileName = "receptioning.log"} , configurationLocation = FileSystemLocation {rootFolder = configFolder, fileName = "receptioning.config"} , eventStore @@ -108,7 +108,7 @@ getProposerMicroservicesSettings , getExecutableSettings Flushing.MicroServiceSettings { nodeId - , executableName = "dolla-consensus-proposing-detecting-starvation" + , executableName = "dolla-consensus-proposing-detecting-tension" , logFileLocation = FileSystemLocation {rootFolder = logFolder, fileName = "starving-detection.log"} , configurationLocation = FileSystemLocation {rootFolder = configFolder, fileName = "starving-detection.config"} , eventStore} diff --git a/packages/zeus/package.yaml b/packages/zeus/package.yaml index 19c53be..02b6ddd 100644 --- a/packages/zeus/package.yaml +++ b/packages/zeus/package.yaml @@ -5,10 +5,10 @@ extra-source-files: ghc-options: - -Wall -## - -j + library: - source-dirs: lib + source-dirs: library dependencies: @@ -17,8 +17,8 @@ library: - dolla-consensus-proposing-simulating-settings - dolla-consensus-proposing-receptioning-settings - - dolla-consensus-proposing-packaging-settings - - dolla-consensus-proposing-detecting-starvation-settings + - dolla-consensus-proposing-staging-settings + - dolla-consensus-proposing-detecting-tension-settings - base == 4.13.0.0 - text == 1.2.4.0 @@ -54,7 +54,7 @@ library: executables: dolla-consensus-proposing-zeus : main: Executables.hs - source-dirs: app + source-dirs: executables ghc-options: - -main-is Executables.executeZeus - -threaded diff --git a/stack.yaml b/stack.yaml index c50990d..96784ab 100644 --- a/stack.yaml +++ b/stack.yaml @@ -1,15 +1,15 @@ # When changing the resolver, also update the docker image in `.gitlab-ci.yml` accordingly! resolver: lts-16.2 packages: - # Packages specific to the proposing pipeline + # Projects specific to the proposing pipeline - packages/receptioning - packages/receptioning/settings - - packages/detecting-starvation - - packages/detecting-starvation/settings + - packages/detecting-tension + - packages/detecting-tension/settings - - packages/packaging - - packages/packaging/settings + - packages/staging + - packages/staging/settings - packages/simulating - packages/simulating/settings