Controlled hallucination

Learning Elm Part 14 - Replication & Notifications

Published: - Reading time: 15 minutes

In my last post, we fixed some issues but did not add any new features. In this iteration, we will fix some bugs by removing features, add replication, and add notifications. We will also add some missing end-to-end tests.

As always, you can find a deployed version here, with the Elm Debugger here, and the source code here.

Table of contents

Removing features

The encrypt/decrypt/re-encrypt feature works, but it does not make much sense if you can export/import. So I am going to remove it. It also introduced a bug into my “Edit Settings” page, where I have to enter the current password 3 times to change my settings (if the DB is encrypted) - not good.

Even though we are changing “only” two things (only allow setting-up encryption on the first run, and group encryption settings behind a toggle), the change, is quite verbose, so I’ll give only an overview:

  1. A new ToggleEncryption message that toggles a new encryption : Bool attribute in our Inputs record.
  2. A new helper function:
isFirstRun : Settings -> Bool
isFirstRun settings =
    settings.version == ""

That tells us if we are editing or creating our settings. We use this to know when to enable/disable part of the UI and also in how we validate the form, as we have to ignore password & password-confirm once the initial settings have been saved.

  1. Delete a bunch of code
  2. Adjust until the compiler is happy

This is the end result:

Encryption Settings

We need to modify our end-to-end tests, to delete the feature and add the new requirements. These are the new relevant scenarios:

Feature: Users should be able encrypt their data

    Scenario: Encrypting is suggested by default on first run
        Then the use encryption toggle is on
        When I set the new password to "secret password"
        And I set the new password confirmation to "secret password"
        And I save my settings
        When I reload the app
        Then I see "Enter password"

    Scenario: A user can opt-out of encryption
        Then the use encryption toggle is on
        When I toggle use encryption
        Then I should not see "Encryption password"
        When I save my settings
        And I reload the app
        Then I see "Add Transaction"

    Scenario: A user cannot change encryption setting after choosing encryption
        When I set the new password to "secret password"
        And I set the new password confirmation to "secret password"
        And I save my settings
        And I go to settings
        Then I should not see "Encrypt local database"

    Scenario: A user cannot change encryption setting after opting-out of encryption
        When I toggle use encryption
        And I save my settings
        And I go to settings
        Then I should not see "Encrypt local database"

Implementing replication

We will implement replication using PouchDB’s two-way sync feature, that is compatible with CouchDB’s replication.

We’ll give the user the option to use replication, and he’ll need to specify the URL, username and password. We’ll group these settings in the same way we grouped encryption settings. To store all these settings we need:

  1. Create our new type alias ReplicationSettings that will store our url, username, and password.
  2. Add our replication: Maybe ReplicationSettings that will maybe store the above record.
  3. Add our JSON decoder for the new ReplicationSettings record.
  4. Add our new Msg variants:
    • ToggleReplication toggles the view and usage
    • EditReplicationUrl String our controlled input for URL
    • EditReplicationUsername String our controlled input for the username
    • EditReplicationPassword String our controlled input for the password
  5. Add the view.

Phew, that is a lot, for this small UI!

Replication Settings

Now we are ready to use these settings! We will make a “one shot synchronization” function that will call the Sync method of our PouchDb data database. In other words, settings won’t be replicated. We might need to make that clearer later.

For our end-to-end tests (and why not, our manual tests), I’ll install the pouchdb-adapter-memory that runs an in-memory PouchDB instance:

npm install pouchdb-adapter-memory

Our most lower-level replication method looks like this:

// src/Db.js
import MemoryAdapter from "pouchdb-adapter-memory";

PouchDb.plugin(MemoryAdapter);

function getSyncUrl(settings) {
  const url = new URL(settings.url);
  url.username = settings.username;
  url.password = settings.password;
  return url.href;
}

function remoteDb(settings) {
  if (settings.url.startsWith("pouchdb://")) {
    return new PouchDb(settings.url.substr(10));
  }
  if (settings.url.startsWith("memory://")) {
    return new PouchDb(settings.url.substr(9), { adapter: "memory" });
  }
  return new PouchDb(getSyncUrl(settings));
}

class Db {
  oneShotSync(settings) {
    const remote = remoteDb(settings);
    return new Promise((resolve, reject) => {
      this.db
        .sync(remote)
        .on("complete", (info) => resolve(info))
        .on("error", (error) => reject(error));
    });
  }
}
  1. We load the memory adapter.
  2. We create a couple of helper functions to create the remote PouchDB:
    • If the URL starts with memory:// we create an in-memory instance and ignore username/password.
    • If the URL starts with pouchdb:// we create a default PouchDB that will persist during reloads.
    • If not, we create a normal PouchDB that will work against a remote CouchDB instance.
  3. We create a onShotSync(settings) method that will start synchronization and finish when it fails or succeeds.

Our DbPort will only have this new addition:

class DbPort {
  sync(settings) {
    return this.dataDb.oneShotSync(settings);
  }
}

And our ElmPort will tie this to our Elm app:

app.ports.sync.subscribe(async function (settings) {
  try {
    const info = await dbPort.sync(settings);
    if (info.push.ok && info.pull.ok) {
      app.ports.syncFinished.send({
        sent: info.push.docs_written,
        received: info.pull.docs_read,
      });
    } else {
      console.error(info);
      app.ports.syncFailed.send();
    }
  } catch (e) {
    app.ports.syncFailed.send();
  }
});

This last change will crash our app until we implement these new ports:

  • sync is for Elm to initialize the synchronization
  • syncFinished communicates back to Elm the results
  • syncFailed communicates back to Elm the failure (right now, with no information)

To initialize the one-shot replication, we will add an icon in our bottom bar to start the process. We will need:

  1. A new replicating: Bool attribute in our model, that will be True while replicating
  2. Three new Msg variants:
    • SyncRequested that is fired when the user clicks the synchronization icon.
    • SyncFinished (Result Json.Decode.Error SyncResult) will be the callback from the syncFinished() port.
    • SyncFailed will be the callback from the syncFailed().
-- src/Main.elm

type alias Model =
    { replicating : Bool
    -- (...)
    }

type alias SyncResult =
    { sent : Int
    , received : Int
    }

type Msg
    = SyncRequested
    | SyncFinished (Result Json.Decode.Error SyncResult)
    | SyncFailed
    | NoOp
    -- (...)

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        SyncRequested ->
            let
                ( replicating, cmd ) =
                    case model.editSettingsState.settings.replication of
                        Just settings ->
                            ( True, sync settings )

                        Nothing ->
                            ( False, Cmd.none )
            in
            ( { model | replicating = replicating }, cmd )

        SyncFinished (Ok results) ->
            let
                cmd : Cmd Msg
                cmd =
                    if results.received > 0 then
                        getTransactions defaultGetTransactionsRequest

                    else
                        Cmd.none
            in
            ( { model | replicating = False }, cmd )

        SyncFailed ->
            ( { model | replicating = False }, Cmd.none )
-- (...)

viewBottomBar : Model -> List (Html Msg)
viewBottomBar model =
    let
        syncItem =
            case model.editSettingsState.settings.replication of
                Nothing ->
                    []

                Just _ ->
                    let
                        ( icon, msg ) =
                            if model.replicating then
                                ( "loading spinner icon", NoOp )

                            else
                                ( "sync icon", SyncRequested )
                    in
                    [ div [ class "item" ]
                        [ div
                            [ class "ui icon button"
                            , classList [ ( "disabled", model.replicating ) ]
                            , cyAttr "sync"
                            , onClick msg
                            ]
                            [ i [ class icon ] []
                            ]
                        ]
                    ]
    in
    div [ class "item" ]
        [ div [ class "ui icon button", cyAttr "settings", onClick (SetPage EditSettings) ]
            [ i [ class "settings icon" ] []
            ]
        ]
        :: syncItem
        ++ [ div [ class "right menu" ]
                [ div [ class "item" ]
                    [ div [ class "ui primary button", cyAttr "add-transaction", onClick (SetPage Edit) ]
                        [ text "New"
                        ]
                    ]
                ]
           ]


-- (...)
port sync : ReplicationSettings -> Cmd msg
port syncFinished : (Json.Decode.Value -> msg) -> Sub msg
port syncFailed : (Json.Decode.Value -> msg) -> Sub msg

We also added a new NoOp message, that should never fire because the sync button is disables during synchronization.

If we use memory://something to try it out, we will see this message in the Elm Debugger:

SyncFinished Ok ({ received = 0, sent = 48 })

Great! But it would be better to see some more feedback…

Implementing notifications

I’ve been avoiding showing error messages, and now would be a good time to implement them. I’ll use a custom type in our src/Misc.elm to model the notification data. I will be using the Message component of Semantic UI. I’ll use its attached variant for a more prominent message, and some custom CSS to simulate a popup notification.

-- src/Misc.elm
type MainNotification
    = MainNotification NotificationData


type PopupNotification
    = PopupNotification NotificationData


type Notification
    = Notification MainNotification
    | Popup PopupNotification


type NotificationType
    = NormalNotification
    | NegativeMessage
    | PositiveMessage


type alias NotificationData =
    { type_ : NotificationType
    , header : Maybe String
    , message : String
    }


viewNotification : Notification -> msg -> Html msg
viewNotification notification onClose =
    let
        ( messageClass, data ) =
            case notification of
                Notification (MainNotfication notificationData) ->
                    ( "attached message", notificationData )

                Popup (PopupNotification notificationData) ->
                    ( "message", notificationData )

        typeClass =
            case data.type_ of
                NegativeMessage ->
                    "negative"

                PositiveMessage ->
                    "positive"

                NormalNotification ->
                    ""

        mainClass =
            "ui " ++ typeClass ++ " " ++ messageClass

        header =
            case data.header of
                Just txt ->
                    div [ class "header" ] [ text txt ]

                Nothing ->
                    div [] []
    in
    div [ class mainClass ]
        [ i [ class "close icon", onClick onClose ] []
        , header
        , p [] [ text data.message ]
        ]

We will store two maybe references to these notifications in our main model:

-- src/Main.elm

type alias Model =
    { , notification : Maybe MainNotification
    , popupNotification : Maybe PopupNotification
    -- (...)
    }

type Msg
    = DismissPopupNotification
    | DismissNotification
    -- (...)

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        DismissNotification ->
            ( { model | notification = Nothing }, Cmd.none )

        DismissPopupNotification ->
            ( { model | popupNotification = Nothing }, Cmd.none )
-- (...)

view : Model -> Html Msg
view model =
    --(..)
    div [ class "container" ]
        [ viewMainNotification model.notification
        , viewPopupNotification model.popupNotification
        , div
            (class "ui attached segment main-content" :: scrollListener)
            [ mainContent ]
        , div [ class "ui bottom attached menu bottom-bar" ]
            bottomBar
        ]

viewMainNotification : Maybe MainNotification -> Html Msg
viewMainNotification maybeNotification =
    case maybeNotification of
        Nothing ->
            div [] []

        Just notification ->
            viewNotification (Notification notification) DismissNotification


viewPopupNotification : Maybe PopupNotification -> Html Msg
viewPopupNotification maybeNotification =
    case maybeNotification of
        Nothing ->
            div [] []

        Just notification ->
            div [ class "popup-notification" ]
                [ viewNotification (Popup notification) DismissPopupNotification ]

Now we need to add our CSS for our popup notification, courtesy of ChatGPT:

.popup-notification {
  position: fixed;
  top: 10px;
  right: 10px;
  box-shadow: 0 2px 5px rgba(0, 0, 0, 0.2);
  z-index: 999;
}

And the last piece of the puzzle is to add the message to the model:

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        SyncFinished (Ok results) ->
            let
                message =
                    "↑ Sent: " ++ String.fromInt results.sent ++ "  /  ↓ Received: " ++ String.fromInt results.received

                popupNotification : NotificationData
                popupNotification =
                    NotificationData PositiveMessage (Just "Synched OK") message

                cmd : Cmd Msg
                cmd =
                    if results.received > 0 then
                        getTransactions defaultGetTransactionsRequest

                    else
                        Cmd.none
            in
            ( { model
                | replicating = False
                , popupNotification = Just (PopupNotification popupNotification)
              }
            , cmd
            )

        SyncFailed ->
            let
                message =
                    "Could not synchronize"

                notificationData : NotificationData
                notificationData =
                    NotificationData NegativeMessage (Just "Synchronization Error") message

                notification =
                    MainNotification notificationData
            in
            ( { model | replicating = False, notification = Just notification }, Cmd.none )
        -- (...)

It ends up looking something like this:

Notifications

Auto-closing the popup notification

In plain JavaScript, we could use the setTimeout function to schedule the dismissal of the popup notification.

To do something similar in Elm, we’ll use Process.sleep. I’ll create a small helper function that will “emit” a message after a certain amount of milliseconds:

delayMillis : Int -> msg -> Cmd msg
delayMillis millis msg =
    Process.sleep (toFloat millis)
        |> Task.perform (\_ -> msg)

And I’ll use it to automatically close my popup notification after 5 seconds:

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        SyncFinished (Ok results) ->
            let
                message =
                    "↑ Sent: " ++ String.fromInt results.sent ++ "  /  ↓ Received: " ++ String.fromInt results.received

                popupNotification : NotificationData
                popupNotification =
                    NotificationData PositiveMessage (Just "Synched OK") message

                dismissCmd : Cmd Msg
                dismissCmd =
                    delayMillis 5000 DismissPopupNotification

                cmd : Cmd Msg
                cmd =
                    if results.received > 0 then
                        Cmd.batch [ getTransactions defaultGetTransactionsRequest, dismissCmd ]

                    else
                        dismissCmd
            in
            ( { model
                | replicating = False
                , popupNotification = Just (PopupNotification popupNotification)
              }
            , cmd
            )
-- (...)

Small UI adjustment

In this iteration, I added the bottom bar in the “empty list” view. I will also replace the “Add Transaction” button with the Import JSON one:

New Empty List

Before, there was no way of importing JSON without first adding a transaction, which was annoying.

End-to-end tests for Import JSON feature

I looked a bit into how to make Cypress selectFile work with Elm’s [elm/file] package, but did not get far. From what I can tell by looking at the package’s source code, it adds the <input type="file"> to the DOM and immediately “clicks” on it, which doesn’t play nice with what Cypress is doing.

So I ended up creating a e2e port just for this:

-- src/Settings.elm

type Msg
    = GotE2EJsonLoaded (Result Json.Decode.Error String)
    -- (...)

update : Msg -> State -> ( State, Cmd Msg, Bool )
update msg model =
    case msg of
        GotE2EJsonLoaded (Err _) ->
            ( { model | working = NotWorking }, Cmd.none, False )

        GotE2EJsonLoaded (Ok string) ->
            let
                newModel =
                    { model | working = Importing }
            in
            update (JsonLoaded string) newModel
-- (...)

port gotE2EJsonLoaded : (Json.Decode.Value -> msg) -> Sub msg

And using it in my DevApi:

// src/DevApi.js

class DevApi {
  // ...
  importJson(json) {
    this.appPorts.gotE2EJsonLoaded.send(json);
  }
  // ...
}

I can now create my import-json.feature as follows:

Feature: Users should be able import data from JSON files

    Scenario: Importing transactions from valid JSON
        Given I save my settings
        And I go to settings
        And I import the sample JSON file
        Then I see "Dental cleaning"


    Scenario: Importing transactions missing id or version
        Given I save my settings
        And I go to settings
        And I import a JSON file containing
            """json
        [
            {
                "date": "2021-01-10",
                "description": "Utility bill payment",
                "destination": {
                    "account": "Expenses:Utilities",
                    "amount": 6000,
                    "currency": "USD"
                },
                "source": {
                    "account": "Assets:Bank",
                    "amount": -6000,
                    "currency": "USD"
                }
            }
        ]
            """
        Then I see an error message "Error decoding JSON"

This is the import-json.js step definitions:

const { When } = require("@badeball/cypress-cucumber-preprocessor");
const buildSampleData = require("../../src/SampleData").default;

const sample = buildSampleData().map((t) => {
  t.id = "";
  t.version = "";
  return t;
});

function importJson(json) {
  cy.window()
    .its("ElmExpenses")
    .then((ElmExpenses) => ElmExpenses.importJson(json));
}

When('I click on "Import Json"', () => {
  cy.get('[data-cy="import-json"]');
});

When("I import the sample JSON file", () => {
  importJson(JSON.stringify(sample));
});

When("I import a JSON file containing", (text) => {
  importJson(text);
});

I would have preferred to use exactly the same code-path as “in production”, but this is definitely better than nothing, for the time being.

End-to-end tests for replication

To make these work, I added a couple these new methods to our DevApi to manipulate our application state:

class DevApi {
  async deleteDataDb() {
    await this.dbPort.deleteDataDb();
    const dbPort = new DbPort();
    await dbPort.openDbs();
    this.dbPort = dbPort;
    this.onNewDbPort(dbPort);
  }

  syncWithRemote(settings) {
    return this.dbPort.sync(settings);
  }
}

These are used in this feature file:

Feature: Users should be able to replicate their local database

    Scenario: Replicating to an empty remote
        Given I have saved the default settings
        And I have saved the following transactions:
            | date       | description  | destination                  | source        | amount | currency |
            | 2024-02-29 | Fill-up tank | Expenses:Auto:Gas            | Assets:PayPal | 1999   | USD      |
            | 2024-03-01 | Pizza        | Expenses:Eat Out & Take Away | Assets:Cash   | 3999   | USD      |
        Then I see "Fill-up tank"
        And the sync icon button is not present
        When I go to settings
        Then the use replication toggle is off
        When I toggle use replication
        Then the use replication toggle is on
        When I set the replication URL to "memory://remotedb"
        And I set the replication username to "any-username"
        And I set the replication password to "any-password"
        And I save my settings
        Then the sync icon button is present
        When I press the sync icon button
        Then I see "Sent: 2"

    Scenario: Replicating an empty DB from a previously synched remote
        Given an empty app but I had previously synched the following transactions::
            | date       | description  | destination                  | source        | amount | currency |
            | 2024-02-29 | Fill-up tank | Expenses:Auto:Gas            | Assets:PayPal | 1999   | USD      |
            | 2024-03-01 | Pizza        | Expenses:Eat Out & Take Away | Assets:Cash   | 3999   | USD      |
        Then I see "Import Sample"
        And I see 0 transactions
        When I press the sync icon button
        Then I see 2 transactions
        And I see "Received: 2"

The above feature file is implemented with the following JavaScript:

// ./cypress/e2e/replication.js
const { Given } = require("@badeball/cypress-cucumber-preprocessor");

Given(
  /^an empty app but I had previously synched the following transactions:$/,
  (table) => {
    // create settings with replication enabled
    cy.createReplicationSettings();

    // add the transactions
    for (const txn of table.hashes()) {
      txn.amount = parseInt(txn.amount);
      cy.addTransaction(txn);
    }

    // sync with the remote
    cy.syncWithRemote();

    // delete the local database
    cy.deleteDataDb();
  }
);

// ./cypress/support/commands.js
const replicationSettings = {
  url: "memory://remotedb",
  username: "anything",
  password: "anything",
};

Cypress.Commands.add("createReplicationSettings", (password = null) =>
  cy
    .window()
    .its("ElmExpenses")
    .then((e) =>
      e.saveSettings(
        {
          version: "",
          defaultCurrency: "USD",
          destinationAccounts: [
            "Expenses:Groceries",
            "Expenses:Eat Out & Take Away",
          ],
          sourceAccounts: [
            "Assets:Cash",
            "Assets:Bank:Checking",
            "Liabilities:CreditCard",
          ],
          replication: replicationSettings,
        },
        password
      )
    )
);

Cypress.Commands.add("syncWithRemote", () =>
  cy
    .window()
    .its("ElmExpenses")
    .then((e) => e.syncWithRemote(replicationSettings))
);

Final thoughts

This was a long iteration, and now that we have notifications, we still need to add lots of missing error messages. So as always, lots to be done!

Next up, I’ll probably to a couple of iterations of bug fixes, code cleanup, add missing messages, etc.

After that:

  1. Export transactions (this one should be short)
  2. Support multi-currency transactions
  3. Support more than two entries per transaction

Until next time!