Learning Elm Part 5 - Persistence

Published: Mar 1, 2024 - Reading time: 15 minutes

In part 4 of the series we added form validation, and now, finally, we are ready to store some data!

As always, you can find a deployed version here, with the Elm Debugger here and the source code here.

Adding some tests
Making some small UI adjustments
Choosing the persistence software
Final thoughts

Adding some tests

What? I thought we were going to persist some data! As you might have realized by now, when performing this type of application, I rarely do “test-first” development. I like clicking and using the UI, etc. Then adding tests.

As it stands, we are missing quite a lot of tests. We’ll begin with the most boring one, which tests our validateForm function, which you might recall has this signature:

validateForm : FormInput -> Result FormResult Transaction

I’ll add two blanket tests for both error and success cases:

testValidateFormError : Test
testValidateFormError =
    let
        badFormInput : FormInput
        badFormInput =
            { date = ""
            , description = ""
            , destination = ""
            , source = ""
            , amount = ""
            , currency = ""
            }

        expected : FormResult
        expected =
            { date = Err "Expected a date in ISO 8601 format"
            , description = Err "Description cannot be blank"
            , destination = Err "Destination cannot be blank"
            , source = Err "Source cannot be blank"
            , amount = Err "Invalid amount: "
            , currency = Err "Currency cannot be blank"
            }
    in
    test "validateForm where all are errors" <|
        \_ ->
            badFormInput
                |> validateForm
                |> Expect.equal (Err expected)

Nothing out of the ordinary… just a normal unit test.

I’ll also add a test to verify that the new SubmitForm message creates the formValidation attribute in the model:

submitFormValidatesTheForm : Test
submitFormValidatesTheForm =
    let
        badFormInput : FormInput
        badFormInput =
            { date = "2024-03-03"
            , description = ""
            , destination = "Expenses:Groceries"
            , source = "Assets:Cash"
            , amount = ""
            , currency = "USD"
            }

        expected : FormResult
        expected =
            { date = Ok (Date.fromCalendarDate 2024 Mar 3)
            , description = Err "Description cannot be blank"
            , destination = Ok "Expenses:Groceries"
            , source = Ok "Assets:Cash"
            , amount = Err "Invalid amount: "
            , currency = Ok "USD"
            }

        model =
            { initialModel
                | formInput = badFormInput
            }
    in
    test "When the form is submitted, we validate the current input and set the result in the model" <|
        \_ ->
            model
                |> update SubmitForm
                |> Tuple.first
                |> .formValidation
                |> Expect.equal (Error expected)

And I’ll add a test to verify that all errors are rendered:

editPageRendersValidationErrors : Test
editPageRendersValidationErrors =
    let
        badFormInput : FormInput
        badFormInput =
            { date = "2024-03-03"
            , description = ""
            , destination = "Expenses:Groceries"
            , source = "Assets:Cash"
            , amount = ""
            , currency = "USD"
            }

        formResult : FormResult
        formResult =
            { date = Ok (Date.fromCalendarDate 2024 Mar 3)
            , description = Err "Description cannot be blank"
            , destination = Ok "Expenses:Groceries"
            , source = Ok "Assets:Cash"
            , amount = Err "Invalid amount: "
            , currency = Ok "USD"
            }

        model =
            { initialModel
                | formInput = badFormInput
                , formValidation = Error formResult
                , currentPage = Edit
            }

        expectations : List (Query.Single Msg -> Expect.Expectation)
        expectations =
            [ \q ->
                q
                    |> Query.has
                        [ all
                            [ tag "div"
                            , classes [ "field", "error" ]
                            , containing [ tag "input", attribute (name "description") ]
                            ]
                        ]
            , \q ->
                q
                    |> Query.has
                        [ all
                            [ tag "div"
                            , classes [ "field", "error" ]
                            , containing [ tag "input", attribute (name "amount") ]
                            ]
                        ]
            , \q ->
                q
                    |> Query.has
                        [ all
                            [ tag "div"
                            , classes [ "ui", "error", "message" ]
                            , all
                                [ containing [ tag "p", text "Description cannot be blank" ]
                                , containing [ tag "p", text "Invalid amount:" ]
                                ]
                            ]
                        ]
            ]
    in
    test "Edit Form renders validation errors" <|
        \_ ->
            model
                |> view
                |> Query.fromHtml
                |> Expect.all expectations

I am not very happy with these tests. I see a lot of repetition. At least I took some time to read the test selectors documentation this time. I’ll try to address these issues later, after I have a better grasp of what I’d like to test, and at which layer.

Making some small UI adjustments

A few months ago I finally gave in and signed-up with Google on Medium, and now I periodically receive their email digests, adding to my ever growing inbox of 17,478 unread emails.

However, once in a while I will glance at them, and I read an article titled 16 little UI design rules that make a big impact just because I have no idea about UI design. I had already noticed that the transactions list seemed very “condensed”, and that it would benefit from more line space.

The first thing I did was to find an online tool to address Rule 8: Ensure text has a 4.5:1 contrast ratio, because the date headers had very little contrast. After bouncing off the Figma something the author mentions, I googled once, and found the WebAIM (web accessibility in mind) website that worked just fine.

I also added the header class to the transaction description, to make it more prominent, which I guess would fall into Rule 4: Create a clear visual hierarchy.

I used the relaxed list variant in Semantic UI to address Rule 16: Use at least 1.5 line height for body text. But that CSS class only adds padding between elements, and I wanted more padding in between my description, and the accounts involved. I ended up adding a new CSS class to my description that increased the padding.

This is the before and after:

Before And After

I think I will make a couple of changes later. It kind of bothers me that the amount is not vertically aligned in the row. But enough for now.

Choosing the persistence software

The webapp will be offline-first, so data will live in the browser. In my React webapp, I use pouchdb, and I wanted to see what else was out there. A few months ago, I asked ChatGPT what it recommended and in the list it spewed out, RxDB caught my eye. Its target is a much more reactive/online application than mine - imagine a chat or collaborative tool. Still, why not try it out? But alas, I npm installed it and suddenly my elm-app could not bundle the project because of some syntax error I had no intention of investigating. So PouchDB it is!

The goal is to store transactions locally, and then replicate them to a server for backup and/or sharing with another device or person. Transactions are rarely edited, so PouchDB is actually a very good fit for this.

As it says in its homepage:

PouchDB is an open-source JavaScript database inspired by Apache CouchDB that is designed to run well within the browser.

And as CouchDB, documents have an ID field named _id and a version field named _rev. Elm does not allow that naming convention:

❯ elm repl
---- Elm 0.19.1 ----------------------------------------------------------------
Say :help for help and :exit to exit! More at <https://elm-lang.org/0.19.1/repl>
--------------------------------------------------------------------------------
> doc = {_id="My Document", title="Hello"}
|   
-- PROBLEM IN RECORD ------------------------------------------------------ REPL

I just started parsing a record, but I got stuck here:

2| doc = {_id="My Document", title="Hello"}
          ^
I was expecting to see a record field defined next, so I am looking for a name
like userName or plantHeight.

Note: Field names must start with a lower-case letter. After that, you can use
any sequence of letters, numbers, and underscores.

Note: If you are trying to define a record across multiple lines, I recommend
using this format:

    { name = "Alice"
    , age = 42
    , height = 1.75
    }

Notice that each line starts with some indentation. Usually two or four spaces.
This is the stylistic convention in the Elm ecosystem.

>

So I will just use id and version in my Elm code. We’ll need to add both of them to our Transaction and our FormInput.

We will start very simple:

We always send all documents to Elm
Saving a transaction will save it and then send all documents to Elm

Before we even run our npm install, we will create our new JavaScript port. And before we do that, we will need a new Elm type! The Date type we are using is not “serializable” as it is, but if we use normal Elm types, it is ok. So we’ll create a new JsonTransaction for that. Some relevant parts of the changes:

-- NEW id AND version attributes
type alias Transaction =
    { id : String
    , version : String
    , date : Date
    , description : String
    , destination : Entry
    , source : Entry
    }

type alias JsonTransaction =
    { id : String
    , version : String
    , date : String
    , description : String
    , destination : Entry
    , source : Entry
    }

-- (...)

-- NEW FUNCTION TO CONVERT
transactionToJson : Transaction -> JsonTransaction
transactionToJson txn =
    JsonTransaction txn.id txn.version (Date.toIsoString txn.date) txn.description txn.destination txn.source


-- NEW PORT ELM => JAVASCRIPT
port saveTransaction : JsonTransaction -> Cmd msg

-- (...)

transactionDecoder : Json.Decode.Decoder Transaction
transactionDecoder =
    Json.Decode.succeed Transaction
        |> required "id" Json.Decode.string
        |> required "version" Json.Decode.string
        |> required "date" dateDecoder
        |> required "description" Json.Decode.string
        |> required "destination" entryDecoder
        |> required "source" entryDecoder

You might have noticed that I expect to receive id and version in Elm. That is because I will do some small transformations in JavaScript.

And of course, the SubmitForm will now save the transaction:

SubmitForm ->
            let
                formValidation : FormValidation
                formValidation =
                    case validateForm model.formInput of
                        Ok transaction ->
                            Valid transaction

                        Err error ->
                            Error error

                cmd : Cmd Msg
                cmd =
                    case formValidation of
                        Valid transaction ->
                            transactionToJson transaction |> saveTransaction

                        _ ->
                            Cmd.none

                page : Page
                page =
                    case formValidation of
                        Valid _ ->
                            List

                        _ ->
                            Edit
            in
            ( { model | formValidation = formValidation, currentPage = page }, cmd )

If the form is valid, we send the JsonTransaction to our JavaScript through our new port, and we go back to our list view.

We will use the pouchdb-browser version, and this adds our first JavaScript dependency:

npm install pouchdb-browser

We will have to import it, create our database. Eventually, we will have a more complex initialization. For now, I will move all to a new async main() function. This is the initial JavaScript with no try/catch:

async function main() {
  const db = new PouchDb("elm_expenses_local");

  const app = Elm.Main.init({
    node: document.getElementById("root"),
  });

  app.ports.saveTransaction.subscribe(async function (elmTxn) {
    const txn = mapTxnFromElm(elmTxn);
    setRandomId(txn);
    await db.put(txn);
    await sendTransactionsToElm();
  });

  async function sendTransactionsToElm() {
    const result = await db.allDocs({ include_docs: true, descending: true });
    app.ports.gotTransactions.send(
      result.rows.map((row) => mapTxnToElm(row.doc))
    );
  }

  await sendTransactionsToElm();
}

function mapTxnFromElm(doc) {
  doc._id = doc.id;
  doc._rev = doc.version;
  delete doc.id;
  delete doc.version;
  return doc;
}

function mapTxnToElm(doc) {
  doc.id = doc._id;
  doc.version = doc._rev;
  delete doc._id;
  delete doc._rev;
  return doc;
}

function setRandomId(txn) {
  if (txn._id != "") {
    return;
  }
  txn._id = txn.date + "-" + window.crypto.randomUUID();
  delete txn._rev;
}

const sample = [
  /* AI GENERATED SAMPLE  - to be used later */
];

main();

// If you want your app to work offline and load faster, you can change
// unregister() to register() below. Note this comes with some pitfalls.
// Learn more about service workers: https://bit.ly/CRA-PWA
serviceWorker.unregister();

I named my database elm_expenses_local and I open it before we initialize our Elm application
I moved the “send transactions to Elm” to its own small sendTransactionsToElm() and we call it at the end of our main() function
Our save transaction uses mapTxnFromElm() to move id => _id and version => _rev. The mapTxnToElm(), unsurprisingly, does the opposite.
Our save transaction calls setRandomId() to create our document primary key, if the _id is a blank string.
We call our main() function.

After all the commands and subscriptions involved in obtaining the current date, I opted to generate the ID in the JavaScript world, using the browsers native UUID primitive.

This is what gets stored in PouchDb:

{
  "date": "2024-02-29",
  "description": "Supermarket",
  "destination": {
    "account": "Expenses:Groceries",
    "amount": 1099,
    "currency": "USD"
  },
  "source": {
    "account": "Assets:Cash",
    "amount": -1099,
    "currency": "USD"
  },
  "_id": "2024-02-29-34b011fe-bd98-4cf8-9b3e-6d6fb726fa36",
  "_rev": "1-eb01eac86587edef9b1dd66855632a00"
}

The id field is in the format of ${date}-${uuid}, not to avoid collisions of course, but simply because I will use it to query the DB more efficiently, as that is the pattern encouraged by CouchDB. In my personal implementation I do the same, and I leak that information (i.e. - it is not encrypted, as we will see in further posts).

So great, now our small app finally persists our data-entry!

First Persist

It would be nice to:

Edit and delete transactions - we all do mistakes
Add an “import sample” and “delete all data”

So on to these new requirements!

type Msg
    = GotTransactions (Result Json.Decode.Error (List Transaction))
-- (...)
--  OUR NEW ACTIONS
    | EditTransaction Transaction
    | DeleteTransaction String String
    | ImportSample
    | DeleteAllData
-- (...)
    | SubmitForm

-- (...)
update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
    -- (...)
        EditTransaction transaction ->
            let
                formInput =
                    transactionFormInput transaction
            in
            ( { model | formInput = formInput, formValidation = None, currentPage = Edit }, Cmd.none )

        DeleteTransaction id version ->
            ( { model | currentPage = List }, deleteTransaction ( id, version ) )

        ImportSample ->
            ( { model | currentPage = List }, importSampleData () )

        DeleteAllData ->
            ( { model | currentPage = List }, deleteAllData () )
-- (...)

-- NEW onClick
viewTransaction : Transaction -> Html Msg
viewTransaction txn =
    div [ class "item", onClick (EditTransaction txn) ]
        [ viewDescription txn
        , viewAmount txn.destination
        ]
-- (...)


-- NEW BUTTONS
viewForm : Model -> Html Msg
viewForm model =
--(...)
            , viewFormValidationResult model
            , button [ class "positive ui button right floated" ]
                [ text "Submit" ]
            , div [ class "ui button", onClick (SetPage List) ]
                [ text "Cancel" ]
            , maybeViewDeleteButton f
--(...)


maybeViewDeleteButton : FormInput -> Html Msg
maybeViewDeleteButton f =
    if f.id /= "" then
        div [ class "negative ui button", onClick (DeleteTransaction f.id f.version) ]
            [ text "Delete" ]

    else
        span [] []

-- (...)
transactionFormInput : Transaction -> FormInput
transactionFormInput txn =
    { id = txn.id
    , version = txn.version
    , date = Date.toIsoString txn.date
    , description = txn.description
    , destination = txn.destination.account
    , source = txn.source.account
    , amount = toFloat txn.destination.amount / 100.0 |> String.fromFloat
    , currency = txn.destination.currency
    }
-- (...)


---- PORTS ELM => JS ----
port saveTransaction : JsonTransaction -> Cmd msg

port deleteTransaction : ( String, String ) -> Cmd msg

port importSampleData : () -> Cmd msg

port deleteAllData : () -> Cmd msg

We add an onClick in to edit transactions
The EditTransaction uses a helper function to populate the FormInput
The edit form renders the Delete button if the FormInput has a non-blank id
We have new JavaScript ports that are used for each of these

Our JavaScript code is pretty straight forward:

app.ports.deleteTransaction.subscribe(async function (idAndVersion) {
  const [id, version] = idAndVersion;
  await db.remove(id, version);
  await sendTransactionsToElm();
});

app.ports.deleteAllData.subscribe(async function () {
  await db.destroy();
  db = new PouchDb("elm_expenses_local");
  await sendTransactionsToElm();
});

app.ports.importSampleData.subscribe(async function () {
  const toImport = sample.map((t) => {
    setRandomId(t);
    return t;
  });
  await db.bulkDocs(toImport);
  await sendTransactionsToElm();
});

We delete the transaction
We destroy the database and create it again
We generate the random IDs and bulk-insert the documents

But we still need to add the UI for these 2 functions!

For this, I will add the third “page” of this application, all in this one file, before I start considering splitting it off.

We will need a place to edit settings, so I’ll use that for the time being:

type Page
    = List
    | Edit
    -- OUR NEW Page variant
    | EditSettings

--(...)

renderPage : Model -> Html Msg
renderPage model =
    case model.currentPage of
        List ->
            viewListItems model

        Edit ->
            viewForm model

        -- OUR NEW VIEW
        EditSettings ->
            viewSettings model


viewListItems : Model -> Html Msg
viewListItems model =
    div [ class "ui celled list relaxed" ]
        (List.map
            (\item ->
                case item of
                    T transaction ->
                        viewTransaction transaction

                    D date ->
                        viewDate date
            )
            model.listItems
            ++ [ button [ class "massive circular ui blue icon button fab", onClick (SetPage Edit) ]
                    [ i [ class "plus icon" ] [] ]

                -- OUR NEW SETTING BUTTON
               , button [ class "massive circular ui icon button fab-left", onClick (SetPage EditSettings) ]
                    [ i [ class "settings icon" ] [] ]
               ]
        )

-- OUR SETTINGS VIEW
viewSettings : Model -> Html Msg
viewSettings model =
    div []
        [ form [ class "ui large form" ]
            [ div [ class "ui button", onClick (SetPage List) ]
                [ text "Cancel" ]
            , div [ class "ui positive button", onClick ImportSample ]
                [ text "Import Sample" ]
            , div [ class "ui negative button", onClick DeleteAllData ]
                [ text "Delete ALL Data" ]
            ]
        ]

We add our new EditSettings variant
The compiler will tell us to handle that variant in the renderPage function
We create a new fab-left settings button
And finally, an empty form with 3 buttons

And we are done!

One last thing we’ll do is add a “blank state” view:

renderPage : Model -> Html Msg
renderPage model =
    case model.currentPage of
        List ->
            if List.isEmpty model.transactions then
                viewEmptyState ()

            else
                viewListItems model

        Edit ->
            viewForm model

        EditSettings ->
            viewSettings model


viewEmptyState : () -> Html Msg
viewEmptyState _ =
    div [ class "container" ]
        [ h2 [ class "ui icon header middle aligned" ]
            [ i [ class "money icon" ] []
            , div [ class "content" ] [ text "Welcome to Elm Expenses!" ]
            , div [ class "sub header" ] [ text "This is a work in progress" ]
            ]
        , div [ class "ui center aligned placeholder segment" ]
            [ div [ class "ui positive button", onClick ImportSample ]
                [ text "Import Sample" ]
            , div [ class "ui horizontal divider" ] [ text "Or" ]
            , div [ class "blue ui button", onClick (SetPage Edit) ]
                [ text "Add Transaction" ]
            ]
        ]

We still have lots to do, but now at least you can store your transactions safely in your browser.

Final thoughts

We already have around 1,000 lines of Elm code in a single file, so in the next post we will probably split it up before we continue with the rest.

This simple approach of firing-off a Save/Delete/Edit message to JavaScript without waiting for a response, results in small UI glitches, that we will have to address at some point.

See you next time!

Table of contents

Adding some tests

Making some small UI adjustments

Choosing the persistence software

Final thoughts