Controlled hallucination

Learning Elm Part 12 - Import JSON & Infinite Scroll

Published: - Reading time: 15 minutes

Now that data can be encrypted at rest, I need some way to import all my data into the app. In this iteration I’ll do just that, add a simple “Import from JSON” feature.

But once I do that, I’ll need to change how we render the transactions list, because right now it brings all data into Elm, and I plan to import a subset of my transactions (those that I can render properly now), which are about ~7,900.

As always, you can find a deployed version here, with the Elm Debugger here, and the source code here.

Table of contents

Adding a bottom bar

I’ll start by adding a bottom bar. This seems strange, and it is. I’ll move my FAB buttons there. Change some CSS, and make the transaction list “scrollable” in the sense that it will emit “scroll” events. That will allow us use an infinite scroll library later on.

The code that renders the app is this single function:

view : Model -> Html Msg
view model =
    div [ class "ui container" ]
        [ renderPage model ]

After some trial and error, I decided to move all the buttons to the bottom bar, so I changed the view to this:

view : Model -> Html Msg
view model =
    let
        ( mainContent, bottomBar ) =
            renderPage model
    in
    div []
        [ div
            [ class "ui attached segment main-content" ]
            [ mainContent ]
        , div [ class "ui bottom attached menu" ]
            bottomBar
        ]

Each page can now return the content to populate the bottom menu bar. I added the following CSS to make the main-content use all the vertical view-port space minus the size of the bottom bar:

.main-content {
  overflow-y: auto;
  height: calc(100vh - 50px);
}

I am sure there are more correct ways to achieve this, but this works, and ChatGPT recommends similar ways. I tried to read about the Flexbox and Semantic UI’s Grid system, and I gave up after a few hours.

This is the before-and-after for the main list:

Bottom Bar

Hardcoding a page size

During the implementation of the import JSON, I’ll hardcode a limit in our src/DbPort.js:

async getTransactions() {
    const result = await this.dataDb.allDocs({
        include_docs: true,
        descending: true,
        limit: 200
    });
    return result.rows.map(row => mapDocToElm(row.doc));
}

Reading files in Elm

To read files in Elm, we need to install elm/file:

elm install elm/file

This gives us an access to the browsers File API in Elm.

We need to separate the process in three steps:

  1. Tell the Elm runtime we want to choose a file
  2. Read the contents
  3. Parse the contents

Then, we’ll need to import them through our JavaScript port.

So lets get to it!

Just to keep things “tidy”, I’ll initially add this feature in the Settings page, so we will be modifying our src/Settings.elm:

-- OUR NEW MESSAGES
type Msg
    = JsonRequested
    | JsonSelected File
    | JsonLoaded String
    | TransactionsImported
    | TransactionsImportedError (Result Json.Decode.Error String)
    -- (...)

update : Msg -> State -> ( State, Cmd Msg, Bool )
update msg model =
    case msg of
-- (...)
        JsonRequested ->
            -- Triggered the user clicks Import JSON
            ( model, requestImportJson JsonSelected, False )

        JsonSelected file ->
            -- Read the file to String
            ( model, Task.perform JsonLoaded (File.toString file), False )

        JsonLoaded jsonString ->
            let
                ( cmd, error ) =
                    case Json.Decode.decodeString transactionsDecoder jsonString of
                        Ok txns ->
                            let
                                errors =
                                    txns |> List.map isBalanced |> List.filter isError
                            in
                            if List.isEmpty errors then
                                ( List.map transactionToJson txns |> importTransactions, Nothing )

                            else
                                ( Cmd.none, Just "Some transactions are not balanced or not supported!" )

                        Err _ ->
                            ( Cmd.none, Just "Error decoding JSON" )
            in
            ( { model | error = error }, cmd, False )

        TransactionsImported ->
            ( model, Cmd.none, True )

        TransactionsImportedError _ ->
            ( model, Cmd.none, False )
-- (...)


-- (...)
requestImportJson : (File -> msg) -> Cmd msg
requestImportJson a =
    Select.file [ "application/json" ] a

-- (...)
-- This view renders our settings form
viewFormDecrypted : State -> ( Html Msg, List (Html Msg) )
viewFormDecrypted model =
    let
        f =
            model.inputs

        otherButtons =
            if model.showCancelButton then
                [ div [ class "item" ] [ div [ class "ui button", cyAttr "cancel", onClick Cancel ] [ text "Cancel" ] ]
                -- (...)
                -- OUR NEW BUTTON
                , div [ class "item" ]
                    [ div [ class "ui basic button", cyAttr "import-json", onClick JsonRequested ]
                        [ i [ class "folder open icon" ] []
                        , text "Import JSON"
                        ]
                    ]
                -- (...)
-- (...)


port importTransactions : List JsonTransaction -> Cmd msg
port transactionsImported : (Json.Decode.Value -> msg) -> Sub msg
port transactionsImportedError : (Json.Decode.Value -> msg) -> Sub msg



-- src/Transactions.elm

type BalanceError
    = AllZeroError
    | MultiCurrencyError
    | NotBalanced Int


isBalanced : Transaction -> Result BalanceError Transaction
isBalanced txn =
    let
        diff =
            txn.destination.amount + txn.source.amount
    in
    if txn.source.amount == 0 && txn.destination.amount == 0 then
        Err AllZeroError

    else if txn.source.currency /= txn.destination.currency then
        Err MultiCurrencyError

    else if diff == 0 then
        Ok txn

    else
        Err (NotBalanced diff)


transactionsDecoder : Json.Decode.Decoder (List Transaction)
transactionsDecoder =
    Json.Decode.list transactionDecoder
  1. We add a new “Import JSON” button which triggers the JsonRequested Msg.
  2. When we receive that message, we call the Select.file function to have the Elm runtime handle this. It will call us with the JsonSelected which contains the File instance.
  3. When we receive the JsonSelected message, we read the whole file into a String using File.toString. The Elm runtime will call us with the JsonLoaded message which contains the String.
  4. When we receive the JsonLoaded message, we parse it into a list of Transaction.
  5. We also do some basic checks that the amounts in the transaction make sense, with a new isBalanced function.

After hooking the subscriptions in our src/Main.elm program and some few adjustments, we still need to implement the JavaScript part in src/ElmPort.js to make it all work:

app.ports.importTransactions.subscribe(async (transactions) => {
  try {
    await dbPort.saveTransactions(transactions);
    app.ports.transactionsImported.send();
  } catch (e) {
    console.error(e);
    app.ports.transactionsImportedError.send(e.message);
  }
});

Adding some UI polish

In my development build, it takes about 1.5 seconds to import the complete 7,900 transactions. When encrypting the documents, it takes about 2.5 seconds. And this is only in the persist phase (i.e. not counting the reading and parsing). It would be nice to give feedback that something is going on.

We’ll replace our half-implemented saving : Bool model attribute for a new custom type. In our settings, we will either be not working, saving, or importing:

type alias State =
    { encryption : EncryptionStatus
    , settings : Settings
    , inputs : Inputs
    , results : Maybe Results
    , showCancelButton : Bool
    , error : Maybe String
    , working : Working
    }


type Working
    = NotWorking
    | Saving
    | Importing

We will transition from NotWorking => Importing when we receive the JsonSelected message, transition back to NotWorking on any error or on success. If we are Importing, we change our Import JSON button icon to a loading spinner, and disable it:

viewFormDecrypted : State -> ( Html Msg, List (Html Msg) )
viewFormDecrypted model =
    let
        f =
            model.inputs

        importIcon =
            if model.working == Importing then
                "loading spinner icon"

            else
                "folder open icon"

        otherButtons =
            if model.showCancelButton then
                [ div [ class "item" ] [ div [ class "ui button", cyAttr "cancel", onClick Cancel ] [ text "Cancel" ] ]
-- (...)
                , div [ class "item" ]
                    [ div [ class "ui basic button", classList [ ( "disabled", model.working /= NotWorking ) ], cyAttr "import-json", onClick JsonRequested ]
                        [ i [ class importIcon ] []
                        , text "Import JSON"
                        ]
                    ]
-- (...)

Great! I won’t do the same for the “Import Sample” because I plan to deprecate that feature (and it is quite fast).

Choosing a pagination method

A few years ago I read API Design Patterns, by JJ Geewax, and I found the discussion about pagination very interesting. In particular, about the benefits of using an opaque “next page token”, which boils down to not leaking the implementation details so you can be free to change it later, if needed.

In this pattern, our getTransactions() will now accept a request, with a maxPageSize and an optative pageToken which was supplied by the previous response. If the response has a nextPageToken, then it means we have more data to retrieve1.

So we will start there, with the Elm types, and go modifying our code to use them:

-- src/Main.elm

type alias GetTransactionsRequest =
    { maxPageSize : Int
    , pageToken : Maybe String
    }

defaultGetTransactionsRequest : GetTransactionsRequest
defaultGetTransactionsRequest =
    { maxPageSize = 50
    , pageToken = Nothing
    }


type alias GetTransactionsResponse =
    { results : List Transaction
    , nextPageToken : Maybe String
    }

-- Our modified GotTransactions Msg
type Msg
    = GotTransactions (Result Json.Decode.Error GetTransactionsResponse)
-- (...)


-- Our new decoders
transactionsResponseDecoder : Json.Decode.Decoder GetTransactionsResponse
transactionsResponseDecoder =
    Json.Decode.succeed GetTransactionsResponse
        |> required "results" transactionsDecoder
        |> optional "nextPageToken" (Json.Decode.nullable Json.Decode.string) Nothing

decodeTransactionsResponse : Json.Decode.Value -> Result Json.Decode.Error GetTransactionsResponse
decodeTransactionsResponse jsonData =
    Json.Decode.decodeValue transactionsResponseDecoder jsonData



-- our modified port
port getTransactions : GetTransactionsRequest -> Cmd msg

-- our modified subscription
subscriptions : Model -> Sub Msg
subscriptions _ =
    Sub.batch
        [ gotTransactions (decodeTransactionsResponse >> GotTransactions)
-- (...)
        ]

Pagination in PouchDB

Of course, we need to modify our JavaScript to accept it. And we will do that next, but before, a related tangent about paging!

Several years ago, when I decided to write my old React app, I had read the Pagination Recipe of the CouchDB documentation. It has the typical limit=10&skip=20 pattern, and an “Alternate Method” that leverages on the default sorting by ID in CouchDB. That is the original reason why I leak information about the date of the document, as explained on my previous post about encryption, so I could get the documents sorted by date, even if they are encrypted.

However, I did not end up using that approach in my React app, because of reasons I won’t get into. So I’ll try that approach here.

The idea is simple:

  1. The request has maxPageSize = n
  2. We fetch n + 1
  3. We use the ID of the n + 1 document as the place from where to continue

We will use the browser’s native Base64 functions to encode our JSON-encoded “cursors” as our nextPageToken.

If there are no more documents, the nextPageToken will be null.

So, on to the implementation!

// src/ElmPort.js

// We now accept a request
app.ports.getTransactions.subscribe(async (request) => {
  try {
    const transactions = await dbPort.getTransactions(request);
    app.ports.gotTransactions.send(transactions);
  } catch (e) {
    console.error(e);
    app.ports.gotTransactionsError.send(e.message);
  }
});

// src/DbPort.js

const emptyCursor = () => ({ nextId: null });
function parsePageToken(pageToken) {
  if (null == pageToken) {
    return emptyCursor();
  }
  try {
    const json = atob(pageToken);
    return JSON.parse(json);
  } catch (e) {
    console.error("Error parsing token", e);
    return emptyCursor();
  }
}

function createPageToken(cursor) {
  return btoa(JSON.stringify(cursor));
}

class DbPort {
  // (...)
  async getTransactions(request) {
    const opts = {
      include_docs: true,
      descending: true,
      limit: request.maxPageSize + 1,
    };

    const cursor = parsePageToken(request.pageToken);
    if (cursor.nextId) {
      opts.startkey = cursor.nextId;
    }

    const result = await this.dataDb.allDocs(opts);
    const results = result.rows
      .slice(0, request.maxPageSize)
      .map((row) => mapDocToElm(row.doc));

    let nextPageToken = null;
    if (result.rows.length > request.maxPageSize) {
      // more results!
      nextPageToken = createPageToken({
        nextId: result.rows[request.maxPageSize].id,
      });
    }

    return { results, nextPageToken };
  }
}
  1. The code in src/ElmPort.js forwards the call and adapts the response. Nothing much.
  2. We have a pair of parsePageToken(pageToken) and createPageToken(cursor) to convert between our internal representation and the opaque representation.
  3. Our getTransactions(request) now:
    • Reads the maxPageSize and translates it into limit parameter, adding the extra 1.
    • Reads the pageToken and recovers the nextId, if any.
    • Uses the nextId as the startkey option in db.allDocs().
    • Creates a nextPageToken - if any

In later iterations I plan to add a feature of “get transactions for this account”, and I’ll be able to switch between paging mechanisms in the JavaScript “backend” without modifying my Elm code.

Implementing infinite scroll

Instead of manually going to the next page, like in the good old days, we will use the FabienHenon/elm-infinite-scroll package to add well… “infinite” scroll. First, we need to install it, of course:

elm install FabienHenon/elm-infinite-scroll

And then, follow the examples/documentation:

import InfiniteScroll as IS

type alias Model =
    { infScroll : IS.Model Msg
    , nextPageCommand : Maybe (IS.LoadMoreCmd Msg)
    -- (...)
    }

initialModel : Model
initialModel =
    { infScroll = IS.init (\_ -> Cmd.none)
    , nextPageCommand = Nothing
    -- (...)
    }

type Msg
    = InfiniteScrollMsg IS.Msg
    -- (...)

buildLoadMoreCommand : String -> (a -> Cmd msg)
buildLoadMoreCommand pageToken =
    \_ -> getTransactions { defaultGetTransactionsRequest | pageToken = Just pageToken }


update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        InitOk (Ok settings) ->
            let
                s =
                    model.editSettingsState

                editSettingsState =
                    { s | settings = settings }
            in
            ( { model
                | editSettingsState = editSettingsState
                , settingsStatus = SettingsLoaded
                , currentPage = List
                , infScroll = IS.startLoading model.infScroll
              }
            , getTransactions defaultGetTransactionsRequest
            )

        GotTransactions (Ok transactionsResponse) ->
            let
                newTransactions =
                    transactionsResponse.results

                nextPageCommand : Maybe (IS.LoadMoreCmd msg)
                nextPageCommand =
                    transactionsResponse.nextPageToken |> Maybe.map buildLoadMoreCommand

                infScroll =
                    model.infScroll
                        |> IS.stopLoading
                        |> IS.loadMoreCmd (nextPageCommand |> withDefault (\_ -> Cmd.none))

                transactions =
                    model.transactions ++ newTransactions

                listItems =
                    buildListItems transactions

                accounts =
                    buildAccounts transactions

                frequentDescriptions =
                    buildFrequentDescriptions transactions
            in
            ( { model
                | accounts = accounts
                , frequentDescriptions = frequentDescriptions
                , transactions = transactions
                , listItems = listItems
                , infScroll = infScroll
                , nextPageCommand = nextPageCommand
              }
            , Cmd.none
            )

        InfiniteScrollMsg msg_ ->
            let
                ( infScroll, cmd ) =
                    IS.update InfiniteScrollMsg msg_ model.infScroll
            in
            ( { model | infScroll = infScroll }, cmd )
-- (...)

view : Model -> Html Msg
view model =
    let
        ( mainContent, bottomBar ) =
            renderPage model

        scrollListener : List (Html.Attribute Msg)
        scrollListener =
            case model.nextPageCommand of
                Just _ ->
                    [ IS.infiniteScroll InfiniteScrollMsg ]

                _ ->
                    []
    in
    div []
        [ div
            (class "ui attached segment main-content" :: scrollListener)
            [ mainContent ]
        , div [ class "ui bottom attached menu" ]
            bottomBar
        ]
  • We add the IS.Model to our model.
  • We also add an optional “get next page” command.
  • We initialize the infinite scroll and the command to “nothing”.
  • When our app is ready, we manually transition the IS to start loading with infScroll = IS.startLoading model.infScroll, and manually fetch the first page.
  • When we get the transactions, we look our response to see if we have a nextPageToken. If we do, we set the nextPageCommand to use it. If not, we use the Cmd.none.
  • We also transition back to stopLoading and set the next command.
  • We need to hookup the internal messages of InfiniteScrollMsg msg_ so that it can listen to the Elm runtime events, etc.
  • And finally, we attach it to our scrollable view. We only do it if we know we have more data to load

I must say it took me a couple of trial and errors to understand it, and I am still not sure if this is the best way to do this. My doubt is about the end of the list, how to disable the functionality. At first I always called the \_ -> Cmd.none but it seems pointless. It seems better to just “unhook” the listener.

End-to-end tests

I’ll test three scenarios: more than two pages, one page, two pages:

Feature: Pagination should be automatic when users scroll down

    Scenario: A user with many, many transactions
        Given I have saved the default settings
        And I have 200 test transactions with description Transaction1, Transaction2...
        Then I see 50 transactions
        When I scroll to the bottom
        Then I see 100 transactions
        And no transaction is repeated

    Scenario: A user with not many transactions
        Given I have saved the default settings
        And I have 39 test transactions with description Transaction1, Transaction2...
        Then I see 39 transactions
        When I scroll to the bottom
        Then I see 39 transactions
        And no transaction is repeated

    Scenario: A user with exactly two pages
        Given I have saved the default settings
        And I have 200 test transactions with description Transaction1, Transaction2...
        Then I see 50 transactions
        When I scroll to the bottom
        Then I see 100 transactions
        When I scroll to the bottom
        Then I see 100 transactions
        And no transaction is repeated

And these are the step implementations, in my infinite-scroll.js:

const { Given, Then } = require("@badeball/cypress-cucumber-preprocessor");

const sources = ["Assets:Cash", "Liabilities:CreditCard"];
const destinations = ["Expenses:Groceries", "Expenses:Eat Out & Take Away"];

function random(arr) {
  const randomIndex = Math.floor(Math.random() * arr.length);
  return arr[randomIndex];
}

function buildRandomTransaction(n, date) {
  const amount = 50 + Math.floor(Math.random() * 4000);
  return {
    id: "",
    version: "",
    date: date.toISOString().substr(0, 10),
    description: `Transaction ${n}`,
    destination: {
      currency: "USD",
      account: random(destinations),
      amount,
    },
    source: {
      currency: "USD",
      account: random(sources),
      amount: -1 * amount,
    },
  };
}

Given(
  "I have {int} test transactions with description Transaction1, Transaction2...",
  (targetCount) => {
    const perDay = 3;
    const transactions = [];
    const today = new Date();
    for (let i = 0; i < targetCount; i++) {
      if (i % perDay === 0) {
        today.setDate(today.getDate() + 1);
      }
      transactions.push(buildRandomTransaction(i + 1, today));
    }
    return cy
      .saveTransactions(transactions)
      .then(() => cy.sendTransactionsToElm());
  }
);

Then("I see {int} transactions", (number) => {
  cy.get("div").filter(".txn-description").should("have.lengthOf", number);
});

Then("no transaction is repeated", () => {
  cy.get("div")
    .filter(".txn-description")
    .then(($el) => {
      const descriptions = new Set();
      $el.each((_, el) => descriptions.add(el.innerText));
      assert.equal($el.length, descriptions.size);
    });
});

This is the result:

And with that, we have finished this small iteration!

Final thoughts

The next iteration will probably add replication. Then, I still need to handle multi-currency transactions (we do quite a lot of USD/ARS conversions here in Argentina 😬) and finally add support for transactions with more than two entries, and then I’ll be sort of feature par with the old React app. Still lots to do!

Until next time!

  1. The pattern described in the book allows for an empty results and a nextPageToken - i.e. {"results" [], "nextPageToken": "AZ..SSD"}. This allows the server to communicate “I could not retrieve that amount of data in the SLA of 200ms, but here you have a token to try again”. Of course, all of this is really too much for this small application! ↩︎