Elm Codegen Quirks with Servant-Elm and Persistent

Developing further on the previous post, building some slightly less trivial examples with the combination of servant-elm and persistent (with Sqlite backend) yields some interesting problems in the generated Elm code. 

1. Phantom Types for Keys

Due to the way persistent handles database primary keys, servant-elm (via the underlying elm-bridge) currently generates Elm code with types such as Key World instead of WorldId, where, for Sqlite, ID variables are of type Int, used for database primary keys. 

But what is Key World in terms of Elm types? And what happens when you try to sequence a JSON encoder / decoder for Key (which doesn't exist!), along with a JSON encoder / ecoder for World -- something altogether different from the desired Int? 

type alias User  =
   { userEnv: (Key World)
   , userName: String

    [...]

jsonEncUser : User -> Value
jsonEncUser  val =
   Json.Encode.object
   [ ("userEnv", (jsonEncKey (jsonEncWorld)) val.userEnv)
   , ("userName", Json.Encode.string val.userName)
 

    [...]

There's some discussion of handling phantom types in elm-bridge, as well as some existing support for custom type replacements in the elm-bridge library.  To get it working, I ended up needing the custom type replacements, as well as some direct manipulation of the generated Elm code. (I also found some fairly extensive replacement ideas here).

The elm-bridge replacements are defined here: https://github.com/tkuriyama/spacemonkey/blob/master/spacemonkey/make/Make/Spacemonkey.hs

That takes care of the instances of Key World and the like in Elm type definitions, but not in the API call functions, and also introduces a new problem of missing definitions for Id type encoders/decoders. 

type alias User  =

   { userEnv: WorldId

   , userName: String

   [...]

jsonEncUser : User -> Value

jsonEncUser  val =

   Json.Encode.object

   [ ("userEnv", jsonEncWorldId val.userEnv)

   , ("userName", Json.Encode.string val.userName)

   [...]

getWorldByWid : (Key World) -> (Result Http.Error  ((Maybe World))  -> msg) -> Cmd msg

getWorldByWid capture_wid toMsg =

    [...]

There may be some way to do replacements in a more integral way with the Template Haskell codegen, but I ended up just doing appends and naive text replacements on the final Elm output: https://github.com/tkuriyama/spacemonkey/blob/master/spacemonkey/scripts/PostCodeGen.hs

2. Missing URL String Encoders

I also couldn't get servant-elm to generate correct URL string encoders, despite defining instances of toHttpApiData, which seems like it should be the thing to do. 

 

        Http.request
            [...]
            , url =
                Url.Builder.crossOrigin "http://localhost:8080"
                    [ "cellColor"
                    , (capture_worldid |> String.fromInt)
                    , (capture_x |> String.fromInt)
                    , (capture_y |> String.fromInt)
                    , (capture_color |> String.fromInt)
                    ]
             [...

... where Color is defined on the Haskell side as:

data Color
  = White
  | Yellow
  | Red
  | Green
  | Blue
  | Grey
  deriving (Show, Read, Eq, Generic, Enum, Bounded)

instance FromHttpApiData Color where
  parseUrlPiece :: T.Text -> Either T.Text Color
  parseUrlPiece = myParse

instance ToHttpApiData Color where
  toUrlPiece = T.pack . show
  toQueryParam = T.pack . show

So for each sum type instance, the post-code-gen script from above also takes care of the string encoders for use in URL building, appending the functions to the generated Elm code.

 

Wrapping Up

With the above edits, the Elm code gen + persistence layer stack works again. There was a lot more research and work(arounds) required this time, and I needed some help from friends at the Recurse Center, but it still (mostly!) satisfies the property of writing data definitions only once for the whole stack. It's not hard to see that some of the type mapping issues across languages and data boundaries are non-trivial to resolve, so I hope more usage and development goes into these kinds of type-safe stacks. 

Hello Server + Persistent

 The previous post covered setting up a Servant server in Haskell, with automatic codegen for Elm bindings and using the acid-state package for persistence. 

An obvious benefit of the stack is that one only ever needs to think in native Haskell data structures. It's relatively easy to use and achieves the goal of minimizing work across data boundaries (in this case, between the server and persistence layer). It's also self-contained, insofar as it doesn't require an external database server / process / engine.

On the other hand... many external database servers have been developed and refined for years -- if not decades -- yielding stability, tools, and all sorts of best practices and optimizations. 

Yesod's answer to the problem is persistent, which provides a type-safe and unified way to interact with a variety of database backends (primarily Sqlite, MySQL, Postgres, and MongoDB). Although the library was designed for Yesod, it is not tightly coupled and seems to work for Servant.

This file contains the data definitions for a revised version of Hello Server.  persist also uses Template Haskell, so we adapt the data declarations accordingly:

share
	[mkPersist sqlSettings , mkMigrate "migrateAll"]
	[persistLowerCase|
	ServerState
	state String
	counter Int
	deriving Eq Show Generic
	|]

Note that the servant-elm template invocations come later and works as expected -- i.e. first, persistent generates the ServerState data type and associate code it needs; then, servant-elm uses the generated ServerState data type for its purposes. 

This file contains the server instance, following the example in the Servant repo. 

And, with a few small changes on the Elm client side to accommodate API tweaks, it works!

There are a number of fundamental things I still haven't figured out (like... shouldn't the Sqlite connection pool be explicitly closed in the bracket pattern?). Overall, there are more things to think about compared to working with acid-state -- which seems reasonabl, since it corresponds to the relative power of the database engine (or at least, I assume so). Leveraging the "migrations" feature from persistent, though, I've still only written a single data definition in Haskell for the entire stack!

The repo with all examples: https://github.com/tkuriyama/spacemonkey

Hello Server + Acid-State

 The previous post covered defining a Servant server with automatic code generation for Elm bindings.

Are there libraries that allow for type-safe persistence in Haskell?

Again referring to the state of the Haskell ecosystem, one interesting library is acid-state, which provides ACID guarantees for de/serializing (serializable) Haskell data structures directly, with no external DBMS.

acid-state also uses Template Haskell, and it's relatively straightforward to augment the prior Hello Server data definitions. We define some simple operations we want (query, write), as well as the template invocations:

    $(deriveSafeCopy 0 'base ''ServerState)

    $(makeAcidic ''ServerState ['writeState, 'queryState])

On the server side,  this file has the updated logic to interact with the data store. Since acid-state uses monadic state, we can use ReaderT and again follow the Servant tutorial to thread the data state through the server calls. Although it doesn't seem strictly necessary given the acid-state guarantees, we can use the bracket pattern from Control.Exception to ensure that the data store is closed normally in the event of unexpected server failure.

That's pretty much it! As promised by the stack, generating the Elm code is just a matter of changing some filename configs in the code. In this case, the REST interface didn't change, so changing the backend had no impact on the rest of the client code. 

I haven't thought too much yet about the implications of this approach vs using some typed-SQL library like persistent, which I intend the try next.

Hello Server: Haskell + Elm Across Data Boundaries

 I started with the thought that it would be fun to write a small, private social network app in Elm, for family and friends to share. This immediately begs the question: what should the backend be? 

Setting aside for the moment the class of problems associated with running infrastructure with real users and data on the internet (e.g. security, availability, load, etc), there is the problem of type consistency and safety.

Maintaining safe, static data typing across data boundaries (say, client and server, or server and persistence layer) seems inherently tricky. Ideally, one wants to define data types (or data structures) somewhere,  exactly once, and have some abstractions and / or tools that propagate the type guarantees throughout the stack. That would safe effort, minimize boilerplate, and prevent typ(e)o-related errors!

The Elm ecosystem actually has Lamdera, whose the promise is that the server and persistence layers are abstracted away, leaving the developer to reason only about Elm data structures and what data maintain logically in the front vs back end. That's pretty much exactly what I want in this instance! Sadly, it remains in alpha, and doesn't appear to be open source for the moment.

Browsing through the State of the Haskell ecosystem, there are a variety of options for running servers and handling persistence in Haskell (the most obvious fit with Elm). Some of them look like they need project status updates (like Spock... people want to know the status, as it's been dormant for ~three years!). But of course I'm not the first to encounter this problem... so there are a few promising libraries. 

Haskell Server + Elm code gen + no persistence

The first is servant-elm, which wraps the elm-bridge library for use with the Servant webserver. 

Following Servant's tutorial and the servant-elm README on Github, we can:  

1. Define ADTs of our choice in Haskell, with corresponding typed REST APIs for Servant
2. auto-generate Elm code for corresponding data, including JSON serializers / deserializers that match the REST API (!)

This file defines the Haskell ADTs and API endpoints, with the Template Haskell line that servant-elm needs: deriveBoth defaultOptions ''ServerState

This file then defines the server logic, with an IORef to maintain some simple in-memory state.

Finally, this file actually generates the Elm module (well, the binary that generates the Elm module).

The resulting Elm code works as expected, albeit with some long function names!

To actually use the Elm client with localhost, I needed to start a temporary version of Chrome with some web security options disabled to account for CORS (borrowed from here).

Next post... adding persistence.

Game of Risk Elm Visualization

Building on the previous post... an interactive(ish) Elm visualization of the game's probability trees: 

https://tarokuriyama.com/projects/risk.php

Trying TCR

A fellow at the Recurse Center hosted a workshop on TCR (see Kent Beck's post for an introduction to the idea), using these templates. 

The short of it is: a watcher will automatically run tests upon file save; if any tests fail, it will revert to the prior commit; else, it will make a new commit. 

Despite some initial mental (and practical workflow) resistance, I've warmed very rapidly to the idea. 

By far the biggest change is the revert part of test && commit || revert. Some initial complaints:

• I made a typo, and it revert to the last commit
• I just lost a whole block of code
• I can't just try this simple thing because it keeps failing a test and getting deleted 

The first one maybe retains some legitimacy, but in general, the idea is: 

• slow down
• write fragments that you understand
• write smaller fragments if necessary 

The process also highlights the fact that tests may evolve with the code; as an interface changes, so too do the tests need to change (or TCR won't let you make the commits!).

A few implementation notes for macOS users, building on the Python example from these templates. 

This gist contains the edits discussed below.

watch.sh

• Instead of inotifywait, on macOS fswatch can be installed from Homebrew (brew install fswatch)
• fswatch should be called with the --one-event flag, since it's already called in an infinite loop (otherwise it blocks the next line)

commit.sh

• By default, TCR will auto-commit with the same message; as one possible alternative, call AppleScript and ask for a commit message via a prompt

    

test.sh

• The default example has some assertions in the main Python file; modify this to call a testing framework instead (in my case, py.test)

There's a good video here of doing TCR-style testing in Elm. An interesting aspect they mention is squashing a bunch of TCR-generated commits ("what changed") at a later point into more of an explanatory commit ("why the changes").

P.S. for best results with editors, autosave features should be disabled, and autoreload features should be enabled, e.g. in emacs: 

(global-auto-revert-mode t)

(setq auto-save-default nil)

Sudoku Visualization

The Sudoku solver visualization in Elm is done! 

https://tarokuriyama.com/projects/sudoku.php

Writing Elm feels similar to writing Haskell, with a bit of syntax that's more like F# (e.g. the pipeline operators <| and |>). Between the familiar format of Learn You and Elm and the examples in Beginning Elm, it's relatively easy to pick up given a bit of functional programming background.

All of the code for the solver and visualization is on GitHub.

 

Elm JSON Decoder

In the vein of functional programming, I've been learning Elm to visualize the progress of a Sudoku solver.

The JSON decoding tutorials are clear, and the JSON.Decode library is well-documented once you know how to read it (in general, the library documentation in Elm seems to be on the terse side). 

• https://elmprogramming.com/decoding-json-part-1.html
• https://elmprogramming.com/decoding-json-part-2.html

Two things were not very clear from initial read of the docs: how to parse lists to tuples, and how to parse strings to custom types. 

Leaning on SO for the below example,  tuples apparently used to be supported as first-class decoders, but in the current version of Json.Decoder, index seems to be the official way to construct decoders for tuples. 

arrayAsTuple2 a b c =

    Decode.index 0 a

        |> Decode.andThen (\ aVal -> Decode.index 1 b

            |> Decode.andThen (\ bVal -> Decode.succeed (aVal, bVal)))            

boardDecoder : Decoder Board

boardDecoder = Decode.list <| arrayAsTuple2 Decode.int Decode.int 

Elm 0.19 only supports up to three-value tuples (somewhat surprising coming from Haskell, though with records and custom data types, the decision to limit unnamed tuples has a certain logic...). So the helper construction isn't too bad.

Parsing strings to custom data types turns out to be more straightforward, though a bit repetitive (especially if you need to later get the string representation back out of custom data type -- there is no deriving Show in Elm). First decode the string, then chain a pattern match using andThen:

data Transform = Rows | Cols | Boxes

transformDecoder : Decoder Transform

transformDecoder =

    Decode.string

    |> Decode.andThen

       (\ s ->

            case s of

                "Rows" -> Decode.succeed Rows

                "Cols" -> Decode.succeed Cols

                "Boxes" -> Decode.succeed Boxes

                _ -> Decode.fail <| "Unknown transformation.")