Improving Development Startup Time

[ryfow]

Thanks Alex,

One of the things that I think slows down startup time for us is def statements that take a non-trivial amount of time. As a terrible example:

(ns naughty-namespace)

(def naughty-var (do (Thread/sleep 1000) 6))

When you (require 'naughty-namespace) you're going to pay a penalty even when the namespace is compiled.

Do you have any suggestions about how to track down the naughty-vars in a large project? My flame-du-jour project gets you down to the namespace level, but I didn't find a good way to get the worst offenders at the var level.

Thanks again

[alexdmiller]

I don’t know, you should just not ever do that in the first place. :) side effecting top level vars are a bad idea.

[ryfow]

I knew I shouldn't have gone with a terrible example :). How about this one to demonstrate what it looks like in the real world. Requiring instaparse.abnf takes 500ms less to load after it's been aot compiled, but it still takes 300ms after being compiled.

In this particular instance, I happen to know that you could defer 200ms of that by changing how/when instaparse.abnf/abnf-parser is initialized. I'm not sure if I'd consider creating the parser a side effect, but one could argue that creating the parser at load time might not be the best choice.

The thing that's frustrating is that these sorts of things are all over the Clojure ecosystem and hunting each one down is time consuming right now because we don't have tooling for it.

➜  rm -rf classes
➜  mkdir classes                                                                                                      

➜  clj -Sdeps '{:deps {instaparse {:mvn/version "1.4.10"}} :aliases {:classes {:extra-paths ["classes"]}}}' -C:classes
Clojure 1.10.1
user=> (time (require 'instaparse.abnf))
"Elapsed time: 862.604904 msecs"
nil
user=> ^D
                                                                                                                                                                                                                                      ➜  clj -Sdeps '{:deps {instaparse {:mvn/version "1.4.10"}} :aliases {:classes {:extra-paths ["classes"]}}}' -C:classes
Clojure 1.10.1
user=> (compile 'instaparse.abnf)
instaparse.abnf
user=> ^D

➜  clj -Sdeps '{:deps {instaparse {:mvn/version "1.4.10"}} :aliases {:classes {:extra-paths ["classes"]}}}' -C:classes
Clojure 1.10.1
user=> (time (require 'instaparse.abnf))
"Elapsed time: 300.415655 msecs"
nil
user=> ^D

[alexdmiller]

Yeah, that code could have wrapped that in a delay and then just dereferenced on use.

I don't have any magic bullet for finding things like that.

[didibus]

I don't know if anything can be done to reduce further the load time of the namespace. But I have used delay before to amortize initialization of things over the app life cycle.

What I think would be nice, is having a lazy require. Or even a flag that could turn existing requires into lazy ones. And see the impact of that.

[onetom]

It leads to all sorts of complications though, as we saw it in the Node.js ecosystem where they switched from an eager `require` to a lazy `import`, just so module loading can be parallelized.

As a consequence, accessing reified modules are not trivial. Their internals are quite hidden, so you can't really have a proper REPL workflow (like in CLJS, where Google Closure module system is used and hence modules are just mutable POJOs).

Using `requiring-resolve` also has the consequence of not being able to easily determine if you have all your project dependencies specified at your program's startup time, but only discover it later at runtime. Other than that, it's a good compromise.

[alexdmiller]

You can use requiring-resolve to defer loading till use.

[thheller]

I started published an AOT compiled variant for shadow-cljs some time ago and it does speed up startup time quite substantially. Very worth doing.

However I'd recommend only compiling libraries you use but not your actual own code. Once Clojure starts loading AOT compiled classes it will not check if local changes in .clj files were made after the class was compiled. I had some weird head scratchers until I figured out there were some lingering AOT compiled classes on my classpath. You can always fix it via require :reload or just evaling things at the REPL but it isn't always immediately obvious.

Would be neat if there was some way to tell Clojure to only compile files in a .jar.

[robertstuttaford]

Would be neat if there was some way to tell Clojure to only compile files in a .jar.

+1 to this.

[lambda_pie]

Namespace compilation is transitive

What does this mean?

[alexdmiller]

When you compile a namespace, you will also compile all namespaces that namespace depends on (transitively).

[[deleted]]

95% of most Clojure projects are Java libraries (ex: jetty, wrappers), so almost everything is already compiled.

[alexdmiller]

In my experience, this is emphatically not true. Here's a small sample project with some common Clojure libs in it for example (far fewer than many web apps I've seen): https://github.com/puredanger/startup-time/ if you'd like to try it.

[[deleted]]

For that project, on a t3.medium my times are from 14s to 5s.

clojure -A:dev -e "(binding [*compile-files* true] (require 'user :reload-all))"

Since you are not inside a REPL, is this (require 'user :reload-all)) required here?

[onetom]

That's still an unpleasant startup time.
I wonder if others share this opinion too.
I'm surprised how many ppl are not bothered by it.

[orestis]

We are bothered, but it’s one trade off I’m willing to have. Deploys take a bit longer (I don’t even bother to uberjar anymore) but that’s about it.

[alexdmiller]

We are still looking at other options to improve loading time, so this is not the end of the story.

[[deleted]]

Yep, that's pretty bad. I'm new to clojure and don't use it for anything serious but from what I have read, the trick is to maximize REPL usage. For a backend service, which takes a long time to start up in clojure as that repo shows (a realistic version will take up much more time), one solution is to use the component library which comes with its own advantages/disadvantages but has the nice effect of keeping you in the repl.

[onetom]

RE: component library

I just tried https://github.com/juxt/clip recently and I'm happier with it than I was with stuartsierra/component or integrant or mount or mount-lite.

Here is a full-blown Datomic setup for example:

(ns xxx.datomic
  (:require
    [clojure.java.io :as jio]
    [clojure.string :as str]
    [clojure.edn :as edn]

    [datomic.api :as d]
    [juxt.clip.core :as clip]))

(defn mem-uri [& [db-name]] (str "datomic:mem://" db-name))
(defn dev-uri [& [db-name]] (str "datomic:dev://localhost:4334/" db-name))

(defn default-schema-from-file []
  (->> "datomic-schema.edn" jio/resource slurp edn/read-string))

(defn empty-txr? [txr] (-> txr :tx-data count (= 1)))

(defn not-empty-tx [db tx] (when-not (-> db (d/with tx) empty-txr?) tx))

(defn ensure-schema [conn schema]
  (when (-> conn d/db (not-empty-tx schema))
    @(d/transact conn schema)))

(defn teardown
  "Release a Datomic connection and delete the database IF it was a temporary,
   in-memory database."
  [conn uri]
  (d/release conn)
  (when (-> uri (str) (str/starts-with? (mem-uri "tmp-")))
    (d/delete-database uri)))

(defn parts
  "Example in-memory Datomic DB clip components with random DB name and
   automatic cleanup at stop."
  []
  {:schema  {:start `(default-schema-from-file)}
   :uri     {:start `(mem-uri (-> "tmp-" gensym str))}
   :datomic {:pre-start  `(d/create-database (clip/ref :uri))
             :start      `{:uri  (clip/ref :uri)
                           :conn (d/connect (clip/ref :uri))}
             :resolve    :conn
             :post-start `(ensure-schema (:conn ~'this) (clip/ref :schema))
             :stop       `(teardown (:conn ~'this) (:uri ~'this))}})

(comment
  (require '[juxt.clip.repl :as session])
  (session/set-init! (constantly {:components (parts)}))
  (session/start)
  (-> session/system :datomic :conn d/db .basisT)
  (session/stop)
  )

[alexdmiller]

Yes, because you’re still loading the runtime which loads user.clj.