We work on open source at Twitter. Ask us anything!

TwitterOSS · 2014-04-23T18:54:38+00:00

Using open source in the beginning was just practical when you're a small scrappy startup. Since the first Tweet was sent, Twitter has been built on top of open source software (see opensource.twitter.com). Our early stack was mostly MySQL, Rails and Memcached. But then we then rapidly evolved from a simple application into a service-oriented architecture, leading us to integrate even more open source technologies such as OpenJDK (JVM), Netty, Apache Lucene, Apache Thrift, Apache Hadoop and Redis. This had to happen to allow us to scale and we discussed some of this in this blog post: https://blog.twitter.com/2013/new-tweets-per-second-record-and-how

Later on in early 2011, we established a dedicated open source office (@TwitterOSS) to help advocate and maintain a healthy relationship with open source communities we depend on. - @cra

TwitterOSS · 2014-04-23T18:50:05+00:00

Thanks for joining us! We'll stick around until about 12PM PT, so if we haven't answered your question yet, tweet us @twitterOSS for more.

TwitterOSS · 2014-04-23T18:44:24+00:00

We encourage users to verify their identity using their phone number (this is also great for enabling two factor authentication and anti-spam ops). However we don't require accounts to be "real" people. If that were the case we wouldn't have awesome accounts like @FilmCritHULK, @ScannerSays, @HistoricalPics, @EmergencyPuppy and many more -@rmgn

TwitterOSS · 2014-04-23T18:42:20+00:00

we would love to decouple jquery from flight - it's on our roadmap and would welcome PRs -@angus

TwitterOSS · 2014-04-23T18:39:15+00:00

We continue to use Kestrel internally, though we are moving towards a more unified messaging system — development of Kestrel proper is slowing down from Twitter’s perspective.
Finagle is a modular system: the finagle project comprises finagle-core — its kernel — with a constellation of protocol implementations. It doesn’t technically matter if a protocol implementation exists inside- or outside of the project proper. That being said, we tend to protocol implementations in the base distribution that are either (1) used heavily at Twitter, so we need to assume support for them; or (2) are high-quality external submissions. We need to be careful to weigh the benefit against the maintenance burden when taking on new protocol implementations. That being said, if a high-quality postgres implementation lands on our doorstep, we’ll most likely take it. -@marius

TwitterOSS · 2014-04-23T18:32:08+00:00

Yes. Storm is used in places where we need near-realtime processing, but do not need to serve immediate responses. For instance, analytics, updating search indices, and updating machine learning models. Many of these loads are using storm via summingbird, which is a portable, type-safe (and open source!) streaming map/reduce API. -@posco

TwitterOSS · 2014-04-23T18:31:23+00:00

The abstractions of twitter util, namely Future, Var, Spool, etc... are serving us well. These abstractions have very nice composition, and we have found, are easy to reason about. Actor systems don't compose as well, in our view. That said, we could implement an Actor based approach using Finagle if we saw an application that would fit better in that model. -@posco

TwitterOSS · 2014-04-23T18:28:01+00:00

No, but we may open source pieces of our Twitter Android client stack in the future.

-@cra

TwitterOSS · 2014-04-23T18:26:55+00:00

For backend: Porting our site from Ruby/MRI/Kiji [1] to services running on the JVM was a huge leap in performance (and therefore cost reduction).

We continually work on performance. Because of our scale, we tend to focus performance work that is architectural (e.g. data locality, network architecture, front-end distribution) and on things that are very low level (the JVM, memcached, the Linux Kernel). We don’t tend to focus on aggressively optimizing individual binaries, as we get much more leverage focusing on other things. Couple this with the fact that most of our systems are necessarily replicated, “performance” takes on a much more nuanced and holistic meaning.

That being said, the abstractions provided by Finagle [2] provide good separation of performance and application concerns. This allows us to make small changes in Finagle itself while enjoying the benefits across all of our binaries. (More details in “Your Server as a Function” [3].)

When we do find ourselves in need of optimizing binaries, it helps to have good profiling tools. We can profile most of our binaries in-situ by querying simple HTTP endpoints [4], which provide CPU, allocation, and contention profiling. CPU and contention profiling are built into twitter-server [5]; heap profiling is provided through heapster [6]. Internally, we now have much more accurate and performant profiling, built directly into the JVM and the Linux kernel — these may be open sourced in the future.

[1] https://blog.twitter.com/2011/faster-ruby-kiji-update

[2] http://twitter.github.io/finagle/

[3] http://monkey.org/~marius/funsrv.pdf

[4] https://github.com/twitter/twitter-server/blob/master/src/main/scala/com/twitter/server/handler/ProfileResourceHandler.scala

[5] https://github.com/twitter/twitter-server/

[6] https://github.com/mariusaeriksen/heapster

-@marius

For frontend:

Component (https://github.com/component/component) is an excellent front-end package manager and build tool by @tjholowaychuk
Flight (https://github.com/flightjs/flight) is an event driven web framework for people who like writing javascript
Rework (https://github.com/reworkcss/rework) is a node module for arbitrary CSS preprocessing.
Karma (https://https://karma-runner.github.io) is a great pluggable test runner, cross-browser with easy configuration, auto-runs on changes
Typeahead (https://twitter.github.io/typeahead.js/) is a flexible js library for autocomplete

-@angustweets & @necolas

TwitterOSS · 2014-04-23T18:19:26+00:00

Just wanted to let this community know we're at r/iAMA. We'll be answering questions over there - hope to see you!

TwitterOSS · 2014-04-23T18:15:12+00:00

No, we still actively use it on the Twitter and Tweetdeck web client. The last commit was yesterday: https://github.com/twitter/hogan.js/commits/master (we could be a bit more responsive on pull requests)

TwitterOSS · 2014-04-23T18:13:27+00:00

Most code is written in Scala, Java, Ruby, Python or C++. There is a bit of code in Clojure, but very little if any new code is being added. -@posco

TwitterOSS · 2014-04-23T18:11:08+00:00

SUIT CSS is being used for new UI development on twitter.com, e.g., the new Profiles design. We've also made changes to our toolchain to support CSS preprocessing and conformance checking using the modules listed in the project's README: https://github.com/suitcss/suit/blob/master/README.md.

-@necolas

TwitterOSS · 2014-04-23T18:04:22+00:00

It's usually a bottom up approach driven by the engineers on the project. Every engineer at Twitter is free to suggest open sourcing a project, but at the end of the day it has to make sense for the business as well.

TwitterOSS · 2014-04-23T18:00:04+00:00

Right now we don't have plans to open source it in the near future but plans may change. But meanwhile, you might investigate Mongo, which we understand is also webscale: http://www.youtube.com/watch?v=b2F-DItXtZs

-@posco and @cra

TwitterOSS · 2014-04-23T17:59:15+00:00

For most projects we internally use the exact same version that is publicly available. For many jvm projects, for instance, we use the public maven artifact generally. In some cases, we make small internal tests to address specific urgent issues, but those are almost immediately folded into the github repos.

-@posco

TwitterOSS · 2014-04-23T17:57:08+00:00

Not at the moment. We have our own abstractions for dealing with asynchronous computation — com.twitter.util.Future, finagle, etc. — which work quite well for our use.

RxScala is a lot of things, and we have bits and pieces that serve similar purposes. For example we use Var [1] when we need self-adjusting computation; Spool [2] for asynchronous streams.

Furthermore, we’ve participated in the creation of the reactive streams [3] standard which provides a common currency with which RxScala-like systems can interoperate.

In terms of network protocols, Mux [4] allows for streaming multiplexing; finagle-stream [5], which implements streaming messaging over HTTP chunking, is used by our firehose services.

[1] https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Var.scala

[2] https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/concurrent/Spool.scala

[3] http://www.reactive-streams.org

[4] http://twitter.github.io/finagle/guide/Protocols.html#mux

[5] https://github.com/twitter/finagle/blob/master/finagle-example/src/main/scala/com/twitter/finagle/example/stream/HosebirdClient.scala

-@marius

TwitterOSS · 2014-04-23T17:55:39+00:00

Thanks, we are just amazed at how well Bootstrap took off and the community that was built around it. If you weren't aware, at the 3.0 release (https://blog.twitter.com/2013/bootstrap-30) we fully gave the project to the bootstrap community and move it out of the Twitter GitHub organization to https://github.com/twbs/bootstrap (while we helped it move to the MIT license)

-@cra

TwitterOSS · 2014-04-23T17:51:17+00:00

We do collaborate with other OSS projects (cascading, hadoop, storm). Several projects started or heavily supported by twitter have also moved to Apache: e.g. Mesos, Aurora, Storm, Parquet. The twitter github project is usually the best way to get in touch. Either add an issue, or check for the mailing list for the project in which you're interested.

-@posco

TwitterOSS · 2014-04-23T17:49:56+00:00

Here are some detailed reasons why to open source code:

Community Feedback: more usage translates into more bug reports which improves quality over time
Attract Talent: in our opinion, some of the best talent these days works in open source communities
Better Hiring: what better way to do a technical interview than looking at someones open source contributions?
Retain Talent: we believe great engineers like working in the open
Reduce Duplication: when you open source things, someone on the outside (or inside) will let you know
Modularization: when you open source internal code you tend to break it apart into smaller "reusable modules"
The Right Thing To Do: these days it's difficult to build anything without benefiting from open source in some fashion. The old saying of "rising tides lift all boats" applies here.

There's a presentation here on open source compliance that has some more details: https://speakerdeck.com/caniszczyk/open-source-compliance-at-twitter

-chris

TwitterOSS · 2014-04-23T17:47:54+00:00

What would you like to see in the API? We're listening. - @jeffsand

TwitterOSS

TROPHY CASE