I’ve been occasionally following the development of rust ever since its 0.4 back in 2015. It didn’t really need huge amount of efforts to catch the eyes for someone like me who has been intrigued by the gateway drug in the programming language design, Haskell, to invest some time to take a closer look at it when Mozilla said they are developing a new programming language to replace C++. And the language would be used for the next generation of rendering engine, Servo. The ambitious pitch made you raise your eyebrows, after all so many predecessors has failed. I did spend some time to play with the language back in 2015 and did a presentation internally. However, I stopped to devote major share of my time to get involved due to the immaturity and lack of echo chamber to push myself to get more involvement.
Things started to change after the release of rust 1.0, especially with the proven cases like TiDB by PingCAP, and the release of Quantum. Even Fuchia and Dropbox has rust running as the real world applications. It’s hard not to notice the momentum, though not on par with the growth of golang, more and more companies open to jump on the board of rust when they face the use cases where traditionally C++ would be a no brainer.
Earlier this year I read the information of Rustcon Asia 2019 from Rust daily news’ zhihu publication. It’s a good chance to me to attend since it is a long weekend in Singapore. The distance is not close between Singapore and Beijing (6hr of flight time), but it is definitely much closer for me to attend Rustcon in U.S.A. and I don’t need to adjust for the jet lag before and after the conference, which seems quite ideal to me. I asked if anyone would be interested to attend as well in the company’s internal rust interest group slack channel, and it turned out that three of us Grab’s SRE, myself, Alex and Lincoln decided to go to Beijing for the Rustcon Asia.
The conference’s talk session is hosted at Hyatt Regency in Wanjing. The conference is well prepared, the reception is quite smooth and it’s sweet to have the live translator for the whole conference session for people that is not Mandarin native speaker. I didn’t attend any international tech conference in China before so I don’t have a baseline to compare with. As a first-timer I found it is good step to make the conference in China truly international. For sure there are domain terminologies that were not translated well and it could be left for improvement.
There are many good talks, from beginner level to the hard-core level. Here are the list of the talks that I am more impressed with, and along with my notes.
Futures in TiKV
I didn’t follow closely to the Futures in rust, this talk is quite new to me. However, I can understand in order to make distributed system development in Rust easy and possible. Futures would be a strong building block and everyone would like to have it. It is the same for most of the mainstream multi-paradigm languages from what I can tell. PingCAP as the pioneer and the biggest user of Rust, this talk given by their engineer seems to be quite convincing on the use cases and its interface design. At this moment, the Futures seem to still at the nursery phase (you could tell that from the repo name), but the core-team seems to would like to have it stablized and ship it in the upcoming future.
Stackful coroutine based rust async story
This is another hard-core talk. It reminds me of the stackful coroutine tutorial in C++ I read on internet before. The speaker didn’t mention about what the concrete problem he is trying to resolve and why his rust implementation fits the problem. It sounds to me it is more like I want to do it because I can do it. It’s quite ok for the proof of concept, just that by missing the link to the real problem you face in the production system, I would lose track of the motivation behind as an audience. I still learned something from the Q&A session where a few of questions about the tradeoff and comparison between golang’s goroutine implementation and speaker’s implementation. I need to dig into the source code to understand a little bit more about the concrete implementation to digest.
Re:Zero — writing a custom derive
It’s a quite hard-core session on how to write custom derive for rust compiler. The speaker first introduced what are the common circumstances where you would like to have your own custom derive. In his case, it is the internals of Servo CSS rendering engine. In Servo it had a lot of data type conversion operator you would like to support, you would have to write a lot of ToXXX and FromXXX methods in your code. If you hand-writing all of them, it is not only requires lots of laboring but it is also error-prone, and you could not simply avoid that due to the intrinsic complexity in the design of CSS engine. This is the typical case where you would like to have custom derive. By putting at [derive(ConvertA)] then compiler would do all of the heavy lifting for you. However, writing the code for custom derive is not an easy task and requires special care. The input and output of your code are both code, it is not the raw input where you could verify against and easy to tell it is correct. You are doing a code transformation and rewriting, logical reasoning is probably the best way to verify it is correct or not. The examples put in the slides are written from scratch so that they are easier to be digested for the beginners, it is sweat to have a clean example for the beginners to put their hands on.
Distributed Actor System in Rust
This is one of the talks that I was impressed with, mainly because the problem domain they are working on is quite similar to some of the projects that Grab is also planning to do. And the approaches are similar in that both are aiming a Distributed Actor System, but for sure what has been done at Alibaba is way ahead. Alibaba is building a tick-based actor system, in the Q&A session it is explained that it has the flexibility to support their use cases and therefore they built their own in-house but not using the open-sourced like Akka. The talk addressed the issues they faced while doing the message layout in Rust. The major issue is the message typeID set has to be consistent across machines and across compilation, and a pending pull-request already submitted to the main branch to resolve the issue. I recalled that the project of Cloud Haskell faced the same problem where the stable typeID need to be guaranteed across the machines and compilation therefore this problem sounds quite familiar.
Be Fearless Using Rust in Production
This talk is about a real-time banner ads image stitching. It would generate the final banner ads from the provided image elements and stitch them together. The very first version of the service was in NodeJS, however, its latency was 200ms and the QPS was only a few thousands. This metrics was unacceptable to the team and they need a low level language to optimize the computation-intensive part. They tried out to rewrite the computation-heavy part in Rust as a NodeJS module and successfully reduced the latency to 20ms, but the QPS didn’t improve significantly. Due to the major blocker of high latency was removed so this version is actually deployed. In Alibaba there isn’t paved road for rust, to workaround this organizational issue, the team wrap the rust as a NodeJS application and did the deployment. This is the major problem that most of the developers would face if they would like to pitch the rust application internally to the company I suppose. There would definitely push-back from the operation team and management. With the most recent version the service was completely rewritten in rust, the QPS and the latency are greatly improved. The QPS was 20k-ish if I remembered correctly. I asked a question about the tooling to improve rust application’s observability, if there is anything comparable to golang’s pprof. It seems that there is still lack of the same thing but only something like failure to log the necessary stuff. The speaker mentioned that they debugged the deadlock of concurrent map offline.
Rust at Bilibili
Bilibili is the China’s niconico, the niche streaming website for Japanese anime. With the fast growth they faced the CI/CD performance issue and they need a distributed cache server for the building workflow. Initially the speaker said he was trying to implement it in golang, but it turned out the performance is not so well. Therefore he used his spare time to rewrite the cache proxy in Rust as a proof of concept. The performance is much better and they code-named the project as Aster, and the whole talk is basically sharing the pitfall they faced during the project development. It included the flame graph generated by prof was not fine-grained enough, missing key information so that the developers could identify the hot path to continue to optimize. With the help of the community, they resolved the flame graph granularity issue and move the thing forward. The other difficulty they faced was something about parser, and it was eventually resolved by using FSM. I didn’t pay my full attention at that part of the talk therefore I couldn’t recall most of the detail.
Search Engine in production with Rust
The speaker shared about the experience of rewriting Lucence (code name Rucene) in zhihu, roughly done in a year. With the growth of zhihu they need to build their own search functionality. Due to fast iterations in the past they hacked some version of Elasticsearch. However, the change could not be merged to the upstream therefore the in-house version stucked. They could not upgrade to the latest one and they unfortunately was using a version that was unstable and often had avalanche failure for the cluster. The search is a core functionality and therefore the team considered it is a good investment to build their in-house solution. They definitely would like to have good performance and did consider C++. However, they worried about the memory safety problem as well. They did evaluate the rust but was also worried about the lacking of the experience for the team would create uncertainty. Luckily, the PingCAP was already the pioneer by adopting rust in the Chinese community, they consulted the CEO of PingCAP and felt confident enough to go by rust, and therefore the project. At the start of the project, for sure it requires about a month for the developers to get comfortable with rust, but if it is amortized to a long project cycle, the learning investment doesn’t seem to be much, at least to zhihu team. They decided to rewrite the Lucene in rust due to practical reason. They did want to have their own index library, but due to the project timeline, they need to put the requirements like it is easy to verify the result is correct, and the ranking/scoring should be comparable to what Elasticsearch has provided. It would be a challenging task if you don’t have an implementation reference. By being compatible with Lucene, that would make it possible to cross validate with the de-factor Lucene implementation. The speaker didn’t share too much detail on what other difficulties they faced by fighting with the compilers etc, but it is still interesting to see that zhihu decided to have their one of their core infra in Rust.
Workshops
In order to get some time to do sightseeing in Beijing, I skipped most of the workshop but one, Nick’s workshop on intermediate rust. In the workshop you would know how to get yourself familiar with the ownership concept and how to write idiomatic rust. I already knew most of the stuff but it is still good to know how the slide was organized. It is very systematic and very easy to digest. I think I would use the slide as the base when I would like to pitch the rust to others.
Summary
Overall it is a good experience to attend the conference, nice speakers and good audience. The lunch on the first day is very nice and it was hosted at Hyatt. Also, you get the perks if you raise your hand and ask question, to get a free rust book. I am looking forward to see Rust getting momentum so it is much easier to get a full-time job. Right now I can feel the traction but far from the heat like golang. I am carefully optimistic.
