Watermill: from a hobby project to 8k stars on GitHub

Banner image

Quick takeaways

  • Solve real problems first - successful open source projects start by addressing actual needs, not by looking for problems to fit a solution
  • Keep breaking changes minimal - Watermill stayed on v1 for 6 years with no breaking changes in the core library, building trust with users
  • Examples and documentation are crucial - provide real-world examples with automated tests, not just simple “hello world” demos
  • Promotion matters - creating a great library isn’t enough; you need to actively share it through conferences, blog posts, and communities
  • Be patient with growth - Watermill took 7 years to reach 8,000 stars; overnight success in open source is rare

Introduction

In this episode, we share the story of how Watermill, our event-driven library for Go, grew from a side project to a popular open source library with over 8,000 GitHub stars and 100+ contributors.

We discuss the key decisions and strategies that helped make Watermill successful, from focusing on solving real problems to maintaining backward compatibility and building a community around the project.

Show notes

Quotes

My idea was that it should be possible to make building event-driven applications as easy as HTTP handlers. And if you think about building HTTP APIs, actually you don’t need to care about many things when you are building it. So you don’t care about connection pooling, handling TLS, HTTP protocol, TCP protocol, DNS resolution, partial reads, network failures.

Robert

It’s not as easy as upload anything to your GitHub so some recruiter can tell you know how to code. But rather, if you spend time and maintain the library over time and people find it useful, then it can work like this.

Miłosz

If you would like to create a library that people like to use, create something that people need. I know that it sounds obvious, but I’ve seen a lot of libraries that, well, the use case may be clear for the author, but if the author is posting it somewhere, people are like, yeah, it’s cool, but we don’t really know how to use that and how it can be useful for us.

Robert

If someone sees a project that has not been updated since a few months, they might be afraid to use it again. Like, is it deprecated? Is it maintained? Because I’ve seen there is no updates. So regular updates are often a good idea.

Miłosz

If you are building your library and you have pretty complex examples, it’s a really good idea to start with some end-to-end tests for them from day zero.

Robert

Simple examples are fine to start with, but then when you try to apply this in your own projects, you will see it’s not that simple. I guess it’s better to have some more complex examples and see how it works in real life.

Miłosz

Timestamps

Transcript

Robert [0:00]: Did you ever wonder what’s the real key to build a successful open-source project? Today, we are going to share what helped us to grow Watermill from a small sideproject to a popular open-source library with over 8,000 stars on GitHub. What have we learned along the way? I’m Robert.

Miłosz [0:19]: And I’m Miłosz, and this is No Silver Bullet Live podcast, where we discuss mindful backend engineering. We have spent almost 20 years working together in different projects and teams. We learned that following advice like always do X or never do Y doesn’t work and can limit your growth. In this show, we share multiple perspectives that will help you make smart choices and grow into the principal engineer level.

Robert [0:46]: So if you have any questions, leave them on the chat, so we’ll surely address them at the end of the live. or maybe during, if it will be also relevant to any place that we’ll cover during the discussion.

Miłosz [1:01]: Right, so let’s start with what is Watermill for anyone who is not aware. Watermill is a code library we maintain for building event-driven and message-oriented applications, and building it the easy way. And it has currently over 8,000 stars on GitHub, more than 100 contributors.

Robert [1:25]: Probably somebody’s using that. So yeah, obviously GitHub stars is some kind of proxy, probably to understand how popular the library is. But, well, we’re not buying them, so probably it means something.

Miłosz [1:37]: Yeah, and GitHub also shows, I think, above 1,000 repressories that use it. Some of them are also quite popular. So this is kind of a social proof, I guess, the best we can have that it is in fact used.

Robert [1:51]: The outside is that we don’t really know what companies are using that. So we know some, but probably most we don’t know because we have no way of backtracking who is using Watermill. So if you are using Watermill in your company, let us know, because it will be cool to know.

Miłosz [2:07]: Sometimes we hear from people on conferences or just randomly on the internet that But yeah, we use Otremio, we love it. But it happens after someone uses it for a few years, let’s say. So it’s not always the best way to know.

Robert [2:21]: So it’s always cool to hear from you that it’s useful. So we always appreciate that.

Miłosz [2:27]: Yeah, so Watermill supports 12 officially supported Pub/Subs like Kafka, RabbitMQ, NATS, but also Postgres or MySQL. And we have SQLite support coming soon.

Robert [2:40]: Yep, exactly.

Miłosz [2:42]: So whatever infrastructure you use, you might find it useful.

Miłosz [2:49]: Before we go further, maybe let’s focus on what Watermill solves. Why did we create it?

Robert [2:56]: Yes, probably it will be easiest to start with actually how Watermill started. Because I think that this is interesting and it may be also useful for you to try to create some library. And, oh, it was like probably many libraries that are there. So we’re working on some project where we had one challenge. So in this project, Event-Driven Architecture would be a perfect fit because of resilience requirements and some tasks that we could do in the background. Upside was that I was already familiar with Event-Driven Architecture, but most of the team wasn’t at this point.

Miłosz [3:35]: Yeah, and I remember the meeting when we sat in the room and you draw the Event-Driven Architecture on the whiteboard and you were very confident it would work. And the rest of us were like, I don’t know, it seems complicated. And we never used those pops-ups, Kafka is scary, what if we mess something up? Exactly. What would be useful is to have something easy to use, to abstract away those complicated details.

Robert [4:06]: Yeah, and I would say that the bar was set pretty high because my idea was that, it should be possible to make building event-driven applications as easy as HTTP handlers. And if you think about building HTTP handlers, HTTP APIs, actually you don’t need to care about many things when you are building it. So you don’t care about connection pooling, handling TLS, HTTP protocol, TCP protocol, DNS resolution, partial leads, network failures.

Miłosz [4:37]: Reconnect.

Robert [4:38]: Yeah, it’s a lot of things that can happen in the background or it’s happening all the time, But you are not thinking about this normally. You are defining, I would like to have handler at this URL with this method, and that’s all. And it’s really high level, and you don’t need to think about all the details. And, well, Watermill is a similar idea. So you should be able to create your event-driven services, but with really high-level API. And adding support, for example, for one event should be super simple. You don’t need to really think about topics, Kafka, or any message broker that you are using under the hood. It’s not an only thing, because there are a couple more things that Watermill is solving. The next one is handing for you a marshalling and un-marshalling of messages, because usually you are using some transport.

Robert [5:36]: So JSON, protobuf, whatever, and just doing it over and over, it’s just boring and a waste of time. Another thing that also is cool and Watermill can support is allowing you to swap PubSub implementation pretty quickly. So if you have seen some presentation that I was doing about Watermill over the time, I often show an example when I was using, for example, Kafka, and I wanted to switch it to Google Cloud PubSub or vice versa. And I was able to do it during live coding, and it took like one minute or less to do that. And it’s cool about Watermill because it’s putting high-level abstraction on this PubSub. And thanks to that, you can switch it pretty easily, like with ORM, basically, with databases. basis.

Miłosz [6:27]: Yeah, so those are the basics we started with. And then we built more features on top of them. Some of them are quite complex, but with a high-level API, so you can use them easily, like CQRS pattern with commands and queries. Sorry, not queries, but commands and events. Or Poison Queue or the transactional outbox pattern, which is super useful and very common. are commonly needed, not everyone uses them, but probably more people should. And the important part of those features are that they should be easy to use for everyone in the team, even if they don’t know how it exactly works under the hood, which is something we would need in that project back then, because we, not all of us were experienced with even the architecture, but it would be a nice approach for the project. So if we had Watermill back then, it would be much easier to start.

Robert [7:30]: Yeah, and probably today we also don’t want to go that much into details about some Watermill specifics and things like what’s the difference between PubSub and Q, because I know that for many people it’s not clear what’s the difference, what are use cases for PubSub, what are for queue. It’s something that we’ll cover more in the episode that will be in two weeks. Today, I would like to more focus on what we believe that made Watermill successful. Because, again, it’s probably not the most popular library of all times in Go, but it’s probably the most popular library for building event-driven applications in Go. So I think it’s a pretty big achievement. And there are a couple of things that we believe that helped us with that. And I think it’s also worth mentioning that it wasn’t overnight success. And as we mentioned, so it was after some project that we started after work. And we started to spend more and more time on that. And it took us time basically to make it.

Miłosz [8:37]: And the first comment was seven years ago?

Robert [8:39]: Yeah, I think it was seven years ago and seven days, something like that. So yeah, super long time ago. and it took us half year to issue version 0.1 so not 1.0 0.1 so it was some kind of alpha that we were able to raise to people so and i think i didn’t really write down how many comments it was but it was a lot of work to at least prepare this 0.1 version i remember i’m actually not sure but I think before version 1.0, we spent 500 days to release that. And what also helped us during that time is that we didn’t have a daily job, because at this point we, for the first time, tried to go full-time on our own. That time it didn’t succeed, but at least we released Watermill 1.0. But it took a lot of effort to do that.

Miłosz [9:41]: It was just going over time, steadily, instead of this huge success right away.

Robert [9:49]: Like in previous episodes, we’re not giving you any magical solution that will make your library super popular in one month. Because probably it’s often about a lot of luck because you can create a great library, that for some reason didn’t take out because there are, for example, 10 other libraries that are doing the same. So there is just a lot of competition. And if you are not lucky, well…

Miłosz [10:23]: It’s difficult with regular products and open source is more specific than that. It’s probably worth asking why someone would create a open source project it’s probably not to make money you can see many companies and projects try to monetize on open source, it usually doesn’t end well all the issues with licensing and so on

Robert [10:50]: I think it’s not fully fair when you’re starting an open source project and making it closed source after time after more people are using that but unfortunately it’s often happening. I guess that often it’s some pressure from investors that invested in some company and it kind of makes sense. You can expect that if you are using VC-baked open source that someday investors will come and ask where is my money back.

Miłosz [11:21]: I can count on sponsors as such but it’s probably not the idea to do it just for this. We found a way to connect it with our commercial projects, but we will touch on this later. But so far, I think we can summarize that we learned a lot doing it. That’s probably the primary reason I would try again to build an open source library. Just to learn something new and publish it. And also gives you some credentials if it succeeds. Like if you’re looking for a new job or just want to have some proof of your competence. That’s a good way to do it.

Robert [12:06]: I think it’s interesting sometimes to hear on the interview, on both sides actually, because sometimes when we’ve been interviewed, some people know Watermill, but I also remember interviewing some people and they were like oh, okay, you’re a creator of Watermill and you work here, nice.

Miłosz [12:22]: But that’s the difference between a project you just upload to GitHub and no one uses, and some project that some company actually found useful. So it’s not as easy as upload anything to your GitHub so some recruiter can tell you know how to code. But rather, if you spend time and maintain the library over time and people find it useful, then it can work like this.

Robert [12:48]: And I think it’s also nice when you’re recruiting somebody and this person can send you his GitHub or her GitHub and you can see that the person is working on some open source project after hours. And you can have some understanding on how this person is working and you can also you know feel that this person is able to, spend some time after job to learn something maybe or maybe create something cool that other people can also benefit from yeah it’s.

Miłosz [13:19]: Also a nice way to give back to the community because we’ve been using many open source software before and this way other people can use our ideas which also It also feels nice, you can do it for the satisfaction you get from it.

Robert [13:37]: All right but it all sounds cool but many libraries are like that and they’re not that successful so i think it would be good to discuss some things that really makes some projects successful and, some of my biggest observation from other projects and i know it may sound obvious but, if you would like to create a library that people like to use create something that people need because I know that it sounds obvious, but from other side, I’ve seen a lot of libraries that, well, the use case may be clear for the author, but if the author is posting it somewhere, people are like, yeah, it’s cool, but we don’t really know how to use that and how it can be useful for us. And, yeah, it also doesn’t mean that it needs to be some solution for a problem.

Miłosz [14:36]: A very generic problem, right?

Robert [14:38]: Yeah, because solving generic problems is more harder because it requires more effort. So probably it’s good to find something in the middle, like, for example, water mill. So, creating event-driven applications, it’s not something that every company does, but a lot of companies does. So, it’s big enough, let’s say, market, if we can say it this way. But, okay, we can say that if you’re building open source library, it’s pretty close to creating a product. So, you can say that it’s some kind of market for that. And it’s big enough that there are many people that can use that and benefit from that. but it’s also kind of niche so there is no competition around there so it helps a lot.

Miłosz [15:25]: Yeah, and sometimes if you start a project and you have a problem you want to solve and then you look for a solution and you find an open source library so you might feel discouraged to try and build your own I think it still makes sense to try because you may have a fresh approach. You can provide a better API or better design or less dependencies. So there’s definitely a place for more libraries that solve the same thing.

Robert [16:01]: Yeah, definitely. So yeah, I think it’s a good…

Miłosz [16:04]: In Go ecosystem, it was clearly visible because in the very beginning there were only a few libraries to do everything. So everyone sticked to one, let’s say, HTTP router. Because there were no other options. But as other libraries appeared, many people moved on. Aside from routers, maybe CLI libraries are also a good example. So there’s still a place for better APIs or a different approach that will help people to do something in a better way.

Robert [16:47]: Again, try to focus on the problem that you are solving and avoid trying to find a problem for your solution. So I’ve seen always that people found some cool technology or some cool idea, but they didn’t have a real use case for it. I think it’s probably easy to do AI now. So you can try to put AI in many places, and it will be helpful in many places. But in many places, it will be more looking for problem, for solution that you have. So you should also really watch out for that. I think it’s one interesting observation about this focus at the beginning on solving the problem. I think at the beginning, we didn’t know what word terminal is. So we had, as we said earlier, we had a problem that we had people that didn’t build any event-driven application earlier. And we didn’t have an idea, okay, we need to build a library for abstraction.

Miłosz [17:51]: All use cases everywhere.

Robert [17:52]: Yeah, yeah. Fun fact is that actually Watermill at the beginning, when it was alpha, it was named GoDDD. So it was probably something that, okay, it was around these domain events that we had there. And it started with that.

Miłosz [18:07]: I’m still not sure why.

Robert [18:11]: Probably we’ll never know. And yeah if you’re experimenting with time you are experimenting with time, it’s important to not keep this for you forever so create some version that will be useful for people and try to share it ASAP for feedback.

Miłosz [18:40]: Just mark it as experimental so people are aware that the API can change so they are not annoyed. It’s normal to have some proof concept and then keep changing the API if you find a better way. That’s fine. Just make sure people are aware. But this early feedback can be super helpful though.

Robert [19:02]: Yeah, and one of the best early feedback that you can have, and it can be also a nice hint how to find a, problem that people care about is using it in your daily job as long as you can because i know that it depends it’s maybe hard in some teams but like in our case so we’ve been building it after hours but we’ve worked in a company that it was fine to just use it in a real project that we worked in so thanks to that we had real-time feedback for that and we’ve been able to dog foot it with it but Obviously, it’s just a starting point. At some point, you need to share it outside so other people can give you feedback, and you can adjust to that, because you will be not always right from day one.

Miłosz [19:54]: And after a while, it’s probably a good time to start with documentation. We have some docs in place, and the go doc we have is not enough, usually. Because the go documentation is not that bad, but it’s still… It’s sometimes hard to get started with just the code documentation. So, we found it very useful to have this getting started page, so someone can easily see how to install the library and some minimal example to get started, also some minimal theory. And then lots of real-life examples are also super useful, which is sometimes… I’m super happy when I see some library and I see the examples directory, I often go straight there, instead of trying to figure out how it works. Because you can just copy some snippets and it gives you a better idea of how it works, rather than trying to understand from the code itself.

Robert [21:02]: Or clone it and run it locally and play with that. So it’s also pretty useful. And I think it’s also pretty close to our way how we believe that learning is the best. So actually learning by doing. So it’s really close to it.

Miłosz [21:21]: And lots of our examples we have for Watermill are real-world examples. So they are more complex than a simple hello world. So you should be able to copy them to your solution and change them, and it should be what you need.

Robert [21:36]: Yeah, so if you’re longer with us, you should know that we don’t like examples like to-do apps or some fake examples, because usually it’s just hiding a lot of complexity. Okay, maybe some Watermill examples aren’t that simple, but they’re as simple as they can to show how to solve these problems.

Miłosz [21:59]: Simple examples are fine to start with, but then when you try to apply this in your own projects, you will see it’s not that simple. I guess it’s better to have some more complex examples and see how it works in real life.

Robert [22:14]: Yeah, for example, we have the example with handing ordering in your messages. And this topic is not simple. There are a couple things that you need to do right. It also differs a bit for different PubSubs. But again, it’s easier to give you more complex example rather than you need to figure it out yourself. Because we know that it’s not easy because at some point we needed to figure it out as well. At the end, your life will be easier with those examples. One tip. So if you are building your library and you have pretty complex examples, it’s a really good idea to start with some end-to-end tests for them from day zero. So every example that we have, it’s included in the documentation, but we have also automatic tests that are checking if those examples are working properly. So it’s cool because when we are releasing a new version of WaterMill, we’re updating dependencies, we can just run tests on all examples and ensure that all of them are working previously.

Miłosz [23:18]: It’s easy to break them without a version bump or something like this.

Robert [23:26]: But it’s worth mentioning that it’s not that easy, unfortunately, to create such tests. Sometimes it’s taking a pretty long time to run them. But again, it’s still better than not having an automation for that. Because doing it by hand is 10 times slower. And it’s nothing worse than examples and documentation with snippets that doesn’t work. Because it’s just a super bad experience. Because if somebody is trying to use your library and your examples are not working, Well, probably the person may just leave your library and don’t use it at the end.

Miłosz [24:03]: Yeah, but assuming they stay, you usually will have some early adopters of your library. It’s nice to have a place for communication, for building the early community. So in our case, it was issues and then Discord. Our Discord and dedicated channel. So it gives you a place to listen for this feedback from people and your suggestions. And also sometimes they contribute, which is super nice. You have your first contributors, so it’s not just your project, but you start to feel like you maintain it, actually. It starts to get more real. What other things help?

Robert [24:53]: The next thing that is coming to my mind is the fact that Watermill is not a framework. So, we’ve covered it more in the previous episode about frameworks, but TLDR is that. We are not big fans of frameworks in Go for multiple reasons. And I think it also helps with building a library when this is not a framework.

Robert [25:14]: For one reason, because it’s easier to adopt if it’s just a library. Because when you are starting to use some kind of framework, it requires more changes in your code. And I guess that many people may be also not…

Miłosz [25:34]: So it locks you in.

Robert [25:36]: Sorry?

Miłosz [25:37]: It locks you in, so it’s harder to migrate out of it. And we chose this way of being more independent. So it’s easy to throw a water meal away, if you choose to do so, and replace it with something else. And that’s my design, because we prefer working with smaller libraries that are not as heavyweight as frameworks.

Miłosz [26:13]: Maybe something similar is to how public API design matters. I think in Watermill the publisher and subscriber interfaces are quite good examples of how having a simple interface and the complexity hidden behind it can lead to nice API of the whole library. And we have the higher level components built on top of them. So it’s… quite… easy to use and fun to try. Because you don’t need to dig through the documentation to understand how it works. Use those two functions together and you say, oh yeah, okay, this is how this stuff should work.

Robert [27:09]: And I think it’s also cool because it’s giving you a lot of flexibility. Because we are giving you some building blocks in Watermill, but we know that there are a lot of different kind of projects. And it’s just impossible to create a library that will fit everybody’s needs. But if it’s built in the modular way, it can be adopted much easier to totally different use cases. So you can just use some of the components, use some more lower-level components, for some cases, some higher-level components. So to give you some examples, so you have in WaterMill, you have high-level interface for handling events that is handling marshalling for you, but you have also lower-level API that you can use, interact with PubSub more directly. So it just will have message with slice of bytes payload, and that’s it. And it may be a great choice for some low-level processing or data pipelines. It was hard people that they were using it in some strange cases, let’s say.

Miłosz [28:17]: So you can choose what to use, basically.

Robert [28:20]: Yeah, and it’s just making a list of use cases that we can support much, much bigger.

Miłosz [28:28]: And what about testing strategy? That’s also something we use, a unique approach.

Robert [28:34]: Yes, it’s interesting because if you think about the abstraction over PubSub, how it works in Watermill, so basically PubSub is implementation of one interface with two methods. PubSub publish and subscribe. So you can guess that publish method is responsible for handling publishing of your message, and subscribe. Yes, you guessed. It’s for subscribing for those messages that you just published to your topic. And every PubSub implementation that we have basically needs to implement this interface. And at some point, when we’ve been thinking about how to test that and how to implement many of the PubSubs, we found out that actually, okay, if every PubSub should implement this one interface, basically every PubSub should work in a similar way. So your publishing message with publish and subscribing, and it should run back. Obviously, it may sound like a pretty simple thing. At the end, I think we have probably like 30 different use cases, or 30, 50, I don’t remember.

Miłosz [29:41]: You can configure them depending if this PubSub supports them or not. The funny thing here is how you can adjust and adapt many different systems to implement this interface. Probably one unusual thing is the HTTP-based PubSub. So you can just do it over HTTP requests and responses.

Robert [30:03]: It’s basically webhook.

Miłosz [30:05]: Yeah.

Robert [30:07]: We have also PubSub, for example, based on the file system. I mean, io.Reader, io.Writer.

Miłosz [30:14]: So you can do it over files or whatever else you… We should probably build one over email.

Robert [30:25]: Yeah, it would be cool.

Miłosz [30:28]: But this goes back to this design of the API, right? If you choose this nice interface, the rest kind of falls in place automatically. And I see many libraries that don’t care that much about it. And you have these public methods or functions that take many arguments, and you have to check how they work before you start using them. And they could probably use some more high-level APIs that would be easier to use. So if someone is wondering if there’s still a place for some other libraries, I think there’s plenty.

Robert [31:14]: And there’s one cool thing about having this one test suite for every PubSub that we support. This is the fact that if you’re implementing a new PubSub implementation, basically you don’t need to write a test because tests are already there. And to be super exact, we have implemented, for example, Google Cloud PubSub at some point without having access to any production system that was working on Google Cloud PubSub. But we know that, OK, it’s cool because it’s hosted. And we implemented it basically just based on test. And I think a couple months later, we joined a company that was using Google Cloud PubSub. And we think, OK, let’s try if it works with that. And we used this PubSub that we just implemented based on tests. And it just worked on production without, and a big adjust. So I think it was just some context cancelling or some really minor thing, but everything else, it just worked as expected. So it’s a cool thing because it’s ensuring consistency about multiple Babsops. So we can say that we have one standard of how PubSub implementation should work. And when you’re implementing a new one, it’s just much faster to do that, and even if you don’t have access to any system that works on this PubSub.

Miłosz [32:29]: It’s just a nice exercise to try to bring your own PubSub implementation because of those tests. So now you have this nice system that tells you what the code you wrote is correct or not.

Robert [32:45]: It’s a bit like a quiz. You have a quiz and you need to solve multiple challenges to make it working.

Miłosz [32:54]: It’s also quite easy to do it, right? I remember the NATS PubSub we created. We did it over one day on a hackathon with a couple of friends. Yeah, because it’s just one interface. If you can implement it, then you’re done.

Robert [33:10]: So if, for example, you are using some PubSub that we are not supporting in Watermill, well, we are open to accept contribution, obviously. But the most important thing, your tests needs to pass and be stable. If they’re passing and they’re stable, well, we can talk. But we’re also happy to help with that because we know that in some systems, debugging is a bit harder. It’s also useful to know how to use grep and pipe your logs to text files because sometimes… Oh, implementing PubSub may be not that easy when SDK is not best.

Miłosz [33:49]: But it’s doable. Yeah, and you can start with the happy path. Because there are many different configuration options, usually. And once the implementation is successful, you have people asking for more features or changing the default behavior, stuff like this. But the HappyPath scenario is usually quite easy to do. So you can focus on this and later add the configuration.

Robert [34:20]: There’s another thing that I think is also important when you are building a library that people should trust in. And this is some policy of how you are introducing breaking changes. For example, Watermill is since, I think, six years in the major version V1. So in other words, we didn’t make any breaking changes in the core library. We did some in some PubSub implementations just because some SDK was updated or we had some good reason for doing that. But in general we are avoiding that and in the core we didn’t make any breaking change so obviously it has a downsize because we need to even if we’re deprecating some methods we need to keep them backward compatible keep old functions still working but from another side it’s very, easy to update for every people and i remember that i was using some libraries that were doing some breaking changes without bumping a major version. It’s not the best experience.

Miłosz [35:21]: Even with major bumps, people often tend to not migrate at all. Even if it’s just a couple of methods that change, it’s some effort. Especially if you use this library in some part of the system that’s critical, like messaging, or maybe it’s a part that is not used much, so you don’t know if something breaks. Then you are more likely to postpone upgrading. So, yeah, we want to keep the v1 promise.

Robert [35:52]: And important disclaimer, so no breaking changes, it doesn’t mean only that the public interface doesn’t change, because it’s also about keeping behavior as earlier. Because I would say that it’s even worse, because if you are breaking your API in a way that the compiler will say you that, okay, it doesn’t work, it’s fine because you will see the compilation error and you can fix that. But if you’re changing your library behavior, but without breaking interfaces, it’s pretty bad because you may expect some, consistency in how it works. And in general, you don’t want to surprise your library users.

Miłosz [36:35]: And probably the best way to avoid breaking changes is using configuration structs for the top-level methods. I think we had to bump the Kafka PubSub implementation to v2 very early, because we did more positional arguments and changing them is a breaking change always. Or you need to introduce a new constructor, which is annoying.

Robert [37:03]: Yeah, I think we also made the similar thing with SQL PubSub because we needed to add more parameters in some interfaces that we have in that library. PubSub. And by the way, if somebody is surprised, you have SQL PubSub. So yeah, we have SQL PubSub. So if you are in the stage that you cannot effort, let’s say, to have extra infrastructure piece or Kafka, RabbitMQ or whatever, you can just use Postgres or MySQL. We have also benchmarks and we checked how fast it works. And for many systems, it will be more than enough. but it’s just a bit off topic.

Miłosz [37:52]: On this topic, one more thing to add is to keep the V0.1 before V1 so you can experiment with the changes and people are aware this is an experimental version. But at some point, you probably want to release the 1.0 because it builds trust. If someone says your software is always in a beta release, they might not be eager to use it in the project they run. They will keep asking, is it production ready or not? But this buffer, which in our case was two years or something, quite a long time basically. But it was useful to experiment with different approaches, and then once we were ready to finalize the API, we could just froze it, and that’s how it stayed over time. And we kept adding more features, of course, more high-level components, but they all worked with the low-level API.

Robert [39:03]: So the tactic that we often use is just keeping the old functions or old types that we had there, marking them as deprecated, and suggesting to use another one. So a good example is our SecureS component for handling events and commands. And at the beginning, we have some super weird, crazy, complex interface. Now, it’s still there, but it’s deprecated. And we have nicer API using generics now that you should use. But if you are using the old one and you don’t have time to migrate, okay, that’s totally fine. It’s still working as it worked five years ago. So I think it’s pretty cool.

Miłosz [39:43]: And it wasn’t huge efforts to keep it backwards compatible. Most of the time, it’s not. You just need some deprecation commands and maybe keep some old tests in the repository.

Robert [39:56]: I think the biggest challenge is often how to name the new version. Because often you use the good name already and deprecate that and have some good names. I think we didn’t do that, so we didn’t put any suffix like v2 or whatever, but in probably worst case, you can do it. I think we have also this with the protobuf marshaller that we have in Secure S component. So I think the old one was the protobuf and the new one is proto. Not perfect, but probably a bit better than v2 suffix or something like that. Yeah.

Miłosz [40:35]: Naming and choosing a good API is one of the challenges definitely. Another challenge that’s kind of specific to Watermill is we support 12 PubSoaps right now. And most of those systems is quite complex. Like Kafka can be configured in many ways and used in many different scenarios. And we are not able to run all of those systems in production. We use some of them. So yeah, that’s one of the challenges here is to keep up with the community. So you have to keep it in mind that once the library starts growing and future requests are coming and you need to keep up and review them. And if you are not using some Pub-Sub, in our case, in production, it’s sometimes hard to tell if the changes make sense.

Miłosz [41:41]: In general, maintenance is important, because if someone sees a project that has not been updated since a few months, they might be afraid to use it again. Like, is it deprecated? Is it maintained? Because I’ve seen there is no updates. So regular updates are often a good idea.

Robert [42:06]: Yeah, I think it’s often also challenging when your project is kind of feature-complete. Like, okay, I wouldn’t say that WaterMill is feature-complete because we have still many ideas what we can add there. But from the other side, the features that are there are feature-complete. I mean, we can probably just not touch WaterMill for the next 10 years and it will be enough. We maybe just can update other PubSub implementations, but that’s it.

Miłosz [42:33]: It’s stable.

Robert [42:35]: Yes, yes.

Miłosz [42:36]: Yeah, but very often some security updates also make sense or library bumps, version bumps.

Robert [42:44]: Or license changes.

Miłosz [42:46]: Yeah, that also happens. So that’s one of the challenges. So it’s not only challenging to start, but then you also need to maintain it over time.

Miłosz [42:58]: Maybe to change the topic a bit, we should talk about marketing. Which is probably not the first thing you think about when you consider starting an open source project.

Robert [43:11]: Yeah, but as we already mentioned, running open source project library, it’s pretty close to building a product. So I think we’ve been lucky to already build a couple less or more successful products. And I think it also helped us to learn multiple things. But I think one of the most important things is that it doesn’t work. So just creating great product, great library isn’t enough for people to use that. So you need to let other people know that it’s great product, it’s great library, because it doesn’t work in the way that people will find it. No, they won’t. The chance is really, really low.

Miłosz [43:54]: Yeah, and there’s no easy way to track how popular the project is. GitHub stars are one way, which can be a vanity metric. It doesn’t tell you very much.

Robert [44:07]: You can buy them.

Miłosz [44:08]: You can buy them, yeah. But it also gives you some social proof. And in general, if your library is used, people will be more eager to use it as well. I remember once, when we joined the company, at the time when we released the 1.0, I think. I remember we wanted to introduce it internally to other teams. And there was some kind of internal library like this before, but it was kind of limited and didn’t have a nice API. So there was this meeting of senior engineers and this kind of meeting where you have 10 or 15 people that gather to decide a thing in one hour

Robert [45:02]: And it costs a lot of money.

Miłosz [45:04]: Yeah it never ends with a decision it often ends with another planned meeting and I remember we drifted from the main topic and just kept talking about which library we should use and then one of the principal engineers starts reading mail on his phone because he’s bored and he finds Golang Weekly newsletter and water mirrors featured in there and he says out loud guys you are in Golang Weekly and somehow this ended the conversation it was like a proof that oh yeah this library I’m sure it’s worth something Which is funny, because it didn’t mean some company used it on production or whatever. Maybe they did, but basically social proof is something you need to succeed in open source too. That’s why often projects also add some logos of companies that use it to the readme or website. Because then you go like, oh, okay, this company uses it, so it must be a good library. But of course, they don’t tell you that this is used by some obscure testing framework this company uses or something like that.

Robert [46:33]: Or it’s just used in some service that two people are using in this company. But, you know, it’s often like that.

Miłosz [46:41]: But long story short, you need to promote it, whether you like it or not. And the obvious answer is where to do it? Is Hacker News and Reddit? What other places you suggest? Maybe some conferences?

Robert [46:59]: Yeah, so this is the thing that, for example, I like to do a lot. Maybe some of you had the chance to see my presentation on some conferences. I really love to talk about watermelon conferences. Also on every bigger release that we have, we’re trying to write some blog posts, that we can share what’s new there.

Miłosz [47:24]: Also the examples are one way to promote it sometimes we have more complex examples and some blog posts around it, like the one with servers and events which also features Watermill, but it’s not only about Watermill, it’s how to do something very specific in your application.

Robert [47:47]: Yeah, and some of you may ask, all right, but I’m posting that and nobody cares and nobody’s avoiding that. And I think it’s often all about going back to the one of the first points that we have. So maybe it’s something that doesn’t resonate with people, because maybe they don’t understand how to use that, or maybe they don’t have use case for this library. And yeah, I think that this is part of why Water on You is that popular. It’s solving real problems that people have and is the key. Again, if people are not happy about using that, maybe it’s not solving problems that they have. Maybe you should try to find another idea that you can help people with a form. Maybe people don’t understand what this library is doing, what problems it’s solving. Quite to focus on use cases, on emphasizing what problems the library solves. And it will help a lot when you’re posting it to Hacker News or to Reddit.

Miłosz [48:46]: I was browsing GitHub today and I found a library with README and there was a big banner that says share your ideas for your use cases so I think that might be a sign you don’t know why this library exists so if you can come up with some real world examples featuring your library, that’s probably a good sign yeah,

Robert [49:14]: So watch out for looking for problems for solution that you have, because often it may be not the best thing yeah, All right, so we’ve shared a couple things about watermills, what we believed it was successful. I hope that it will be pretty useful if you’re building your own project or maybe you’re looking for some inspiration. If also watermill and eventview application sounds useful for you, but you didn’t have any chance to build such kind of system and you don’t know how to do that, Obviously, we have our examples, we have our documentations, but we also know that just reading examples, reading documentation is not enough for learning. So we already covered it in the previous episode, I think. But if you are looking for something about event-driven applications, you’re pretty lucky, probably, because twice a year we are running our online training about building Go event-driven applications in Go. And actually we’ll be starting in next three weeks. And it’s also a pretty special edition because we have a lot of updates. So it will be V2 of Go event-driven training. Breaking change. Yes, but yeah, it’s exactly breaking.

Miłosz [50:43]: What didn’t happen to Watermill. But sometimes it’s better to do it.

Robert [50:47]: Yeah, yeah. So what we have made, we’ve gathered all the feedback that we had from people from previous editions. and we spent a lot of time to implement them, and we believe that it will be better than it was earlier. This is also the first time when we will be doing it in the cohort way, so we will go with the content over four weeks with a group of other people. So let’s see how this form will work. We believe that it will be much better.

Miłosz [51:23]: And everyone who bought it before will have access to this new one, like we promised. So we get all updates for free.

Robert [51:34]: Promises are important for us, so we’re keeping that. One of the promises is that when you’re buying at this point, it’s the best price that you have all the time. So you can join in-house here, probably more or less, but it will be more expensive. So it’s definitely we encourage to join earlier. And you have also lifetime access for that. So it’s not like you need to finish it with this cohort now. You can do it in your own time. When you have time, you can do it now, in one year, whatever works for you.

Robert [52:09]: But yeah, it’s just available from time to time. It’s not available for the time. So if it sounds cool for you, just Google it. It’s named Go Event Driven. There’s a page about that. And then again, it will be available in three weeks. So stay tuned.

Miłosz [52:26]: And actually, Go Event Driven is also a nice synergy with Watermill. Because we feature Watermill in the training. Because it’s about event-driven applications. And it lets us show how to build event-driven applications from a high-level perspective. But it’s not a training about watermill so we don’t make money on watermill directly but it’s this synergy where we have this incentive to keep improving watermill to make the training even better so I think it’s a quite unique example of how you can take an open source project and connect it to something commercial but also So it’s not about sponsorships, or changing the license to closed source, or maybe even running an enterprise edition. Which often doesn’t work, because some cloud takes your source code and does it cheaper.

Robert [53:30]: And it’s also worth mentioning that we’re also not stripping documentation just to force you to buy the training. I mean, if you already build event-driven applications in other programming languages, Watermill documentation will be totally enough for you. You can go over the documentation, understand how Watermill works, and that’s it. Because some people may think that, oh, you need to buy training to understand Watermill. So it’s separate. And it’s also separate in the other way. So it’s not training about Watermill, it’s about universal patterns that works in every programming language. So we’re using Watermill because it’s making stuff easier, but we’re planning in the future also to create additions in other programming languages. Because again, there’s no training about Watermill. You can replicate it in any programming language and it’s universal knowledge that works now, worked 20 years ago and probably will work in 20 years.

Miłosz [54:28]: Well, we will have even given AI systems.

Robert [54:32]: Maybe AI will learn from our training. And as always, when we’re mentioning WaterMill, we always are sharing a big thank you to all contributors. We mentioned that we have more than 100. It’s hard to say because it’s across almost 20 repositories, and it’s hard to deduplicate and count them, but I think it’s more than 100 people across all repositories. Hard to say the exact number. But yeah, real thank you, because without you, Watermill wouldn’t look like you now, because we’re just two, we’re 100. So it’s really, really helpful.

Miłosz [55:14]: Right. Let’s move on to the Q&A then.

Robert [55:17]: Yeah, so if you have any questions about Watermill, about anything, just leave it on the chat, and we’ll be more than happy to answer them. Yes, I see that there is some comment about focus on me. Yeah, so we’re fighting with cameras to get the good settings, but yeah, I see. Probably it’s pretty good now, but light may be not perfect. Although, yeah, we are trying to improve every episode. Adrian is also asking, when creating a project, do you consider performance at the start of the project? Is it wise to make it run first, then think about performance later? Because performance plus design do go together right. What do you think, Niosh?

Miłosz [56:14]: Yeah, good question.

Robert [56:16]: It depends.

Miłosz [56:19]: So for what I mean, we have benchmarks. and that’s kind of objective way to measure how fast it is. But we never explicitly focused on performance, because most of the time it’s not that critical. Even for the SQL PubSub, which is super slow, we calculated that you can process, I don’t know, a few million messages per day or what was the count?

Robert [56:55]: Per hour probably, I don’t remember.

Miłosz [56:57]: It’s slow but still pretty fast for most use cases and I think we never had anyone complain about performance but it’s a good comment about the design. I guess one example could be if the design didn’t support batch requests, that could be pretty bad. That’s probably something to keep in mind. And that’s probably different in open source projects than some internal libraries. Because if you finalize the API and then it turns out it’s a blocker for some performance, action, then it’s a problem. So, for example, the publish of Watermill Publishers supports multiple messages in one batch. I guess that might be one example how it can help. If all you could do was publishing one by one, that might be a big factor for performance reasons.

Robert [57:58]: Yeah, but I think the most important thing is about thinking about use cases for the projects, because in some libraries it will just doesn’t matter and in some cases it may bother yeah.

Miłosz [58:09]: Very often it won’t matter

Robert [58:11]: Yeah for example if you’re building i know some ai stuff probably doesn’t matter because okay you can optimize every memory allocation order your struct fields to use as least memory as you can but when you just call lm it will be super slow and what you can do Oh.

Miłosz [58:29]: I know. Good example. Very popular in Go ecosystem is doing benchmarks for HTTP router libraries. And each library claims they are the fastest. And they have charts where you see that they are, I don’t know, faster in nanoseconds than the other ones. But the truth is, if you use HTTP, then you communicate over a network and each request takes, I don’t know, 50 milliseconds or 100 milliseconds roundtrip. Then this micro-optimization is not that important really.

Robert [59:14]: And probably you’re using JSON for communication. So come on, if you’re using JSON for your API and you’re talking about optimization, probably it’s not the issue.

Miłosz [59:25]: Someone will now say they have a very specific use case for this and they need to process billions of messages per second. And yeah, maybe it’s important. But for many projects, it won’t be important. Yeah, I guess advice is keep it in mind in terms of API, but probably don’t focus on it as the priority for the first release.

Robert [59:52]: So in other words, if you can optimize performance later without breaking changes, maybe it can wait for a while. Because it’s all about not introducing breaking changes. Like in Watermill, we can probably optimize a couple of things in terms of performance, but it will not make any breaking change, basically.

Miłosz [1:00:13]: Unless maybe you want to create a similar library, like one that exists already but that is faster or makes zero allocations or something like this, then it might be useful.

Robert [1:00:31]: All right, so the next question, when creating a wrapper around, say, GCP pops, what separates from lower-level implementation versus higher-level implementation? What typical feature should be included?

Miłosz [1:00:58]: Higher level implementation. Do I get it right that it’s about higher level GCP pubsub features? So basically, the Watermill wrapper around GCP pubsub is this… You implement this low-level interface only. So you implement the Publisher and Subscriber, and then all other components of Watermill work with those two interfaces. So you get the rest working out of the box. I’m not sure if this is the question, but the typical features are basically publish and subscribe. That’s the short story, but there are often very a PubSub-specific configuration you need to include. Okay, probably one very often feature that appears in most is consumer groups. Some kind. Or in the case of GCP PubSub, this is subscriptions. So you need a way to configure those. I’m not sure if this is the answer. So if you have a Yeah, if you have a follow-up question, you can let us know if this is what you meant.

Robert [1:02:24]: Yes, Adrian mentioned, true, at the end of the day, which HTTP package we use, it may not be that important. Yeah, it’s also worth mentioning that, you know, in this benchmark wars of HTTP routers or whatever, often the routers that are winning those wars are not fully compatible with HTTP protocol. It’s also worth mentioning. It’s cheating a bit, but for specific use cases, they are faster in it. But again, we are mentioning it very often that you can optimize allocations, struct, order, but just making one more SQL query will just kill all your optimizations. Again, there are cases when you’re not doing SQL queries, you’re just doing some CPU intensive operations.

Robert [1:03:20]: In this case, it totally makes sense to optimize. And if you’re a big library, just think about value kpi in this case and you can optimize later. And I think it’s also useful to if the library is when performance is important, just share benchmarks with people because, if they care about performance, it may be good. It may be just using the Go benchmark that is built-in. For these cases, it may work. For example, in Watermill we have special benchmark, because we are not measuring it in go-run time. We are more interested in how many messages we can publish or subscribe in one second.

Miłosz [1:04:01]: Yeah, I agree that the HTTP package is probably one of the least interesting things about the project I want to decide. I mean, I want a nice API and probably middleware support. But other than that, I want it to be like Watermill, so I can replace it with anything else if I want.

Robert [1:04:24]: If you feel that maybe you need to optimize your HTTP communication, maybe use gRPC, for example, that will be faster if you really need. Again, optimization when you’re using JSON for communication is a bit… Yeah.

Miłosz [1:04:42]: Yeah, I wouldn’t spend too much time deciding on the library, basically. Just pick something you know or you like, and that’s it.

Robert [1:04:52]: Right, so it seems that this is the last question so far. So let’s wait for a couple more seconds for questions.

Miłosz [1:05:06]: So probably a good time to remind you about the subscribe button or the five star writing button on

Robert [1:05:15]: Podcast applications or even better joining our newsletter so you will be sure that will not miss an episode notification so you can have notification before life one week before or summary and you know when you’re clicking subscribe it’s also cool but you are never sure that your notification will be there with our newsletter you are always sure plus you also know what all other updates that are doing uh what else so yeah five star reviews uh share with your friends so it would be also cool when this knowledge can go wider and if.

Miłosz [1:05:55]: You have any open source libraries share with us in the comments so we can take a look

Robert [1:06:01]: Or if you’re using watermill production also please let us know in comments it would be cool to know if it’s useful for you because again we don’t know basically we just know from some people that we’ve seen on conferences but, we don’t have any telemetry or whatever so we don’t know if you are using watermel.

Miłosz [1:06:22]: Right i think no more questions thank you everyone for joining us today

Robert [1:06:27]: Yeah thank you Miłosz for today. So see you in two weeks. In two weeks we are discussing what’s the difference between sync and async architecture. When we recommend to use that, we’ll also go deeper into the topic what’s the difference between PubSub and Message Queue.

Miłosz [1:06:46]: So wehave a very related topic in two weeks.

Robert [1:06:50]: Thank you for being with us and see you in two weeks.

Miłosz [1:06:52]: Bye-bye.

Let's stay in touch

Never miss a new episode: get notified directly, without relying on unpredictable social media algorithms.

You'll know when we're live, get updates on new episodes, and receive exclusive content.

Subscribe via RSS
Tags:
Last update:
  • May 14, 2025

Related content

Learning Software Skills fast: what worked for us best in the last 15 years

Learning Software Skills fast: what worked for us best in the last 15 years

Quick takeaways Focus on applying what you learn - reading books or watching videos isn’t enough without practice Build real projects that challenge you - trivial examples don’t expose you to the hidden complexities you’ll face in actual work Expect and embrace frustration - feeling stuck often means you’re learning something valuable Learn timeless concepts over framework-specific details - aim for universal software skills like modularization Mix theory with practice in small chunks - read a bit and code a bit, rather than consuming large amounts of content at once Introduction In this episode, we discuss how to learn effectively as a software engineer.

Is Clean Architecture Overengineering?

Is Clean Architecture Overengineering?

Quick takeaways Clean Architecture is most beneficial for complex projects with larger teams - for small teams or simple projects, it can become overengineering. Separation of concerns is the core benefit - keeping domain logic separate from implementation details makes code more maintainable. It’s easy to go too far - using too many interfaces or too many layers without a clear reason creates unnecessary complexity.