In our last blog post we talked about the upcoming feature smart data transfer, in pursuit of moving applications around a cluster and getting them up and running faster – this is now released and available in our SDK! One of our customer has seen scale up time reduced from 30 minutes to 5.
We have some deeper blog posts in the pipeline to go into some of the technical detail around how we’ve done this, but the result for you is that you’ll see far more responsive scale up behaviour in your application, especially where your deployment size is large (e.g. if you’re running a Hadean Simulate simulation with GigaBytes of large assets bundled). This speedup gets even better the next time you deploy too – only changed data will be transferred between machines.
There’s still some work remaining, specifically to make the smart transfer apply when you’re first sending your application from your local machine to the cluster – currently it only kicks in when using spawn() within a running application. That will land within the next few releases and you’ll just magically see things get faster across the board.
Our logging work is making progress too – advancing past just using vector.dev as our backend (which is approaching release), the architecture is being put in place to have overridable log backends, as well as an easy to use development logging experience. Want to use datadog, Loki, Elasticsearch or something completely custom? No need to wait for Hadean, it’s all doable in userspace!