(smooth guitar music)
- Thank you so much for coming.
So yeah, a bit about me.
My name's Jade, I'm a Software Engineer from London.
I got really interested in optimizing code
at my second dev job,
and that's been an ongoing interest since then.
This is me back at home with my dog, Cubby.
She is a Shiba Inu.
So I'm gonna start out with a bit of background
for this workshop,
and then we'll look at how to find slow pages,
and then we're going to investigate why they're slow.
And then at this point as well, we'll have the exercises.
So this workshop is called,
How to use Flamegraphs to Find Performance Problems.
But first up a bit of background,
let's think about why we actually care about the performance
of our applications.
So firstly, a website that loads more quickly
has a better user experience.
You don't want to just be waiting around forever
for something to load.
The impact of this for customer-facing pages
is that slower load times tend to decrease conversion rate,
at the point where you've got issues of conversion.
The easiest way to get a read on this
is if you've got pages that are slow in your sales funnel,
or signup process,
to measure the, sort of, dropout rate on those.
Cool, so the next thing that performance is relevant for
is server costs.
So if you have slow code in more heavily used areas
of your site, then naturally,
you're gonna have higher server costs.
So if you can make that code faster
then the impact will be to sort of lower those costs.
As well as this,
if you have performance problems in your system,
then eventually under peak traffic, peak load,
you're going to have errors.
I'm gonna talk more about this
in the second half of this workshop.
At this point,
I just want you to sort of bear this in mind, basically.
So this is my dog in profile.
Let's talk about why we need to profile applications.
So we need to get a realistic view
of what is causing slowness.
So for speed, intuition can sort of fail.
You might have had the experience that I have had,
of reading some code and thinking,
oh, I bet that's gonna be really slow,
and then it's actually not at all.
We also might not, for those reasons,
catch slowness in code review,
and also because products can slow down
as they get more successful,
you are not necessarily gonna see slow code
being slow immediately,
like the point at which it becomes slow
is gonna be ahead of that merge really,
after that merge, sorry.
So you could ship something that works perfectly well
with a couple thousand rows in your database,
but it could start slowing to a crawl
when that scales to like, 100,000 or a million rows.
So we want to get data on what part of the page load
makes it slow.
So I talked a bit about why we need to profile applications.
Let's talk more about tools.
So many of you will know this.
We use something called a profiler to collect profiles
of all code that's run on your server,
when loading a page or calling an endpoint.
And what a profiler does is it tells the computer to like,
stop what it's doing for just a second
and say what it's up to.
So you might say like, oh, what are you up to you now?
And it says, okay,
so I'm in the middle of initialize method,
or, oh, I'm making a database query right now.
So this happens over time throughout the operation.
It keeps asking for the like, stack frames,
the details of what's going on right now.
And then these are collected up
into something called a profile,
which is what we use flamegraph tools to visualize.
So some common Ruby profilers include Rbspy by Julia Evens,
ruby-prof, StackProf and rack-mini-profiler,
which is what we're using throughout this workshop.
So we've covered the why, let's take a look at the how,
so this is what you're here for, essentially. (laughs)
The first thing is that for any performance work like this,
I would say, realistically,
you're going to have limited time to do this,
so you want to focus your efforts
on where they're really going to be useful.
You can follow an elimination process
to find causes of slow code,
and then any code change you make locally,
you measure the result of that change on your machine
to see if you sped things up,
made no impact, or sometimes to be frank,
slowed things down,
of which, please don't merge that. (laughs)
So you follow an elimination process.
I've gone backwards. (laughs)
And then once you've made that change,
you raise a PR and you wanna test the impact
of that change in production,
in case it's like different in a bad way.
So yeah, prioritizing areas to speed up.
To prioritize areas to speed up,
you want to find pages that are both heavily used and slow.
So say you skip ahead a bit and you just imagine
for a second that you've already raised a PR
speeding up a page.
You want to be able to say on that pull request,
why, like not just what you did,
but also why you've done that.
This might not be super-clear to people.
Like if it hasn't always been a priority
to have pages load quickly, which it should be,
but now it is,
then that won't be necessarily clear to people.
So I do want you to to be like, very confident in explaining
why you're improving something, not just like,
oh, I thought it'd be cool. (laughs)
So yeah, the reason I say this is,
is as I kind of hinted at,
you've probably got a lot of other work to do on your team,
so you want to be looking for something
that is not too optimized, that it's not only heavily used,
but it's also slower than it could be,
so it takes up too much server time.
A bit about like the impact of speeding something up,
if you, say you find that like one internal report
that's generated once a month, by hand,
by one person, using your software,
if that takes 100 seconds to run,
I wouldn't start by speeding that up,
unless that becomes, somehow becomes a priority.
On the other hand,
if your homepage or your payments pages
are taking a 100 seconds, please speed them up, definitely.
And also you wanna think about
the context of the business that you're in.
So I've put some examples here on the fourth bullet point
of pages that are generally key.
I've seen this in a couple of pages, places.
If these are slower, these pages,
it's gonna become a problem,
like actually a business problem, not just this is uncool,
we should make it faster,
but like it's a problem for your work.
And you can also think about,
I suppose I've said customer base,
but I really mean the user base.
So can you rely on one type of user
continuing to use the site, but not another?
So the example here I'd give was,
if you've got an internal admin application, it sounds bad,
but internal admin users can tolerate
slightly slower load times than customers
who haven't signed up yet,
because they can go switch to another tab
and just look at your competition
if they get bored waiting for your site to load.
So if that's happening in your signup or conversion process,
that's pretty bad.
And this would all inform your prioritization.
Most of this is very dependent
on the business that you're in,
but what is going to hold
is how you find those pages
that are taking up the most time.
So let's have a look at that now.
So you want to put together an ordered list
of the kind of worst offenders,
and to do this, you want to look at real results
for real users.
You can do that with a tool like New Relic.
So you go in New Relic, or whatever APM you're using,
and look at web transactions taking the most time.
The reason I'm saying look at web transactions
and not non-web transactions, is that if you've said,
we are going to process something in a background job,
you've already decided you don't need that
to complete immediately,
whereas the stuff in web transactions
you have, effectively do.
So I'm not gonna go into New Relic right now,
instead I've picked out some examples of lists
of most time-consuming endpoints.
So this is like a factor of how often these get called
and also the amount, when they do get called,
how much time they take each.
So I just wanna make a point that the endpoints
are ranked relative to each other.
So the cost is action from like the middle diagram,
for example, could be faster than the fourth cost list
on the left or right.
You may find after putting together your rank list,
that when you actually look at the average response time,
it's not that slow, it's just called a lot, for example.
So then you can also think about what will have the highest
impact for your product and start there from that list.
And you might wanna think about this
from an architectural perspective, if that is relevant.
So if you've got like a bunch of services
and one is the outstream application,
or the one where sales happen,
you might think about that as well.
So you've got your list.
You will now wanna work your way down that list
and work out where the time is being spent.
If you've got New Relic, this is quite straightforward.
I think Scout APM is also good
and it's free for open-source projects.
So if you've got a public repo on GitHub
I think it's free.
I would double-check that, but I think last time I checked.
So for when those times when in the frontend,
you're all probably using different frameworks,
so I'm not gonna cover this.
However, you can be in a situation
where you spot that your backend is very fast,
but your frontend is slow, and that is just as important.
This would be a very long workshop
if I tried to cover everyone.
So instead I've put together a bunch of resources,
which you can dig into,
for your particular framework
and I'll share these at the end.
You can also use flamegraphs in backend languages
other than Ruby, they didn't originate in Ruby,
so I'm gonna share resources for those
if you want to look at that a bit more.
I'm also just telling you everything
I'm not gonna do in this workshop.
So I'm also not going to dig into database performance.
Again, you could be using Mongo, you could be using SQL,
could be using multiple options.
For the purposes of this workshop
we're going to skip over it entirely.
So yeah, let's look at how we can find
where in the backend is responsible for slowness.
We are going to use rack-mini-profiler to do this.
So yeah, the first exercise
is to check it's installed and working okay.
So yeah, if you've done the setup, you're ready to go.
If you haven't, it might be a good idea to see
somebody who has,
this is really about recognition at this point,
so kind of like,
hopefully you'll have access to something that's running.
So yeah, I want you to start up your server
and go to local host 3000 in the browser.
So once you've done that,
can anyone see anything extra on the page
you wouldn't expect to see?
Has anyone got it running?
Okay, great.
Can you see anything you wouldn't, oh sure, go ahead.
(man faintly speaking)
Oh yeah, sure.
Yeah, sure.
So they're on, hang on.
Oh, that's not.
It's this, can you see that okay?
Has anyone got it running?
Okay, great.
Can you see anything on the page
you would not expect to see?
(man faintly speaking)
Yep.
Yeah, great.
So you should see a little badge at the top left
of your browser with some boxes,
each reading out a millisecond value.
I'm gonna show you it,
so if you haven't got things set up, don't worry.
This badge, so if you run your app
with rack-mini-profiler switched on,
you're gonna see a badge in the top left.
When you click on it,
it shows you where the time is being spent on your page
and what database stuff is happening.
So this is actually,
I said, I wasn't gonna talk about databases,
but I just wanna make a point that
though this table says SQL,
it actually works for all active record queries.
So this diagram is an example of it
showing Mongoid queries.
It just says it's SQL still.
But yeah, this is a different topic.
Let's talk more about the flamegraphs tool.
Yep, so rack-mini-profiler has a very good flamegraph tool,
which you can use
by adding the query parameter, dot pp equals flamegraph,
question mark, sorry, pp equals flamegraph
to your query string.
And it gives you a flamegraph of what happens
when you load your page.
So the way it works is in the same way
I talked about profilers working,
every millisecond or so it sends a signal
to your Ruby process to interrupt it,
and then it records the current stack,
and then it squishes them altogether,
and then it visualizes that.
So the height of the flame is how deep we are in the stack,
and at the bottom,
you're gonna see stuff like Ruby or Puma,
like the very base stuff.
The X axis on this flamegraph represents time.
And you can also click on any parts of the graph
to see what line of code has been reached at that point
and all the code that it calls.
So, yeah, next up,
we're gonna look at making rack-mini-profiler
show a flamegraph for your page.
So if you go to this address and notice the query parameter
on the end.
Let me know if anyone can, if you can all see it,
and what it looks like.
Nelson, any luck?
(man faintly speaking)
(laughs) Okay, good, it's working.
So, yeah, that's cool.
You're gonna see a pretty graph, like I showed you before.
It's cool, but it's not super-obvious
what you can do from there,
so let's talk about how we can identify
the patterns in a flamegraph.
So if you've got a big horizontal section,
that's the same color,
which is taking up a lot of space,
that indicates a lot of time spent in one method.
And a pattern like hedgehog spikes,
I'm assuming you all know what a hedgehog is,
it's kind of a UK thing. (laughs)
If you can see a pattern like hedgehog spikes
that are all the same color that indicates a loop,
then if you've got a random peak in a different color,
that is just changing color from everything below it,
but indicates calling out to a dependency,
and with that one, those can add up, so yeah.
Finally, if you see a big chunk cut out of a flamegraph
with a flat line at the bottom edge,
that tells you that garbage collection is happening.
We're about to see this on the next slide.
So yeah, this is an example of a big horizontal section.
And I've kind of like outlined it in a rectangle at the top.
This is from what was formerly the third slowest controller
action in one of the lists that I showed you earlier.
This indicates we're spending a lot of time in one method,
and if you see something like this,
you want to look at why that is
and if it's actually necessary,
because that's quite a lot of time,
it's taking up most of the horizontal space.
You can also, as I said, you can see horizontal, sorry,
you can see garbage collection,
which is the light green line at the bottom
where literally nothing else is happening in the center.
That's garbage collection.
So that's useful to notice as well.
So yeah, so it might be a good idea
to get mics at this point, maybe,
if that's okay?
Could we get some mics for people?
Cool.
So yeah, for the next exercise, if you go to this address,
I'd like you to try and figure out what kind of pattern
you can see here.
And when you've got an idea,
if you wanna put up your hand
and we can get a mic round to you.
Great, can we get a mic up here at the front please?
Is that okay?
Yeah.
Oh, cool.
- [Female Audience Member] I was just seeing
like, a big horizontal.
- Yes, absolutely, you got it, sweet.
(laughs) Thanks, Catherine.
So, yeah, let's try another.
So if you go to /animals with the query parameter,
and then see what pattern you can see here.
And then again, if you wanna put your hand up
when you have it.
- [Female Audience Member] Seeing a lot of
garbage collection with the chunks cutout.
- Oh, okay, cool.
Nice, yeah.
That's really cool.
If you refresh it a couple of times,
you might see that change as well.
You should also be seeing the hedgehog spikes,
so showing the loop as well, but yeah, that's really good.
So yeah, with this, you kind of wanna think about,
maybe you don't need a loop,
maybe there's another way you can do this.
Is that a more efficient way, for example?
Okay, yeah, so for the last example in this section,
I want you to just go to the root root as I've given here.
So just forward slash nothing
and take a look at what you can see.
And can you notice any patterns here?
- [Female Audience Member] There's a lot of cutting off
and a lot of gaps,
so I'm assuming that's all garbage collection.
- Yep.
Okay, cool.
And there might also be calling out
to dependency somewhere there,
but I thought it might not be super-prominent,
so I wanted to show you an example as well.
I do wanna state something as well for this,
the colors of the flamegraph are not going to be the same
every single time you load the page.
That doesn't really mean anything.
It's just, so don't kind of take any meaning from that.
So, yeah, so this is an example
of calling out to a dependency.
It's not super-clear here, so zoomed in, can you see it?
There it is, okay.
So this is calling out to the New Relic gem.
That is just literally one time, not really an issue.
If you see it a lot, look at it,
if not, probably fine.
You can also click on the flamegraph
to see the stack trace from an individual frame.
So you can see like all the code being called.
Another thing you get is a table
showing you the percentage of time spent in each gem.
And if there's like too much, then look at it.
Okay, so I wanted to talk a bit about investigating further.
The demo app I've provided you with is not really slow,
it's just, I found it a good example.
And you can also take a look at contributing to it
'cause it's an open-source project,
that's not necessary for this workshop.
So yeah, this is an example from work of a show action.
So this was the third slowest controller action
in the entire main application.
From looking at the flamegraph,
I saw one of those big, long chunks,
So that's a lot of time spent in one method,
and I found it.
So this is in the RFQ class
and the questionnaire method calls out to
another questionnaire method on another object, vertical.
So this is really just a worked example.
What I did next was I stuck a binding in
to the verticle's questionnaire method.
And then from that,
I noticed that the values passed in for the answers,
and the two merged parameters in this lower diagram here,
were identical every time,
which results in an identical result.
And that meant that most of the time
spent calling out to, just calling out from there
was completely unnecessary.
And I also counted and the method was called 176 times
from the RFQ questionnaire method.
So that's like, you need to do that once
for the first result,
which is the same every single time,
you don't need those like 175 or so extra method calls.
So yeah, since I knew that call
was returning the same result every time,
that meant I could memorize the result,
the first time rfq.questionnaire was called.
So yeah, this is what the flamegraph looked like before,
big long chunk, not what you want,
and here's what it looks like after,
so just go back and forth a little bit.
It is better.
So we can see the long flat section is no longer there,
the big horizontal chunk, not there.
So we've eliminated those unnecessary method calls.
All right, so that's a worked example.
Imagine you've got this far locally,
you don't want that to go to waste.
I want you to think about how you get that into production.
So first up, you're definitely
going to be raising a fear, sure,
but you can't just put,
it is faster, in the PR description, can you?
So you want to show locally that the change you've made
has improved the speed of the page.
So you wanna test that change locally.
You can use Apache bench,
I've put an example of how to run it at the top.
Talk to me afterwards if you're having trouble with it,
it does, it's a little bit weird.
So yeah, you might need to grab the cookie.
You'll need to make a request.
You can do this by essentially making the request
in your browser and copying the entire thing out.
So I'll give an example of how you do this in Firefox.
So you would go in the network tab and then you right-click
on the request, and then you click Copy Request Headers,
and then you get
everything you need with the cookie in there.
So the results here were from continuing to make
the same requests that the page makes
for 30 seconds continuously,
and seeing the average time per request.
This is called load testing.
You can measure your results before and after your change,
locally, and present them in your pull request.
And you could also present the flamegraphs
and describe the changes evident on these.
Cool, so, yeah, welcome back, everyone.
Hope you had a good break.
And it's the last day of the conference,
so there are gonna be a lot of pictures of my dog
on the second half of the slides.
So yeah, so for the second half, first off,
I'm gonna talk about adapting your approach,
what to do beyond getting your PR into production.
And then we're gonna really kind of,
look in detail at flamegraphs.
Bless you.
So we talked about how to find slow code
and get a PR making an improvement into production.
And we talked earlier about the process
for convincing your team that it's useful to do this work.
So depending on what stage your product is in,
that you work on,
you'll need to adapt your approach a little bit
in order to think about making it fast.
So in my personal experience,
there are three stages of dealing with performance
within an organization.
So I'm gonna describe these stages
and the slightly different attacks, you might,
sorry, tacts not attacks. (laughs)
tacts you'll want to take in each situation.
So you can recognize what stage you're in, probably,
and then adapt accordingly.
Briefly, these are, the ones that I'm gonna cover,
are early stage, rapid growth stage,
and the point at which growth has stabilized.
So,
so,
my slides aren't quite right.
Sorry about this.
Yeah, so I think I can remember it.
In a early stage product,
you're probably finding product market fit.
Frankly, I think it's unlikely
that you will have slow pages
unless you've got some very computation-heavy operations.
The idea here is that you want to hold the advantage
that you have at this point.
So, in a talk that my old boss,
the CTO of Carwow did about really, really scaling
a startup from very, very small to a larger business,
He said that, "If you can establish performance targets
early on, then that's a really good thing to do."
So I think this makes sense
when you don't have any slow pages,
because then you can decide what you want your goals to be.
Then if things start to push out of control,
you can then start to address that,
alongside building out new things.
Next up, rapid growth stage of the business.
So in this stage, you may know how many unique users
to expect at a given time,
however, you might not.
This can be a very exciting time to work on a product
because business is growing
and you also got to work on scaling challenges.
However, there is a downside.
So to talk about this,
I'm gonna talk a little bit about retail.
So in retail, some customers are going to buy,
pretty much no matter what, but almost,
so if they walk into a shop
and they know what they're looking for, and you have it,
they will buy it.
So if for example,
if it's raining and someone needs an umbrella,
this happens a lot in the UK,
then if they're in your shop,
and you have them in stock,
they're pretty much guaranteed to buy that.
But you can make a mistake and end up losing the sales
to say, like, you refuse to tell them where they are,
you don't have any, they're like,
everyone else has taken them ahead of them,
then you're gonna lose that sale.
So it's kind of the same thing as this.
With too much demand, you might not be ready to scale
to that volume of traffic quickly enough,
or even serve that volume of traffic, full stop.
So situations this can happen in
are, you could run a TV ad
and suddenly you got a lot of users visiting at once,
or it could happen around a special occasion
that drives users to your business.
So, I mean, Christmas could be relevant.
It could be some other special occasion.
It really depends on the business.
It could be in-house marketing.
So you send an email to lots of potential customers
and they all hit the site at once.
Or you could be really lucky
and it could be like word-of-mouth
or like a buzz is building around your product,
and so you get a lot of people come into your site.
The problem here is that the,
the problem here is that if you have spent money
on trying to acquire users and then you can't serve them,
then you are effectively losing money that you can't recoup
'cause they're not gonna buy your product.
So at this point,
you wanna start thinking about site reliability.
And you also wanna start thinking about
making sure things are performing
under like the highest traffic that you get.
So yeah, this is actually my friend's dog, Wilson,
and he's in front of a honeycomb dashboard,
so I thought this was relevant. (laughs)
So yeah, this is a talk in itself,
and that is not this workshop.
And I have actually reigned myself in a bit.
I do literally have a half hour talk on this
and I'm not gonna do it,
but these are the key points about site reliability.
But yeah, if you're interested,
please talk to me afterwards because I have got
like a lot of opinions about this.
So your first priority with site reliability
is keeping the site up, or your service up,
and this is known as availability.
Your next priority is preventing slowdowns.
And in extreme cases, if it's hundreds of seconds,
if your timeouts actually allow this,
then that's known as brownout.
So this is where your site has become so slow
that it actually appears to be down.
With this, I would say people have limited patience,
and they do expect sites to load
in a reasonable amount of time.
After that they will leave
and you won't get the gain in revenue
from them using your product.
So to work around this,
engineers who either specialize in SRE or rotate onto it,
depending, they will see a mixture of proactive
and reactive work.
So proactive work is the kind of thing
we're talking about in this workshop,
so it's performance optimization,
and it might involve planning
how you scale your site as well.
And then reactive work is the point at which it's too late,
you've already got a problem and you need to deal with it.
So that might be a major incident.
So what this means is that a major part of your product
is failing in a way that affects business.
So like the homepage isn't working,
payments isn't working for all users, that sort of thing.
That's obviously bad and you don't want that to be happening
at your work for a very long time.
So that's the kind of reactive work.
There's also triage, which is another aspect,
which essentially handing out bugs
to the teams that can fix them.
Another thing is you can have,
put in place performance targets,
or service level objectives,
which are those sort of performance targets
that I discussed before.
So those are either, okay,
so basically like these are targets that you agree with
your product manager, or other stakeholders,
for how quick you want your product to be.
And essentially like these,
you kind of agree these,
and you decide where you wanna be from where you start at.
And then if you aren't meeting these,
then that kind of gives an importance to prioritizing work,
to deal with that.
So, for example, I mean, this is, this is an example.
So don't take what I'm saying is like something
that you need to make an alert of.
But for example, you might say,
and you wanna be talking in averages.
You might say, if the home page, if it takes,
we want it to take one second
or we want it to take three seconds, your decision.
And then if that's for all users on average,
taking more than that for a distinct amount of time,
then you want to alert one of your engineers
to take a look at it.
That's it, I'm not gonna do a half hour talk on this.
That's it for now on site reliability.
So yeah, I talked about three stages,
when growth has stabilized,
this is another stage.
So when growth has stabilized,
there could be a really nice reason for this,
it's super-unlikely,
but you might have gained the entire market
for your product.
I do find that very unlikely.
Otherwise you have some kind of competition.
So for an idea of how quick you want to be,
I would recommend you look at your competitors product.
So Jonathan Nightingale talks about this idea
of getting a better understanding
of the business that you're in.
And you can do this from a performance standpoint as well.
Incidentally, he wrote a book with his wife, Melissa,
and it was recommended to me,
and I'd recommend all engineers read it.
I've put the front cover up on the slide.
So yeah.
So whatever type of product your work is in,
what you would do here is you look at your competition
and you get an idea of how fast their site is
by visiting the homepage, going through a sales funnel,
and then obviously if the final step is paying,
you don't actually have to do this, like it's not necessary.
But then this gives you a good idea
of what you want to be aiming for.
You don't need to be exceptionally fast to start with,
you just need to be a bit faster than your competition,
because it's possible that your competition is slow enough
that people get bored and start looking for other options,
and maybe that can be your product.
This is also a good point in which it might help
to think about bringing down your server costs,
but I would talk to people in your team
and see if that's a priority before you do that.
And that you can do from this kind of work.
So yeah, we're pushing on.
So it talks about how you raise a PR
and I'm gonna assume you've gone and done this.
So I'm going to talk a little bit about the concept
of marginal gains, staying in the lead,
which I've already sort of talked about,
and bringing your team along with you.
So something I've observed is that with work like this,
you can make an impact doing optimization work on your own,
but after a few PRs, you're going to find
that there's only so much impact you can make on things.
So as we discussed earlier,
we're kind of assuming there's only so much time
you have to spare for this,
because there's other work that you have to do.
So you all time limited on this.
So in the long run, I would say,
you're gonna have more of an impact
if you can get other people you work with involved.
So yeah, marginal gains, just briefly,
concept taken from sport science.
The idea is that when you want to speed up,
rather than trying to make huge, massive changes
with respect to speed,
you can make little changes, lots of them, and those add up.
This ties into working with your team
and bringing your team along with you.
So, say for example, you have hundreds of slow pages,
it would be super inappropriate for one engineer
to try and fix all of them.
So if you can get other people involved
and try and make little improvements for each one of those,
then that can add up to a massive improvement over time.
Okay, so moving on to the flamegraphs detail section.
So in this section of the workshop,
we're gonna learn more about flamegraphs.
We're gonna look at how they're invented,
and some alternatives for visualizing profiles.
Then finally, we'll do an exercise
where you can get into profiling your own apps.
Cool.
So why do we need flamegraphs?
So to explain why we need flamegraphs,
I'm going to show me an example
of just one of those stack traces.
So you've seen this one before.
If you take a look at this,
that's just lines and lines of code.
This is for one point in time only.
So if you were going to try and get information
for an entire request, just for one,
you would have to get every single stack trace
for every point, like for the points in time
for that request,
and try and read them yourself
and then figure out from that, manually,
what's going on and where time is being spent.
And I don't know about you,
but that's pretty much impossible for most people.
So this is actually the reason
that flamegraphs were invented.
They were invented by Brendan Gregg,
who I think is at Netflix at the moment.
So in an article in 2016, announcing flamegraphs,
he wrote, "The output of profilers can be verbose,
making it both laborious to study and comprehend."
He then explains how flamegraphs can be used
for much faster comprehension.
If you wanna read the paper, that's the link to it.
Okay.
So we've established why we need flamegraphs,
and how they were invented,
let's look at other ways that profiles can be visualized.
So first up, what we call flamegraphs
in rack-mini-profiler,
and in most writing about Ruby performance,
are not actually originally what were called flamegraphs.
So earlier I mentioned how the X axis of a flamegraph
represents the time,
but actually when he coined the term,
Brendan Gregg was calling a flamegraph,
a visualization where profiles are sorted alphabetically.
So that's actually the first alternative way
of visualizing profiles, flamegraphs.
So yeah, what I've been calling a flamegraph
was first called a flamechart.
I've kept it this way in this workshop
because this is what they are consistently called in Ruby,
and I just thought it'd be very confusing otherwise.
However, if you look outside to other languages
and Google for it,
you might notice this term instead
for visualizations of profiles sorted by time.
Sunburst charts take the same information
as profile sorted alphabetically,
and they wrap it around in a circular graph.
They do look very nice,
I'm not sure they're super-easy
to get information out of though.
The last type I'm gonna cover is Icicle charts.
So if you think of a flamegraph as starting
at the bottom edge of your graph like that,
an Icicle chart flips that,
so the instead, you've got the information hanging off
the top edge of the graph.
Which format you find easier to read
is quite a personal thing.
And interestingly, we are now going to see
that the new visualization, sorry,
the new visualizer for flamegraphs in rack-mini-profiler
it uses an Icicle layout.
So,
the eagle eyed among you may have noticed
in the setup instructions that I locked rack-mini-profiler
to a specific version number.
This is because it has started using a different renderer,
Speedscope, for visualizing flamegraphs in the browser.
So Speedscope was designed by Jamie Wong
to be a language agnostic renderer
that's more performant than other options,
more performant, that seems weird.
Like the flamegraphs we looked at earlier
loaded pretty quickly, right?
So like why did rack-mini-profiler need a new renderer?
Right, so for slow pages, so for example,
I ran a profile for a page that took 10 seconds to load
when I started,
I did this before the new renderer was introduced
and the browser tab would crash quite a lot.
I wanted to introduce you to redo flamegraphs.
I find it easiest to get the whole picture
if you can see the whole flamegraph in one thing,
but realistically you could be looking
at some pretty large profiles,
and there's every chance your browser tab could crash
using the old renderer.
So I want to introduce you
to using Speedscope instead as well.
Right, so what I want you to do,
this might be difficult on the wifi to be fair.
What I want you to do is to stop your rail server,
check out the branch speedscope-renderer,
and bundle install again.
Once that's completed,
I want you to start the server up again
and then navigate to localhost 3000
with the query parameter for flamegraphs at the end.
And then tell me what you notice.
- [Male Audience Member] I guess what it looks like for me
is there's, most of the screen is like a few
pastel bars and then the left
is like this really smooshed area.
- Yep.
- [Male Audience Member] That's kind of hard to read,
but that's what - Cool
- it looks like for me. - Nice, thank you.
So yeah, a different view entirely.
The flamegraph is upside down.
You can zoom using the plus or minus keys.
You can't zoom out and see the entire graph in one go.
You don't have the percentage of time spent
in each gem in the layout as well.
Don't worry if you've not got it working,
like we'll see some examples as well.
So yeah, so rack-mini-profiler started using Speedscope
in December, 2020,
and there are a few differences between Speedscope
and the old renderer,
which was Sam Saffron's flamegraph project.
So firstly Speedscope has been designed for performance.
Like I said, we haven't seen an example of this
in the exercises,
but for pages where there's a lot of information,
just a lot, the old renderer can crash the browser.
I've had success, if you can call it that,
shutting down everything, opening one tab,
restarting my computer and just being persistent.
But I don't think you really want to do this
and it can waste a lot of time
when you're already time limited.
So Speedscope will render flamegraphs
for these kinds of profiles without any problem.
Yeah, you'll notice the graph is upside down,
so that's the Icicle layout.
I do think it would be cool to support both
the Icicle and the flamegraph layouts.
And that's something I would personally like to look into.
You can also see other options as well.
So we're gonna talk a little bit about these
and then I'm going to get you
to take a look at these yourselves.
So I'm gonna talk about the left heavy view,
the sandwich view, and the export and import options.
So the first additional option in Speedscope
is the left heavy view.
This group stacks, which are the same together,
and it means it cannot sort along the X axis by time.
Instead it sorts each group,
so that the biggest stack for each group is on the left,
so therefore each group is left-heavy.
So you can see where the slowest code is running.
Next up, we have the sandwich view.
This is a little bit like the percentage of time
spent in each gem table from the old renderer,
but it's done by individual methods rather than by gems.
Finally, there's some very nice
export and import options in Speedscope.
So in the top-right of the view,
you will be able to see an export button,
this exports the profile in a JSON format.
So I've put an example on here
of the format that it exports in,
but you can learn more about the format
in the Speedscope docs, which I will link to.
Also because Speedscope was designed
to be language agnostic,
you can also import profiles for lots of different things.
So Chrome, Node.js, Instruments for Apple Operating Systems,
Pastel, .NET, there's a lot of options.
You import these using the import button
in the top-right right corner of the UI.
So for this exercise,
I basically want you to play around with Speedscope
and take a look at the other options.
So if you've not got stuff set up,
like try and power up with someone who has,
and you can do this on the root root,
or one of the other pages.
So I'll give you a couple of minutes,
just try looking at the left heavy and the sandwich views,
and try exporting a profile to JSON.
Okay, cool.
Hopefully everyone's had a chance
to take a look at those options,
if not, you can take a little look after as well.
So yeah, I'm just gonna recap.
So we talked about reasons we care about performance
in our applications, how to prioritize
and rule out other causes,
how you read flamegraphs,
looking at changing some code to speed it up,
and getting that change into production.
I talked about adopting to what stage your work is in,
and the history of flamegraphs,
the Speedscope visualizer and where it's most useful.
So yeah, the final exercise, I'm not gonna pause, folks,
this will be after the end of the workshop,
but I want you to install rack-mini-profiler
in a personal or work project
that you'd like to take a look at,
and then check it works locally,
and then go through the steps that we've talked about.
I'll share the slides afterwards as well.
And then you just wanna see
if you can find something to speed up.
So yeah, I'm gonna stay on for a bit after as well.
So yeah, if you need help, do let me know.
So you get references,
once you found these areas that are slow
that you wanna speed up,
other people have written very good material
about how you approach this.
I want to do a particular shout out to Nate Berkopec
because I've been following his stuff for years
and he's written really well about this.
These are my references,
references continued,
even more references. (laughs)
So yeah, thank you very much.
You've been a great audience.
I'm Jade Dickinson, _JadeDickinson on Twitter.
And yeah, I wanna hear from you after.
How did you get on, what did you find?
Did you change something in the application
you've picked out and I'd like to hear about results.
- [Male Audience Member] Hi, have you done
any stress testing with these techniques as well
where you're like generating a lot of traffic
artificially to see where things are slow?
- Okay, so yeah,
so a friend of mine has written a very good article
about doing this.
It's an open source project.
I'll just try and get the details to you,
but essentially doing shadow requests.
And like you said,
just making fake traffic to a site
based on, just really cool,
'cause it's based on real traffic.
It's called Umbra.
And if you go, I think get GitHub.com/carwow/umbra,
I think that's the repo.
This is off the top of my head though, so yeah.
Cool, you're welcome.
So yeah, thank you very much.
Hope it's okay if I take a picture.
(audience applauds)