Angie Jones Chats With Kent About Automated Visual Testing

Kent C. Dodds: Hello friends, this is your friend [Kent C. Dodds 00:00:02], and I'm joined by my friend, Angie Jones. Say hi, Angie.

Angie Jones: Hello.

Kent C. Dodds: Super happy to have you here with us, Angie. Angie, and I ... Actually, you know what, I don't remember how we got introduced to each other. It was definitely over a year ago though. Was it ... it was through Applitools, I'm guessing, right?

Angie Jones: I think it was just through Twitter verse. Just kind of running in the same circles or whatever.

Kent C. Dodds: Yeah, yeah, exactly. I make most of my friends these days on Twitter now. But Angie works at a company called Applitools. I'm super psyched about what Applitools is doing and I'm sure we'll talk about Applitools a little bit more today. But I would like for everybody to get to know you, Angie, as I do. And so could you introduce yourself a little bit, tell us things about yourself, whether it's your technical things that you're interested in, or personal or whatever you'd like to share.

Angie Jones: Yeah, sure. So I'm a senior developer advocate. Again, I work at Applitools, which is a startup that focuses on automated visual testing. So that's my niche, testing is my niche, specifically automated testing. So I've been doing this for the bulk of my career. Had a couple of stints in feature development, but I really, really love automated testing and feel that, you know, I just get this bigger, overall picture of the product and what we're doing and our customers. As well as gets a flex my development muscles more, because I get to architect frameworks and solutions and stuff. So I've been in the industry for quite a while now. Worked at some really big companies, such as IBM and Twitter. So I'm enjoying the startup life now. Something different for me, developer advocacy is also something different for me. So a couple of years ago I started just preaching the gospel, which is testing, and did that at conferences all over the world and workshops and blog posts. So developer advocacy was a natural next step for me. Loving it so far.

Kent C. Dodds: Awesome. So when you say like preaching the testing gospel, that makes me think of something that somebody once said, it was Ryan Florence, he said, "Sometimes it feels like testing is a religion, and if so then I belong to the church of Kent C. Dodds." Yeah, it's kind of funny, but it really ... There's a lot about testing, a lot of opinions about that. I went to the Assert JS conference last year, and it was great, all about testing JavaScript. And what was interesting about it was how every talk seemed to contradict the one before it. So many opinions there. But, you know, there's not a whole lot ... I don't hear people talking quite as much about visual testing, and there are some really compelling arguments for investing time in getting the confidence out of visual testing. So, Angie, could you tell us a little bit about what even is visual testing, and I'm curious why you got so into a visual testing.

Angie Jones: Yeah. Okay, so visual testing is essentially like snapshot testing with images. So when your application is in the state that you want it to be in, and you verify this as a human being, you can utilize tools to take a picture of your application in that state. So this is mostly page by page basis. So let's say you're on a certain page, you've done some scenario, and you want to capture this. You can capture this with an image. And then as part of regression tests, every time you run this test, then it'll take another picture and compare the two. So it's not something that's really new, it's actually been awhile for ... been around for a while now. However, the technique that was used in the past was pretty flaky. So it would do like a pixel-to-pixel comparison of these two images and determine if there's any difference between them.
Well, you know like I know, on different resolutions things can change. There's also things that are happening in our application that can change from one screenshot to the next, such as cursors blinking or things like that. So Applitools has used a new technology, they're using AI and machine learning to be able to only detect the things that we care about as human beings. So I've been doing animation for a very long time and I've been doing it the traditional way. I use tools like Selenium WebDriver and JUnit for assertions and things. And I thought I was pretty good at it, you know, got to principal level as an automation engineer. And then I was introduced to visual testing. So I'm always looking at new tools and, you know, because I speak a lot and I write a lot.
So I'm always looking for what's hot on the market and what can help solve some of the challenges that lots of automation engineers and developers face. So I looked at visual testing with Applitools and I realized, Kent, how much I was not covering in my tests. And I thought it was pretty hot stuff, but when I looked at all that it looks at, you know, basically the saying; a pictures is worth a thousand words. This is literally like a picture's worth a thousand assertions. Say, for example, let's take a scenario where we added something to the shopping cart and maybe change the quantity to three, right?
So if I was writing a test for this, I would make sure that, you know, probably the subtotal in the final total is okay. That would be my two assertions there. But then, when you look at visual testing and how it's looking at the entire page, what if there were any exceptions or errors that were thrown on that page? I don't have anything in my code that's looking for that. What if the thumbnail on the image is not there, what if by me increasing the quantity, that distorted the way that it was displayed and now it's kind of all jumbled and overlapping. None of that would be caught by my scripts because my script is simply looking at the Dom. With visual testing I get basically anything I could think of plus more that's being validated. And it's usually like one or two lines of code.

Kent C. Dodds: Yeah. Yeah. And there's something that I like to say a lot, which is, the more your test resemble the way your software is used, the more confidence they can give you. And that fits in perfectly with visual testing because with traditional unit and integration tests where you have specific assertions, like you're saying, you can structure your things so you ... Okay user click on this and then do that, but maybe the user can't click on that thing because it's hidden behind another Dom node. Or maybe there's some color contrast issues because some CSS is applying, you didn't expect it to. All of those things that you just said.
And so, you can get a huge amount of confidence with visual testing. So my next question is, and this is often when we're talking about these different types of testing, there always are trade offs. Or are there? That's, I guess my question is, it seems like visual testing gets you a huge amount of confidence and so why shouldn't I just do visual testing for all of my tests? Is there a reason to try, or to mix in visual testing with my current testing strategy or should I just replace my current strategy with visual testing?

Angie Jones: Yeah, what I love about the Applitools API is that it integrates with whatever you have. So if you're using Cypress or Selenium or [Jest 00:08:35] or whatever you're using, then there's all of these various SDK's that will just integrate with what you have. So I do some workshops and I talk about the pros and cons of these different strategies. Do I replace everything or do I mix it in? And I show how you can couple it with traditional assertions as well in some cases. For example, that visual check that's going to check the entire page, that's great in a lot of cases. But there may be some cases where I just don't want the entire page validated, right? Maybe it's under construction and so, you know, I don't care about the whole right side of the page. Or maybe we have advertisements right there and every time that's going to be different, right?
So you can do lots of different strategies using this API. You can narrow your scope down and only verify a certain DIV or other element on the page, so maybe I only want to verify that shopping cart part and not the whole thing. Or you can ignore certain regions of the page as well. So if I know that the ad is always shown in this particular part of the page, then I can mask that and say, "Don't verify this part," right? But when I do that, then there's some give and take there. So me, narrowing the scope of what I'm verifying, then maybe my picture doesn't have a thousand assertions. Maybe now that's 50 assertions, you know what I mean? So in cases like those, you can couple it with other things that you want to assert on. I often tell people to go down the test automation pyramid so you don't have to do everything at the UI. You can verify a lot of data points and things like that at maybe the API level, and then do one visual assertion that just makes sure everything looks right.

Kent C. Dodds: Yeah, I think that that makes a lot of sense. And one thing that I like, I think snapshot testing is a good parallel to this, as you were saying earlier. And one danger that I've seen that people have is they'll make giant snapshots for their stuff and those snapshots break a lot. With Applitools, when it breaks, normally you're glad that it broke because it's preventing you from something thanks to all the smarts that Applitools has. But another, I guess, trade off that you can have with visual testing is, there's a little bit of indirection between where the assertion lives and where the assertion fails. Right? Or at least where the specific thing that you're trying to assert. So like some people would make the mistake with snapshots to say, "Hey, that cart number total should increase by quantity of one," whatever the total amount should increase by whatever.
And so they'll take a snapshot of the whole thing, and that assertion lives within that snapshot. But when it fails, people don't know whether or not that failure is a bad thing because they're not sure what the intended version is. And so there's a little less of a, you know, a specificity there. Which is why, from where I sit, I think it makes a lot of sense that Applitools is integrated with the existing testing tools that people are using. So that they can have those specific assertions, but then also get all the benefits of the visual testing, where most people aren't testing anything that has to do with the way that their applications laid out, you know, CSS-wise or whatever. Or whatever the case may be.

Angie Jones: Right. And that's a huge miss when people don't do this. There's so many people who don't. And because of this, I see these visual bugs all over the place. And a lot of times I'll talk to people and they think, oh, that's pretty cool, but you know, that is not applicable for their application. Or that is a nice to have, but as we're developing for more and more devices and view ports, that's where those visual bugs come in. So, yes, your app looks great on the web, but how does it look on the phone or the tablet or all these different things? And when you have all these different permutations it's really, really difficult to test for all of those things. And that's where stuff starts breaking. So I see it all the time. Companies, big and small, all of your favorite tech giants have these visual bugs all the time.
I was looking at one not too long ago on Instagram, and this was a sponsored ad, Kent, and all of the text was jumbled up and overlaying on top of each other in the upper left corner, and everything else was white space. And this was sponsored. Somebody paid for this. Right? And so, I think about their testing strategy. Anytime I see things like this, as a tester, I start thinking about, well what was the testing strategy?
And it's hard to blame them because I bet they probably had automated tests for them. But your automated test is, is this text present? Yes it is. Is this one? Is this one? Is this one? And they could have had 10 assertions in that test and all of them been true, because the stuff exists in a Dom and so they missed this and this went out to production where they're losing money as a sponsored ad. They have to refund this customer. And not only lose money but also trust. I don't know about you, but I probably would be a little bit hesitant to do another ad on Instagram because I think they don't test their stuff.

Kent C. Dodds: Yeah, that's a very good point. And so, Angie, I hope you allow me to just geek out a little bit on Applitools for just a second, because there's one thing ... I'm not sure how to lead you into this. And so I'm going to just mention probably my favorite thing about Applitools, just conceptually. And you can stop me and correct me if I miss any of this, or miss-explain this, but one of the things that I love about Applitools is the cross browser support. So let's say that you just want to use Jest for your testing and that's all in JS Dom. And so you're, first off, you're not actually testing in a real browser, which can be problematic, and you can't do any cross browser testing either. But as soon as you throw in Applitools assertions in there, this is what you get.
So Applitools, when you say, "Okay, I want you to snapshot this Dom," Applitools will say, "Okay, let me take all of that Dom HTML, and all that CSS, everything that's in our document right now, and I will upload it to Applitools servers." So it's not actually taking a snapshot on your local machine, it uploads, it's Applitools servers. And then Applitools pulls all of that up in all of the different browsers, and a bunch of different screen sizes all over the place, and then snapshots that. And so here are the two things that makes this so awesome. First off, it makes your test still run really fast, because typically visual testing is really slow.
But because we can offload that to Applitools then it's very fast. The second thing that's awesome about this is, now you're getting a huge amount of confidence that your code or that ... Yeah, what's produced the Dom output, that's produced by your code, looks good across different browsers and different screen sizes. And you only had to run that test one time, in one place. You didn't have to run it on SauceLabs in 30 different permutations of different browsers and stuff, which takes a lot of time. And so those are two things that I just absolutely love about Applitools. It's fantastic. Did I mess any of that up Angie, or do you have [crosstalk 00:16:36]

Angie Jones: No, that was great. That was great. So it's called the visual grid and, yeah, it does just that; executes it one time. Let's say it runs it on like Chrome, for example. And let's say that your scenario had 10 steps, right? So after the eighth step you maybe do some assertions, this is where you call Applitools in, everything before that was set-up type of stuff, getting the system in the certain state. And then maybe your last two steps are just cleanup. So instead of running that across all of the configurations that you support, you just run it the one time, it executes those eight steps, and then when you call Applitools, it takes all of the artifacts like you mentioned; so the Dom, CSS, anything that's pertinent to the application. It grabs that and then splashes it across all of the configurations that you've specified you like to run against, in that state.
So instead of executing those eight steps across 20 different configurations, we don't have to do that. Let's just take the state of the app and see how it's displayed across all of these different things. So yeah, like you said, it's lightning fast. It saves so much time on test execution. And so this could then be a part of your CI jobs where you need all of your tests to be really fast. Before, a lot of people were doing the visual testing, but they would do it in a separate build, maybe that ran once a day or something like that. But with this visual grid, that now speeds up the test, it takes exactly the same amount of time to run on one config as it would on, say, like 50 configs. Because they're all running in parallel and you're not executing those same steps across all of them. So, yeah, it's really amazing. Magic even.

Kent C. Dodds: Yeah, it's honestly a stroke of genius there. That just so awesome. So I do have a question about that, though. So when I have that assertion that says, "Okay, this should match," now I have this whole awesome dashboard on Applitools that I can go look at and compare and approve new snapshots and that. What that experience? Can you describe that experience in the tool? So if I'm using Jest or Cypress, for example, do I see that failure locally or do I have to go to this, to the dashboard to see that? And if I see it locally, what does that error message look like? How long does that assertion take, typically? If I have a test that takes two seconds to run and then I add one Applitools assertion in there, is that going to impact the speed of my test? Those kinds of things?

Angie Jones: Yeah. Okay. So let's take a scenario. So you've written the test, you run it. Let's say that this is a regression run and it finds a failure. So the test will fail and it will throw differences found exception, and it'll give you a link in the error message. So you'll see this on the console, that takes you to the Applitools dashboard. So all of the images are stored in the Applitools cloud, and then there's this nice little dashboard that keeps all of your test runs and all of your visual DIF's. So you click on the link that takes you to the dashboard, you can then see that test and it shows you the baseline image, which is the one that you ran the very first time, and you're okay with how the app looks. And then it'll show you, that'll be on the left side, and then on the right side it'll show you the new image.
And then it'll be like some highlights on it that shows you what was different between the images. So you can see exactly what changed. Oh, okay, so they moved ... the logo is not there, or something like that. So say like your logo is not there or something is just kind of jumbled and distorted, you would then fail the test in the dashboard. So I really like this because a lot of times people ask me about machine learning and AI and do I think AI is going to take our jobs and stuff like this. So I really lik that AI is assisting me right here, and saying, "Hey, human, we found some differences, but we're not really smart enough to know if this is bad or not. Can you come, oh smart human being, and they look at it?"
And that just, I don't know, it's an ego boost for me. So I go in and, yeah, if it's a failure then there's this thumbs up, thumbs down. So I'll do a thumbs down, and then that'll be marked as a failure. There's also all of these integrations with things like [inaudible 00:21:28] and stuff like that. So I can annotate the image and assign a bug or assign it to someone or make comments on it. So it's really cool. But let's say that it was a change that we intended. So we moved the logo from the left side to the right side of the application. Okay. All right, so this is become a change in my app. I then press the thumbs up to let Applitools know, oh, okay, yeah. Thank you for letting me know about the change. I would like to update my baseline.
When I click the thumbs up, it replaces the new image with ... The old image, with the new one. So the baseline is then changed to the new run. So that's how it works. There is a little bit extra time added to your test if you're going to put in a visual check. But that decreases by a lot if you use the visual grid. Because that approach is just much faster. But if you're not using the grid, let's say you only want to run against one environment, like, "Oh, I only care about Chrome," or whatever, then it will add a little bit of time to your tests. And I've talked to some people, because you know we like things fast as developers, so I've talked to some people like, "Do you not care about this time? This extra time?" And they was like, "No, because it's just checking so many other things." So it's kind of like a price that people are willing to pay because it gives them that confidence to deploy.

Kent C. Dodds: Yeah, absolutely. I find it funny ... So I like to lean more heavily to the integration test side of things. And I find it funny when people say, "Well, that's going to make your test take a little bit longer." And yeah, okay, it'll take a couple of seconds longer I guess? And I'm more confident with what I'm shipping. I'd be willing to spend a couple extra minutes waiting for my test to finish to be confident I'm shipping. And so I'm actually a little bit curious on the process because I've never used Applitools in a team setting before. And so, if you just answer this question really quick, if I'm working on a new feature, like moving that logo over, and I'm running all the ... Do you find people typically running these tests locally? And if you do, then I say, "Okay, I'm going to say update that baseline," but then anybody else who's currently also running these tests on their machine, they don't have my changes yet. And so now, are their tests going to start breaking? Or what's that workflow typically?

Angie Jones: Yeah. So in the dashboard, the tests are identified by a number of criteria. So the name of the test as well as the browser or device, so this works on web or mobile, and the resolution. So all of those things coupled together is the identifier for a test. So if I run this test locally using that same dimensions or whatever, then that will correlate to the test that Applitools already has. So if you running that same thing, then we're talking to the same test, if that makes sense. Right?

Kent C. Dodds: Yeah.

Angie Jones: So if I update that with the new thing, then you'll have those changes as well, because we're sharing this team space in the dashboard. What's also great, like that whole logo example, so this would be like a maintenance nightmare if you think about this. Because if you move the logo from the left side to the right side, all of your tests that took pictures where the logo was, all of these will now fail. So this is another way that Applitools uses AI to find that failure and then look through all of your failures to see if there's the same failure across multiple tests. And then I only can, I can just change it one time. I can say, "Okay, oh yeah, the logo moved, great. Can you just fix that in all of my baselines that have that problem?" And it'll go through and do that. So that's really cool.

Kent C. Dodds: Yeah, that's brilliant. So the process of getting that change applied across my application, I know how that works with code because I make a pull request, people review the pull request, they merge it. But how do I say, "Okay, now that this has been merged, that new baseline is correct." What's the process there? Do you integrate with GitHub and say, "Okay, when this pull request is merged, all of its snapshots are the new baseline," or how does that typically work [crosstalk 00:26:23]

Angie Jones: Yeah. Yeah. So we keep what's called a UI version control. So think of version control for your code before your baseline images. So we keep a history of all of the baselines for a particular test, and we do integrate with GitHub, so you can go in and like say, "Yeah, okay, and once this is merged then is my new baseline." Let's say that you were doing some kind of AB testing or something like that, and now you want to go with A, so you want to go back to a previous baseline. You can also restore that as well.

Kent C. Dodds: Wow. Boy that's powerful stuff.

Angie Jones: I know, it's so mature.

Kent C. Dodds: I remember trying to set up visual testing a couple of years ago and it just was nowhere near this. A tough failure, one after the other because some pixels were off, it was just outrageous, bananas. So, Angie, it has been a pleasure to chat with you. We're coming down on our time. Is there anything else that you wanted to talk about that we didn't get a chance to?

Angie Jones: No. So I do have more information about this on my blog, as well as other testing strategies and techniques. So I don't just talk about visual testing, but the whole shebang. Testing in CI, all of this good stuff. So my blog is angiejones.tech and also Director of Test Automation University, which is a platform that we launched this year. And it contains free courses on all things test automation. So I do have a course there on visual testing in different languages, Java, JavaScript, C Sharp, whatever. So whatever fancies your boat, you can go and check that out. We also have courses on all of the tools and techniques that have to do with test automation.

Kent C. Dodds: Very cool. So, yeah, for our homework, for everybody, the call to action for this episode, Angie has this awesome visual testing course on Applitools and it's totally free and it's called Automated Visual Testing: A Fast Path to test Automation Success. And it's just about an hour, 18 minutes long, I think is what you said, Angie. So it's a pretty quick one and it'll, like, if you want to get into this a little bit more than you can go through that and learn a lot more about visual testing. Fantastic stuff. Gives you tons of confidence, so I definitely recommend that people give that a look. So, Angie, where's the best place for people to reach you online?

Angie Jones: Yeah, I'm on Twitter all day long, so you can hit me up @techgirl1908, or on my website, angiejones.tech where I blog about test automation strategies and techniques.

Kent C. Dodds: Great. Awesome. So thank you, Angie, for giving us some of your time today to chat about testing, something near and dear to my heart as well. And I hope everybody has a wonderful day and takes this opportunity to get better at getting confidence with shipping their apps. So thank you so much, Angie.

Angie Jones: Thanks, Kent.

Kent C. Dodds: See ya.

Angie Jones: All right. Bye bye.