Speaking at Selenium Conf 2022
Joel Rosario & Hari Krishnan spoke about Contract Driven Development
Summary
Tired of dealing with seemingly innocuous data type mismatches, missing parameters, etc. in API signature that wreak havoc on MicroServices deployments late in the development cycle?
Join us in this talk where we will share our experience about how we leverage API Specification Standards such as OpenAPI to identify such compatibility issues early in the development cycle and in the process effectively eliminating integration tests. We will also be sharing our journey about how necessity drove us to come up with the โContract Driven Developmentโ approach and Specmatic, an open source tool that embodies the same to promote collaboration among teams, lower cycle time and deploying services independently.
Transcript
Good afternoon everyone. My name is Pyal and thank you so much for joining in. Today we have with us Joel and Hari and they both will be sharing their insights on a very interesting topic which is contract driven development. And they’ll be also letting us know how we shall be able to deploy our microservices independently with doing any integration testing. So, quite an interesting topic to hear. So, without wasting any time, over to you, Joel. Thank you.
Hello everyone. Welcome to this talk. Thanks for Pyal for the great introduction. So this is about contract driven development. Before we jump into the topic a little bit about ourselves. My name is Hari Krishnan. I’m a consultant and a coach. I help both Unicorn startups and enterprises. Large enterprises with their cloud transformation, extreme programming, agile and lean. My interests include distributed system architecture and high performance applications. I’m a regular at most of these conferences and I love contributing to the community. So these are some of the conferences I’ve spoken at. So that’s about myself. Over to you, Joel.
I’m Joel Rosario. I’m a consultant and coach as well. I have about 19 years of experience under my belt. I’ve worked in development, I’ve worked in testing. I sometimes find myself with a foot in boot camps, which I think is a great place to be. And these days I help tech teams improve their quality and engineering capabilities across the board. And to start this session off, I’m going to take you to a small demo. Let’s dive right in. I think this is about contract driven development. Let’s dive right in and see a contract of an application. This is a sample application. Here. We have an API for ecommerce, some orders, products, et cetera. I won’t go into this too much quick look at the contract. There’s a product API with an ID and so on and so forth. I’m sure most of you have seen an open API specification before. I’m just going to dive in and run some tests. Let’s see where this gets us. Here we go. We have some contract tests that are running. We are using a tool called Specmatic to run these contract tests. And there you go. We have twelve contract tests.
Let’s take a quick look at the first one. We are saying the test sends get product ten to the application and gets a response back. This response is supposed to match the contract, right? And maybe we can take a quick look at the contract now and see what we have. So you have Get product ID that’s over here. You have the response that’s been defined. So this is what I showed you at the start. This is the structure, right? Specmatic has of course validated the contract. Test here validates that this response matches the contract and that’s how we know that the application is in sync. We are trying to make sure the application is in sync with the contract. Here’s another test in a similar fashion. We post product ten. The test sends this in, the payload, gets the response back and checks that for no content. Actually, no content is as per the contract. There are twelve such tests. Right. Where’s the code? Let’s take a quick look at the code for this. And that’s it. Look, no code. This is just a starter sort of helper class. What essentially happens is we pull Specmatic.
This is a spectrum class that comes into the picture. Specmatic lifts the contract, reads each and every operation in the contract, turns it into contract tests, executes the contract tests. There are twelve of them that come out of this and you get this all for free. No code to be used or to be written except for these few lines. You get twelve contract tests pretty much for free. And this is good as it stands. We know that the contract tests from the contract pass and that means the application is in line with the contract. But what if you send requests which do not match up with the contract? Will the application behave effectively? Well, let’s try that. We just turn negative testing on setting it to true. I’m going to run these tests as well, so just take a second. Spec matrix is running again. Spring application is starting. Here we go. Suddenly we have a lot of failing tests. Right? The test failed 12:26. What’s happening here is that the original tests, the happy path tests that I showed you before, those are passing, but now we have a whole bunch of negative tests.
Right, let’s take a quick look at the first one. There’s some no element exception, so on and so forth. What’s actually happening under the hood? And here we see the first example of a negative test. We are passing ideas null. But according to the specification, the open API specification, this is not supposed to be null, this is supposed to be a number. And Specmatic was able to take this specification, know that it’s not nullable, pass a null in its place and see how the application responds. The application here has returned a 500. It’s not supposed to because this was an invalid request. We should have seen a 400 or something like that. And all of these 26 tests that you get essentially for free, schematic test generates it for you out of the contract. Right? It’s the same line, the same set of code that I’ve got here. I just added a flag. And with this flag on, we get 26 tests. That’s a total of 38 tests, which gives you a solid idea about how well your application adheres to the specification that it’s supposed to. So with that, I’d like to hand it over to Hari, who will help us understand how we got to this point from scratch, how we got to this point where we are able to so magically generate tests from an open APS specification.
Over to you, Hari.
Thanks, Joel. That was an awesome demo, wasn’t it? That was just a teaser, actually, right? Of what is to come. And I guess you guys are ready for the main show here before we get into the deep dive of what we achieved, just to get this context set right, to continue from where Joel left off, like, Joel did something awesome. He just took an open API specification and with practically zero code, he was able to run it as an executable contract test against an application. And he also got it to show where the application’s weaknesses lie, wherein if you send a null, the application did not have a null check. Right? So now let’s actually look at it in the frame of reference of how does it all fit in into killing integration tests. So, to start with, we have a mobile application. Let’s assume it’s an ecommerce app and it’s got a view product screen for showing the product. It requests the details from the back end and the back end provides the details for the requested product. Sounds fairly simple, right? Like what could possibly go wrong with this application? Let’s take a little bit more deeper look.
How would we write the component test for the mobile application? This view product screen. So just so we are clear on the vocabulary, because the application is requesting the data, we’re going to call it the consumer. And since the back end is providing the data, we’re going to call it the provider, right? So we have the terminology in order. How does consumer component testing look? Any test has three parts to it, right? We all agree that there is a test system under test and then there is a dependency. And a good component test ideally isolates the dependency. In this case, the dependency is the real backend application. And as most of you would realize, we don’t want to talk to the real application in a staging or a production environment. It’s kind of messy, right, to go across the network if you want to get the provider running on a local machine, that’s again, may or may not be possible, depending upon how complex that stack is, to bring it up on our local machine. So the tried and tested approach to that is to sort of have a handrolled mock or a record and replay tool, sort of simulate the provider for us, like stubborn out, so that we don’t have to have the real provider and we can isolate and test the consumer and develop start writing code for the consumer.
While this looks quite all right, like this is all familiar, right? We use this with wire mock or other record and replay tools. There’s a fundamental problem here. The mock is usually not in line with the provider, right? If you are using a handrolled mock or if you did a record and replay, it’s very likely that the provider made a code change or an API change and your mock has not made that change, right, like we did not. How often can we keep re recording and playing or keep maintaining this stuff? It’s quite expensive. And this fundamental issue can lead to huge problems, like how I, as the developer of the consumer application, may assume that I could send the product ID as a string, right? And I set up my mock like that. However, the actual backend might be expecting the ID as an integer and likewise the provider might be returning the name and SK of the product. However, I have wrongly assumed while setting up my mock that it’s going to be returning name and price. Right? What does this lead to? Broken integration. Is this the worst of it? No. Where do you find such issues?
It’s not possible. On local, which you just saw. If you’re isolating with handrolled mocks, it’s not possible. And if you are taking the same setup into your continuous integration environment, it’s going to repeat. Right. And for the provider, the handicap is that there is no true representative emulation of the consumer, right? So again, it is depending on a common deployment location. So the first time you see such issues is when you actually deploy to an integration testing environment and boom, you have a bug and it says these two things are not compatible. Now there’s a double whammy of an issue. This compromises your integration testing environment. Most often it’s not likely that you just have two apps in your integration testing, right? You have a deployment of microservices. Even if two or three have compatibility issues, the entire environment may get compromised and that may block your path to production. Right? And that’s unhappy users, which we don’t want. There’s another problem with regard to cost, effort and time. So this heat map that you see at the bottom obviously represents the reality, right? If you found an issue on your local machine, it’s cheapest and easiest to fix.
Cycle time is really slow, I mean small. If you found it in CI or if you found it later on, it’s going to be harder to fix and the resolution time is much higher, right? So that’s also not desirable. What we really want here is to be able to shift left, find these issues on the left hand side, and sort of avoid integration testing. However, we do want to find compatibility issues, right? So that’s the problem statement. Can we identify compatibility issues without integration testing? So with that, I’ll hand it over to Joel again to see if he can solve this problem. Over to you, Joel.
So taking off from where Tori left off, we are trying to shift left. We are trying to identify compatibility issues without these compatibility issues often come up because of some misunderstanding between the consumer and the provider, right? And the reason for that usually is that they’ve come to an agreement about how the API should behave and what the request and response should look like. But this agreement might be over email. Very often we’ve seen teams collaborate over email a lot on this topic. There might be multiple emails, there might be multiple Word documents, there might be multiple Excel files. And so there is a common, but really a fragmented in some sense understanding, scattered across different inboxes and desktops. So what if, as a first step towards fixing this problem, we could actually have a rock solid industry standard specification that actually contains all the details?
Right?
For rest, this is open API and it contains everything from what the headers should look like to what the JSON payload should look like, right down to is a key, mandatory or not, and lot of rich details. It’s too much to go into right now. And essentially once you have a specification with that level of detail and clarity, it becomes much easier for consumers and providers to adhere to the specification. What’s even more interesting is that the specification is machine readable. And because the specification is machine readable, you suddenly get superpowers because now you can feed this thing to tools. And when you feed the specification to tools, you start getting early feedback and you can now get early feedback on your laptop in your development environment rather than deploying it into integrated environments and getting errors there. And now the tools locally can help to keep the consumer and provider accountable to the contract. And then the interaction between the consumer and provider then become governed by the specification. And so I’m going to actually get a little deeper into that. Let’s actually start off with an exercise, a quick exercise. I’d like you to download this contract.
I’d like you to download Specmatic and just get this thing running. I’m going to open my chat window briefly and just tell me whether you’ve been able to get the contract downloaded and running. If you’re following along, I’m going to be doing this myself as well. The interest of time I’m going to carry on. We have a small contract here. It’s okay if you don’t know. In case you don’t know open API too well. I’ll just quickly run you through it. This is a ProductID API. The ID is parameterized. There’s a number which essentially means this path matches something like product ten or 20, whatever. It’s got to be a number. It’s compulsory. When the application receives this request, it’s supposed to get this response back. The response has got to be a JSON object. The object has a name, a skew. Both of them are strings. Right? Pretty simple. We are now going to try to start it up as a stub use of the command on screen. And I’m going to do the same thing. Java minus jar jar, stub and API. There you go. If you’re following along and you’ve gotten this far, you should be seeing something like this.
Stub server is running on port 9000 control system. That means your Stub is running. And then we are going to take this to Postman. I was saying that since you have a machine possible specification, you get superpowers. One of them is that tools understand, tools like Postman understand that you can just drag and drop it into the import button and Postman just opens it. So I’m going to do that again so you can follow along if you are. You just need to have Postman open just where my mouse is moving. At the top you have an import button. You click that, you get this text. This area a blank area. Here you can open Explorer or find out if you’re on a Mac. Just drag and drop the contract there. Postman just sucks it up in a jiffy. And here you go. Get products. Right, so I’m going to click on this. I’m going to go ahead and one send. Spectric has returned something. Right. This is of course looking a little random. Right? We see. Name some random string. Firstly, this itself is useful because this will help you get started. Secondly, Specmatic did this with absolutely no further code involved from you.
Thirdly, given that this is random, you actually do want sometimes to tell much of them to tell Spectric what to return. We never told Specmatic what to return for product one. Right. And so Spectric just took it back to the contract. Checked to the contract, is it? Right, fine. It doesn’t know what to do with it, so it looks at the response in the contract and generates a response. I have done something a little different on my laptop, so if I pass five, Specmatic actually returns something because I’ve told it what to return. Okay, so this is a quick demo of how we can get Specmatic to use contract Open API specification based service virtualization. Just a standard.
Just wanted to check if everyone’s following along what Joel just showed could type in. Yes, you’re following along if you’re having a hard time, if you have difficulties, you could put that in the chat. We’ll try our best to help you out. Sorry for the interruption, Joel.
Go ahead, please. Yeah, great. I’m glad to see that folks are able to follow this. Pretty cool. So I’ve shown you an example of simple service virtualization without the simplicity of doing it without any code. I’ll show you how to how to specify something explicit to Spectric further down the line. But this is how we can and Specmatic validates this. So I’ll show you how Spectric validations work as well. Specmatic would never have returned something like this if it was not matching the Open API specification. And we do have to take this essentially to the other side as well. And so what typically happens with a test, if I can go over that quickly, is you have the Open APS specification, talking to a system under test and Specmatic reads the contract, turns it into contract tests and basically fires those requests against the provider, gets a response and checks if the response is actually valid. Let’s see how this works. Let’s actually try this with a real test. We are taking this over now to the provider side. I’d like you to download one more file. This is the provider sample and download it and get this running with Java minus Java products, et cetera, like before.
I’m going to give you a minute to do this. I’m going to move ahead now. We shall try to run this as a test as well. Let me actually start the application. The application is kicked off. I’m going to start a new tab. And we have a contract test running. This is something similar to what you would have seen at the start. We’ve just got one contract here the contract test. Specmatic has generated the contract test from the contract. We have a request get products 579. We have a response returned from the application that is for the contract and the test has passed because Specmatic realizes that the response is according to the contract, which is great. What would happen if the response does not match the contract? So I’m hoping folks have been able to follow along, right? Can I have a quick check if people are able to follow? All right, thank you. So essentially, let’s make a change to the contract. So we are now trying to figure out what would spectrumically if the contract is out of sync with the application. To do that, we’ll make a change to the contract here because we don’t have the application source, which is normally where the source of the errors is.
So we change the contract. So you just change the sq to a number to supposed to be a string. We know the application is returning a string and we run the same command again. And now Specmatic actually gives you the same contract expected number. But response contained book SKU forward, which is a string. And essentially what this means is this is pinpointed feedback, response to a body or SKU. Spectric was expecting a number. The response contain the string. You get that feedback right here and this thing can be integrated with CI and other things. We talk about that later. The main thing to take away is both in the previous demo and in this one, there was no code that I had to write to get this test running. There was no code that I had to write to faithfully represent my provider. I was able to faithfully represent each to both. This is symmetrical. As a consumer, you need to faithfully represent your provider, right. And you use the contractor for that. As a provider, you need to faithfully represent your consumer. And so you generate contract tests out of the consumer. This is how this would typically work.
The consumer and the provider have only the contract when they start off. The consumer essentially starts off on the local development environment. Your laptop consumer doesn’t have the provider running locally, obviously, but we have the contract using which we can faithfully simulate the provider. The way we do this is we pass it to Specmatic. Specmatic generates a contract as Stub, which allows the consumer to run a faithful high fidelity service virtualization. Specmatic also generates contract as test, which allows the provider, without having the consumer to run a high fidelity simulation of the consumer. And this means one does not have to wait for the other. So the consumer can start development without the provider being available. The provider can start development without the consumer being available. The two of them can then deploy with confidence into an integrated environment because they have both ensured that they stay faithful to the contract. This is a very brief demo of how things work. As a next step, I want to talk about, well, smart service virtualization. I’d like to do a deep dive. So I showed you a little bit about how service virtualization could work. I’m going to take it further now and we are going to actually see can this switch over to the other one?
We are actually going to see service virtualization in action. I don’t have anything for you to try here. This one is going to be a demo. So let’s take a look at this contract briefly. I’ve shown you some of this before. I think you will be familiar with this portion of the contract. There’s a new piece to this, just a small additional API for creating products. Essentially, you have slash products. The request body accepts a JSON with two fields, a name and an SKU. And in the response to this API, the provider is expected to respond an ID, which is an integer. Now essentially, we are going to start the stub again. This is a vanilla stub. I’m going to import this into postman as well, old one, and import is imported. Both of them. We’ve done this before. We’ve seen this random response. We’ve done this before as well, passing five. But this time we are getting a random response because I haven’t told Specmatic what to do with it. This time I’m going to do that in front of you. So you can see how that works. So I’m going to just create a folder about it.
The name of this, the contract is Products underscore API. So this is called Products underscore API. Products Hyphen API underscore Data file. There Batteries JSON because it was returning some information about batteries. Let’s take this small, flip it and make some changes here. This is exactly what we saw at the start with the call to Slash Products. We get this data back. And let me start this off now. Right, there you go. You have batteries back and this is how you tell Specmatic what to return when you fire product five. The very first question that we will want to know after this is what happens when we try to stub out. The whole point of using this is what happens when you try to stub something out that does not match the specification. Let’s say instead of SKU, ABC, one to three just said one to three. This is a number. We know this does not match the contract. Specimen is going to take a moment to load that and says contract expected string but stop contained one to three, which is a number. And that basically is pinpointed feedback right there. We are basically Specmatics telling you that you were expecting a string, but you got a number.
You can’t do this. And if you go back to postman and fire this request, you get the random response back. Basically, Specmatic discarded that, so it will never load a stuff that does not match the contract. I’m going to switch this back to ABC one, two, three and let’s just add a new one. So now let’s say we didn’t want to just specify batteries. Something like soap as well. Changes here, changes here. Save that spectrum. So spectric follows along, right? Spectric just watches the file system and loads things back again. Whenever this change, go back postman ten five still works. I realize I have not changed this SKU ID, right? Skuid for Soap and batteries is same. And honestly, I don’t care for the purpose of this demo. So let’s just tell Specmatic you generated for me. I really don’t want to have to bother to change it. I just want to be different every time. Give Specmatic a chance to sort of reload that and then we go back and try this again. This SKU is not randomly generated. We really didn’t care, right? Now let’s say there’s something more towels. We change this to 15 towels anyway.
We didn’t care about SP. We just leave that alone. Life is easy. Now, essentially, you don’t have to care about the things you don’t care about. Leave that to Specmatic and specify the values that you really want to see. Specmatic will do it for you, right. And Specmatic will not load a specific stub unless it matches the contract. Now, let me try something a little different. We’ve seen how we are putting payloads in the response. Let me flip that to the other side. Let’s take a payload in the request. For example, this time we are going to create, right, I showed you there was another API. Let me just quickly check the chat and see how folks are doing. So there was another API for creating products, which is a post. I’ll show you that changing the surround. Now, there was a body section here. So now this is notebook. The SKU ID was something like and the return value was an ID. Now that we’ve created it, we need say ID ten. Let’s just save that. And it loaded it right? It’s loaded. It over here. Notebook is here. So we know that this matches the specification create.
Let’s double check that this is working the way we intended it. I’m going to the Add product request which got imported with postman a moment back notebook and send that. And we get a 200. So that’s spectrum stubbing or something in the request. Let’s try that again. We should get the same kind of feedback in case SKU is a number. Immediately we are told that contract expected string, but stuff contained ten. That’s a number. So that’s wrong. Let me revert that. Let me revert that. I do wanted to have this for a moment spectrum just rewrote that in a second. What if, for example, we passed the number here. So Specmatic has loaded the stub, but we are passing the wrong value in the request itself. What would happen? You’ve seen this message before, right? You get a request for the SDU expected string, but requests contain ten. So Specmatic holds you accountable both when you’re setting the expectation up, but also when the application makes a request to the service virtualization to the virtualized product API. Matic would hold the application accountable there as well. So this this is gonna work. Interestingly. Sorry, forgot to mention one thing.
This and this also comes in handy. This error response came back with a 400 bad request, right? And the fact that it came back with a 400 bad request is useful because we can check the status message and you immediately know that something is amiss. We’ll see how this comes in handy later when I show you a demo of how to use this in an actual test. So this is good. I think we’ve got pretty far. The next thing that we need to look at is all of these stubs that we’ve created here are statically created. Statically, meaning they are files on the file system. Once they are created, they can’t be changed. And when they are created, all before spectratic loads, right? If you create a new one, you create it as a file, spectrum starts and detects it. This is already very useful, but sometimes that might not be enough. Sometimes you might actually want to create a resource on the fly, right? And then simulate the fact that that ID is going to be detected. So, for example, what if I need to simulate the idea ahead of time? What if I need to create if I have a get API for products, for example, and let’s say the ID is something I can’t control.
I can’t control that ID. So product ID is something I cannot control. I really need to create the resource in the test on the fly. And while that’s going on, I need to pick that ID up and send it to Specmatic. Basically, at that point, Specmatic has started, the tests are running. And I didn’t know what the ID was. I need to be able to specify the ID and in fact, this entire content to Specmatic dynamically. So let me show you how that works. Okay, I’m going to take this let me just start this back once again. I’m going to take this to a postman request. Let’s post here, say http localhost 9000 expectations, right? Let me get into the body, get into Raw, right? And here I’m going to tell Specmatic what to or let me let me do that with one of the others, something like this. So I’m going to tell Specmatic, 30 product, 30 return. Something new happens or whatever it is with some random skill, and I fire this here. This was not done with the file. Specmatic returns at 200. 200 just means Specmatic has validated this. The contract validating.
This essentially means we know the contract contains an API with products as product slash 30, right? Spectmic would find the API where ID is a parameter. It would know it’s a get to check that that API has a response that is shaped like this. So this is all validated and it’s accepted, and Spectric returns 200. So when we now go into get products and say 30, we get tablets back. Let’s quickly try and see what happens if we try to simulate something that is not for the contract. Suppose, for example, we want to say SKU ID of 200. Will spectic allow this? Specmatic will not. We say no match was found, error from contracts in scenario, get products, so on and so forth. We’ve seen this error message before. Specmatic is holding you accountable this time. It’s happening in the test. So when the test attempts to set this expectation up with, Specmatic will return a 400 batch request, and your test can die right there. I will show you more in depth how this works.
This is kind of just to add to what Joel just mentioned, right? This is super powerful, because if you have a sequence of tests, right, like you’re making API calls as part of a test, the result of the first API call feeds into the request for the second API call. So obviously, you cannot know ahead of time what to send. So in those cases, these are really powerful. And that’s just pretty awesome demo, which just Joel did. Go ahead, Joel. Okay.
Essentially, this is the anatomy of a test, right? So I’m going to actually take this ahead. I’m going to take this further and show you how this works with an actual piece of code. So you have a test. You have system under test, and you have dependencies. This is typically how it works, right? On a component developer’s laptop, there is a test which invokes a system under test which hits a dependency, the test would be some component test using some testing framework, say, selenium, say, Appium, right? For example, you could be hitting some mobile app or some product screen you would be hitting some dependency. And in the case of the component test here the dependency would be spectratic which would be started up and would be fed the contracts like I’ve shown you before. Now there are three parts to a component test that’s basically what we call a range which is the first piece. Arrange essentially is responsible for telling spectic what to do. This is setting expectations where we are setting Specmatic up so that the system under test can talk to it. And if the Specmatic validates the expectation, if the expectation adheres to the specification then that’s great, it will accept it.
We will act then the test will act by hitting the product screen. Product screen would hit spectric spectrum would return a response and the product screen returns a response which would then be asserted, right? The return value would be asserted by the test and the test actually passes or phase as the case may be. And I’m going to show you how this would work. Just a quick note that anything that has dependencies is a consumer. So the most visible consumer that we have these days is a mobile phone. And so we’ve been showing mobile phones. But that could be a mobile phone, that could be a website, that could be a microservice, right? Microservices may have to make API calls too. And the test framework could be anything. Could be Karate, could be Selenium, could be Apple, could be whatever it is. The main thing that I want to emphasize here is the Arrange, act and assert components of a test. Where Arrange is you set up the service virtualization so that the test can run you act which means that you actually invoke the application and then you assert which basically means that you check the application’s response.
So let’s quickly see how that works. This is a sample Karate test, right? As I was saying, microservices could be consumers too. This is an API test but we are testing a microservice that itself has dependencies. And so the very first step is to set up Specmatic so that the dependency which Spectric behaves as expected. You’ve seen this before. This is the expectations URL. You’ve seen this request, this shape of it at least send a get. When you get a get to slash products with query parameters, type and gadget return this response and then this is the point in the test where we send the request out and then this statement is an assertion. We’re saying then status 200. That means Specmatic would have returned a response. And if the response was 200 that means Spectatic accepted the request. And if the response was not 200 which means it’s a 400, right, has not accepted. It means that for some reason this did not match the contract. And the test dies right here. Specmatics response is locked. Of course the test dies right here. And we don’t move ahead to the act section. So this itself gives you early feedback that the way you say you want spec matter to behave, it’s not how the actual API is going to behave.
We tell you right then and there, the test doesn’t even run the act and the server don’t even run. But assuming that spec matter accepted this, we come to the act section, which means we make the call to the microservice. The microservice internally will call its dependency, which is Spectra. And then once that’s done, we assert in the microservices response. And this essentially is how a typical range act assert test would look. And this helps the consumer to stay in line with the provider. Somewhere down the line. I’m not sure if you would have noticed I started using the Word contract instead of open API specification. The reason for that is that a specification is great because it contains a lot more detail than a Word document. So it’s already a step up, but it doesn’t actually hold either side accountable to staying in sync with it. But once you start executing the specification as service virtualization on the consumer side and as contract tests on the provider side, at this point, you are actually using it to hold both sides accountable. And now it starts functioning as a contract. And so that’s why we start thinking of them as executable contracts.
So with that, I am now going to end this session. Back over to hurry. Hurry is going to show you how it works on the provider side. And I’ve shown you a good deep dive of how service virtualization works to hold the consumer accountable to the contract. The next step is how do we hold the provider accountable to the contract. Over to you, Hari.
Thanks, Joel. That was a super nice deep dive into the service virtualization work. Let me just present my screen. All right, I think the important part to understand here is not just about stubbing, right? Like what Joel earlier showed, you have two sides to the coin here, wherein unless you take the same specification and make it as a contract for stubbing out the provider on the consumer side and on the provider side, we need to run the specification as a contract test. And that’s how we keep the two sites balanced, right? So I’m going to go do a deep dive of the contract as a test approach now. So let’s look at what we’re trying to achieve. So this slide is already familiar to you from earlier when Joel mentioned this. Like essentially Specmatic is able to pull an open API specification, convert it into a bunch of contractors, and based on that, generate Http request and verify the response. And this was done with an existing provider application, right, the Jar file, which you guys tried. Now what if the application does not exist in the first place, right? And we have to begin to build this from scratch.
So how about we attempt doing some live coding and we let Open API specification and Specmatic guide our development itself for the provider. Let’s try it. Okay. So by now, I think you must be all too familiar with the specification. It’s got just one endpoint here, which is the product ID, and there is the Get operation on it. And it’s supposed to give you the product pack. And I will just repeat the command which Joel already ran earlier. Just so we are all on the same page here. And connection refused, obviously, because the application does not exist, we haven’t built it. So let’s start building out the app. So I’m going to be using Springboard for this today. However, Specmatic itself is language and technology stack Agnostic. It works on top of Http, right, in this case. So it really does not care in which stack you’re building your provider application. It is still able to test it out and give you feedback. So, as a good first step, I think I want to show you that this is a blank Springboard application that I just generated out of Start Spring IO. It’s got nothing here, just one empty controller.
And I’m going to boot it up. And once the app is running, I’m going to go back and repeat the command again. spectic test this time. What do you anticipate? There’s no connection refused. However, it says, I was expecting a 200. Okay. However, I got back a 404 not found. And this is understandable, right? Because we do not have any path for this URL. We don’t have this Get operation at all defined. So let’s go ahead and define it. So I’m going to copy paste in some of this code, which I have as a snippet, put that here, and like any good developer would do, I’m going to start with hello world. This is a perfectly valid endpoint and I can reboot the app and then see what happens. However, this is becoming a little bit of a hassle, right, which is keeping on restarting the application, running it from the command line. Is there a more integrated approach? Yes, there is. Specmatic provides JUnit support. JUnit five support. And note that this is a test implementation dependency only in Gradle, which means it does not have to ship as part of your production code.
It’s only in the context of your test code. Right? So added this dependency in and then set up contract as test, which you are also familiar with from earlier, which Joel showed. I have just extended the spectrumatic JUnit support and pretty much just done simple plumbing, right? Start the app and stop the app and set up and tear down respectively. Apart from that, whatever is in the command line here, the coordinates to where the application is running and the location of the Open API file itself. Those are the pieces of information I’m providing Specmatic through system properties. That’s pretty much all. So now all I need to do is run this test and that should be good enough. So let’s try it out. I’m going to switch to running the contract test and run. And like earlier, Naresh said, this is money for nothing and test for free. So free tests. Who wouldn’t want one? And this time around we don’t see the expected 200 but got 404. This time it’s a different issue, right? It says I got a 200, but instead I was expecting a JSON object. But you gave me hello world.
That makes sense, right? Because our API specification says we’re looking for a schema that looks something like this product, which has got a name and an SKU and a Hello, I’ve returned as a dumb hello world. That’s not necessarily helpful. So let’s actually comply with the specification now. So I’m going to drop in the product data object and try returning something that complies well with the specification. So I’m going to say I will return this product, which is a book with this SKU. Looks about. All right, everyone following along. Okay, so let’s see. When in doubt, run the test, right? Always. Let pass hooray first. Green for the day. Okay, so which means you have potentially gone from zero code to some code. And this whole thing was guided by the open API specification. There is an important point I want to call out at this moment, which is, look how Spectric really does not care whether you actually wired it up with the database and pulled the information out and then returned it. Specmatic is only worried about your signature, your API signature, right? So which is why when you’re talking about contractors, you have to understand that contractors verify your API signature and therefore they are not a replacement for component API tests which are actually about logic.
And there’s an important differentiation that you need to understand that this just separates the concern of verifying the signature and gives you early feedback. This supplements and sort of increases your ability to find out issues early on without actually writing too much code, right? If your signature itself is not in line, then why bother writing the rest of the logic? I just want to call that out quickly and then let’s get back to our development. Now this is not interesting. This is like very simple app. The fact that you have been able to get to this point guided by specification is pretty good. But then I want to make this app a little bit more complicated and real life, like so in real life, let’s say you would have a service call to fetch the data from database and then return the product. So what if your seed data or your test data in your database only has product ID two and nothing else? However, Specmatic is just auto generating the product ID, right? It’s just a number. And like what joel already showed it could be any number that is sent. And you cannot potentially have all those random numbers sitting in a database.
Right? So let’s assume that’s going to happen. So if your service says if the product ID is not found in my database, I’m just simulating that if it’s not equal to two, assuming I have only a product with ID two in my DB, then I’m going to throw a runtime exception. Yeah. So the service is doing its right thing. Like, I don’t know what you’re talking about. For every random ID that you give me, I cannot give you a product. So I’m going to throw this error. So what do I do now? Let’s run the test and see what happens. It failed. What do you guess will be the error? 400? No, it is a 500. Oh, my. You’re not supposed to get 500.
Right?
It’s not a good place to be. It was expecting a 200, but got a 500, obviously, because it’s an unhandled exception. The web framework your spring handled the caught that error unhandled exception and then converted that to a 500 and returned it to us. So now the issue is Specmatic does not know that it needs to send two because that’s the only row available in the DB. So we need to help spectic by giving it a clue that, hey, don’t send a random number, send a number that’s in my DB. So how do I do that? I can use pragmatic a clue through open API examples. So what I can do here is say for a 200 okay response, I want the value to be two. And there’s one more thing I’m going to do, which is for this particular request where I’m sending the product ID as two, I want a corresponding response also. And I’m going to say this guy right, remember this data. This is the exact same data that you saw in the controller here, book 123-12-3123. And that’s what I’m saying here. And notice how I have named in line number 24, 200.
Okay? And line number 39, 200. Okay. Both these examples on the request and the response right. Are named same open API as such, does not really correlate examples, but Specmatic is able to correlate and give you that convenience, which is to say, for this request, I want this response, and I’m going to verify it. So will this pass? Let’s try it out. You went from red to green to red to green. It’s a good rhythm to be right. However, I’m not happy with what happened earlier, right. Which is we saw an ugly 500, and that’s not a right place to be in. Whenever there is a not found error, you should potentially what is the right error to be throwing? You should be giving back a 404 status code. Correct. So let’s make that change. So where do I think I should make it? Should I start writing the code right here doesn’t make sense, right? We are driving the implementation through the specification. So first I need to make the API design right. So I’m going to put the four or four response here into this guy and say apart from the 200, I also have another response for when there are no products found with that particular detail.
I’m going to give back a four four. And this is my error response situation. It’s going to give me my status, the error type, and what is the path that the issue happened on. There is a problem now, just like I’ve given an example for 200 and I said send ID two now for 404. Specmatic does not know what to do, right? So you know the drill. It’s just a clue. I need to give it, right? So I’m going to say give it a meaningful name and set up an example that in order to generate a four or four response, send the idea zero. Not likely that zero is going to be in my DB, right? So I’m just going by that and you also guessed it right. I need a corresponding example on the response so that we know what to map it to. This time around, I put the example here and I say four or four earlier. If you noticed, I put in the exact details of the book they SKU. This time around, I don’t care because it’s error, but I just definitely worry that it is in line with what is the actual schema.
So I’m just setting up the same data types here. So I have done this specification change, which means now my open API operation looks something like this, which is I have products, I have get, and I have two responses, 200 and 404. We have the example set up for it. All right, let’s run the test now. What do you expect will happen? Red again. Okay, this time around it’s an interesting error. Expected 404 got 500. Why do you think this happens? Because Specmatic, rightly, sent out zero right for the second example, but expected a 404. How have we got a 500 and why is that happening? Quite obvious. We haven’t written code to handle the not found exception. Right. So how do we do that in springboard? Fairly straightforward. I just need to create an exception class and map it out. I’m going to do that straight away. I put in product not found exception, and when the product is not found, instead of throwing a runtime exception, I’m going to throw a product not found exception, which is indeed the right thing to do, and thereby I give a spring a clue that I’m throwing this.
So if you handle it, return a not found, which is a 404 Http status code. What do you think will happen now? Green. Excellent. So we went from zero code to having actually build out the basic application, realizing that we have a 500 and then designing the API in the process. Like, we went ahead and added the 404 response and only wrote the code after we wrote the specification. Isn’t this very similar to test driven development? And specifically this practice called tracer bullet approach, right, where you use the test to flesh out your system? It’s a very powerful way of thinking about it. And just like how test driven development is not about testing, it’s about designing your code. Contract driven development here is not about contract testing your API. It’s about designing your APIs well and designing your architecture itself. Well, that said, I’m not saying that this is the only way to write code with the contract driven development. I just am particularly fond of writing tests for code. And I like this traceable approach because it gives me a nice design longevity for my application. Whatnot Specmatic as a tool itself is flexible to fit in with any setup.
For example, when Joel showed earlier, it worked with an existing application also, right? But with contract driven development as an approach, I would personally recommend that you try this approach. And it’s definitely quite interesting to learn from and see what sort of mistakes we sort of make and how it can guide us through designing better APIs and eventually designing better applications. So that was Trace, a bullet with open API specifications. And let me do a quick recap. What we did is we started with the contract as a test. We showed how we can use Pragmatic JUnit support to have really quick iterations. And then we showed how we can use open API examples to give clues to open API so that it sends the right kind of data out to this system so that according to the test data, it behaves well. And very important point is that contract tests are not a replacement for your components. However, they are super important because even with API tests today and component tests today, you still have integration issues identified only in integration environment. Right? Much later. But contractors are going to find that much earlier in the cycle and thereby have them both.
They’re not a replacement for each other. All right, so with that, I’ll hand it over to Joel for another interesting topic. Over to you, Joel.
Thanks, Hari. So we’ve just seen Hari show us in a very interesting demo how we can keep the provider in line with the contract. This means we have seen two sides to two pieces to this so far. How do you keep the provider in line? How do you keep the consumer in line? That means that given a contract, we know that the two of them can stay in lockstep. What we now need to take a look at is what happens when you have to change the contract. The contract can also be changed. This can happen for a variety of reasons. There might be some business existencies. Will change, new pieces will have to be developed, the contract may have to accept new APIs or accept new input, so on and so forth.
Right?
And what is the number one fear that we might have when you change the contract? You’ve got everything working today, you’ve tested, you’ve got your consumers working. Providers and consumers are integrated, everything’s working fine, we change the contract. Is that going to break consumers today? And that is a founded fear. And I think firstly consumer and provider will stay in sync with the contract and that’s great. But if you only want to make this change to the provider, you want to make a change only to the provider. You want to make sure that consumers don’t break as a result. How do you make that change? We don’t want to touch the consumers in the environment. This is called backward compatibility. Backward compatibility means I’m changing the contract in a way that would not break consumers. This means I don’t have to touch my consumers at all and then my provider changes to match the contract. So the first step in that process is make sure that the contract itself is backward compatible. Just quickly revisiting what this looks like without a contract is consumer would send some request to the provider. The provider is updated, the provider sends back maybe a response that the consumer does not understand.
Consumers break.
Right?
You discover this feedback in an integrated environment. What we have shown you is maybe you can discover this feedback while the developer is building the application. Right? But that still doesn’t serve the purpose because in fact we don’t want anything to break at all. We would ideally like to make a change that the provider can change, can gain new functionality and consumers don’t even break. That’s desperate work for everyone. And this means that we got to figure this out even before handing the contract off to a developer. And we have an interesting way to do this which I’m not going to demo right now. But it essentially turns out you can run contract tests out of the existing contract. We’ve seen how that works. You can run service virtualization out of a contract. We’ve seen how that works as well. But what if you run the two of them against each other? So you take the contract of the existing the existing contract basically run contract test against the changed contract. And the reason this is interesting is contract tests simulate the provider. So contract test running from today’s contract means this is how today’s consumers sorry, my bad.
Contract test simulate the consumer contract testimony the consumer. And this means that contract tests running out of an existing contract simulate the consumers as they would look today. Service virtualization simulates a provider. Right? And if you run service virtualization from the changed contract, this is how a provider would look with the updates. And so when the contract tests from the existing contract pass against a Service virtualized new contract, right, service virtualization or a stub of the new contract. It means that existing consumers will understand the provider even after the change of the contract, which automatically means they are backward compatible. Essentially, we are running contract versus contract. This is a pretty interesting approach. We haven’t seen anyone do this report statement pending. And by doing this, it is possible to identify backward compatibility problems even before the contract changes reach the developer. And we are therefore able to make sure that backward compatible changes alone reach the developer and the consumers don’t break when these changes are made. Of course, there may be reasons to break backward compatibility, but then that will be an explicit choice. And Hari will talk later about how to handle this in a seamless way.
Just to add to what Joel was mentioning earlier on that slide, the Contract contract is really powerful, like what he mentioned is because when you are just verifying the backward compatibility with zero code, how much does it cost? Really? If you think about it, you don’t even have to have your provider or consumer change a single line of code. All you’re doing is just verify if the contract changed itself. You could experiment with it. What if I change this? What’s going to happen? Am I going to break backward compatibility? You could ask such questions to Specmatic and Specmatic could tell you those. That’s why it makes it interesting again. Yes. Just wanted to add that point to what Joel was sharing earlier.
Yeah. Thanks, Hari. And with that, I will hand it over back to Hari for the next section, which is contract as code. Over to you, Hari.
Thanks, Joel. All right, so very quickly, so that we are all well into this workshop right now and you’re in the deep, right? So we have done three different three major things, right? Which is we’ve done Service virtualization, which is Contract as stub, we’ve done Contract as test and we’ve done contract versus Contract. Essentially all in the interest of making sure that we do not break compatibility between the consumers and providers. That’s the goal. Right? And that’s the problem statement with which we started. Also, however, there is a fundamental practice or a fundamental aspect which we need to consider. If not, that could break everything and bring us back to square one. How is that? Let me just go over that. So right now you have the Open API specification practically acting like the glue between the consumer and the provider while they are developing in isolation. And there could be a potential situation where, let’s say I am the provider engineer, I made a change to the provider application. However, I forgot, for whatever reason, to upload the latest version of the Open API file into the location where it is shared. Right? Or I may be the consumer engineer and I’m referring to a stale version of the.
Open APS type. Maybe the provider engineer emailed it to me. I did not notice the latest version. I’m still referring to the old one, and thereby I still am on the old version of the truth. So what happens? Isn’t this looking very familiar to the initial slide that we shared? We’re back to square one, right? Like we are on different pages. So how are we going to get beyond this? All the fancy contractors testing contractors stubbing, and still this doesn’t make sense, right? The only solution to this is if we start treating Open API as a single source of truth, or any specification to that matter should be treated as a single sort of source of truth and stored in such manner. Why I’m calling this out very specifically is because with teams I’ve worked with, I’ve seen scenarios where you store certain you share Open API over emails or you potentially have it in some shared folder and we look at it and there’s not much rigor around it, right? What we found in our experience is the best place to store Open API is a version control system. And in our case, we’ve been using git.
And what better place to store it? Open API is code, right? It’s machine parsable and it rightly belongs in a version control system. And if you are choosing to keep your specifications in a central location, in a version control system, across teams and across organization as a single source of truth, you also want some process around it in order to get it there. So the process that we’ve been following on some of the teams is to first have a style check or a length check on the specification itself to see if it’s, number one, adhering to the industry standards, and number two, if you have any specific standards within your organization. Are you in line with it basically circumventing some of the manual review of all these processes. We try to codify the review as much as possible into a Lint or a style check process. We use Spectral as a tool for this, but it’s not necessarily like the only tool we recommend. But any tool that can do a style check is a good idea. And once you pass that basic point, then comes the backward compatibility contract versus contract checking, which Joel just spoke about.
And what Specmatic does here is it just needs the two version two files, right? Like to say this is the old file, I have this API, and this is a new file with a minor change in it. And I’m going to compare and say if the change is backward compatible or not. Yes or no? It’s a binary answer, right? So in this case, it’s not comparing two files, but rather because it’s a pull request or a merge request. It can take the version of the file that is modified from the branch and it can take the corresponding specification file from the central repository, run a zero code comparison like wherein you don’t have to write any code. Stagnate is going to run the comparison for you and let you know if it’s backward compatible. If and only if that is compatible, then you move on to the next stage in the process, which could potentially be a manual review if needed. And then you merge the pull request and thereby the change flows into the central repo. So now you must be asking what happens if the specifications are not backward compatible? Sometimes we do need to make a change to our API which is going to break backward compatibility because we want to evolve the features, right?
So that’s when we sort of version bump to communicate. So the way we version we’ve been trying to leverage the semantic versioning practice where I’m going to use the major version in the file name itself to say this is products 20 to 30 because it’s a completely backward, incompatible change and it’s communicated very clearly to the consumer through the versioning and a minor version of Grid. Is it’s a compatible change nevertheless? But we do want to communicate it, right? So I could go from 20 zero to two 10. And patch version is mostly indicative that there’s a change, but not necessarily in behavior like a structural change, right? So if you have a large Open API file, it’s very common practice to extract some common data structures into reusable components inside the component section in this schema. So if you’re doing only structural change and no practical change to endpoints or their behavior, then that could be a patch version upgrade. And this is again just a recommendation like this is what we’ve been following with some success. But that said, it’s up to individual teams and organizations in terms of how to manage it.
With that, I just want to show you how contract repository itself looks. So, have three applications here. This is a sample project on our GitHub. We’ll share links later. So we have the order contracts which are the central repo, the API which is the provider, and the UI which is the consumer. Just the three participants in this thing, right? And the order contracts itself is a fairly straightforward repository. We organize our files more like Java package names or C Sharp package names, just like how we manage code. Because that way instead of having all the Open API specifications flat in the parent directory, having some sort of package naming gives you that control of how to locate easily. So that’s what we do here. And if you look at the files themselves, this is all very familiar to you. This is open API file. And then you have the corresponding static steps which Joel demonstrated earlier. So we have all of that sitting here and that’s your central contract repo as an example. Just wanted to show you that. Now back to our deck, right? So if you have your code in the central repo and then you have your consumer application and provider sorry, the consumer team and the provider team sitting and developing on their local machines, how do we reference this file, right?
Like every time, do we have to get clone git clone? Is that going to be scalable? It is possibly. But then for the purpose of convenience and also to make sure there is consistency and correctness in how this works, Specmatic has this config file called Specmatic JSON, right? And all it does is it kind of gives you a list. You can put in what are the Open API as that your application needs because it’s the central report to an organization, at times there could be hundreds, if not thousands of specification files sitting there. So you don’t want to list everything. You don’t want everything on your machine. You just want what interests you as a microservice or as a smaller application. You just say, I want this, this and this. And then Specmatic will fetch it for you and make it available to you on your local. And how does the Specmatic JSON look? Let’s take a quick look. So, going to show you the syntax that really matters in the context of this workshop. So, first thing is the coordinates to where the repository itself is. Here we are saying that the repository is a git repo.
And here’s the location. It’s an azure git. Repo. And then we are saying this contract is to be used as a test and this is to be used as a stub. And why is that important? Again, calling your attention to this, we had these three projects, right? The API being the provider and the UI being the consumer. And the Order YAML is going to be used differently by both of these people, right? The UI obviously wants to stub out the one moment, the UI obviously wants to stub out the provider. So thereby it will try to list it under the stub category, right? It says, I want to stub the Order API and that’s why it’s listed under Stub. And for the provider, which is the API, it wants to run the order specification as a test. So that’s why it’s listed under the test section. So that’s very quickly into how a Specmatic deeply integrates into your setup. And you don’t really have to point to files manually. Specmatic will do all that for you and give you the convenience of it. All right? So with that knocked out, let’s get into the details of it.
And what does it take that we’ve been running all of this on our local machines? How can we embrace contract driven development? And how does it affect your CI pipeline itself? So, we know for now, the facts are that Open API or any other specification to that matter is sitting in a central repository. So it’s a single source of truth. And Specmatic is reading that at the top you have the consumer team, which is the View product screen, and at the bottom you have the Products API screen, which is the provider team. So Specmatic, as you already seen on the local environment, is able to stop the provider for the consumer and run the contract as a test for the provider, right? So now what remains is what happens in the CI. So for the consumer, you run unit tests. As usual, nothing changes there. But in the component test section, where the dependency comes into picture, in order to support the dependency, all you need to do is use the spectrumatic dependency, which is the contract, as a stub, which you have been using on your local. So it’s pretty much the same. Specmatic is just a jar file, right?
So it runs pretty much anywhere. That’s the only change on this consumer side. On the provider, again, you run the unit test. However, you then run the contract test before you run your component test, right? Obviously, you want to verify signature before you verify logic. And with that, you can deploy with confidence to the integration testing environment. And you know for sure they’re going to be working well together. And that means you have an unblocked path to production and a happy user. And from the heat map point of view, you are identifying bugs much earlier in the green part of the heat map. You’re not identifying the issue here, you are identifying it on your local or worst case, maybe CI. So that’s pretty much how you embrace contract driven development. With that, we open it up for QA. Thank you all for being a very patient audience.
Thank you so much, Hari and Joel, for such an insightful session. It was really very knowledgeable session. So thank you so much for sharing your insights with us today. And thank you audience, for having this patience because it was such a long session. But I think they managed to connect us with this session and we all were able to get these concepts very well. You.