Search for a command to run...
Professional Services Team Lead
AWS Solutions Architect
Well Architected Team Lead
Certs:
AWS Certified Solutions Architect - Professional
AWS Certified DevOps Engineer - Professional
AWS Certified Solutions Architect – Associate
AWS Certified Developer - Associate
AWS Certified Cloud Practitioner
ITIL v3 Foundation
SVQ 3 Management
No comments yet. Be the first to comment.
I'm in the process of working through some Machine Learning Training at the moment, with the aim of taking the AWS Machine Learning Speciality Exam and I've had some thoughts on Machine Learning training and learning as a whole that I'd like to shar...
Welcome to Part 3 of my EC2 section of my Intro to AWS series. This is a continuation of part 1 and part 2, which you can read here: https://blog.johnwalker.tech/elastic-compute-cloud-ec2-intro-to-aws-series and here: https://blog.johnwalker.tech/ela...
Welcome to Part 2 of my EC2 section of my Intro to AWS series. This is a continuation of part 1, which you can read here: https://blog.johnwalker.tech/elastic-compute-cloud-ec2-intro-to-aws-series. In this post, I will continue with some topics I me...
Keeping track of which external software libraries you have used in your programs and what the requirements are for running your code is a key part of good coding. Being able to get your code up and running on a new install or just knowing which depe...
Microservices are a software development pattern that breaks down a large application into smaller independent services.
• Small subsets of functionality but self-contained
• Size is variable but should generally be small enough to fit in a single developers head
• Functionality of the Microservice itself shouldn’t depend on other Microservices. User flow within the application should use multiple microservices
Some examples from Amazon and Netflix of Microservices and their communication paths.
Microservices, like other software development methods, have had various ways of working developed over the years and choosing the right method of development that matches your business needs is critical.
Event-Driven Microservices
Domain Driven Design
Communication Methods:
API Communication
Decoupled Messaging
Pub/Sub
Microservices have their own design considerations, but they also come with several benefits.
The team size and codebase for each microservice can be small. It should be able to be entirely understood by one person.
Focus on one specific business outcome or aspect and perform 1 task
Development autonomy. Each microservice is separate, so having different programming languages, technologies, and development methods is possible.
Allows alignment with the bounded context of the problem domain. The problem a microservice is trying to solve can align with business outcomes.
Loosely coupled architecture, allowing for separate scaling and consideration of each microservice.
Microservices are a good software development model for many use cases, but they are not always the best option.
It's important to consider the best method for your software development. Sometimes a hybrid of monolithic development and microservices is the right option.
Microservices should be independent single tasks. In some cases, this isn’t possible, for an overall task to complete, multiple tasks may need to be completed, and they can’t easily be separated.
Separating services adds some latency as each microservice needs to communicate via API or Events to proceed to the next step. This might be a blocker in latency-sensitive applications or even cause data synchronisation issues.
With distributed systems, debugging and tracing becomes more complex
There are methods for resolving or mitigating many of these issues, but these are also considerations when deciding on a software development method. Microservices give many benefits, but there are challenges and complexity considerations.
One of the ways Microservices are used is through event-driven design. In this pattern, a producer is either a microservice or some other source of events, ranging from S3 to 3rd Party applications, that produce an event when something happens, providing a stream of events.
Once these events are produced, they can be put into a queue or an event bus, and other services or microservices interested in dealing with that event can process them.
This can take various forms:
Single queue, where an event is produced and consumed by usually a single target
Pub / Sub pipes or Event buses, where all events are put into a pipe, and multiple different interested targets can process the events or be notified of the event happening
Streaming, where all events are passed through a data stream where they are processed as they pass through, usually in chunks of events rather than one at a time
Single Queue:
Streaming:
Pub/Sub:
Eventbridge is AWS’s service that provides an event bus that can be used to handle events from various sources.
It can be used to:
Send requests to multiple consumers
Create an event lifecycle record
Integrate between different components, including 130 event sources and over 35 targets, as well as third parties.
Create cross-account event pipelines by sending events to Eventbridge buses in other accounts.
Provide filtering of events to trigger consumers.
Provide a schema registry for events to track and version events, allowing for producer-consumer contracts.
Driving microservice design takes the same amount of consideration as other software development methods, with relevant requirements for both your business and application to be considered.
Required scalability and use (Are you over-architecting something with 20 microservices used once a year?)
Business structure – can your development and operations team support a move to microservices?
How closely aligned to your business processes should microservices be? Are those processes well-defined?
Do you want to follow the domain-driven design, where your business needs define the outcomes of the application and its boundaries?
Is your application suitable for microservices, or are there parts that should be in a distributed monolith?
Each microservice, whether API-based, Event-Based or otherwise, should adhere to a contract for each version so that consumers of the event or requesters know the expected inputs and outputs for communicating with the microservice. Contracts should not be changed in a specific version. Any required changes should require a new version, and old versions should run alongside to allow consumers to update to the new version. Breaking changes in microservices have a larger impact than in other development types.
While microservices can provide a whole host of benefits, there are potential pitfalls to watch out for when developing microservices.
Assuming that just because you now use microservices, it solves all your problems
Making microservices the goal in itself, rather than using them as a tool to solve business problems. It's no use changing everything to microservices at the expense of developing features and fixing customer issues.
Piecemeal adoption, having just one team work on one aspect without co-ordination isn’t likely to work.
Uncoordinated adoption. If teams creating microservices don’t collaborate, different approaches and implementations could result, such as one team expecting streaming and another expecting pub/sub.
Focusing more on the technology than the overall service design and how the services communicate
Not changing organisational processes or structure to follow microservice patterns. For example, moving to microservices won't work if you still do six monthly deployments.
Not separating tasks correctly and just building multiple monoliths instead of single-focus microservices
Planning how your microservices communicate is a key step when moving to microservices.
Above is an example based on architecture by AWS, where the planning method has two main sections:
Planning what the microservices are and how they fit together by looking at the following:
What input events trigger this microservice
What the service does
What actions the service takes
What output events the microservice produces
Then you can drill down into each event and list:
What attributes are in each event
Which service owns/produces which event
What the action that each Microservice was part of
Eventbridge Scheduler is a newer service from AWS that allows you to invoke tasks based on defined parameters. This allows you to perform tasks within your application and scheduled background tasks.
Integrates with 200+ services and over 6000 AWS API calls
Schedule tasks such as customer reminders, delayed actions, prompts to continue if the customer hasn’t performed an action
Perform cron-based tasks, one-time events, fixed-rate events, time-specific events
Add retries, dead letter queues and more
Microservices are designed to be loosely coupled, in terms of their dependencies on each other, so that each service can act independently of another and not have to consider what is happening in other services.
However, this isn’t 100% the full story. Microservices are still coupled in certain ways, but mainly through semantic coupling.
This means that a microservice depends on events coming in being in a specific format so it knows how to deal with them, and similarly, any microservice that consumes events produced by another microservice needs to know the event format won't change.
Service Contract – events produced by a microservice shouldn’t change without warning, so contracts are important, with versioning being key
Schemas should be available for each event so that other consumers of these events know what the format is and what data is available.
AWS Eventbridge uses a defined format for Events to be put onto the event bus.
An example of an event is shown above, and it is a JSON format document with specific minimum information to be considered valid:
A detail JSON object. (Can be blank, using “{}” )
A detail-type string identifies the type of event
A source string identifies the source. This cannot start with 'aws', but other values are ok.
However, adding in extra fields is recommended:
Id – a unique id per event
Version – event version number
Time – time the event was created
And more detail to help identify the event and provide the information required by the consumers about the event.
https://github.com/jbesw/eventbridge-sam-example
This GitHub repo from James Beswick – Principal Developer Advocate for AWS Serverless demonstrates setting up an event bus, and multiple microservices that outputs an event and consumes it, using the AWS SAM framework.
AWS Samples:
https://github.com/aws-samples
Building Event-Driven architectures on AWS Workshop:
https://catalog.us-east-1.prod.workshops.aws/workshops/63320e83-6abc-493d-83d8-f822584fb3cb/en-US
Microservices.io: (Patterns, explanations of terminology and more)
https://microservices.io/index.html
Software Architecture: The Hard Parts:
https://www.oreilly.com/library/view/software-architecture-the/9781492086888/
Building Event-Driven Microservices:
https://www.oreilly.com/library/view/building-event-driven-microservices/9781492057888/