INTEGRATE 2019 – Day 2 | byBrick Development
Tuesday 4th June 2019
The day 1 was packed with lots of information, let have a look at what day 2 had to offer.
08:30 – 5 tips for production ready Azure Functions – Alex Karcher, Program manager – Microsoft
The day 2 started off with a presentation from Alex Karcher where he shared five major tips which included,
- Serverless APIs & HTTP (premium plan) scale
- Event stream Processing and scaling
- Options for Event Hubs scaling
- Inner and Outer loop development ( Azure DevOps CI/CD)
- Monitoring & Diagnostics
- Application Insights ( easy to integrate with Functions )
- Distributed tracing
- Application Map trace and diagram ( view dependencies)
He also spoke about the consumption plan and enabling Auto scaling in premium plan has more control than App service plan.
09:15 API management deep dive – Part 1 – Miao Jiang, Product Management – Microsoft
He spoke about the automation challenges with API management which included below 2 bullet points –
- How to automate deployment of API’s into API management
- How to migrate configurations from one environment to another
He suggested an approach how you can build a CI / CD pipeline using ARM template. He also showcased deployment of Food Truck application to API management.
Creator Tool – It was build to generate an ARM template so that you can be used to deploy it to API management
Extractor Tool – Which can be used to extract existing data from published API’s. He used VS Code and API management extension (private preview) to show case a demo. You can perform all the necessary operations event without moving to Azure portal.
- Use separate service instance for environments
- Developer and Consumption tiers are good choices for pre-production
- Templates based approach is recommended
- Consistent with the rest of Azure services
- Modularizing templates provides wide degree of flexibility
- Access control, governance, granular deployments
09:45 Event Grid Update – Bahram Banisadr, Program Manager – Microsoft
He started the session by speaking about Why is there a need for Event Grid and also explain the basic’s related to its working.
What’s new –
- Service Bus as an Event Handler (preview)
- 1MB Events Support (preview)
- IOT Hubs device telemetry events (preview)
- GeoDR (GA, Generally available)
- Advanced Filters (GA)
- Events Domain (Bundling of topics)
- 100,000 topics per Event Domain
- 100 Event domain per Azure subscription
He also presented a Case Study on Azure Service Notification’s.
What’s Next ?
- Remove work arounds
- Greater transparency ( proper diagnose/debug functionality)
The session ended with a Quick demo on event grid.
10:45 Hacking Logic Apps – Derek LI , Program Manager – Microsoft | Shae Hurst, Engineer – Logic Apps Microsoft
VS Code for Logic Apps –
- Create or Add existing logic App
- Automatic creation of ARM deployment template
- Azure DevOps integration
Tips of Trade
- Sliding window trigger
- Run against older version
He also suggested to when should we go for Inline Code Vs Azure Functions – If you code takes longer execution time, more than 5 seconds use Azure Functions.
He also shared few tips on how to avoid 429s (throttling)
- Use Singleton Logic App to call connector to avoid parallel branches/instances fighting over rate limits
- Use different connectors per action/Logic App
- Use multiple connections per action
What’s on Derek Mind ?
- More Inline & VsCode
- New Designer
- Better Token picker
Emojis generator connector
11:30 API Management: New Developer portal – Mike Budzynski, Product Manager Microsoft
This was the session where he shared the news about new developer portal which will available for all the users on coming Wednesday (12th June 2019). He gave a quick intro about the portal which is either used by the API Providers (design, content editors etc) or the API consumers (app developers).
Some key points to take note off –
- They Portal is built from scratch
- It has a modern look and feel
- Its Open source and DevOps friendly
He also showed a quick demo of the new developer portal. The Look and feel resembled with the Integrate 2019 website.
You can get more information related to the developer portal on the link below https://github.com/Azure/api-management-developer-portal
13:00 Making Azure Integration Services Real – Matthew Farmer, Senior Program Manager – Microsoft
By 2022 65% of the organizations will move to Hybrid Integrations. There are 4 different integration scenarios
- Application to Application
- Business to Business
Each of them have different integration challenges and he quoted few of them which we may face working with these integrations. ( Different Interfaces, Cloud or On-prem, Service oriented, distributed etc).
IPaaS – Below are the 4 key integration components for building a solution
He also showed a quick demo related to Processing Orders and also showed how the all integrate and work together.
https://aka.ms/aisarch (Basic Enterprise integration on Azure)
Free Whitepaper (Azure Integration Services)
Few Slides where he spoke about BizTalk to Azure Integration Services
- There isn’t 1-1 mapping
- Adopting cloud paradigm requires a different approach
- Many new concepts to take advantage of
- API Economy
- Reactive code
- Many Assets can be transformed from BizTalk to Logic Apps easily
- EDI Agreements
All of the above can easily be moved to Log App Integration Account
- Orchestration and Pipelines can be remodeled in Logic Apps
Tricky things that is hard to move,
- BizTalk implementations with huge code bases
- Lots of rules engine (sometimes)
Buy in to the vision of Azure Integration Services
- API Economy
- Logic Apps as application ‘glue’
- Serverless – or dedicated look at integration services
- Pay as you use
Identify use case, Design a target architecture and Create a migration plan.
Making it Real
- Understand the principles
- Strategy over tactics, value over cost
- Build a migration plan
- Don’t under govern
- Don’t over govern
- Think about culture change
13:40 Azure Logic Apps Vs Microsoft Flow, why not both ? Kent Weare, Microsoft MVP – Business Application
The session started off with explanation with regards to Microsoft Flow features.
- ISaaS (Integration Software as a Service)
- Azure Subscription not required
- License entitlement available through Dynamics 365 and Office 365
- Additional standalone license available
- Part of power platform (Power Apps, Power BI, Microsoft Flow)
- Deep Integration with PowerApps
- Over 275+ available connectors.
- Custom Connectors, Standard and Premium (P1 required)
- Cloud and On-Premise
- Tracked in Common data service
- Custom Approval options
- Can respond from
- Flow Approval center
- Flow Mobile application
- Microsoft Teams
- Graduate flows to Logic Apps with few conditionals apply
Later he jumped to describe some features for Logic Apps
- IPaaS (Integration Platform as a Service)
- Azure Subscription required
- Consumption based billing
- Part of Azure Integration Services platform (API Management, Service Bus, Event Grid)
- Over 275 + available connectors
- Around 95% symmetric between Flows and Logic Apps
- Enterprise connectors ( SAP, IBM MQ, AS2, EDI EDIFACT)
- 3rd party custom connectors + custom connectors
- ISE – Vnet support
- Cloud and On-premise
- Editing Experience
- Web Browser
- Visual Studio 2017/2019
- Azure Logic App tools for Visual Studio 2017/2019
- Enterprise Integration Pack
- Visual Code
- Continuous Integration / Continuous Deployment
- Azure DevOps
- Integration Pack
- Integration Accounts
- Typed Schemas
- Flat File encoding/decoding
- XML Transformation
- BizTalk tooling
- JSON Transformation
- Third Party management
- Partners and Agreements
- ISE (Integration Service Enviornment)
- VNet connectivity
- Private static outbound IPs
- Custom inbound domain names
- Dedicated compute & Isolated storage
- Scale In/Out capabilities
- Flat Cost – whether you use it or not
- Extended limits
- Webhook integration with Logic Apps for event orchestration
- 3rd Party support for Serverless360
As we can see there are lot of shared capabilities between Microsoft Flows and Azure Logic Apps. There are also some subtle differences between the two, but they can play a significant role in determining which is the best tool for the job. Ultimately, tooling should be selected based upon Organization design and the complexity of the requirements.
The Winner is …
- The Organization which leverages both tool to address their needs
- Innovation doesn’t happen while waiting in line
- Implement governance and education that allows your business to scale
He later shared a link to his blog post along with links to Middleware Friday and Serverless Notes
14:20 Your Azure Serverless Applications Management and Monitoring simplified using Serverless360 – Sarvana Kumar, Founder – Serverless360 / Microsoft MVP – Azure
He started off by sharing an Agenda for his presentation
- Management of your serverless Apps
- Improving DevOps for your serverless Apps
- End-to-End tracking for your serverless Apps
- Customer Scenarios
What is Serverless App ? Similar to LEGO blocks
Azure Logic Apps, Azure Functions, Azure APIM, Azure Service Bus, Azure Relays, Azure Event Grid, Azure Event Hub, Azure Storage – Queue, table, files, Azure Web Apps, Azure SQL Database
He later showed few examples of Serverless Apps along with what problems we face while managing these Apps
- No Visibility and hard to manage
- Complex to diagnose and troubleshoot
- Hard to secure and monitor
We see similar problem in modern application which he explained by making us understand the Lifecycle of BizTalk Application. With great power comes great responsibilities, what are the best solution to manage your serverless apps ?
- Composite Applications and Hierarchical Grouping
- Service Map to understand the architecture
- Security and Monitoring under the context of Application
He later should and navigate through product Serverless360
- Templated entity creation
- Auto process left over messages
- Auto process dead letter messages
- Remove storage blobs on condition
- Replicate QA to Production
- Detect and Auto correct entity states
BAM (End-to-End Tracking) was also demonstrated during the session. He also showed how we can use the functionality and use the serverless360 connector in our Logic Apps for tracking purpose.
The session end with 3 different customer scenarios who are currently using Serverless360. You can book a demo to know more about the product on – https://www.serverless360.com
15:30 Monitoring Cloud and Hybrid Integration Solution Challenges Steef-Jan Wiggers, Microsoft MVP – Azure
He started the session by speaking Cloud native integration solutions – AI. He also spoke about the challenges we faced while creating either a hybrid solution or cloud solutions. He also showcased a real time scenario related to Hybrid Integration.
He also spoke about different types of Monitoring and challenges faced.
- Health Monitoring
- Availability Monitoring
- Performance Monitoring
- Security Monitoring
- SLA Monitoring
- Usage Monitoring
- Application Logs
- Business Monitoring
How users can use different Azure Artifacts to perform Monitoring activity ( Azure Monitor, Log Analytics, Application Insights, Power BI , configure Alerts.
How can people build an effective hybrid integration scenarios ( Training, Hands-on-labs, Learn through mistakes, Knowledge base, Forums, Guidance, Mentoring, Exams (AZ-103, AZ-900). When the Solution is in place, you must have a strong Support Model and well define processes.
There are different products available in market and based on the requirement select the right and perfect tools which suites your need. Few tools are,
- Atomic Scope
- Invictus Framework (BizTalk, AZURE)
- DynaTrace and the list goes on…
There are also different ways of Monitoring your Services in Azure. He also made a walkthrough of Cloud Native Solution – Order Processing revisited and gave inputs on which Products or Services to be used.
Finally he ended the session by share few Resources for learning.
- Azure Administrator AZ-103
- Azure Cost Management
- Microsoft Azure Monitoring
- Codit Invictus for Azure and BizTalk
- Codit D365 Whitepaper
- Serverless360 Blog
16:10 Modernizing Integrations – Richard Seroter, Microsoft MVP Azure
He started the session by going back to year 2009 where he wrote a book called – SOA Patterns with BizTalk Server 2009.
Modernization is a spectrum
Various tools which follows this spectrum include BizTalk, SSIS and Azure Service Bus. Later he spoke about few concepts related to integration and what would have been his take based on below
- My advice in 2009
- My advice in 2019
- Benefits of 2019 advice
- Risks with 2019 advice
Content based routing
- BizTalk Server with send port subscriptions
- Use BizTalk Server on-premise and Service Bus and Logic Apps for cloud based routing
- You message engine is scalable and flexible
- Explicit property promotion needed for Service Bus or you need Logic Apps to parse the messages. Cloud based rules are not centralized
Later he shared links to some blogs
De-Batching from a database
- Configure in the BizTalk SQL Adapter and de-batch payload in receive pipeline
- For bulk data, de-batch in a Logic App. Switch to real-time, event-driven change feeds where possible
- With change feeds, process data faster, with less engine based magic
- De-batching requires orchestration (LogicApps) versus pipeline-based de-batching. Can be a more manual setup.
This was a moment when my Blog was shared and it made my Day !
Stateful Workflow with correlation
- Use Orchestration and take advantage of dehydration, correlation and transaction with compensation
- Use durable functions for long running sequence along with Logic Apps and Service Bus apart giant orchestrations into choreographed sequences
- Easier for any Developers to build workflow
- You may come across limits in how long a workflow can “wait” and there is centralized coordination and observability
Complex data transformation
- Use the BizTalk mapper in transformation data structures and take advantage of functoids and inline code
- Map data on the way out of it all, and use Liquid templates for transformation but not business logic. Also consider transforming in code (functions)
- Avoid embedded too much brittle logic within a map and leave it up to receivers to handle data structures changes
- Not suitable for flat files or extremely difficult transformations. Put new responsibilities on client consumers
Integration with cloud endpoints
- Call cloud endpoint using http adapter and custom pipeline components for credentials or special formatting
- Use Logic Apps and connectors for integration with public cloud services. Use Logic Apps adapter for BizTalk where needed
- Any developer can integrate with cloud endpoints and you have more maintainable integrations
- More components from more platforms participating in an integration
Strangling your legacy ESB
- Put new Integrations into the new system, and rebuid existing ones over time
- Similar to 2009, but avoid modernizing to a single environment or instance. Use Event storming to find seams to crave out
- Get into managed systems that offload operational cost and are inviting to more developers
- You will have a lengthy period of dual costs and skillsets
Getting Integrations into Productions
- Package up BizTalk assemblies, libraries, scripts and policies into MSI and deploy carefully.
- Put On-premise and Cloud Apps onto continuous integration and delivery pipelines. Aim for Zero downtime deploys
- Reduce downtime, improve delivery velocity and reliability. Introduce automation that replace human intervention
- Complicated to setup with multi-component integrations. Risk of data loss or ordering anomalies when upgrade rollouts.
Building Integrations Team
- Invest in training and building center of excellence
- Integration experts should coach and mentor developers who use variety of platform to connect systems together
- Fewer bottlenecks waiting for Integration team to engage and more types of simple integration get deployed
- More distributed ownership and less visibility into all integrations within the company
16:50 Cloud Architecture Recipes for the Enterprise – Eldert Grootenboer, Microsoft MVP Azure
The final session for the day, Eldert started by explaining how on-prem infrastructure has to be managed by the enterprise as compared to the cloud IaaS you don’t have to worry about the infrastructure (patches, OS updates etc) all will be taken care by the Microsoft once your environment is in place. He also discussed how serverless can be useful where your primary focus is on what business needs and nothing else.
- What is the right size of server for my business needs ?
- How can I increase server utilization ?
- How many servers do I need ?
- How can I scale my app ?
He also suggested before we implement any solution we must draw a line and set guidelines.
- What must be a desired architecture
- Have a proper involvement from all the teams (Business units, architect and everyone involved) so that all are on the same page.
- Look at various options available and utilize it rather than building your own. In case you do go that path try to use PaaS offering.
- Other key things to look our for included
- Event Driven approach
- Loosely coupled solution
- Integration patterns
- DevOps strategy
- Look out for something what suites your needs and do buy something which is hyped or attractive
- Your environment must be Secure and monitored.
Explore Azure components while deciding on Architecture (Serverless – LogicApps, Event Grid, Functions ) Containers and AppInsights for monitoring purposes. There are endless possibilities to choose from. He also shared few of his customer experiences.
Cloud native start from PaaS and DevOps is also something to think of. Few take away from the session –
- Understand and make sure all the scenarios are captured
- Have a good governance and security model
- Look out for endless possibilities available out there
- After understanding your needs consider a cost effective method.
You can check more insights on Day 3 here.