Niall's Data Blog

A Data Engineer / Architect writing about Tech, Data and the Community

Build & Deployment Warnings in Azure DevOps

It's all in the details...

I use Azure DevOps with clients a lot, mostly for deploying data platforms. Sometimes have a task within our pipeline that fails, but the failure is actually something that is a problem, but not something that should stop a build or a deployment. Some examples of this situation might be Deploying to an environment and configuring a user or group to have access to the table and the user or group does not exist Sending an alert from a build or deployment, and the code to send the alert fails Catching specific known errors, and dropping them down to a warning.

Posted by Niall Langley on Sunday, September 15, 2024

Azure Data Factory: Dev Mode vs Published Code

I’ve worked with quite a few people new to Azure Data Factory, and one thing that seems to confuse new users is the difference between the developer sandbox where we build pipelines, and the published/deployed code. Understanding this is key to working with Git and using CI/CD pipelines to deploy your code, and getting other Azure services to integrate nicely to call your pipelines. Connecting to ADF A good first place to start is to understand the different ways we can interact with a data factory.

Posted by Niall's Data Blog on Thursday, September 2, 2021

Azure DevOps: SqlPackage Deployment Timeouts

I recently looked at an Azure DevOps pipeline for a client that was timing out whenever an index change or other long running task was deployed using a DacPac. All deployments that ran within a few minutes successfully completed, but those taking longer than 10 minutes were failing. A quick google for the issue shows a few helpful pages, a SqlPackage.exe bug from 2016 & a blog post from 2018. These suggested using either a couple of parameters when running SQLPackage, or setting a registry setting.

Posted by Niall's Data Blog on Sunday, August 29, 2021

Niall langley | Data Engineer, Speaker, Blogger

FEATURED TAGS

adlsv2 associative grouping azure azure devops data cleansing data factory tsql