Posts

The Power Query Formula Language (informally known as “M”) is a powerful mashup query language optimized for building queries that mashup data. It is a functional, case sensitive language similar to F#, which can be used with Power Query in Excel and Power BI Desktop. A few days ago I got asked to produce a list of users (and their email address) in a number of AD Groups. I already had a SSIS package that had a script task to pull this data from Active Directory and push it into a SQL database and we have a PowerShell script to get the same data in our code repo....

This might sound pretty pointless and silly but when talking about a 20 working day SLA a day can mean alot between a massive fine or compliance. During one of my driving lessons with Gary we were talking about his recent streak of passes. The conversation then lead on to pass rates and us wondering how this broken down – for example, by gender, ethnicity, age, driving instructor type (intense driving school \ chain \ individual driving instructor \ family etc)....

Today my colleague had a problem opening our BI solution, the solution had multiple projects, including 3 SSDT projects. Although the project builds correctly on both my machine, the build machine and another colleague machine it refused to build stating that the reference to the object in another project was invalid. After thinking for a few moments, I remembered I had seem this before. The problem was a bug in SSDT....

Microsoft has released SQL Server 2016 Developer, for free. That’s right, zero, zip, nothing. Completely free. The developer edition is fully featured, it contains the same goodness as the $$$ Enterprise edition. The catch? Its not for production.

Another day, another reference to an old Jamie Thomson blog post – today it was getting the ConnectionString to output to the information. Admittedly I wasn’t looking for how to fire the connection string into an information event, just the ConnectionString, still another great idea and its useful for debugging.

This month saw the release of SQL Server 2016, which from a BI\Report point of view is huge. Once we had access to it yeah, already been on msdn. Can't believe we're being made to wait!!! — Dave Kerby (@davekerby) June 1, 2016 I spotted something odd. Now, in SQL Server terms there are two types of licensing Server license + CALs Cores (processors) Except for Enterprise, you can only get Core licenses....

I had previously created a SSIS package with a simple Process Full on the SSAS MD database, however, as the project has progressed this hasn’t been ideal. Its basically all or nothing. In order reduce the damage a failure can cause I’ve setup a Process Full for each dimension and cube so each is processed independently of each other. I’ve used the data from the Analysis Services Dynamic Management Views, or DMV for short, which I’ve used for the documentation to feed the foreach loop....

So today I went to run a SSIS package on my new laptop and bam, error message. Microsoft.SqlServer.Dts.Runtime.DtsRuntimeException: The package failed to load due to error 0xC0011008 “Error loading from XML. No further detailed error information can be specified for this problem because no Events object was passed where detailed error information can be stored.”. This occurs when CPackage::LoadFromXML fails. —> System.Runtime.InteropServices.COMException: The package failed to load due to error 0xC0011008 “Error loading from XML....

I was at event that Simon Sabin arranged, SQL SERVER TOOLS and SSDT shape the future and one of the questions that was asked, sorry, I forgot who asked (joys of leaving it too long before blogging about it – I want to say Gavin) was, How does (Windows) Containers fit into SQL Server? This is basically Microsoft integrating the Docker technology into the Windows OS, so you in effect, ship a more complete solution....

Power BI is Microsoft new cloud based Business Intelligence platform, one of the cool things about it is it built atop internet standards and leveraging open source libraries like D3.js for its visuals. Microsoft has continued on the path of open source-ness by open sourcing some of its codes allowing Joe Public to submit custom visuals to Power-BI, all via GitHub. Yesterday saw the introduction of 4 new visuals, one in particular looked very interesting – Hierarchy Slicer by Jan Pieter Posthuma....

Another day, another problem. Todays problem was incorrect column length which was causing a SSIS package to fail. Historically I use a freeware tool to compare the different environments schema, but lately I just don’t bother, I have continuous deployment so its easier just to hit the deploy button then it is to try and figure it out. Well today I decided I wanted to it old school – although I’ve been using the SSDT SQL compare in my deployments, I’ve not actually used it in terms of viewing the differences....

Msg 0, Level 11, State 0, Line 0 The connection is broken and recovery is not possible. The client driver attempted to recover the connection one or more times and all attempts failed. Increase the value of ConnectRetryCount to increase the number of recovery attempts. In short, you’ve lost your network connection, either your WiFi has dropped out or your network cable has 🙂

As some of you are aware I currently working for a local authority and anyone who has been following local government news in England will know the bonds between local authority and the health authority, aka the NHS, are growing. As are other parts of the government for that matter, both central, other local authorities as well as charities, but for now, I am focusing on the health part. Specifically the part around what that means....

There are various types of Auditing in the Microsoft BI stack. There is auditing in SSRS, SharePoint, SSAS and not forgetting SQL has its own auditing. Today I am looking at the SSAS auditing – you can find out more about it on TechNet. Olaf Helper has published some TSQL code for querying the audit data -on the Script Center. But first, we need a table to store the data, here is a TSQL script for creating the table:

So eariler this month I switched from my Azure hosted TeamCity server to Appveyor, which freed up some much need Azure credit. This month, it looks like its going to be going on a Visual Studio Ultimate vm for some SharePoint development. Hopefully this won’t take too long to do as I really want to get on with machine learning after a demo from Ric Howe. I’m also looking at building a few test machines locally – old school I know!...

So today I managed to move SIMS Bulk Import over to Appveyor. So what does this mean? Well it means I don’t have to worry about going over my Azure credit limit each month for starters! Appveyor has excellent support for GitHub, so each commit is automatically build, tested, and (when I enable it) a new release created. The next release will include Pupil username creation – you can blame\thank Andrew Mulholland for this, I saw his project PiNet and thought we can make this better....

One of the bad habbits I have picked up from working in Local Government is reviewing your options, regularly. As I approach my renewal for this site’s web hosting, I decided to revaluate the market. (Short version, I’m with Mythic-Beasts for another year) Currently, I have the site hosted with Mythic-Beasts. They are pretty cool folks, unfortantely one of there DC had a power cut recently – ok, only a few hours and I pay peanuts so I don’t expect much in terms of reduancy or fancy SLAs, but it did make me look at my options for a bit longer then normal....

Just thinking out load (because its good to get feedback) Just to expand on my tweet. I’ve lot of SSIS packages for load and building my BI warehouse (SSAS cubes), now currently I have a SQL job that has multiple steps For example Load data from Source system into Staging Build ODS intermediate tables Build Warehouse tables And this works for DataMart where they are a denormalized copy of a single source system, but when the source system, or the intermediate tables, are used multiple times, you don’t want to run them multiple times....

I recently discovered the ability to reference other database project (I know, I know), anyway, this cause my lovely automate build\deployment process to fail. Investigating the error led me to, yes, you guessed it a Jamie Thomson’s blog (from 2012). Thankfully, I managed to figure out a clean solution, the DacPac solution on Octopus Deploy library has an option to pass Profile Name, which is the publish profile XML file. He is the cut-down version I’ve used, just the missing variables

I’ve previously blogged about how I was going to detail how I build and deploy my Microsoft BI projects and to be honest, I’ve been putting it off. The main reason is my work has been looking at Visual Studio Team Services, formally Visual Studio Online and I was looking at using the Build functionality that is included. This should make the whole setup process a lot easier for anyone else trying to replicate my setup....

Anyone working with SQL Server will be well aware of the mountains of training material, blog posts, videos, code snippets, events and other helpful resources that helps anyone who comes into contact SQL Server regardless if they are a beginner or a seasoned veteran (even MVPs!). One of the pools of knowledge is SQL PASS. is an independent, not-for-profit organization run by and for the community. With a growing membership of more than 100K, PASS supports data professionals throughout the world who use the Microsoft data platform....

Today, my pull request for the SSAS deployment script was merged into the Octopus Deploy Community Library. So what is Octopus Deploy and why should I care? Octopus is a friendly deployment automation tool for .NET developers. But don’t be fooled. Octopus Deploy isn’t just limited to .NET developers deploying hip new cloud based applications, it can do much, much more. I currently use Octopus to deploy Jasper reports to our 6 HR environments, I also use it to deploy our HR interfaces which are a series of SSIS packages and SQL objects which uses Microsoft SQL Server 2008....

So I’ve been playing wit mapping data, one of the nice things about GitHub is that it supports GeoJSON and automatically renders it With the mapping data already imported into Geography in SQL Server it was easy enough to convert to GeoJSON (Stackoverflow to the rescue again!)

Just an update as I noticed I forgot to say how to get the names and descriptions of the collations your SQL Server supports

So I was handed over some new servers, before I got started I wanted to check everything was as it was suppose to be, so I wrote a few PowerShell scripts to check each server was setup correctly. I wanted my DEV to 100% match my UAT and so on. First, I wanted to check who has local administrator access on the servers. Next, I wanted to check the drives – I wanted to make sure each drive letter was the same across environments and all had a proper label....