Technical debt

I was talking to a colleague about one of our suppliers and the progress they were making on a new product they were developing. He was surprised to hear how little progress they appear to have made, they had previously been making huge steps in very little time with very few developers, especially surprising when they have been scaling up the development team over the past few months. I explained they are burning some of the technical debt they have collected whilst they still can. This of course then led to me explaining what I meant by using a common product, which then later prompted this tweet (which irony Capita hit the favourite button on)

For secondary schools who use SIMS .net – which is most schools in the UK – this can be simply be explained by Nova-T. Nova-T6 is a perfect example. Technically, it should be written in C#/.NET, however, there is a no business case, so it’s still a Delphi program. From a end-user perspective if they re-wrote it, they would gain nothing. Personally however, this still doesn’t mean it should be ruled out. It’s easier to maintain C# code when you have a small army of C# developers vs only a hand full of aging Delphi coders. Not to mention the advantage of C# over Delphi. The longer Capita leaves it, the more the interest costs them. Just look at some of the technical problems SIMS customers have had, it can all be tracked back to that technical debt. That debt that Capita needs to pay before the interest becomes to high.

Slow progress

I’m currently getting bogged down with other projects and unfortunately my final sprint of SIMS Bulk Import has grind to a halt. On the plus side I managed to swash a few more bugs last month, get a code signing certificate. This will mean all future releases will be signed as coming from me which is great news, it adds a layer of confidence that the code you run is unaltered by a third party. I’ve also started reviewing my other projects, ensure that the code on the public repository is up-to-date, I’ve also switched over to Git and copied them all over to GitHub. At bit more on that later.

Part of the updated process was to use a Continuous Integration server – specifically TeamCity. Simply put, these allowed the builds to occur on a dedicated box which made the whole process quicker and easier – it also added confidence as it ensured that everything required was commit in the repository. The only downside to this is the cost – although renting a VPS from OVH is cheap, it still isn’t cost effective due to the limited amount of time I actually use it, I’m therefore looking at moving it to Azure as you only pay for provisioned resources and your can de-provision servers and only pay for the storage. Alas this means more messing about setting up servers.

WIX installer why you no MSBUILD no more?

Last night I chopped up my WIX installer project – I basically split the actual installer parts into one file – CoreMsi.wxs, the versions into a variable file – Version.wxi and the file list into a separate fragment file – SimsBulkImport.wxs. This worked.

Great I thought, simple PowerShell script to generate the file list, I can even grab the main repository complied bin as a artifact dependency – got to love TeamCity.

Problem was, it fails to build.

error CNDL0104: Not a valid source file; detail: ‘.’, hexadecimal value 0x00, is an invalid character. Line 2, position 1.

bit of trial and error later I figured it out. It was my old friend Mr Encoding. After adding

-Encoding “UTF8” 

to the end of the Out-File and it builds again.

SIMS Bulk Import

Overview

If your in the UK, the chances are your local school is using SIMS .net. Just checkout the stats on their site. 22,000 schools taking 2,500,000 children’s attendance, every day. That’s impressive.

So what is SIMS .net? Well its a MIS system, but what is a MIS system? In the simplest terms its a database that holds the school records, for both students and staff.  So it makes sense that you are going to want to interface with it as it’s your at your core when it comes to data sources.

Now extracting data is pretty straightforward. Capita have created a custom reporting engine that allows simple report creation that can then be scheduled and produce exports of data. The problem is getting data back into SIMS .net. Take for example identity management, it’s easy enough to export a list of students, then write a PowerShell script to generate user accounts, but wouldn’t it be good to get that username added back into your core data source? Or what about adding in new students home email addresses and telephone numbers? Well it isn’t that straightforward to bulk import. Although Capita provide a API it isn’t as straightforward and certainly isn’t as friendly as a RESTful web service and certainly requires a programming background to understand it which rules out many schools being able to use it.

Back in May 2012 I was working on a SIMS support desk doing technical support and one of our customers asked if he could bulk import email addresses back into SIMS .net. This resulted in me asking Capita for documentation regarding the API which they provided, at no cost, along with a few snippets of example code. I then spent the following nights coding away at home to what has become SIMS Bulk Import.

About

So what are SIMS Bulk Import features

  • It’s free – Thank you Phil Neal for waiving the licensing costs 🙂
  • It uses the SIMS .net Business Objects (no large Capita charges for corrupting your SIMS .net database!)
  • Imports from CSV, Excel spreadsheets and XML files
  • Matches the file fields with SIMS .net fields

The future

I’ve since moved away from SIMS support and I’ve been trying to find a suitable new home for SIMS Bulk Import. Whether that’ll be Capita writing a replacement, or someone else taking up the project it isn’t clear. For now I’ll continue to support it. But to as part of this I’ve moved the source over to GitHub, the releases will still be via CodePlex.  I’ve also created a separate organisation on GitHubSIMSBulkImport, which effectively “owns” the code. I fork the code from  SIMSBulkImport into my personal repository, and do a pull request to merge the code back into the main SIMSBulkImport repository. This will help should the “owner” of SIMS Bulk Import change in the future.

I’ve also created a number of sub-projects – the SIMS interface library will be moved from Unfuddle onto GitHub, also the installer element will become a separate repository. I’m looking at setting up TeamCity to automate the build process so building and creating releases is a lot more streamlined and less time consuming. Also a simple web site setup giving easy advise for getting started.

Another change is a web API – this allows checking for newer versions and secure uploading of log files to the developers (OK, me). Mythic-Beasts have kindly donated a server hosted in Cambridge, UK to host this. This written in PHP using the Laravel framework, I’ve started writing it based on some of Bobby Allen original code.

Going forward I would like some (official) recommendation from schools and support teams, ideally I’d like to put together some sort of list or program detailing where you can get support from, for example:

Your Local SIMS Team supports SIMS Bulk Import!

Your Local SIMS Team is a SIMS Bulk Import silver partner!

It’s just difficult making a business case to people when your product is free. The main reason I want to do this is I’m find the odd problem where someone need to remote in, this is difficult without some sort of support agreement and T and Cs – most of the problems are actual SIMS problems like the SIMS client hasn’t be installed correctly which their SIMS support team could resolve. I appreciate working in the dark isn’t easy and you can’t know about every SIMS third party product, so your going to get a element of from pillar to post. Ideally I need a main backer to act as data controller with regards to the log files, at the moment I’m investigating if I could get around it by limiting the data it could spit out but that’s going to limit the usefulness of the logs. It’ll also been good to get the program digitally signed.

If anyone is willing to help out feel free to drop me a email – matt [at] matt40k [dot] uk

SOLUS2, will you not just die already!

Well it appears Capita has announced SOLUS2 death day

 

s2_end_twitter

 

That’s assuming they don’t put it back. Again.

This is bad news. Let me explain why.

I’m using Chrome, it’s updated itself I don’t know how many times. If I go and check, all I know or care about is that it is indeed up-to-date, so at some point, it’s updated itself and its worked.

If I look at my antivirus I can see it’s updated 3 times today alone. Again, this is all in the background without me doing anything.

Now compare this with SIMS and SOLUS2. I wonder how many hours I’ve wasted supporting schools doing SOLUS2 upgrades in my past life. I dread to think.

Before you start waiving the MSI flag, let stop and have a look at the outstanding CR logged on SupportNet today (24/06/2014) for the past year. Extract here.

1,057

1,057,outstanding change request. All requested by end-users. All with positive votes.

1,057. Wow. I dread to think how many have gone unlogged or is older then a year!!

Capita clearly can’t roll them up into a major build. How many re-releases have you seen? Add the fact it doesn’t even seem to be making a dent in CR mountain and you can see why Capita wants to be Agile like the other MIS suppliers.

Not sure I fancy pushing out 1,057 updates… per school…

I’m sorry, but SIMS8 can’t come fast enough

I haven’t really been on SupportNet, Capita SIMS support portal, regularly since changing roles about 20 months ago, the only time I pop on nowadays is to find something when I’m helping out my old team during busyhard times (you never really escape at my work). On this occasion I stopped by the forum and one thing jumped out at me. The lack of progress. Now I appreciate I’m about to unload both barrels into my foot, but I think it needs to be said.

The same issues exist today that exists when I was dealing with SIMS .net on daily bases over 20 months ago. That’s nearly 2 years. There are still people who aren’t using SOLUS3 – you know that free tool Capita created to ease the woes of upgrades and deployments. Despite a large number of users already using it and all the major bugs being resolve. More worrying, Capita has even issued SOLUS2 with it’s marching orders. It’s not a question of if, but when. The last release will be Autumn 2014 unless they are forced to push it back. Again.

I don’t think pushing it back is the answer. Pushing the date back is like putting a plaster over a gunshot wound. OK, it might help stop the bleeding, but it’s not the answer. Now I accept my approach of nagging my boss until he allowed me get on top of SOLUS3 whilst Capita was throwing major development time at it so I can make sure it works for our schools might not be possible for everyone. In fact it’s impossible, there aren’t enough developers to go around. But something has epically failed. Why, why, why hasn’t someone gone. Right, this needs to happen. It’s not going to go away. We have these problems:

  • SIMS Discover needs SOLUS3 – Sorry, you don’t install SIMS manually why? So why is installing Discover manually an option?
  • SIMS .net needs MORE access rights to installupgrade – switching UAC off is just plain stupid
  • Upgrades are just a pain

We need to resolve. Lets get SOLUS3 setup.

Now if your one of the support teams I’m attacking, before you start defending yourselves. Step back. Being busy isn’t an valid excuse. If you weren’t busy, you’ll be at the job centre. Someone, either yourself or your boss should be reviewing whats happening. Regularly. If your trying to do workarounds, you’re never going to have time to implement a fix. See what your doing is a putting a plaster on a gunshot wound and stop, get to the hospital. I know smaller support teams this will be a lot harder, especially when schools are doing their own thing, but I think this actually makes it easier.

SOLUS3 is a bit of a different beast. The way I explain SOLUS3 to new IT staff is, SOLUS3 is your tool. It’s a IT technician’s tool. This tool will manage SIMS .net across your network – it also keeps SIMS .net at a safe distance so you don’t enter the mists of MIS land. At the same time it keeps the MIS manager away from any sort of Windows permissions – you’re not giving any non-IT staff administrator access to YOUR network. System Manager is the DataMIS managers tool, it keeps people out of THEIR database, including IT staff. I’ve found this has been really useful with new IT staff coming from the “real” world and they have found the clear what you do and don’t do has helped keep them out of trouble. You do get the odd, why can’t I get a SCCM package, but when you point out that would require either Capita forcing schools to buy more Microsoft licenses or them reading more Capita documentation, they tend to go quiet.

Now I think the issue is the MIS support people are trying to push SOLUS3, when it should be the IT support people. If you look at the problems people are having with SOLUS3, they are IT problems, firewall and DNS mainly. Which MIS support staff don’t resolve. IT support staff do. If you’re now saying, but we don’t have any IT support people, it’s all third parties. What would happen if something, non-MIS related was required to be pushed out? What I’d do is organise a third party to support it, get the community together and issue training. So, find someone who has deployed SOLUS3 successfully and get them to do some training sessions for those pesky IT support teams. The cost of even getting even a Capita trainer in at £££, booking a venue could be offset by charging fees to delegates and to be honest, it isn’t that much when you compare it to even 1 day of failed SIMS upgradesdeployments. Costs can be cut by using a central school as the venue, by using a non-Capita trainer, by joining forces with other local support teams or even going completely digital. Support forums like EduGeek are a excellent way for the community to support itself with very little effort. I would also not be fussy by who attends, if they support one of your schools, offer them the training. Don’t get caught up in the fact their going to resell your knowledge. Think bigger picture.

So, do I think their will be much progress? Do I think Capita will stick with their Autumn 2014 is the last SOLUS2 release?

Well I also noticed they still had the following documents on the SupportNet homepage as sticky items:

  • Technical Guide – Network Impact of SIMS
  • Technical Guide – Optimising SIMS .net startup speed
  • Technical Guide – Remote Working with SIMS

Sound pretty useful, till you look at the dates, 06 April 2005, 08 April 2005 and 11th June 2004 respectively.

Take the first one, Network Impact of SIMS – to quote a bit of it:

Unfortunately, minimum hardware specifications are set to rise as we cease support for Windows 98.

Realistically Windows 2000 and Windows XP we would recommend a minimum of 256MB.

I’m sorry, didn’t Microsoft and even Capita just drop support for Windows XP? Why haven’t they updated a document that is important enough to warrant being displayed on their support portal? In over 9 years?!?

Looking at Optimising SIMS .net startup speed, why are worrying about turning the antivirus off certain directories? Aren’t our computers these days powerful enough to handle it? Surely it’ll be a good idea if we’re not using SOLUS3 and giving everyone and their dog access to a protected folder location – Program Files. Why are we worrying about SIMSload, it’s does nothing once the SOLUS3 agent gets installed. Why would anyone compress the drive nowadays?

Remotely accessing SIMS .net is a hot topic – the general concession is to use RemoteApp (Terminal Services). But the “official” Capita response is the Remote Working with SIMS document… which says to use a VPN. I’m not a fan of this idea. FMS, the Capita SIMS finance package, for starts doesn’t work well over a VPN, it’s not a speed issue, I’ve got 50MBs and that’s the lowest package I could get, it’s a latency issue. SQL doesn’t like latency. If you think of that 50MBs as the number of lanes and the latency as the speed what the cars are traveling. SIMS .net isn’t immune either, it just more modern, so it copes better.

I’m sorry, but SIMS8, the next generation of SIMS which is rumoured to be in the “cloud”, can’t come fast enough for me to wash away all this silly technical problems. It’s ridiculous that things aren’t further along. Take the file sets for the School Census, I can see 4 file sets that came out in May, it’s still a manual import. SOLUS3 should be deploying file sets. Automatically. The biggest strength of SIMS is the community, it’s also the biggest problem. Viva la revolution!

Cloud revolution

Why I like Twitter

Applications crash and "AccessViolationException" exception occurs when you use System.Data.SqlClient after you install Visual Studio 2013 or .NET Framework 4.5.1

I have to say I’m really liking Twitter. A lot of my friends can’t see the point. As a social tool it allows gossip to flow really well, how often do we hear that a celeb has tweeted a inappropriate comment or photo. It allows us fans to be alot more closer. As a business tool it allows a new way of interaction, take this example – there is a bug in .net framework 4.5.1 that occurs if your using a certain type of third party firewall (often bundled with anti-virus software these days), it breaks SIMS .net, that uses .net 4.0. Capita have spend ages trying to resolve it, then, @timbo343 posts on Edugeek that 4.5.2 has resolve it, a quick tweet to the Microsoft @dotnet twitter guys and it turns out that it is fixed in the new version. Awesome.

DfE big plans for big data

An interesting debate erupted on EduGeek this month after Graham Reed, a technologist, made a post on Eduware Consulting blog about the changing data landscape and how he saw Department of Education (DfE) data exchange and warehouse projects changing the way MIS work and the way they are used. Now of course we need to take it with a pinch of salt, firstly, this is very much early days. At the moment it’s a vision looking to become a project, and we all know how government projects end up post election time. Add the fact Graham Reed is one of those folks trying to push SIF and the interoperability agenda and you start to get the feeling your walking into a sales pitch. With that said, I think the DfE onto a winner.

The crux of the DfE goal is turn the very manual processes they have:

DfE_Current

into something a lot more streamlined:

DfE_Future

(Thanks to Phillip Hamlyn at Capita for sharing the link to DfE documents)

In short, it won’t take them months to produce something useful that by the time they’ve produced it, it’s already out-of-date. On paper this makes complete sense, it’s what happening in industry across the board, it’s what the cloud is all about, automation.

It also starts to solve a lot of other issues, generating ULNs for example requires manually generating a CTF then uploading to S2S and finally manually updating your MIS, with the new system, your MIS could do API call to the DfELearning Records Service (LRS) web service and automatically request and update the record with the ULN all without leaving your MIS application and a few clicks of a button. Simple. Take it a step further and pupil transfers could be done the same way, select the pupil who’s leaving, select the new school and it automatically sends the pupil records to the new school! Dual registered could occur the same way, just add the other school and the pupil records are synced between the two, regardless of MIS supplier, via the DfE.

Unfortunately, like most things, the problems appear when you start digging into the detail, how is this data used and how is the data currently gathered and why is it so manual?

The main issue, in one word, is Census. The Census, to put it bluntly, is used for funding. Now in a world where budgets are on the decrease the last thing you want is to see it drop even further because of an admin error, so you can see why the idea of gathering the raw data in near real-time is a bit worrying. For me, I don’t see this a show stopper. I accept data will be inputted incorrectly or not at all. I know the pains of ethnicity data. I also know the importance of a “approve” button. I know a Local Authority (LA) will most likely have access to most of their schools MIS systems and be able to produce a School Census return, I also know they would have enough data in their own systems to produce a School Census return. The key fact here is that they don’t hit that approve button. That approve button is always hit by the school. Period.

So long as decisions aren’t formed on unconfirmed facts I don’t see a problem and I’m not just talking about funding, many other decisions are formed from the School Census data, just have a look at the free reports from Arbor and your see what data there is on your school in the public domain. The general public are becoming better informed and schools are almost having to sell themselves to get more pupils on-roll, to get more funding or even stay open. This is even before we start looking at how services use the School Census data to support schools such as transportation, and not forgetting how researchers use the data for analysis. As long as that approve button still exists in the new DfE system, it’ll be no different then what happens now.

Personally, I quite like the idea. The buttons will change but core processes will remain. You will still submit data which will be used for funding, performance tables and various research purposes. How you do it will change, but that is always changing. It’s one of the joys of IT. It’s just this change is the DfE getting his house in order, it’s bring all the various siloed services into one roof, like the government is doing with all it’s websites. So really, the only people who have to worry are MIS suppliers and if you are a supplier, I know a company who can help you out 😉

 

 

 

 

Should I purchase Full SQL or is Express enough?

One of the common questions I used to get asked whilst supporting SIMS .net was, do I need Full SQL. SIMS .net uses Microsoft SQL Server as it’s relational database management system (RDBMS) so it’s important to ensure the engine is correctly sized for job at hand. Microsoft SQL Server has come in a number of editions over the years, I won’t detail all the various editions but the two editions schools should be concerned with is Express and Standard. Express is the free edition that Capita bundles as part of their installation media and Standard edition requires purchasing additional Microsoft licenses.

Standard edition can either be licensed by the number of processor (cores) or by users. I personally tend to go for processor licenses so I don’t need to worry about buying new licenses when adding new users later on down the road. It does however come down to pricing – I tend to find processor licenses break even around the 30 user mark (based on Academic licensing anyway).

So why should you shell out of Standard edition? Now to be fair to Capita a number of features that Standard has over Express, like SSIS and the SQL Agent Capita has created tools to workaround the limitations, they’ve created a data transfer tool to extract the data from the SIMS .net database into the SIMS Discover database for example, they’ve created routines within SIMS .net to create Windows Scheduled  Tasks for SQL jobs like B2B. With that said, the one thing Capita can’t get around is the hard resource limits Microsoft has set. Express is limited to 1GB RAM and 1 CPU (max of 4 cores).

When deciding what edition of SQL to use, I normally follow the following diagram

Which SQL Edition is right for me?

I simply believe a Primary school doesn’t need SQL Standard edition. Secondary schools who use Lesson Monitor, a Capita bolt-on module that is used for recording attendance for individual lessons rather then just AMPM, will be using the system pretty much all day rather then just morning and afternoons, so your concurrent users will be alot higher and therefore require more then 1GB RAM. It’s rare that a Secondary school will not be recording individual lessons, however other bolt-ons exist from third parties that will allow the same thing but not have the same impact on the server, so you may be able to get away with Express by using a web based bolt-on.