Tuesday, March 17, 2020

Keep the faith

Here in the Netherlands, coronavirus has been quite big news. The government had been criticised for being too relaxed, and here in Breda - one of the main cities close to Tilburg where COVID19 was first reported in the Netherlands - it has been quite interesting to see it develop.

Slightly stricter measures apply in Noord Brabant, where its recommended to avoid social contact with others, with some events in the province being cancelled.

Saturday February 22 marked the official start of the Dutch Carnival in North Brabant, except I was laying in bed already hungover from the big party the night before. Some students most likely started even way before that - they really do know how to party in Brabant. When the local Breda football team NAC Breda, whose stadium I believe is smaller than that of AJAX Amsterdam, manages to sell more beer a night, you know they have some crazy supporters. I especially love the proudly shown banner "The only thing we fear is running out of beer". My kind of football team :D I've only been to one match, but it is a great atmosphere, filled with truly proud supporters and citizens of Brabant.



Carnival is filled with the same for pretty much a week long - people drinking beer, dressed up in costumes (otherwise you won't even be let into a pub or venue most likely), not caring about anything but having a good time, even in the cold windy or rainy weather- so windy that this year they postponed/cancelled a few of the carnival float parades that people had been building most likely since last year.

It was a very different atmosphere to the Breda I experienced this past Saturday evening, which normally buzzes at 6.30pm on a Saturday night, but I barely passed 5 people on my way to the local Albert Heijn (AH) supermarket, which thankfully had not run out of groceries (unlike others apparently). I guess I must have missed the panic rush by then perhaps - I also managed to buy a bag of toilet paper a couple of days before, so maybe I didn't notice what was missing on the shelves because I had the essentials already.

The relaxed attitude got slightly more serious on Thursday 12 March, when gatherings of 100 people or more were to be cancelled, working from home being encouraged (I luckily had just started doing this the day before), people feeling sick advised to stay at home, and it wasn't yet serious enough to close schools - universities were being encouraged to offer online studies, instead, but primary, secondary and vocational schools to carry on since they're vital to society and the government didn't want to set them back.

I was surprised by that one actually, usually it's the kids that spread things...

Sunday evening though a less relaxed and more strict measure was put in place, giving a closure order to the horeca (hospitality-restaurant-catering) industry, closing down schools, and putting in more strict measures.

Coronavirus started spreading throughout the Netherlands, with Brabant being a centrepoint, with Tilburg and Breda being two of the hotspots still.

It makes me very anxious. Especially since I have had a bit of a cough for a week or two! Makes me paranoid that maybe I have it already.

The prime minister, Mark Rutte, announced today a strategy aiming at maximum control to try stop the spread of the virus. Reading that news article makes me feel quite confident in the government.

Yesterday my home country South Africa was also addressed by our president, who also announced similar measures to be strict about trying to control the virus.

It's a tough time at the moment. I feel quite stressed, and as hard as I'm trying to stay positive, it is hard at times to do so, and to carry on as if it is all ok. Today was a very hard day to try and stay focused, and that's why I'm trying to kind of write down what is on my mind, as a source of more peace and calm.

My girlfriend is in New Zealand at the moment. She has been away since January on a fantastic sounding four month tour of Myanmar, New Zealand, Singapore, Malaysia and Thailand. I was going to be meeting her in Thailand around Easter weekend in April, but those plans seem to be falling through right now. Luckily KLM is most likely going to allow me to get a refund, or a voucher I may use within the next year - still need to hear back from their most likely swamped to over capacity customer support team who usually respond very quickly (haven't heard back from them the whole day).

My nerves have been quite on edge even just because of this, wondering if she would be okay on her travels. Even just getting to New Zealand she had to bypass China when the start of the outbreak was happening - NZ weren't allowing anyone from China in at that time, and she had to arrange a new flight herself at her own cost since the low cost airline or travel insurance weren't able to help her out then. Most likely she won't hear back from them anytime soon now with the virus being a pandemic.

She is currently in NZ in the queue for Qantas, hopefully she can find the safest option for her and her aunt that she is there with. No more trip to Singapore for them I believe, which is a pity.

All I can think of though, is "Is this cough that I have coronavirus? Am I being paranoid? Even if she comes back, if she comes to see me, is that safe for her? I miss her. I love her"

Then there is the stress of my close family, my Dad, Mom, sisters and their families. My Dad was diagnosed with cancer last year, and I returned to SA to visit while he was in hospital in September. It was meant to be a family trip to Australia to visit Debbie, Etienne and my nephews. We were all going to meet up for a year of milestone birthdays - 40th, 50th and 70th birthdays. In the end, all these plans changed. I used half my month's leave to visit SA first to check up on my Dad, the rock in my life that has provided me so much that I have to be grateful for today. To see my Mom who has also always been there for me, and to also support my eldest sister Michelle who was an absolute angel and star for caring for them, even with a whole lot of drama she had of her own.

It was good to be there, but it didn't feel like a proper holiday. I wished I could have stayed for longer than 2 weeks to be with them.

I then returned briefly to Netherlands for the starting game of the world cup, and a great braai spent with friends and "second family" here in the Netherlands, before making my way a couple of days later down to Australia to visit my other sister and her family. My brother-in-law had experienced a SCAD heart attack while in Australia, and it was the first time I was seeing them since they had left to Oz a few years ago.

My youngest nephew was just over 1 years old when they left, and now he was busy playing with a remote controlled crane (I was kind of jealous, I think I would have loved that toy as a kid, even now :D), talking, playing with his brother. Fighting with his brother. And vice versa :D

They drove me a bit insane at times, but I love them. Justin reminds me so much of myself when I was younger. I guess that's maybe the scorpio part of us - his birthday is two weeks apart from mine. He also had been having issues with his own health, where his white blood cell count was indetectable for a while, due to what seemed to be an auto-immune disease. I was very worried about my family down under for quite some time.

These days though, they seem to be doing better. My nephew's WBC count has increased, and he is on a lower dosage of the medicine he needed, and that makes me so happy. My brother in law can also do a bit more fitness, though, I still think he should be resting - I think he knows it too, he is just stubborn, and can't sit still!

My Dad, his chemotherapy sessions are almost done, and he seems to have been doing great with it. Except this week he also was back in hospital with a pain in his side. The scans seemed to show that it might be kidney stones maybe that passed before the scan, or that can't be seen in the scan. He was in the hospital for two nights and I was dreading that he would get coronavirus while there, especially since they kept him in for the two nights because his white blood cell count was lower than it should be. Luckily today he was released to go home.

On the way he sent a whatsapp text on the family whatsapp group, to see if my Mom needed milk. Rightfully so, she told him to avoid the shops! My Mom also has been having to deal with her eyesight deteriorating, and I guess even worse - having to admit that it is age related! :P

Not to mention my eldest sister, who recently was also in hospital due to a spider bite while working at a new place on the coast - they recently had to change jobs from where they had setup their business for the past, I dunno, 15 years or so... which was also tough for them. Luckily the bite's venom/poison (never remember the difference) didn't reach her muscles, but eina, the wound looks very painful, and took off a nice chunk of her shoulder. Seems to be healing though, so that's good.

Especially since she also had to go back to the hospital due to an allergic reaction to the medicine/ointment/gel being used for the wound care treatment. Was very worried about her then. My other brother-in-law was up and down to visit her as well, between having to work until a new skipper arrived, helping to take care of her, and avoiding riots closing off roads in-between - he'll have to wait for that road to clear... no roads with burning tyres, having to test out 4x4 features of his car to do escape manoeuvres from angry mobs chasing him, not since the last time that's for sure!

It's all just a little stressful.

So ja, it has been quite a year or two.

It is tough to stay positive during times like this. A lot of people are finding it difficult. Some more than others. I can't even begin to imagine what it is like for the poor people of South Africa, and the rest of the world, who don't even have electricity or water normally - nevermind load shedding. I try to stay grateful and positive, and focus on what I have. Focus on what I have been through myself as well to get to where I am today.

I hope this post didn't ramble on too much, and I hope that this whole coronavirus pandemic gets sorted out.

Jim Rohn has said it best in his one seminar, and various books, where he describes life and business as the changing of the seasons. There will always be winters, but they always eventually are followed by summers. There are always springtimes, and there are always autumns and harvests. Sometimes the winters are easy and sometimes they are hard, whether it be the actual weather with snow and wind and miserable rain, or the metaphorical winters of financial burdens, heartbreaks, anxiety. These will always be followed by summer - you'll get through the hard times. The one big important part of his message though, is how it is your responsibility to act. You only reap what you sow. You only get a full harvest, if you take advantage in the springtime and plant the seeds (or vice versa, I'm not a farmer, and don't have very good green fingers).

He was also quite spiritual and religious I believe. And I am too. I strongly believe that I am here, because of the power of prayer and the love of God. I had encephalitis (inflammation of the brain) in university, and could have become brain damaged. I blacked out, I felt like I was descending into hell, and I felt like when I woke up and saw my sister's face the next day, that I had been saved and that I was now safe. While I was not fully concious, I was fighting the nursing staff and my own parents, fists swinging, with them trying to hold me down to my bed so I wouldn't hurt myself, surprised by my strength. The drip in my arm, in my delusion, feeling as if I was stuck in my bed, not able to escape it, being attacked whenever I tried to get out of it. They had to give me a lumber puncture while I was under general anaesthetic because of this. While I was out, I was imagining myself descending into a bottomless pit, and being scared and praying to God to help me.

I experienced Him before that time as well, when someone close to my family, part of my extended family, seemed to want to try and commit suicide. I prayed as he appeared to disappear from our sight, and I pointed to the horizon in the hope that he would come back at the same point. I kept pointing to that point on the horizon, and prayed words for him to come back to shore safe. He did, and at that point in my life I was very shy, and didn't tell him this. But later that day, he was sharing the story of seeing a bird, all the way out there, as if it was calling him back to shore. The words he said after that, were the exact same words I prayed.

Even the time I came back from Contiki, after visiting the Vatican, buying a Catholic cross that stood out on a table full of memorabilia, meeting a Catholic girl and visiting a church with her not long after for my first Catholic ceremony, then the next day my previous employer's CEO sending out a company wide email about a Catholic formation class that he was starting. It all seemed to be too much of a coincidence, to be coincidence, and not fate.

I may be sounding crazy and delusional to some right now, and as your read the above, I wouldn't be surprised either. I have felt the same reading similar things in the past before as well.

It makes me think of when I was in Prof. David Block's classes, studying Astronomy in university. Not only was he a passionate lecturer, he also appeared to be both scientific and spiritual. I was amazed with the way the universe seemed to be explained by these mathematical equations and how much we knew about the universe, and how little we knew too. Dark matter, finding the "God" particle, what caused the "Big Bang" - was it God the creator that caused it? Who knows. Nobody I guess. I also found it fascinating in university, learning about computational molecular biology, how genes are sequences of genomes, and how Computer Science algorithms to match strings, can help with detecting things genetically. How biology can show evolution with mutations of genes. How artificial intelligence can make us question if we ourselves are even intelligent, creative, or designed to think the way we do with an illusion of intelligence - how is someone that can paint, any less creative than a machine that can learn how to paint, when both may be shaped around their environments and experiences? Are we created, just as we have created AI? Is coincidence a thing, or are things predetermined? If so, do we have free will?

I didn't study theology or philosophy really, but these kinds of questions have been quite interesting to me for a while now. I also joke sometimes that if you want to become a really good software engineer, first study psychology :D People are the hardest part of understanding software development :D

Does God exist? We can never prove that. But we can always believe it to be true, and have faith. Is the Bible true? Is the Cathechism true? Which version is right? Nobody truly knows.

I do know though, that I believe in God, I have felt Jesus' presense or the presense of the Holy Spirit, or both, in my lifetime. At least, I believe I have.

I pray that the world can be made safe again, healed. That coronavirus can be stopped, that people can be cared for. That the world will unite.

That is quite a big thing to wish for.

For now, I just wish that my cough will get better for myself. And for my love to be able to see me again so that we may comfort each other, and live a long, happy life together.

I love you Alies, and I miss you.

I feel more calm now, so I will stop writing in this blog post. It is one of the first blog posts I've written in public like this (apart from on social media like Facebook etc.) in a while.

I hope that if you read this far, that you too stay positive.

I hope that we can all enjoy the summer together.

Keep the faith.

Amen.

Tuesday, August 21, 2018

git bash - deleted local branches

You can find out which branches have been merged into a branch by running the below:

$ git branch --merged origin/master
* master
  feature/some-branch
  feature/some-other-branch

This will also print out master as part of it, so we can then pipe that into grep to match results without master:

$ git branch --merged origin/master | grep -v master
  feature/some-branch
  feature/some-other-branch

Obviously, you won't want the name master in your branch name, otherwise it'll be excluded too. You can fancier with regex here.

Now we can take this result and pipe it into xargs, to run git branch -d passing the output as arguments

$ git branch --merged origin/master
  | grep -v master
  | xargs -r git branch -d

I'm splitting up the command onto separate lines for readability. The -r will only run the git command if there is output.

I've noticed this doesn't always delete all old branches, but it can help. For example, perhaps you had a branch that was a work in progress, or a spike, with changes that aren't merged - this won't be picked up.

You can do one more trick to detect old branches that have no upstream branch anymore:

$ git branch -vv

This will list branches without an upstream branch with gone. Those can generally be deleted too.

Monday, August 20, 2018

Today I learned - VSTS CLI and jq

Today I learned about the VSTS command line tool from a colleague.

I might actually have heard about it before, but wasn't able to try it out because I was previously using the on-premise TFS and not the cloud based VSTS, but hey, today I tried it out :)

For example, you can query VSTS for pull requests you've created by doing the following:

$ cd /your-vsts-git-repo
$ vsts login
$ vsts code pr list --creator "James Barrow"

This will return a JSON object containing info about what pull requests you have created.

If you don't really care about the entire structure, you can then use another command line tool I've come across before called jq.

For example, if you only care about the title, status and url of the pull request, you could do the following:

$ vsts code pr list --creator "James Barrow" | jq ".[] | { title, status, url }"

That's a long command, but what it basically does is pipes the JSON result into jq, which parses it by taking each element in the array of the result, and then filtering it to only take the title, status and url properties of each item in the array.

And finally, since we all know the best way to use git is via the command line and git bash, you can then also create a bash alias in your ~/.bashrc file:

alias prs='vsts code pr list --creator "James Barrow" | jq ".[] | {title,status,url}"'

So you can just check for your pull requests now by simply running:

$ prs
{
  "title": "RE #1234 - some pull request",
  "status": "active",
  "url": "https://blah.visualstudio.com/.../pullRequests/2345"
}

Which gives you something like that ;)

Much simpler than having to open up Chrome, go to VSTS, find the right tab for pull requests, etc...

I want to dig more into using jq to pull out info on if the pull requests have had votes as well, so I can just run a command from my command line and see if I need to follow up on a PR or not, but this will have to wait for another day :)

One last note is that use of single and double quotes might be important depending on your shell of choice, so keep that in mind too.

Hope this helped someone.

UPDATE: There is also vsts code pr list -o table which prints a nice format

Thursday, June 28, 2018

AngularJS to Angular, JavaScript to TypeScript, Gulp to webpack to Angular CLI

So I jotted down a whole lot of links for when I was learning webpack, and thought maybe someday I'd write a blog post, but that I probably wouldn't really. That day may come, but that day is not now :P Posting this in case others may want to browse through the links that I found useful when trying to convert an application in AngularJS 1.x with Gulp, using ngupgrade with Angular5 and webpack, onto just one build with the Angular CLI.

https://youtu.be/ivQ7HrnBJe8

https://webpack.academy/p/web-fundamentals

https://what-problem-does-it-solve.com/webpack

https://survivejs.com/webpack/foreword/

https://webpack.js.org/configuration/

https://webpack.js.org/api/

https://webpack.js.org/api/loaders/
https://medium.com/@MarkPieszak/using-pug-or-jade-templates-with-the-angular-cli-9e37334db5bc
ng add ng-cli-pug-loader
https://www.npmjs.com/package/pug-html-loader
https://www.npmjs.com/package/exports-loader
https://github.com/webpack-contrib/copy-webpack-plugin
https://github.com/webpack-contrib/mini-css-extract-plugin
DONT USE > https://github.com/webpack-contrib/extract-text-webpack-plugin
https://github.com/s-panferov/awesome-typescript-loader
v4.0.0-beta.0 Release Notes

https://gist.github.com/sokra/1522d586b8e5c0f5072d7565c2bee693
https://gist.github.com/gricard/e8057f7de1029f9036a990af95c62ba8
https://webpack.js.org/configuration/optimization/#optimization-minimizer
https://webpack.js.org/plugins/uglifyjs-webpack-plugin/

http://dmachat.github.io/angular-webpack-cookbook/
https://github.com/dmachat/angular-webpack-cookbook/wiki/Angular-Template-Cache

https://github.com/frederikprijck/angularjs-webpack-starter
https://egghead.io/courses/angular-and-webpack-for-modular-applications



Wednesday, June 13, 2018

PowerShell Tips

I'll likely update this as I go along


Create a file like you would with touch

echo $null > filename

Wednesday, May 16, 2018

git-run for managing many repositories

Some may make use of a monorepo, for example over there at Google, however for those of us that don't and have to deal with multiple repositories, possibly unrelated ones across various projects, then a great tool to have in your toolbelt is git-run, or just gr.

It can keep track of where your repositories are located, and can also be used to tag the various repositories. You can then run a git command across all of these repositories (or based on a tag).

To set it up:

$ npm install -g git-run
$ git tag discover PATH

One way I have been using that is to create a tag per project, so across a project which may have many repositories, I can see my git status across all of those by executing:

$ gr @project status

It can also run normal shell commands as well

$ gr @project cat package.json


One thing I realised it didn't do in my git bash shell, was run my aliases that I tend to use (e.g. I have a complicated bash alias for a git log command). But luckily someone else has already got an open pull request to fix this (at least for bash), which suited my needs. Yay for open source!

Try it out and let me know what you think :)

Update: I also recently discovered when trying it on a new computer - make sure you don't forget to add tags to the repositories it discovers during gr tag discover, because it won't save its configuration and you'll get an error.

Monday, March 26, 2018

Package manager tips

Initially I was just going to post about how to install npm packages while offline, but in case I ever come back to update this page with more tips or package manages, this post is now titled "Package manager tips".

npm tips

  1. Configure initial values for npm init -y
    • npm config set init.author.name YOUR_NAME
    • npm config set init.author.email YOUR_EMAIL
    • npm config set init.license UNLICENSED
  2. Installing a package offline:
    • versions older than npm 5, use the local-npm package
    • npm install --prefer-offline
  3. Update a package, e.g. Angular CLI
    • npm uninstall -g @angular/cli
    • npm cache verify # if npm < 5, npm cache clean
    • npm install -g @angular/cli@latest
  4. View the versions of a package:
    • npm view webpack version --json
    • npm view webpack versions --json

yarn tips

  1. Configure initial values for yarn init -y
    • yarn config set init-author-name YOUR_NAME
    • yarn config set init-author-email YOUR_EMAIL
    • yarn config set init-license UNLICENSED

Friday, January 19, 2018

CommonAssemblyInfo

I'm writing this post since I posted this on Twitter, and within half an hour, was asked to share my knowledge:


In Visual Studio, there's a little known feature that lets you add files as links to a project. Using this trick, we can also ease maintenance slightly by having a CommonAssemblyInfo.cs file where common attributes across the assemblies in a solution can be applied, but maintained in a single file.

Here is the sample demo application to show the steps one normally goes through. We have an existing command line utility to calculate tax based on an entered amount and tax rate. Quite simple.


In the Properties folder there is an AssemblyInfo.cs file listing the various attributes applied to the assembly.


This isn't always set up as expected, because most people forget to look at this file. One thing that stands out for example is the Company and Copyright information is defaulted sometimes to Microsoft or some other vendor based on the system settings. This may be undesirable, and so we can update these settings.


See some links at the bottom of this page for more info on these attributes.



You can see these are shown in the properties on the executable that is built.

Maybe we go back to look at our application and want to separate out the domain aspects of the console application, perhaps because in future we may want to share that with a web application frontend. The first step one would do is creating a new project and moving the classes around, fixing references, etc.


But you'll notice that most of these attributes are the same. The only two that maybe would be different are actually just the title of the assembly and the Guid it uses. If we rename things someday, we'd have to update at least two files (maybe we rename CommonAssemblyInfoDemo to something more useful, for example).

The first thing I usually do when creating a solution with more than one project, is first creating a file called CommonAssemblyInfo.cs that lives in the same directory as the solution file (CommonAssemblyInfoDemo.sln). This is usually done outside of Visual Studio, and then using Visual Studio to add the existing item to the solution, to make it easier to edit. I also setup solution folders that are numbered to keep things nicely organised.


You'll notice that only the common attributes are left in this file. But we still need to incorporate it into the other projects.

To do this, we also need to add it as an existing item to each project, however, there is a trick - we need to add it as a link.




You'll notice in the Solution Explorer that the icon of the file has a link symbol in it, to indicate it's just a reference to an existing file, so that all links refer to the same file.

Now we just update each projects AssemblyInfo to just include the information related specifically to that project.


If we need to update the AssemblyVersion across the entire solution, it's now a simple matter of just updating the CommonAssemblyInfo file to do this, instead of going through each project and remembering to update each one.

Hope this helped someone, the extra effort when setting up a new project can sometimes save time in the future.

For example, if we had a continuous integration build, and we just wanted to update the version number across all these artifacts based on some number set in our CI system, we could just update the CommonAssemblyInfo file before we run the solution through the build.

Links




Thursday, November 30, 2017

bash startup files and aliases

Today I hosted a Beer and Tech session at Entelect on Automating Database Deployments (with Flyway).

During the session, I was using the Bash terminal to edit files or run Flyway, and interacting with git quite a bit - as we all know, using git on the command line is the best way to do so ;D

After the session, a colleague asked me how I created shortcuts for certain commands, and I explained to him how Bash supports aliases and that these can be loaded by a startup file. There are various startup files, but I tend to use ~/.bash_profile and not ~/.bashrc. Here are some of the shortcuts I currently have setup:

alias cls='clear'
alias l='ls -Al'
alias ..='cd ..'

alias gf='git fetch --prune'
alias gl='git log --name-status --abbrev-commit'
alias grh='git reset --hard'
alias gs='git status'

And so on. If you modify the file, you have to re-load these into the current session, or restart a new terminal session. To reload the file, you can just do the following:

$ . ~/.bash_profile

Hope this helped someone :)

Thursday, July 20, 2017

git - exclude files in a diff

In some scenarios you might have a build server that doesn't have access to the Internet, in which case if you're working with something like npm, you might need to commit your node_modules folder to the git repository to get stuff to work.

Of course, it might be better to setup your own artifact repository such as Nexus on an internal network for these dependencies, BUT, I didn't have that! :D The node_modules are included as part of a commit.

Having to do a code review of a pull request can be tricky because of this, due to more than 10,000 files being in the commit, of which only 40 are actual source files of interest. We're using TFS 2017 which has a nice web interface for doing this, however it only supports showing 1000 files at a time.

So back to the best way to use git: the command line :)

To exclude the node_modules from the diff was as simple as doing the below:

$ git diff master develop -- . ':!node_modules'

Of course, I used difftool because I prefer WinMerge but it works the same.

Wednesday, April 19, 2017

Finding currently executing queries and parameters

If you've worked with an Object Relational Mapper (ORM) such as Hibernate, NHibernate, Entity Framework, etc., then you'll know these sometimes convert your queries into parameterized queries that can become quite beastly.

Trying to debug it means either profiling the database to see what queries are executing, and then seeing the query and values from there, or if you can't profile it, maybe selecting from some system tables to do a similar thing.

This snippet below (thanks again StackOverflow) will give you some nice details about the queries on a database, including an XML query plan that contains the parameters that were used when generating the query plan. It can be useful in diagnosing slow queries:

select * 
from sys.dm_exec_requests r 
cross apply sys.dm_exec_query_plan(plan_handle) as qp
cross apply sys.dm_exec_sql_text(r.sql_handle) 
where r.database_id = DB_ID('<dbname>') 

Thursday, March 23, 2017

tail on Windows with PowerShell

One of the well known Unix/Linux commands is tail, which gets the tail of a file, meaning the end of a file and prints it out to the output stream.

I used to try find nice programs for this, including Baretail which has some very nice features like colouring lines in that match certain patterns etc.

But if you just want a simple, and now built into Windows solution, just use PowerShell:

Get-Content -Tail 10 filename.txt

This will show the last 10 lines of the file. You can even follow the tail, or watch it for changes, as below:

gc -Tail 10 -Wait filename.txt

Another nice thing you can do is then pipe this into Select-String to filter the output:

gc -Tail 10 -Wait filename.txt | Select-String -Pattern somepattern

Monday, March 20, 2017

diff PDF files

At a client we have a PDF template that needs to be used for registering users. When this is updated, the template is overriden. But it would be nice to be able to easily see what changed between the two.

Well, this is possible :)

You can use Imagemagick and Ghostscript to do this.

You should be able to just do the below:

magick compare old.pdf new.pdf diff.pdf

But that didn't seem to work well. What seems to work better is to rather convert the PDF to an image first, and then compare each page.

magick convert -density 300 -quality 100 old.pdf old.png
magick convert -density 300 -quality 100 new.pdf new.png

magick compare old-0.png new-0.png diff-0.png

Depending on the options you can get better or worse results. Changing the colorspace to CMYK for example could yield better results, or maybe using options to blur/sharpen/despeckle the image too. Try play around with the options to see better results.

The best results I had was actually via using Adobe Acrobat to first export the PDF to images, and then run the compare. I still need to figure out why, because unfortunately I only have the license at work, so this won't always be an option. I'll hopefully update this post in future to show my favourite variation :)

Tuesday, February 21, 2017

Modern Database Development Practices

Back in November 2012 I presented on this topic at Entelect's Dev Days.

I've also done a blog post previously on setting up Flyway.

I keep referring back to the same links and concepts and this post is here just for that, as a pointer to good resources and brief points about this topic.

I may revisit it and put down my own consolidated content, but for now this will do.

Wednesday, February 15, 2017

C# - Testing that different cultures won't affect formatting

Ever worked on a system where you write code and test it and it all works perfect, but then maybe a unit test starts failing on a build server, or maybe a report looks wrong to the consumers of the report, all because it formatted a number to "12345,67" instead of "12345.67"?

This tip will help you.

First, lets assume we have this code:

public class ReportFormatter
{
  public string Format(decimal value)
  {
    return value.ToString();
  }
}

And a nice little unit test for it:

[TestClass]
public sealed class ReportFormatterTest
{
  [TestMethod]
  public void Format()
  {
    var sut = new ReportFormatter();
    
    var result = sut.Format(12345.67M);
    
    Assert.AreEqual("12345.67", result);
  }
}

This works perfectly fine. Lets even imagine that all our machines all have the same setup, and all are set to use the same regional settings. Great, nothing should ever break.

Until maybe Microsoft releases a patch to Windows that changes our regional settings to be "correct" - in fact, South Africa should be using a comma as a separator... even though none of us use this standard :D

So then it breaks our code, and our business rules that disagree with it.

Well, the good news is we can change the regional settings of the running thread, by changing its CultureInfo details. Here is a little utility class to do so:

public class TemporaryCultureSwitch : IDisposable
{
  private readonly CultureInfo _originalCulture;
  private readonly CultureInfo _originalUICulture;
  
  public TemporaryCultureSwitch(CultureInfo cultureInfo)
  {
    _originalCulture = Thread.CurrentThread.CurrentCulture;
    _originalUICulture = Thread.CurrentThread.CurrentUICulture;
    
    Thread.CurrentThread.CurrentCulture = cultureInfo;
    Thread.CurrentThread.CurrentUICulture = cultureInfo;
  }
  
  public TemporaryCultureSwitch(string cultureName) : this(new CultureInfo(cultureName)) { }
  
  public void Dispose()
  {
    Thread.CurrentThread.CurrentCulture = _originalCulture;
    Thread.CurrentThread.CurrentUICulture = _originalUICulture;
  }
}

We can now update our test to be a bit more specific:

[TestMethod]
public void FormatShouldNotBeAffectedByCultureChanges()
{
  var culture = new CultureInfo("en-ZA");
  culture.NumberFormat.NumberDecimalSeparator = ",";
  using (new TemporaryCultureSwitch(culture))
  {
    var sut = new ReportFormatter();
    
    var result = sut.Format(12345.67M);
    
    Assert.AreEqual("12345.67", result);
  }
}

Now we have a test that will fail consistently! Time to fix the code. One way of doing this is realizing that there is an overload of Decimal.ToString that takes in a CultureInfo object. We actually can use the InvariantCulture as below:

public string Format(decimal value)
{
  return value.ToString(CultureInfo.InvariantCulture);
}

The test passes and we now know for sure that regional settings won't affect our code.

Wednesday, January 25, 2017

Uninstall Windows Service using PowerShell

You can get a list of services using the Get-Service cmdlet, for example, to retrieve a list of all running services you could run:

Get-Service | Where-Object { $_.Status -eq "Running" }

You could even retrieve a specific service and start/stop it:

$service = Get-Service DemoService
$service.Start()
$service.Stop()

However, there is nothing to remove the service on this object.

We can also use GetWmiObject to do the same things:

$service = Get-WmiObject Win32_Service -Filter "name='DemoService'"
$service.StartService()
$service.StopService()

But this WMI object actually also exposes a delete functionality, which will mark the service to be deleted.

$service = Get-WmiObject Win32_Service -Filter "name='DemoService'"
$service.Delete()

If there are resources that are being held onto, it will only be deleted once they're freed up - whether it's the service that needs to gracefully shutdown still, or maybe the services manager snap-in (services.msc) has an open properties window open (I think that's happened to me before), and closing it then puts the delete through.

Thursday, July 7, 2016

Updating NuGet extension on Visual Studio 2015

NuGet is pretty awesome, though for some reason the NuGet Package Manager for Visual Studio 2015 extension was re-done and every time I try to update it I get an error, and then the extension seems to not work when restarting Visual Studio.
Took quite a while to figure out why, but it seems that it's to do with not being able to uninstall the old version of the extension. Extensions are installed into the folder C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions (or similar depending on the VS version). Certain folders in this path relate to an extension per folder, those that look like randomly generated names specifically, as below:
404 - Image Not Found
Each time the NuGet extension tried to update, it would fail to remove the old installation, and then create a new folder for the 'upgrade'. If you tried to upgrade it a few times, you may have quite a few folders that contain the NuGet extension. When you restart VS it seems to have issues loading the extension because of this. Go through each of these to find those folders, e.g.:
404 - Image Not Found
Close Visual Studio, and then delete each of those folders specific to the NuGet extension. Then manually download the VSIX extension installer from the website, and install it outside of Visual Studio. Once complete, start up Visual Studio and the extension should be installed properly. Whether it works or not is another thing, it seems the GUI is quite buggy with this extension, so I prefer using the Package Manager Console.
Hope this helps someone, I've had to come back to doing this a few times, so it'll definitely help me in future I'm sure :D

Friday, March 18, 2016

Set up IIS using PowerShell

When you're setting up a new server there are sometimes steps you need to do before you can configure your website, such as turning on Windows Authentication in corporate environments, or perhaps installing the features necessary for an ASP.NET application. I had to do this recently when setting up new servers at a client:
404 - Image Not Found
404 - Image Not Found
404 - Image Not Found
There was this final screen though where I noticed that this configuration can be exported to an XML file and set up via PowerShell. When inspecting the XML that was generated however, it seemed to include specific host names and so may not be a very generic way of setting up a server. After a bit of investigation, there are other commands that one can run to view what is installed, as well as install specific features.
404 - Image Not Found
404 - Image Not Found
Notice too that it seems to install other dependent features by default, as I just specified ASP.NET 4.5 but the other related features were also installed (in the GUI version you would be prompted about this).
Could be quite useful to have this as a script instead of remembering which features to turn on and off. My PowerShellFu isn’t the strongest, but something along the lines of this should work (I formatted this on separate lines for readability, you will want to run it as a single line):

Get-WindowsFeature -Name Web-*
  | Where-Object
  {
    ($_.Name -eq "Web-Asp-Net45" -or $_.Name -eq "Web-Windows-Auth")
    -and
    $_.InstallState -ne "Installed"
  }
  | Install-WindowsFeature

This may be a bit more generic, and could be used as a script included in source control for setting up new servers, instead of manually doing this via the Server Manager. There are also more advanced options to the cmdlets, more information of which can be seen in the documentation, one example of which is specifying a ComputerName.
One scenario I could think of for this is when setting up a web farm, and perhaps you have multiple hosts you need to do this on. Instead of doing each separately, you could probably pipe in the hostnames to a setup script, and let it perform the setup on each host. Though, as said, my PowerShellFu isn't strong, and I've had issues setting up remote PowerShell admin before, so I'll leave that part up to you ;)

Wednesday, December 2, 2015

Getting started with automating database changes using Flyway

Intro

For those of us that strive to get to a "One Click Deploy" - where we press a magical button that automatically deploys all of our changes - one of the stumbling blocks during deployment is if there is a relational database involved where schema and/or data changes still need to be run manually.
Sometimes this is unavoidable, but assuming it's not, how can we achieve a fully automated deployment when we still need to worry about the database? For the trolls out there, don't say "Just use NoSQL".
I spoke about the general idea behind database migration strategies at one of Entelect's dev days, and the concepts are still the same (see the end of this article for some references). Since then, there are two tools that I've become aware of: Liquibase and Flyway. This will go through using Flyway with SQL Server 2014.

Setup Flyway

Flyway is a Java based library, but also comes as a standalone command line tool. I think that means you don't need to install the Java Runtime Environment (JRE) to use it, but I already had it installed so haven't verified that yet.
  1. Go to the Flyway command line page
  2. Download flyway-commandline-3.2.1.zip
  3. Unzip to C:\flyway-3.2.1
Since we're using SQL Server, we also need to make sure the right JDBC drivers are available for Flyway to use. Although the documentation does show support for using both the Microsoft and jTDS drivers, I found that the jTDS library doesn't support Windows Authentication mode for SQL Server 2014 (it might for older versions), and I had to download and set up the Microsoft driver in any case:
  1. Go to the Microsoft JDBC driver download page
  2. Download either sqljdbc_6.0.6629.101_enu.exe or the sqljdbc_6.0.6629.101_enu.tar.gz tarball (I chose the latter)
  3. Unzip the tarball
  4. Choose the right JAR depending on Java version and JDBC compliance, I went with sqljdbc42.jar, and copy this into C:\flyway-3.2.1\drivers
  5. Since we need Windows Authentication, copy sqljdbc_auth.dll into a folder on the PATH
    • I copied it into C:\flyway-3.2.1 since I'll run the command line tool from that location
There are a few configuration options available, but for now we'll just need to setup our connection string and driver in C:\flyway-3.2.1\conf\flyway.conf:
flyway.url=jdbc:sqlserver://localhost;instanceName=sql2014;database=FlywayDemo;integratedSecurity=true
flyway.driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
Technically you shouldn't need to specifiy the driver explicitly, but I prefer being explicit to avoid surprises. In the above connection, I'm connecting to a database FlywayDemo on a named instance called sql2014, and making use of integrated security (Windows Authentication).
You should now be able to run the command below to verify that Flyway can connect to the DB:
404 - Image Not Found
The above shows that it can connect to the database, and that there were no migrations found (we will start with that next). You shouldn't get any errors at this stage, though you may need to configure your database to allow TCP connections if it hasn't been configured to do so already: 
404 - Image Not Found

Baseline

If you're lucky enough to work on a brand new project with a brand new database, you can just create an empty database, and each change to the database can be created as a new script, each run in order.
But most of us have an existing database we want to work from. For this demo, I'll just use a simple database with a single table as below:
404 - Image Not Found
We can use Flyway to baseline this version of the database, say, version 1.0:
404 - Image Not Found
This creates a new database table schema_version in the default schema of the database, and tags the database as version 1.0. We could create a backup of this database, and use it as our baseline database to work off of, and when another user restores the database backup, they can use flyway to inspect what version of the database they are using:
404 - Image Not Found

Database migration changes

With Flyway, each change to the database should be scripted as an incremental change. This allows us to migrate database changes forward from whatever current version our database is at. Flyway uses specifc conventions for database migration scripts, and we'll follow the default. Scripts are to be placed in the folder C:\flyway-3.2.1\sql, and follow the naming convention prefixVERSIONseparatorDESCRIPTIONsuffix, e.g. V1.1__Some description.sql.
Lets say we want to change our database structure to extract the Type column of the Person table into its own table. This would involve:
  1. Creating a new table, PersonType
  2. Inserting necessary data into PersonType
  3. Adding a new PersonTypeId foreign key to Person
  4. Updating the Person data for the new PersonTypeId
  5. Dropping the old Type column on the Person table
We could lump all of these in a new V1.1 script, however the recommended practice is to split each DDL change into its own version, since some database vendors don't handle transactions very well with DDL statements, and Flyway tries to wrap each migration in a transaction so if any migration fails it doesn't leave the database in an inconsistent state.
Instead, we'll just make each change it's own script, by default:
  • V1.1__Create PersonType.sql
  • V1.2__Populate PersonType.sql
  • V1.3__Create Person.PersonTypeId.sql
  • V1.4__Create FK_Person_PersonType.sql
  • V1.5__Update Person.PersonTypeId.sql
  • V1.6__Drop Person.Type.sql
  • V1.7__Make Person.PersonTypeId NOT NULL.sql
After we've created these scripts in the sql folder, if we re-run flyway info, we will see that we now have pending changes that need to be applied to the baseline:
404 - Image Not Found
We can then use Flyway to migrate our database to include all of these changes:
404 - Image Not Found
404 - Image Not Found
You'll notice that the version and description columns are pulled from the filename convention. The checksum column is actually a checksum of the contents of the migration script. That is to avoid changes to scripts that were already run in - instead of changing existing scripts, prefer creating a new script with a new version to do the necessary change. If we did modify a script, and run validate, we should see the issue being highlighted:
404 - Image Not Found
Sometimes though, it may make sense to modify an existing script, as long as it's functionality doesn't change - lets say the script takes forever to run the migration, and we find a way to optimize it, then it makes sense to change the script later on. In this case, we can repair the metadata stored in the version table using the repair command:
404 - Image Not Found

Development time considerations

This way of working does mean you'll need to be more vigilant with how you make database changes. Changes should be done in small incremental changes. An issue when working on teams may be deciding on how to allocate version numbers - if we're all working in parallel, we may all choose to pick V1.1, whereas my changes might only be deployed after my colleague's changes, meaning his should be V1.1 and mine V1.2.
If you were using feature branches, then you could version your changes for your feature as V1, V2, etc. in a separate folder maybe, and only when merging with the common source repository, would you move the files into the sql migration folder with their actual version numbers, e.g. we merge the development branch, notice that someone has implemented V1.6, we can then decide to rename our scripts V1.7.1 and V1.7.2, etc.

Continuous Delivery

What this all enables is a truly continuous delivery option. This would be the typical flow:
  1. Developer checks out latest version of the code
  2. Developer gets latest database setup
    • either restore latest database backup, and migrate pending changes,
    • or rebuild the whole database using Flyway
  3. Developer makes changes for their feature, both code and migration scripts
  4. Developer pushes all changes to source control
  5. Continuous Integration build runs
    • gets latest code
    • compiles
    • runs tests
    • gets Integration Test database setup as above in (2) using the tooling
    • runs integration tests
  6. Continuous Deployment build runs
    • migrates database
    • deploys artifacts
This could be setup to happen automatically for every commit, maybe to deploy to a Quality Assurance or User Acceptance Testing environment (or both). This could even be done to drive Production environment deployments, though that may be more complicated, and may be better done by clicking a button to initiate the deployment rather than automatically deploying after a code change.

Wrap up

Continuous delivery is a part of the devops community that's getting a lot of focus in recent times. Hopefully this article has inspired someone to automate their own deployment process. For those completely new to the database side of this, I recommend reading the links below. For those that aren't new to this, have you tried this on an actual project, and if so maybe share your war stories in the comments! :)

References

Tooling

Articles