Category Archives: Musings

General musings of a disturbed mind

Why is software such a monster?

One of the questions I ask first when I give talks to prospective students or public groups is “What is Software Engineering?”, and then get their responses. Usually the answers are similar; “programming”, “building an app”, “writing code” – all along the technical development line.

My next step (not that it does much for student numbers) is to tell them they’re wrong, that people focus on the programming and development aspects but in fact software engineering is not just programming in the same way that bridge building is not just bricklaying. If you were to say someone was a “Bridge Engineer” it could be they did everything from designing, testing, model making, laying foundations, sinking piles, suspending cables, or, yes indeed, bricklaying. In fact I say, no doubt smugly and patronisingly, “Software Engineering isn’t a job or a profession, it’s a whole industry”. The next 20 minutes or so are then filled as I labour the point and talk about the roles from requirements engineering through programmer to systems engineer, before they troop dejectedly out for a free lunch.

But this got me thinking; what if we did build bridges in the same way we build software?

Certainly there’s no doubt software engineering has advanced leaps and bounds since the Software Crisis of the 60’s, and a great deal has changed since Fred Brooks’ famously categorised software projects as “werewolves” (monsters that can turn around at any time and bite you!). But, advances aside, software projects can and do still fail, and the world is full of semi-functional legacy systems nobody dares to touch for fear of the house of cards collapsing.

Sure, in the formalised world of big business and large projects, generally we now expect projects to succeed. Good requirements engineering (knowing what you should actually be building!), software design (a sensible design to meet the goals and evolve), iterative development and agile approaches (building it a piece at a time, checking in with the customer on a regular basis, responding to change), testing (making sure it actually works!), and use of fancy modern cloud infrastructure (making it someone else’s responsibility to keep the lights on) has made this happen. But is this how most software is developed? As we’ve democratised software development, made a million open access resources and put a computer in every pocket, we are now in a position where most coding is being done in bedrooms or in small offices by individuals or tiny teams. Is this all being done in adherence with best practice on requirements traceability and agile principles? Probably not.

Are even we as professional software engineers eating our own dog food? How many utility scripts or programs in our home directories could be referred to as “quick dirty hacks”?

So, what if we built bridges the same way we (a lot of the time) build software?

We’d start with a seemingly simple requirement – I need to cross a river.

Bridge-1

A simple solution then – drop some big rocks in which can act as stepping stones.

Bridge-2

Brilliant! This solves the immediate problem and our gallant hero can cross to market and buy some bronze, or a chicken, or something.

However in the cauldron of innovation that is humanity someone goes and invents the wheel, and creates a cart. After trying to attach it to the aforementioned chicken, an unimpressed pig, and some crows, someone tries a horse and the winning formula is found. Except horses and wheels can’t use stepping stones (the requirements have evolved).

Simple answer – lay some planks over the stones.

Bridge-3

Time passes, the wheel is voted “best thing prior to sliced bread” four years in a row, and horses and carts cross our bridge back and forth. But there’s no stopping progress, and some lunatic builds a steam-powered cart on rails known as a train, much to the anger of the horses’ union. The train is much heavier than the cart, can’t go up and down banks, and needs rails so we modify the bridge again. We raise the deck, lay the rails, building on top of the planks resting on the stepping stones.

Bridge-4

Trains, it turns out, are way more popular than horse-drawn carts, and make much better settings for murder mysteries. As their number and use increases they get heavier and heavier, faster and faster. No longer will the rickety beams hold up so the bridge is reinforced over and over, as each new train comes onto the line.

Bridge-5

In time even steam becomes passé, some joker goes and puts the new-fangled internal combustion engine in it’s own little chariot and the age of the motor-vehicle is born.

Quick as a flash the bridge solution is found, and a new suspended roadway is bolted on the top!

Bridge-6

Evolved from stepping stones, still standing on those very foundations, and being used heavily day in, day out. It works, mostly, taking more volume and type of traffic than ever before, outliving by far the stepping stone engineer and many afterwards who cared for and modified it through the centuries.

Here we are, bang up to date, with a legacy bridge that has evolved over the years and with the majority of the engineers dead and gone, some from the black death.

Now Dave is responsible for maintaining the bridge, the hodge-podge of obsolete half-forgotten technologies, built by people long since gone using techniques long since lost to the sands of time.

One day, carrying about his usual bridge duties, he sees a cable he thinks is loose…

Bridge-7A

And being a conscientious sort he gives it a good old tighten up.

Bridge-7B

So the bridge collapses, spectacularly, probably just as a busload of orphans and accompanying nuns are travelling across. There is outcry, rage, incomprehensible gibberish of outrage on twitter, fingers pointed at the government, promises of enquiries, and a desire on all sides to find someone to blame. The burden of blame naturally falls on the obvious suspect, the man with the smoking wrench in hand, Dave.

Bridge-Result

We know who to blame, and everyone can rest assured it was definitely, and uniquely, his fault. He rots in jail, flowers are laid at the old bridge site, and we start all over again.

But is this really how software is developed?

I believe so, yes. Certainly more often than it should, and more often than we would like to admit. All too often software is quickly (and dirtily!) built to solve the minimum problem right now, and even at the time we say things like “when I get time I’ll come back and do this properly” or “well this will do for now until we replace it/the need goes away”, and how often do we do that? We could just as well say “well this will do until pigs fly to the moon and prove it’s made of cheese”.

But the initial quick dirty solution isn’t the problem. The issue comes with uncontrolled evolution. As much as night follows day (follows night, follows day, etc) requirements will change, we in the IT industry exist in a state of constant flux, nothing stands still for long. By it’s very nature successful software and successful systems will be required to face change and evolve to meet new requirements continually. When we “hack” these requirements onto our, only intended for short term use, original “quick and dirty” solution, we rapidly introduce all sorts of unintended complexity and integration issues – we make ourselves a codethulu or screaming tower of exceptions.

Yet we still rarely learn our lesson because once we’ve hacked one new set of requirements in and it works, sort of, in a “don’t press G on the third input field or it’ll crash” kinda way we stride away feeling like code-warrior bosses and not looking back. Until the next inevitable change.

So Brooks is still right?

Well obviously – the man is a genius after all. The essential difficulties he identified with software are still present in most cases.

  • Complex – Check! More complex than ever really. As computer power grows so does our demand on what we can do with it. Of course if we properly decompose a problem down we can reduce the complexity, but that’s doing things properly.
  • Subject to external conformity – yes siree! External stuff is changing all the time and it’s expected that the software can be tweaked to conform; we don’t change the printer spec to make it work with the software, we change the software.
  • Changeable – more than ever! Change is a natural part of software and entirely unavoidable, unless you’re MySpace.
  • Invisible – Unlike bridges we can’t see software, we can’t kick it, or go and marvel at it’s grandeur, prod a few bits to understand it’s function. Yes now we have UML and various design tools but only if we do things properly which, often, we don’t.

But most of the time even unplanned development projects deliver, so haven’t things got better?

Ah yes, what I like to call the “fallacy of the initial success” (catchy name, right?). If I set out to build a software system to do X and I stick at it, I normally expect the system will eventually do X. The project probably won’t fail partway through into a pit of recrimination and blame.

This is because we now have some amazing tools at our disposal. Web scripting languages combined with HTML and a browser mean anyone can build a program with a UI. Memory managed (garbage collected) languages mean who cares about freeing resources, or causing hardware resets by accessing the wrong address in RAM. Stack overflow will answer most queries for us, and the world is full of talented developers who put their libraries and source free and available online for us to use. So to sit down and build something to do X, no problem! Everything is rosy and it’s home for tea and medals!

The problem however is Y and Z, new requirements that follow on with apparent inevitability. I’ve been in the business long enough to know when I say to myself “it’ll only need to do X” that I am blatantly lying to myself.

My hypothesis then is that: rather than solving the problem with software development, we’ve mainly just transferred the risk from the front-end (initial development) to the back-end (maintenance and modification) of the development lifecycle.

It’s true we may no longer have werewolves in every project, ready to turn into beasts and sink their teeth into us – the stuff of nightmares, but we certainly now have our fair share of zombies. Many of these are the relatively benign shuffling-gait “braaaaaains…” type, easily avoided or picked off at our leisure, but there’s also a large number of buttock-clinchingly scary 28-days style vicious running zombies. These are the myriad of legacy programs we’re surrounded by, just waiting for the opportunity to smash their way out of the coffin and chase us down for breakfast brains.

So what’s the solution?

All the above is based on one premise – that the lovely and excellent principles of proper software engineering from requirements gathering, analysis, design (flexible, extensible, reusable, standardised, highly cohesive and lowly coupled), agile development, prototyping, stakeholder engagement, iterative methods aren’t being followed (which sadly I think happens most of the time – of course if you are following good practice then of course nothing can go wrong [ha!], or at least we hope not this particular mess).

So the simple solution is to follow good practice. To try and take a little more time in the design and implementation of solutions, consider the future maintainability and reuse potential; build software to last years not days or months. Listen to the little bit of us that screams with impotent rage when we “just quickly bodge this”.

We also shouldn’t ever be afraid to tear it down and start again. Iterative does not always mean incremental, if something is becoming unfit for purpose can’t we find the time to invest in sorting it properly? How much time do we really spend on preventative maintenance compared to how much we should? Is it fire-fighting or careful fireproofing to avoid the emergency?

If you want to fix the roof, better do it while the sun is shining.

There are attempts at automated approaches to help us manage these zombie programs, and in fact a large part of my research is on this kind of area. But don’t worry, if we all start coding properly – engineering not just programming – tomorrow, there’s still plenty of zombies out there already created, so you’re not doing me out of a job. My confrontational attitude and willingness to tell prospective students they’re wrong when I ask them a trick question will do that for me.

Good documentation is another good idea (assuming you’re not following a good practice that tells you documentation is the very spawn of Satan) but remember; documentation can end up evolving like code, we end up with too much of it telling us too little and unsure where to find the little nugget of information we need amongst the volumes. Dave (the bridge man) had plenty of blueprints, and if he understood them all he would have known not to tighten the fateful wire, but how was he to know?

Dave Blueprint scroll

 

David Cutting is a senior research associate at the Tyndall Centre at UEA, associate tutor, partner at Verrotech, CTO of tech startup Gangoolie, and alleged software engineer (purveyor of much shoddy half-built freeware) – he does not eat his own dog food often enough. He holds both a PhD and an MSc in computer science and by that gives even the poorest student hope. This article is a cut-down version of a presentation entitled “Software! It’s Broken” he gives when anyone will listen. Artwork by Justin Harris and Amy Hunter.

All content is (C) Copyright 2015-2017 David Cutting (dcutting@purplepixie.org), all rights reserved.

A day at Whitespace

On Tuesday I decanted myself for the day down to Whitespace Norwich a “not-for-profit co-working space”. Apparently “with a fast-growing network of entrepreneurs, investors, mentors and influencers, the Whitespace network encourages collaboration and growth within a supportive and like-minded community of tech and digital innovators”. Blimey.

To bolster our enterprise engagement and community profile the School of Computing Sciences at UEA, which is one of the places I hang out, have a desk there. So I thought I’d try it out.

Whitespace Building

Whitespace in Jarrolds Building

Whitespace takes up a floor in a classic old building by the river in Norwich, very close to the city centre and offering some nice views.

2016-09-06-10-10-55

View from the Whitespace stairs

View from the Whitespace stairs

Having had a quick tour of the facilities (well appointed kitchens and a games area) I got setup.

2016-09-06-13-05-122016-09-06-13-05-222016-09-06-13-05-24

Nice comfortable chair and plenty of power, it turned out the 5G WiFi wasn’t bad either.

2016-09-06-10-15-06

The only small issue I had was monitor poverty… used as I am to two 24 inch and my laptop screen as well… And the desk looked a bit empty.

2016-09-06-11-21-41

Other than my screen real estate issues all went well, even met and talked to a few interesting people (which for a computer scientist is quite something). Unfortunately I couldn’t capitalise on networking as I was up against it as usual. So, I’ll be back again when I’m not quite as busy.

There was a nice energetic buzz around and plenty of groovy people doing no doubt groovy tech stuff. It certainly seemed like the kind of place a new startup could grow and thrive.

Being an open-plan shared working place meant there was some noise, but this wasn’t really distracting. I did put headphones in after a while but only because I wanted to listen to some music rather than I needed to drown anything out. It was certainly quieter than a PhD lab.

All in all a really great place to work, good resource for UEA to use, and somewhere I’ll try my hardest to make use of again.

The Good

  • Friendly welcome
  • Nice kitchen and other resources
  • Fast WiFi
  • Chairs
  • Big desks
  • Centre manager on hand for any queries
  • Lovely setting (building + views)
  • Handy setting (so central!)

The Bad

  • The only one issue I had was heat. It was a very hot muggy day and there’s no air-con (it’s been suggested on the whiteboard). Everyone else seemed to have desk fans, I did not… but copious cold diet pepsi helped. Note: my UEA office would have been as stifling, but I do have a fan there.

 

Facebook PHP SDK Tips

Hmm so I wrote this in November 2014… and didn’t publish it. Then found it again in November 2015. I don’t know why it was in draft – usually because there’s something horribly wrong with it. But… seems ok, and I haven’t played with the FB API since then so I doubt I can add anything (or even that it still applies) hence… PRESSING PUBLISH:

Over the last two days I’ve been at a startup event called SyncTheCity and been building a little startup called cotravel (website, twitter, facebook). One of the supposedly easy tasks was to integrate the Facebook SDK in PHP to allow facebook logins.

I blame lack of sleep and time pressure but actually this was a pain in the behind. However I have two tips for future sufferers:

1. Development Domain

Facebook will only accept requests from domains you have linked. In our case this means our live domain. We develop locally on our machines (127.0.0.1 or localhost) but you can’t register those as valid domains.

However what you can do is create an A record in your domain to 127.0.0.1 so you now have local.yourdomain.com pointing to 127.0.0.1. You can add local.yourdomain.com to facebooks’ domains for your app and off you go.

2. URL is required at both ends

The docs say you must pass a request_uri to redirect to when calling – but you must also pass the SAME uri on return or else death and room results.

This means that your two files (http://domain.com/sender.php and http://domain.com/recipient.php) must have code like:


session_start();
FacebookSession::setDefaultApplication('appid', 'appsecret');
$URL="http://domain.com/recipient.php";
$helper = new FacebookRedirectLoginHelper($URL);
$loginUrl = $helper->getLoginUrl($params);
header("Location: ".$loginUrl);

And recipient.php:


session_start();
FacebookSession::setDefaultApplication('appid', 'appsecret');
$URL="http://domain.com/recipient.php";
$helper = new FacebookRedirectLoginHelper($URL);
try {
$session = $helper->getSessionFromRedirect();
} catch(FacebookRequestException $ex) {
echo "Facebook Authentication Error";
} catch(\Exception $ex) {
echo "Facebook Authentication Failure";
}

Note in the above this line is the one that differs from the example:


$helper = new FacebookRedirectLoginHelper($URL);

e.g. you must pass the URL!

cotravel at SyncTheCity 2014: a git history

Coming up for a year ago now I took part in the inaugural Sync the City event in Norwich, and guess what – it’s happening again!

Last year we undertook the challenge to design, build/develop and deploy our business cotravel in 54 hours. This culminated in a working system (with proper back end, usable APIs, web interface etc), several actual business agreements with local taxi companies, and finally an excellent presentation by Rod.

The event garnered quite a bit of press (in which we got a mention!) and has been the subject of some other reflective blogging by team members. Sadly I can’t take part this year (some nonsense about a PhD I have to finish…), but it got me looking back.

B3

During the two days of (frantic) development we used a git repo on BitBucket to manage our source code between the development machines, test, and live environments. This got me thinking it would be nice to visualise the development process mining the repository, if only I had the tools and computing power. Then I remembered I totally do! Analysing software systems, including from repository logs, is kinda what we do. So, waiting for a long experiment to finish, I realised I could quickly bodge a couple of our tools together and see what I could get.

During the development period (evening of 20th November to evening of 22nd November) there were two developers: David (me), and Adam (not me). So I plotted commits by hour both per developer and in total, from the very first commit at 17:28 on the 20th to the last at 16:46 on the 22nd.

cotravel-all

And there we have it, from 0:00 on the 20th to 23:59 on the 22nd. As you can see the main splurge of work (technical term) was on the 21st from 06:00 to 22:00, which is when we built the majority of the functionality. On the 22nd we again got an early start, but nowhere near the commit intensity and it petered off anyway, after go live at 14:00 on the 22nd to the presentations.

Two interesting peaks on the 21st – the first was mainly Adam putting together the agreed UI elements and repeatedly pushing it up for all to see. The second peak was me trying (and generally failing) to master the Facebook API, as it needed domains working etc and so had to be pushed to a live server to test (until I found a workaround which I really should document).

Of course once we went live…

Cotravel Lives!

We then had all manner of other stats available to us, for example a log of visitors to the site and the number of API calls made from our snazzy web 2.0 front-end to my dodgy interfaces. We went live at 14:00 and kept track every hour until 20:00.

Cotravel Live Calls (k)

Here’s the graph showing (in 1000’s) the page views and (much higher) volume of API calls made to the system from 1400-2000 when we finally collapsed.

Overall Sync the City was a great experience, a lot of fun and a great way to meet interesting people and do ninja coding (plenty of opportunities to DevOps the **** out of it, or as we used to call it; make live changes to the code while users are still interacting).

Good luck to anyone taking part this year!

Variable Length Arguments in C++, Java, and PHP

Normally in software development we define methods with a given number of parameters (and their type in some languages). Quite often however we want to be able to deal with different numbers of arguments and there are two widely used approaches; different methods and default parameters.

Different methods relies on the concept that the call to function is matched not just on the name of the method but also the count (and type if applicable) of the parameters. So if we wanted a method that could accept one or two integers in C++ we could define two methods:

So if we called SomeFunction(1) the first would be used, SomeFunction(1,2) would use the second.

Default parameters allows us to define some of the parameters as optional and their default values if not passed so the definition:

Would accept one or two integer parameters. SomeMethod(1,2) would have a=1 and b=2 whereas SomeMethod(1) would use the default value and so b=0.

This is all very good and highly useful in a variety of situations but suppose you wanted to handle any number of parameters, from a very small set (or zero) to a large number. Using either of these techniques would require a lot of additional coding, creating a method for each length or the longest set of default parameters imaginable.

This is where the concept of variable length arguments for a method comes in; we want to be able to define a method and accept an arbitrary number of arguments which it can process (please note in most if not all cases the best option for this would be to pass something like a Vector in C++ or an array in PHP, but best practice is not the point of the exercise).

Let’s consider a problem.

We want to have a LineShape function. This function takes a series of Points (a simple class just containing an X and Y coordinate). In a proper system it would then start with the first point and draw a line to each consecutive one but for our example we just want it to print a list of the points it will draw to/from in order.

This could be two points (a single line) or a complex shape of an undetermined total number of points (again note the caveat above that a Vector/List/Array would be the best and safest way to do this in TRW).

So for our implementation we need:

  • A simple Point class
  • A method (LineShape) that takes an arbitrary number of Points and prints out the coordinates
  • Code to create a set of Points and pass them to LineShape

How to do this varies from language to language and, as you might assume, it’s hardest and most dangerous in C++ (because of it’s lack of type safety), slightly easier in Java and PHP (Java because of it’s high type safety and PHP because of it’s lack of any enforced typing).

Variable Length Arguments in C++

To implement in C++ we make use of the va_ functionality provided in stdarg.h. The function is defined as taking the number of parameters passed (int) and then the parameters themselves represented by “…”.

We read the parameter count and then iterate through reading each in turn with va_arg and specifying the type to be used. Note in C++ you must specify the number of parameters being passed when calling the method.

There you have it in C++ (well actually using C libraries); but don’t do it (see above).

Variable Length Arguments in Java

In Java it’s a lot easier as the functionality is built-in to the language. Additionally you don’t need to pass the number of parameters and also the type is determined for the entire set of parameters (in our case Point).

Note that in order for Point to be instantiated as non-static it must be in a seperate file (Point.java).

So our Point class is:

And the main Var.java contains:

So in Java we just need to declare a method with Type… name and then iterate through the array in a for loop.

Variable Length Arguments in PHP

PHP isn’t quite as built-in as Java (an actual language construct) but PHP natively provides functions to support variable length parameters to methods such as func_num_args (number of arguments) and func_get_args (arguments as an array).

 

PHP Dynamic Factories

Design patterns are common solutions to commonly-encountered problems in software development. Of these the most widely used are creational patterns – methods of creating objects, most notably the factory pattern. The factory pattern is used for the centralised instantiation of related class objects.

Let’s illustrate this with an example using the idea of shapes. We want to have an abstract base Shape class and then concrete classes derived from the base class of specific (or concrete) shapes.

We would, without any of this factory nonsense, instantiate the classes directly:

Using the standard factory method rather than instantiating directly we would create classes to handle the instantiation for us. First we define an abstract ShapeFactory class and then specific concrete factories to create individual shapes. So adding to our previous code:

So now, rather than creating instances directly we first create a concrete factory instance and then use that to create the concrete shape:

This is all very well and good (and design patterns in general are a very powerful and useful tool) but it fails to take advantage of PHP’s powers of runtime adaptability, the ability to change and update code behaviour and functionality during execution (at runtime).

For example having these factories provides us with a standard way to create shapes but how would we add new shapes easily. With the factory pattern we would still need code modification, new shapes require a new factory and for this factory to be explicitly used to create objects.

With the wonder that is PHP however we can be more flexible, more dynamic.

Consider the possibility we want to be able to create any type of shape from a factory.

As an example you could have a Shape Maker class which took a text string for the type and returned an appropriate object (this is known as a paramerized factory):

This would allow the creation of shapes as follows:

But we would really like to go further than this; we want to create a dynamic factory, one in which shapes are simply registered with a type and a associated class name, and then created by passing the type.

For this to work we would need the following:

  • A list (array) of registered types and their class names
  • A method to register new types
  • A method to create an object of the given type
  • A fall-back; what to do if a creation request is made for a type that doesn’t exist

To make this all even easier to use in our example the class will be an abstract class using static members and methods (though of course the same idea holds true for a non-abstract class using non-static members, it would just need to be instantiated first).

So, using the same shape code but replacing the factory code with the following:

We create a ShapeFactory class with the ability to register, check the existence of, and create shape classes. All that remains to do is to register the Square and Circle classes with type names (the lower case versions). Once this is done they can be created with:

Which creates objects of Square and Circle using the dynamic type identifiers.

So what is the advantage of this?

Well imagine now in our code we wish to add a new shape. This shape (Triangle) is contained in a file triangle.php which may or may not be included at runtime with an include_once. We may want to limit the inclusion of triangle to an add-on pack or for just specific users, so we have logic to decide if triangle should be included.

All triangle.php needs now to contain is:

And then if it is included the triangle type becomes automatically available from the ShapeFactory. If ShapeFactory had been extended (very easy) to return a list of possible shapes which was used to populate a UI component then triangle would now appear, could be selected, and used as a type identifier to create a concrete shape of the correct type (triangle) at runtime.

The idea of the dynamic factory can be taken much further with list provision or description fields for example and is with certain purplepixie.org products such as FreeDESK.

Another layer of abstraction could also easily be added; all the functionality of the ShapeFactory would be generic to any dynamic factory, just the list of types and objects being unique (if this was a non-static class these could be just different instances of a general DynamicFactory class). In the static example we create a DynamicFactory and then extend from it product-specific dynamic factories to be used to house the lists of specific types:

Some links to source files for download:

classic.php : Shows the classic factory pattern

dynamic.php : Shows the first dynamic iteration along with the maker class

dynamic2.php : Shows the finalised version