Why do SharePoint Projects Fail – Part 6

Hi again and welcome to part 6 of my series on the factors of why SharePoint projects fail. Joel Oleson’s write-up a while back gave me 5 minutes of fame, but like any contestant on Big Brother, I’ve had my time in the limelight, been voted out of the house (as in Joel’s front page) and I’m back to being an ordinary citizen again.

If you have followed events thus far, I covered off some wicked problem theory, before delving into the bigger ticket items that contribute to SharePoint project failure. In the last post, we pointed our virtual microscope at the infrastructure aspects that can cause a SharePoint problem to go off the rails.

Now we turn our magnifying glass onto application development issues and therefore application developers. Ah, what fun you can have with application developer stereotyping, eh! A strange breed indeed they are. As a group they have had a significant contribution to the bitter and twisted individual that I am today.

The CleverWorkarounds tequila shot rating is back!

for a project manager in denial 🙂

for the rest of us!

Leave us sensitive developers alone already!

Oh please.. I could make insultingly broad, sweeping generalisations about developers to make people laugh, but you guys hate each-other more than you do the rest of us! So any insult that I come up with will be water off a duck’s back and you will no doubt have a good come-back insult loaded up and ready to shoot me down 🙂

Case in point? Go and read this hilarious post by Mark Pilgrim – an oldie but a real goodie. Warning: there is profanity there. I’d dearly like to repeat it here because it’s pure gold, but I wear a suit now and I am supposed to sound all "impressive and wise". Got to keep up appearances 🙂

Mark’s post is mandatory reading for anyone following this series. Stop now and read! (It’s cool – I’ll wait for you).

** cue elevator music **

All done? Excellent. Let’s move on!

Mark is the sort of developer who would be worth his weight in gold. What is great about the above post is that it is actually an excellent example of the social complexity of projects and why it is so incredibly challenging to get a project team (stakeholders and user reference group included) to see a common version of the truth. Although Mark nails his fellow developers to the wall here, an equally witty project manager, information worker or senior executive could write the same sort of post for their discipline area.

In common with what I said during the infrastructure oriented post, a lot of application development issues are simply a manifestation of wicked problems that have not been recognised earlier. Although everything I have outlined in posts 1-4 is where you will find your root causes, in an organisation with a low maturity level the project team likely has been largely oblivious to the warning signs up until this point. Suddenly once we are cutting code, problems start to become apparent and it’s all those damn developers fault of course!

So once again, for the rest of this post we have to do a suspension of belief. In part 5 we went to Homer Simpson’s land of chocolate. So for this post, how about we can all pretend that we live in a world where teenyboppers do not exist and therefore people’s minds haven’t turned to jelly from being bombarded by crap music – so let’s blame Britney Spears for 99% of project failures.

In the world without Britney Spears

In this world, the stakeholders all agree with each-other. They have a tacit and deep understanding of the organisation’s vision, strategy and exactly how that strategy is to be executed. Therefore, they have clearly and unambiguously articulated this vision to the project team. This information in turn has been understood at all levels by the project team and after much investigation, soul searching and deep analysis, the entire team agrees completely that SharePoint is the perfect fit.

After reading all of the insightful information in the CleverworkArounds’ "Learn to speak to your CFO" series :-P, a project management plan has been put together and agreed upon and signed off by all stakeholders. Scope, constraints and assumptions are reasonable and clear. A work breakdown structure has been developed and signed off with well developed budget and accurate, confident estimates.

What could possibly go wrong? 🙂

Thinking your developers are developers

Jeremy Thake wrote about "organisational responsibility" recently, and my new "homie" Joel made his feelings known when he wrote about the folly of "Thinking your developers are administrators". This needs to be analysed further however. I prefer to use the title "Thinking your developers are developers". The world is rife with what one would term "bad programming". The term is something of a misnomer though. I work with a great free WCM system called Joomla, and it has a devoted community of developers writing add-on modules for a dizzying array of options. Most are very well written, but some of these modules I have hacked away at the source and quite frankly they are pretty bad (and coming from a part time developer like me, that’s saying something).

Wishing your developers are admins?

That term "bad programming" is misleading and potentially unfair, because bad programming can contain tight, readable code that would satisfy industry standard conventions. In actual fact I’m not really all that fussed about the source code unless it’s *monumentally* stupid. Instead it is all about the thought process that went into the development. Here is a real world example from my past of what I mean.

A system developed by a former employer of mine relied extensively on DNS aliases, that our customers had to add into their DNS. So when our application looked for its "server", it queried DNS for that alias. I suggested that this was not a good idea, and we would be much better served via a registry setting that could override this, centrally controlled via group policy in Active Directory. The suggestion was met with general disinterest until some time later, we had an install at a site with several physical locations.There would be a server at each location, but the customer’s Active Directory set-up was a single DNS namespace. Thus we could only add a DNS alias for 1 server, yet we had several sites where servers resided. The system couldn’t handle the fact that a DNS can spread over multiple sites.

One quickly convened meeting later and suddenly the penny dropped on the logic of my suggestion. (But it took a real world emergency for that to happen). The next problem was the developers immediately chose the wrong location for registry key. Microsoft have a guide that tells you *where* in the registry to place your modification and they didn’t know there were recommendations and didn’t think to check for such a recommendation. Fortunately I caught that one in time and got it sorted.

That example highlights that "bad programming" was actually more like lack of awareness of important external considerations. I named the title of this section, "Wishing your developers are admins", more for the example that I cited than a statement of fact. Certainly, infrastructure administrators are very crap coders, (despite what they think of their efforts). But they do have a greater affinity for integration oriented governance issues than their pure development counterparts because of the nature of their role.

Sniggering infrastructure people who are feeling superior now should stop and take note. Experience has shown me time and time again that greater awareness about governance and integration rarely translates to actually running their environment to the ideals they espouse however 🙂 .

Read my Choose Your Own Adventure post for a more offbeat and sarcastic take on that issue.

Planting wicked seeds..

In part 5, I had a section entitled "Infrastructure and Product Skills", and I suggested that a lack of product awareness and skills among infrastructure designers did not always manifest themselves into pain and suffering until later down the track. In the application development discipline, this is even more of a problem, and the remainder of this post focuses pretty much on this area, since if things do go down this road, it is simply going to end up being one of the root causes of the next, even more wicked problem to try and solve.

The impact of a bad design decision today can in particular have far reaching consequences into the future. Microsoft have this problem at a level that would blow your mind if you have been around long enough know your history (and your security :-). Remember than in 1973, what Rittel said of wicked problems: "There is no immediate and no ultimate test of a solution to a wicked problem", and by that he meant any solution, post implementation will generate "waves of consequences" that may yield utterly undesirable repercussions which outweigh the intended advantages.

The thoroughly depressing thing about that statement is the implication that a wicked problem can arise from the unexpected aftermath of a perfectly innocent "non wicked" problem!

So let’s see where SharePoint and the systems it relies upon makes life hard for developers and can blow out projects.

The obvious way is usually the dumb way

I am going to start by showing you some C# code. Non programmers, don’t be scared, it’s not too bad and in honour of Ms Spears I have kept the theme going. It is opening a SharePoint list programmatically, looking for anything with a title of "Oops I did it again" and if it matches that term, informs you that you must "Wash your mouth out!".

foreach (SPListItem item in list.Items)

{
if (item.Title == "Oops I did it again") {
console.Writeline("Wash your mouth out!");
}
}

{

font-size: small;

color: black;

font-family: consolas, “Courier New”, courier, monospace;

background-color: #ffffff;

/*white-space: pre;*/

}

.csharpcode pre { margin: 0em; }

.csharpcode .rem { color: #008000; }

.csharpcode .kwrd { color: #0000ff; }

.csharpcode .str { color: #006080; }

.csharpcode .op { color: #0000c0; }

.csharpcode .preproc { color: #cc6633; }

.csharpcode .asp { background-color: #ffff00; }

.csharpcode .html { color: #800000; }

.csharpcode .attr { color: #ff0000; }

.csharpcode .alt

{

background-color: #f4f4f4;

width: 100%;

margin: 0em;

}

.csharpcode .lnum { color: #606060; }

Easy enough – aren’t lists sooo easy to work with! Let’s just say you used this sort of logic in a web part that displays only list items that have "Oops" in the title. A few minutes later, you have coded and tested the web part, and it works a treat. Into the change control it goes, and into production it finds itself.

Fast forward 4 weeks and now the list has 400 items in it. Now a refresh of the page containing your web part is taking 10 seconds and getting worse. Naturally ‘the network’ must be the problem, right? Of course it’s not. In this example every list item is being enumerated and a comparison is made. This is well documented as the most inefficient way to query a list.

Putting on my old Cisco hat for a second, I am going to repeat an oft said mantra that people who know me will have heard many times over the years and what all self respecting Cisco nerds know. It’s not the %$^%$ network alright! It’s never been the network, it’s your $%%$ code!

You think the example above is made up? It’s not! On one SharePoint project where I was the information architect, a developer from a well respected Australian integrator did exactly what I described above, and proceeded to blame the infrastructure, the architecture, the network – basically anything that wasn’t their component. Of course, their development environment only had 10 items in the list and thus it seemed fine to them!

If you go back to my real-world example of the DNS/Registry issue, it was a design decision that was probably given half a thought at best, only to turn out to be a major constraint down the track. My SharePoint specific example above shows that for an inexperienced developer, even a "hello-world" style example is potentially a landmine just waiting for you to innocently step on. The issue is that, to the uninitiated, the SharePoint ecosystem has even more virtual land mines to avoid than being Lindsay Lohan’s publicist.

Simply put, if you do not have a broad depth of knowledge that can only come from time, pain and experience, you are inevitably going to get blown up time and time again. What’s that old saying? There are old SharePoint developers, and there are bold SharePoint developers, but there are never any old-bold SharePoint developers 🙂

If I listed all of the examples that I could think of, this post would get out of hand, so instead, Jeremy Thake sent me his diigo link that aggregates blog posts tagged with "SharePoint and performance". That should give you some idea of what developers are getting themselves into.

Junk DNA

John Kominetz wrote a great blog post that is highly relevent to this discussion. It has nothing to do with SharePoint in the slightest, yet if you substituted the word "Documentum" with "SharePoint" we would all be nodding our heads in collective agreement. It examines Documentum, a product that was already a mature player in the ECM space way before SharePoint had even made it to a little red box on a Redmond whiteboard.

Kominetz looks at Documentum’s heritage and describes it in terms akin to a living organism. He makes the observation that its "genome" has evolved over many years and carries with it plenty of junk DNA and "evolutionary dead ends". This is completely unavoidable, despite what any idealistic whinger about "bloatware" would like to believe. Another Joel (are all Joel’s in this world clever?), has a blog that is mandatory reading for anyone who is prone to making idealistic statements about security, bloat or complexity. I urge you to read his "Martian Headsets", "Architecture Astronaut" (where he picks on Groove 🙂 ), as well as his "Things you should never do" posts. Joel is a wise man, indeed.

Quick history lesson for the uninitiated. SharePoint actually started as SharePoint Team Services (STS) and actually used the file system as the content store. (And I predict they will eventually go back to it too but that’s another story). Next Microsoft trotted out SharePoint Portal Server (SPS) where some bright spark thought that the Exchange "web storage system" (information store) would make a lovely document repository (probably because of the replication potential). Over the next couple of years there must have been a big ugly turf war between the Exchange storage team and the SQL team because, lo’ and behold, with SharePoint Portal Server 2003, SQL Server was now the document store. In this major architectural change, we actually lost some functionality. In addition to this, the 2003 version leveraged the recently released  .NET Framework, and provided a richer object model and platform for SharePoint to sit on (although then it was not a completely cohesive fit).

Roll on 2007 and we have a slew of new technologies and features, too many to name. And just to make it more interesting, Microsoft decided to change terminology (again) as ASP.NET was leveraged much more heavily. Thus, "webs" and "areas" are gone. But their evolutionary signature remains. Whether it’s the SDK, the choose-your-own-adventure Technet documentation, blogs or even the STSADM command, it is very easy to spend a lot of time going down the completely wrong path in the quest for a solution to a problem because you get mired in junk DNA. Worse still, you get there, realise you solved the problem the ‘old’ way, say "stuff it" and deliver it as-is.

Web Parts are a great case in point. There are actually two types of web parts in the object model. The one that was introduced with SharePoint 2003 and the one that comes with ASP.NET 2.0 (which SharePoint 2007 uses). When I was writing my hide control web part, almost all the technet examples as well as blogs on the subject did not make it clear which type of web part and the distinction between them. It turned out of course I was using the right web part class, but following the wrong online documentation. A couple of XML irregularities had be scratching my head for hours.

Infrastructure, security and governance awareness (again)

I sort of covered this at a high level in the "Wishing your developers are admins" section and it’s been said before on many blogs that SharePoint blurs the lines between developer and administrator. Both sides have to shift ground (as do project managers and stakeholders too). Administrators really need to gain more development competencies and visa versa. I won’t spend too much time reinforcing what has already been said, but always remember this: Despite all the advances in Windows 200x, IIS, SQL Server, fancy multi-layered security, richer object model via ASP.NET Version x and the like, every developer has it in his/her power to render even a tightly managed SharePoint farm a train wreck. A lack of a simple null-check can render a site un-viewable. Choosing the wrong way to execute a query on content can chew excessive amounts of CPU and memory, an ill-thought design decision can cause a good idea to be lost in the frustration of the user experience. When it all comes down to it, it is the end user acceptance, adoption and evangelism that determines whether a project has been a success or failure.

But it goes beyond performance. What about security considerations? As developers, do you stop and ask yourself whether your chosen course of action has implications on security? What is your definition of security anyway? (Because it is just as much about smart, re-usable design as it is about using passwords with non alphanumeric characters in them – google "the CIA triad").

A major problem area I have seen that has manifested itself in project management time frames is the difference between the development and production environments. SharePoint development environments tend to have a short shelf-life due to the amount of crap you have to install, remove, reinstall, etc. Therefore, they are best served via a single Windows 2003 Virtual Machine with Office 2007, Visual Studio 2005, SharePoint Designer, Reporting Services, SQL Server and all of the other associated paraphernalia. Give each developer their own so they don’t kill each-other 🙂

Without question, it is exceedingly likely that the developer is an administrator of their VM, and those that have a low security acumen tend to hit problems when they migrate their solution to the test or production environment. SharePoint has that many service accounts, application pools, web applications, site permissions, GAC DLL’s, .Net Trust levels, web.configs, etc that it is very easy for something to break into the transition to production.

Remember, everything works when you run it all as the administrator! You will not get much sympathy from your bitter and twisted infrastructure architect if you complain about it.

Project managers and stakeholders need to also be held to account here, as developers routinely cut corners when faced with unrealistic deadlines based on poor assumptions. But don’t worry developers, I’ll nail their asses in part 7!

Solutions and Features

The next two topics have caused me real-world pain. I had an argument with a developer a while back who felt extremely inconvenienced by my audacity to mandate that their web parts, DLL’s and web.config modifications be packaged up as a solution that activates the functionality via a feature.

But to be fair to the developer, if they have a low governance acumen, their point of view is completely understandable. SharePoint application development governance takes time to learn and is often not accounted for in project planning phases. Tools like STSDEV have gone a long way to improving the situation, but at the end of the day, developers who hate all the additional governance overhead associated with SharePoint are buck-passing risk to a place where it is ill afforded.

Branding…

Don’t get me started! 🙂 (I’m drinking a scotch as I write this so please forgive the vitriol 🙂 )

I’ll come back to branding in part 7 of this series, because to be fair, the developers are beholden to stakeholders who obsess over the location of a pixel. In many SharePoint scenarios (not all) this is mis-focused because their success criteria is not how good it looks, but how much the organisation has gained in bottom line.

But from a developer’s perspective, this is another area where I see serious abuse. My entire branding series came from the experience of having to clean up after a "consultant" who decided he did not have to research into the techniques to customise SharePoint in a sustainable, governable way. He completely butchered the _layouts files with no backup or consideration to the implications of future service packs or updates and ignored built in features like the site collection style library. Instead he reinvented his own solution. As I mentioned in the previous section, I asked him to package his work up into a solution for ease of deployment and rollback and I was met with resistance and tension.

The whole concept of ‘CleverWorkarounds’, comes from the pragmatic acceptance that hacks are a part of life. You can’t avoid them, and sometimes you are forced to resort to them. But there is a huge difference between a workaround and a ‘clever workaround’. Most of the various hacks and workarounds that I describe in the more technically focused articles of this blog (like the JavaScript stuff) always come from the philosophical perspective of leaving a environment where the next administrator or developer doesn’t curse your name for eternity.

Miscellaneous considerations…

Jeez, branding got me riled up – I guess the scars are still fresh on that one! 🙂

No-one can know everything. Everyone has governance deficiencies in one way or another. I remember in 2003, a shattered project manager came into my office. A client meeting had gone very badly, blown up in his face in fact. He came to me and his classic line to me was "All I did was tell them to modify their Active Directory schema!". If you know Active Directory, you would know what sort of landmine he just stepped on. It’s like telling Mr T. that his "bling" really brings out his feminine side!

Thus, the branding hacks manifest themselves in other areas too. Pretty much any time you delve into the guts of the 12 Hive (tech guys and devs know what I’m referring to here), you have to be very mindful of governance, security and re-use. Ask yourself this question

Does the change I am making have any chance of being undone or unduly affected by a future patch or service pack?

Is the answer is yes, then stop and get some advice!

Developers in general get frustrated with their project managers. The PM’s nag, don’t have a deep enough understanding of the intricacies and nuances of application development and set unreasonable deadlines based on what you see as incomplete requirements and assumptions. However, as I have said in the second post, their world is just as frustrating as yours. Shared understanding this is the key to dealing with it.

So, if your project manager gets upset because their deadlines stretch because you feel some more research is required to determine the best course of action, tell them to read this series from start to finish (whenever it finishes). Chances are they will be squirming with embarrassment way before they have even gotten to this post.

At the end of the day, it is in both your collective interest to keep the stakeholders happy and ensure that the project is successful.

Conclusion

This post covered a lot of material, and I appreciate you sticking with me. Judging from the reaction from Joel’s endorsement of the series, it also seems to have tapped into a common vein of frustration felt by many IT professionals all around the world.

The next post will round off the "people" aspects with an examination of the stakeholder and project sponsor and senior management perspective. Like the developer and infrastructure post, the key is to recognise that their perspective of the world is just as valid as yours, and just as much as you see them misinformed and contributing to a wicked problem scenario, you need to accept the fact that you do not have the full picture either.

Having said that, they get paid way more than we do and therefore that makes them fair game for some serious nailing to the wall 🙂 Roll on part 7!