SharePoint sucks at document management – or does it? A metal perspective

This is an opinion piece, a different tack than a lot of my other topics. I’m going to attempt to use heavy metal music as my metaphor to get my point across. No idea if I will succeed :-).

Opeth \m/ \m/

Firstly, SharePoint, in my opinion, is a collaborative platform, more than it is a collaborative product. In the same way that Lotus Notes can be argued as a messaging platform. Both have their core competencies and solve particular types of problems. However, they can also be customised and sophisticated applications can be built upon the foundation they provide.  Note my emphasis on the word collaborative.

Secondly, like heavy metal, the term “document management“, is misused and misunderstood. It is simply a generic concept that needs to be broken down into sub-definitions or sub genres.

For those people who think the idea of listening to Celine Dion is a good way to spend an hour, it is exceedingly likely that they would consider heavy metal to be a single genre. But of course, us metalheads would beg to differ. As an example, you have Black metal, Death metal, Glam metal, Industrial metal, Power metal, Progressive metal and Thrash metal to name but a few. (However I personally would put power metal in the same bucket as a Celine CD 🙂 )

So I am a progressive metalhead, I love Opeth and I would consider them to be the absolute masters of progressive metal genre. But, they are a band that frequently uses the ‘cookie monster’ vocal grunts commonly associated with death metal. Does that make them a death metal band? No it doesn’t. They may share some of the characteristics of death metal but are a very different beast.

So, as part of collaborating, we need to manage documents. We have a team of staff all working to achieve a goal and along the way, they need to work together as efficiently as possible. Time is money. That means timely access to the latest version of a document no matter where you are; the ability to track and maintain versions of documents; the ability to enforce internal processes such as peer review and signoff and so on.

Often, the end goal of this process is a document or set of documents that have a ‘final’ or ‘legal’ aspect to them. For example: A final report to a customer. Such a document needs to have this version, at this point in time, captured. At this point, the document becomes a record.

Now collaborative document management is very different from the more rigid and structured form of document management known as records management. Records are defined as “information created, received, and maintained as evidence and information by an organization or person, in pursuance of legal obligations or in the transaction of business”.

A really, brilliant, insightful post on this subject can be found at joiningdots blog and I am going to quote his list of the activities in relation to these two genres of document management.

Records Management:

• Management of the record is more important than the content of the record
• The record never changes (although its properties might)
• Records require access controls, lots of them

Document Management/Collaboration:

• Without content there is no document
• The document changes a lot, that’s the whole point of collaboration
• Access controls restrict and impede collaboration, the fewer there are the better

Now to continue the metaphor, here is the explanation of death metal vs progressive metal 🙂

So collaborative document management and records management are similar to Progressive Metal and Death metal. They have some common characteristics, but have different goals and outcomes. Let’s now look at some of these characteristics in a SharePoint environment.

SharePoint Document Management Sucks?

No, it does not suck (although I will keep an eye on my website stats to see if a controversial title means more readers 🙂 )

In my opinion, SharePoint is a really good collaborative document management system. It, however, is not as good a records management system.

Collaboration is about teams and productivity. What makes one team productive is not necessarily the same as what makes another team productive. A site with a document library for a sales and marketing team will likely have different columns, views and workflows than a document library for a bunch of IT geeks who are writing system documentation (yeah right – geeks write documents? 🙂 )

SharePoint is great for document collaboration, because it is so flexible in how you structure sites and document libraries. It is common to see a SharePoint farm broken into broad, divisional based portal sites, with team based sub-sites within. There is often one or two columns that are relatively consistent across all document libraries in the farm. In addition to this, the various document libraries and content types within these sites will often have additional different columns, based on specific requirements of the team or division using that library.

There may be some file naming standards, but if not it doesn’t matter much, since the users can easily fill in the metadata columns when they save a document. ‘Folder hell’ is reduced and views allow for easer retrieval. Roaming users have offline copies of document libraries either in Outlook, Groove, Colligo or iORA and regularly sync the latest versions.

What a wonderful picture I paint ,eh? I can see Opeth right now, storing all of their tour info in their tours document library, but using a different document library for their songs guitar tab. The latter has a column that classifies the tab by album and author. Michael Akerfeldt, (Opeth’s amazingly talented singer/songwriter) has a sync version of all of this in Outlook on his laptop.

Compare to this…

A records management system is like a single giant document library. The columns are usually standard, no matter what the content. Usually the classification system (called a taxonomy) is based around a coding system that among other things, indicates record types, and the retention policy and compliance requirements for that record type.

That coding system is called a “File Plan”, and record managers love them, but the rest of us hate them with passion. Why? Well, they have about the same level of intuitiveness as the account codes for a chart of accounts, or time-sheets for that matter.

But they all serve a similar purpose, so I guess that makes sense. So I’ll explain better.

For general collaborative use between a team, record file plans just plain suck. This is because the coding system is based around “file codes” which are numeric representations of the taxonomy classifications. Those numbers mean a lot to the records people, but are not intuitive at all to the rest of us. Usually records are sub-classified 2 or 3 levels as well, as it’s rare to have a single level file plan. So here’s Opeth’s example file plan…

401 110 – Reports and Statistics

1.0 Merchandise

2.0 Tours

        2.1 Sweden

        2.2 Norway

(rock on baby! – file plans are just so metal!)

So imagine the above code along with a file number for the document, a number for its version, etc.

401-110-2_1-12345_003.DOC

Now quite obviously if you are a record manager, this document is a report (401-110) about a gig in Sweden (2_1) that is document number 12345 and version number 3 (003).

I can just see Opeth’s Michael Akerfeldt using that naming convention for all his documents relating to the Sweden tour! Then when the tour is over, he can easily archive off his Sweden stuff (401-110-2_1) onto DVD!

Makes me wanna mosh! … NOT!

So many document libraries is not really a good thing for records management. Also, in SharePoint, a file plan style set of columns is difficult. it’s quite painful to have one column filter the value of another out of the box (ie pick your first category column and the list of values in the sub category is filtered). It’s a custom development job or you have to go with a 3rd party if you want to do that.

But no doubt you are aware that there is a records management capability in SharePoint 2007. But in reality, it’s not that fully featured. For the purpose of this article, I won’t get into that, but consider that the data is all in a big SQL Database. Typically this uses a very expensive, high capacity disk, yet some records, according to the file plan, are not worth the cost of that disk. (I.e. it costs more to store than the files are worth). But how to put them onto cheaper servers/media? if it’s one big document library then it’s one big SQL database.

Now don’t think for a second that I’m anti records management. I’m not at all. It is an essential technology. I’m just dumping on it from the collaborative end-user perspective to help my argument :-).

Confusion reigns!

Now in all seriousness, I have painted a pretty extreme example I know. But many companies do attempt to go down the path of treating all files as a record when they implement a document management system. Obviously record oriented file naming conventions like the one I demonstrated will not work for regular users, but many companies previously had a dedicated document control department for handling them, and only a small subset of the files (the ones that are records 🙂 ) are treated this way . But when you get a system like SharePoint, it’s easy to think, “cool, now we can classify ALL documents via our file plan and make it much easier to manage our documents”

For me, that doesn’t compute. Power metal has the same effect for me.. what’s the point?

Does it work? I’m sure it does for many organisations. Some high end, 3rd party DMS products have been doing it for years. For me though, I was involved in a non SharePoint DMS installation back in the late nineties. Despite a pretty good GUI, with nice drop downs and easy to navigate metadata screen, the classification system was still too complex for the user base and they misclassified all of their files when saving.

Records Management then one day did an archive of a particular file code because the retention schedule told them to. The next day all hell broke loose! Users live files suddenly disappeared and the system happily presented them with a CD number, asking them to call records management department 🙂

I’ve never seen such a quick file plan redesign 😉

So why is it that many companies confuse collaborative document management and records management? I’m not sure, but I sure as hell know that for my money, you should not approach SharePoint 2007 with some preconceived idea that it is going to solve all your records management needs. I can predict it now. You will spend weeks arguing with the stakeholders over a taxonomy, document a file plan that they will all approve, despite not actually reading and then find that SharePoint cannot easily accommodate your requirements out of the box. Your project plan and budget will blow out, senior management will be pissed and you will blame Microsoft for releasing a substandard product.

But who’s fault is it really? 🙂

Thanks for reading..

P.S: Recommended Opeth tracks: Bleak, Deliverance, Master’s Apprentice, Drapery Falls