“Folders are bad” and other urban legends…
Hi all
This post is somewhat related to the whole “Zen and the art of SharePoint governance” idea where the notion of a “best practice” is so culture and context dependent that the worst practice is to insist that your best practice applies universally :-).
Let me explain with this example. Consider two problem/solution scenarios, A and B.
- Scenario A: We have a technically inferior solution with a really deep commitment to that solution by the stakeholders and participants
- Scenario B: We have a technically superior solution with “uneven” commitment to that solution by the stakeholders and participants
Which scenario has a better likelihood of success?
Everyone I have asked this question to, has without fail, chosen scenario A.
Why is this? Because people instinctively know that strong commitment to a course of action will trump technical superiority most of the time?
Now, let me change the scenario to something a little more interesting and potentially divisive.
- Scenario A: We have a folder based document storage solution with a really deep commitment to that solution by the stakeholders and participants
- Scenario B: We have a metadata based document storage solution with “uneven” commitment to that solution by the stakeholders and participants
Which scenario has a better likelihood of success? I bet this time you probably feel like answering “Well, it is not that clear cut” or “Scenario A but…” or “Scenario A only because…”, etc. It seems that putting a specific, tangible example against the scenario creates the urge to question the validity of “either/or” in the scenario. This is because you implicitly recognise that there is a little more to it.
This is interesting to me because it gives away what I think is the more “correct” answer. The ideal outcome is actually a third scenario where we, as SharePoint consultants and practitioners, recognise the technical merits of a solution, and start the process of steering a group toward a commitment to that technically superior solution.
Well…most of the time anyway. 🙂
Considering the metadata vs. folders question above. Let me give you a common use-case where the metadata question actually gets murky. Let’s say you have a document library with four well defined metadata columns and have some views that are leveraging those columns. Some users are happily navigating to your site and uploading their files via the “Upload” options or the “New” option in the document library itself. In short, those users who are browser centric in their productivity habits find this to be a good solution. Let’s also say that some two hundred files have been uploaded to this library and are well classified and easy to find via the views. Below is a typical example that I doubt any SharePoint person would find unfamiliar.
But there is another chunk of the user population who has very different productivity habits, in that they are application centric. Let’s say a user lives their life in Microsoft Word (you might scoff but plenty of people who work with a lot of documents do work this way). They start MSWord up and then decide what document to open or work with. They click the open toolbar button (or the office home button) to open a document and navigate to the document library.
Uh, oh…
- Where is our metadata?
- Where are our views?
Well isn’t that interesting. We don’t get to leverage columns and views when accessing documents in this way. All we see is a large listing of files with no metadata. If there is one thing worse than too many folders, then it is surely none at all (with several hundred files in the root folder to choose from).
So now we have a tricky balancing act. Let’s consider what to do, based around the lens of user engagement and commitment.
If you chose scenario A earlier, the technically inferior yet deeply committed option, then you may argue against bringing out the big stick and trying to force users out of their productivity habits (i.e. switch from application centric usage to browser centric usage). You would be concerned that the big stick approach may result in an erosion of commitment, leading to a lack of adoption, leading to the project being deemed a failure.
But then, if we leave it as it is, users will continue to create and manage folders to compensate for the sheer number of files. We have failed to make use of some of SharePoint’s great document management features. We risk creating a chaotic installation, where the same poor information management habits that have likely created the interest in SharePoint in the first place, actually work against SharePoint and devalue it as a platform. This would also lead to user frustration, leading to lack of adoption and project failure.
Interesting dilemma…two options and both with significant risks of an undesirable outcome. What would you do here?
For me, it depends completely on the organisation, department and user tolerance for productive distress. (Don’t worry I will define productive distress in a minute).
Learning by opportunity
When sitting around the bar with Best Practice Conference attendees and presenters in DC, a rigorous debate occurred over aspects of SharePoint branding. One particular person made a fairly generalised sweeping statement and everyone else disagreed quite strongly. The person making that statement actually knew his stuff and everything said was technically and logically correct. But the instinct that I think the rest of us at the table had developed is that the notion of “best” and “right” is actually extremely fluid. No-one disagreed with his frustration that led to his big catch-all conclusion, but we all instinctively rallied against his single “best” solution that would apply to all scenarios.
So I will tell you openly that we have clients for whom we have developed relatively sophisticated information architectures around content types, metadata, search and logical architecture of site collections and sites (Particularly for BMS/QMS scenarios). Yet we also have clients where we have not used metadata at all at first, or else it is quite piecemeal across sites or site collections. What I have come to realise is that sometimes you have to allow people to work with the technically inferior (some would say wrong) option, before you can take them to the technically superior solution with the commitment required to see it become successful.
You have to remember that you, in some way, are pushing someone out of their comfort zone. Depending on the nature of the project, those people did not necessarily ask for you to do this. Thus, push too hard and you will have chaos and pullback. Don’t push at all and you remain with the status quo.
“Obtain management buy-in” is the standard catch-cry for most methodologies, right? It is often cited as if it is the be-all and end-all critical success factor for a project success. To me, it now sounds like a cliché that you say because it is the right thing to say. The same goes with vision and mission statements. Just because you have done those things in your project charter or plan does not mean that everyone is suddenly on board. It goes deeper than that.
I believe that real commitment from the users requires them to believe that a new system will indeed make things better. If they are not convinced that the change initiative will make a positive difference to them, then you are looking forward to a chaotic project that has the odds stacked against it.
Productive Distress
If, like me, you sometimes opt for doing things that go against what SharePoint blogs or books tell you, you simply have to provide the right sort of oversight to support your course of action. This oversight is part of the “how” that I spoke of in the secret to understanding governance post. I *know* that “client X” may only be scratching the surface of SharePoint’s capability, and it may frustrate a seasoned SharePoint professional that they are not making optimal use of what is available. But based on the nature of questions asked, we assess where they are in terms of SharePoint maturity and decide that they are not ready to jump to the deep end. Instead, we help them to work within their level of maturity and allow them to better understand their problem by learning about the solution. This small step approach allows the user base to be pushed just enough out of their comfort zone, where they are still productive and not pulling back. (Hence the term “productive distress”). Pretty soon, that productive distress is forgotten, becomes the new “business as usual” and we now move up a notch.
Repeat process as necessary. Fast forward a few day or weeks and suddenly the nature of the questions change, reflecting the improved understanding of SharePoint and its capabilities. There is a certain maturity around the questions asked, the sophistication behind the scenarios conceived and you can tell that they are now ready to move to the next level of productive distress.
So it is not really that “folders are bad” or the equally common “Keep SharePoint Designer out of the hands of mortals!” because, to repeat an oft abused cliché, they are just tools. If you can convince your users that SharePoint will improve the situation and allow them the time to adjust to the new paradigm, then they will work out for themselves whether folders are bad or not. As the scenario above demonstrates, it can sometimes depend very much on how you look at the problem. Also remember that most of the time, commitment trumps technical superiority.
Thanks for reading
Paul Culmsee
..and then there’s cases where you aren’t allowed to choose because the product locks you into one model, like with the Publishing feature and page libraries. It doesn’t even matter if the limitation comes into play in only certain scenarios. When the customer hears : “You can use option A, except when blah blah blah” it’s easy to favor option B instead. That way you don’t need to think about both A and B.
Publishing feature is a good case in point where one of my clients, after around 30 minutes of exposure to it, wanted it customised to a point where we refused (and we copped some heat). However 4 weeks later when I asked the client again, they accepted that their initial opinion had been wrong and as they learned more, they realised why we initially said no.
Good post Paul.
There are times when the folder option can be a better choice even from a technical perspective depending on the requirements and direction the organization is going. Illustrating that some functionality (such as explorer view, working from the office client (especially ones without document information panels), etc) works better with folders than it does with metadata is a good way of getting people to remember there are many situations in which folders can be advantageous over metadata or as a good path towards metadata while general acceptance, understanding, and integration of metadata improves. (Windows 7 incorporates more metadata into files on your desktop etc).
On your other point I completely agree that it’s always important to try and be objective and not biased to your own interests, likes, dislikes and even experience when evaluating a clients need. This is something that we should always remind ourselves before, after and during meetings because it is so easy to fall into the trap of wanting to feel like “I have heard/done/worked through this before” or “this is the same need as this other need” before listening and evaluating what the full need, motivations and comfort level are.
Looking forward to more posts as always,
Richard Harbridge
While I won’t make a blanket statement that folders are never useful because they can be (especially in collaboration), but SharePoint as you know presents a different paradigm. One that the more I work with I find far superior.
I’ve actually written reasons why you should not use folders.
Your points about metadata being better than folders (I checked your post) are fine, but that really wasn’t the point I was making. Tell me, how would you deal with the user who’s application centric in their document habits transition to the folderles SharePoint paradigm that you are talking about?
That is really what this post is about. If you force it on users, it is very likely that mny users will reject it – even after training. So how do *you* tranisition an organisation to the new paradigm?
Fortunately, on the Folders vs Metadata debate, we don’t always have to choose. Views can ignore folders – it’s one of the view settings. We have customers who have their documents ‘stored’ in folders, but they also have views on the same library that rely solely on the metadata, too. That way they get the benefits of Hierarchy and Metadata.
Sometimes folks ask about why they have to add metadata to a document when it is in the order number ‘1234’ folder, but we just tell them it’s for search. And we can help automate that in different ways too.
But the point of your article, particularly that the ‘best’ approach depends on context, and that change is best when it’s incremental, is a good one.
Andy, this is also how we approach the situation, and we have an event handler that maps the folder path to a site column on the library (called filepath) so that CQWP can slice and dice if necessary.
But your event handler link is an excellent idea! Great work.
Also for what its worth, content types that inherit from the folder content type make for some nice flexibility around views.
regards
Paul
Hey Paul – good stuff! Here’s a screencast I have for “emulating folders” in doc libraries: http://go.meetdux.com/c9hg
Ahh my head hurts… folders, views, metadata and such. Why can’t the folder tag (define metadata) to the contents… and stop the madness. SharePoint 2013 can’t wait 🙂
Nice writeup mate, I’ve also implemented custom folder content types with event receivers copying down field data from the folders to docs for a project or two.
Folders can be useful and aren’t always evil as everyone states, it depends on the documents you want to manage, the clients requirements etc. I’ve built specific solutions that rely on custom folder content types down to a very specific depth in the past, you can even prevent users creating further nested folders my changing the content type visibility on specific folders. Folders do allow you to synch individually into outlook too, and yeah the point about application centric access from Office client apps is a very important one, views and metadata don’t help there.
and yeah, even if you go with folders, you can go ahead and create folder-less views based on metadata too, the best of both worlds.
As always… “it depends” 🙂 One size doesn’t fit all.
“the worst practice is to insist that your best practice applies universally”
Brilliant! – I think the whole best practice bandwagon has got a little out of control and almost to the level of ‘religious crusade’ recently.
Don’t forget that Windows Explorer can be customized to display metadata columns in Details view. Just click View | Choose Details.
Sezai, like that event handler idea a lot. Would be interesting to see if that could be made into a repeatable feature, configured by an application page 😉
Ryan, check out the post entitled “the secret to undertanding governance” for more on that notion.
thanks guys
Paul
I found myself having this very same debate over a beer (during TechEd not just with some random stranger) only a couple of weeks ago, and I think you have articulated it perfectly. It really comes down to the context of the solution and increasing the liklihood of acceptance by your the users. Great to read Paul.
Hey Nick, good to hear from you again.
Isn’t it always the way that insight always comes from drinking a beer or two. Such a pity that after two beers, insight is replaced by beer goggles 🙂
Nice work on this post Paul, I remember chatting via your Plugoo window on this concept of context and wondered how you’d go at communicating it in a post, you’ve done a good job.
You surface a very good point about people and the way in which they work with information systems, the browser centric or application centric insight you write of is a reality, this futher highlights the importance of training and adoption strategies to build buy in from users, which brings me to the old chestnut of how to convince a organisation not to remove training to reduce the cost of the project.
Hey Andrew
Check out this post for a potential answer. http://bdld.blogspot.com/2009_11_01_archive.html
On a selfish level, the author linked to me which is nice, but he also quotes Geary Rummler, whos process improvement book that I have mentioned previously. Fully how these things all connect up.
Anyway to quote from that link that quotes Rummler, “80% of performance problems reside in the environment, such as processes and systems, so ensure the problem is really a learning/training problem, not some other performance problem”
I like how you structure the post from the perspective of user acceptance.
I’m of the opinion that both folders and metadata (including content types) have their place in a complete SharePoint information architecture. It is not a question of which one to use, but rather a question of when to use each. As consultants, we have to educate the users as to why it is appropriate to use a folder or metadata in a particular case.
The post below compares the trade offs between organizing information with folders and metadata and when it is appropriate to use each:
http://thingsthatshouldbeeasy.blogspot.com/2010/01/sharepoint-folders-vs-metadata.html
The comparison categories are security, content type ordering, navigation, url, tools support, searching, sorting, filtering, and grouping.
Thanks Euguene
Watch this space. Early next year I have a series of posts on this topic in a fashion. I agree with your “when”, but your “when” is across technical and feature grounds and I think this is half the pictire. I have found another way to look at it that has worked very well in my classes and consulting gigs.
regards
Paul
Excellent post Paul. I am currently designing a 2007 rollout of about 25 site collections across a lareg corporate team that are migrating from eRoom. Did I mention custom code is out of the question in all shapes and forms? Challenging. I have always been an advocate of using metadata instead of folders in SharePoint and so far this has paid off but I am now seeing this method is not going to work in this case. Using explorer view to drag and drop (a feature these users are more than familiar with in eRoom) creates an enormous bucket of documents and this is not going to be acceptableto end users. So, everything in moderation perhaps (this tends to apply to most SharePoint designs). A mixture of folders and metadata may be what the DR ordered.
I keep reading folders are bad, but they prevent something that metadata doesn’t, filename collisions. My client has thousands of documents and many of them have the same name, although the content is very different. As much as I would like to steer them towards metadata I can’t find a good way around the naming collisions without sitting down with the users and asking, “What do you want to name this one?”, for a few hundred documents.
Great post. I think that jeunesse chile the way is the main way are you focussing but there are some point that could be discussed to center the main ideas and to improve and reforce yours