Gentoo Website Survey 2012: Results and Conclusions

Last year, I invited visitors of the Gentoo website and Wiki, readers of the gentoo-user mailing list, and subscribers of our Google+ page to participate in the Gentoo Website Survey 2012. The survey ran for a whole week and I’m glad that 1392 participants took the time to answer a few questions about our websites. This post sums up the results and I’ll note my (personal) conclusions drawn from these results.

Looking through the handed in surveys, I recognized a few “stereotypical” user profiles: There were many Gentoo veteran participants who are accustomed to the site and don’t want it to change (too much) as well as new and prospective users who are a little more open to change. Thus, a few of the questions below are grouped by developer/user status, and how long people have been using Gentoo. Details on the user group definitions are listed at the end of this article.

The big question: Do you think gentoo.org would benefit from a visual update?

In the first 50 questions of the survey, we sort-of circled around the question that affects our future efforts the most: Should we even bother to work (on yet another try) to bring changes to www.gentoo.org? The answer is yes!
Gentoo Website Survey 2012: Do you think gentoo.org would benefit from a visual update?

As you can see on the graph by user group, developers are very certain we need a change of scene—and most users agree.
Let’s look at a few of the other questions in detail.

Basic ratings: Content, Navigation, Usability, and Appearance

One thing you notice right away are the comparatively bad ratings of the the visual appearance: For no user group, the ‘red zone’, indicating a rating of poor or very poor, ends below the 20% mark. Prospect users and developers are the groups to least like the look of gentoo.org: 40% of these groups give a negative rating. Following the graph to the top, we reach the yellow alright bar at 60-70% of users which means that only 30% to 40% actually like the visuals of our website. This gives us a clear motivation to change the visual appearance.

Things look better in terms of Navigation and Usability: Less than 20% of the participants gave a negative rating here. There is still a rather large yellow ‘meh’ zone. Reading the free-text comments gives an indication why there aren’t more positive ratings: The lack of a consistent gentoo.org-encompassing navigation makes it hard to move around. More on that later in this article.


Content is where our site shines: About 60% of the participants think that the content we offer is of a high quality, well over 50% think the selection is good as well. While they don’t like the look, new and prospect users dig the information and documentation you can find on gentoo.org: These groups give the most (very) good ratings. The quality is slightly better rated than the selection. My personal guess is that people are looking for more information for very recent technologies and software that are up to now only documented on our Wiki.

Asking the devs: On the backend, GuideXML, and using the Wiki

The survey also included a set of questions that were only displayed if you ticked the “I’m a developer” option on the first page. These questions pertained to the website backend which is pretty much only known to developers so it wouldn’t make much sense to ask users about this.

As you might know, gentoo.org currently uses a custom XML dialect called GuideXML for publishing content. The system was conceived a good decade ago and is tailored—as the name tells you—to provide a markup for documentation that yields consistently looking results. Where it falls short sadly is publishing non-documentation websites, as the subset of HTML that it covers is too restricted. Evaluating the acceptance of possible alternatives was one goal of the survey.

Let’s look at the results: First, we wanted to see what people thought about the most obvious alternative: Plain HTML. No majority in favor or against it, so let’s look somewhere else. As we finally got an official Wiki, maybe we should use that for project pages. It turns out that a good 50% of developers would like to do this, with only 20% against it. So, I guess I will go about implementing a move, after getting more feedback on the usual developer discussion venues. This move is also supported by the answers to the other questions: WYSIWYG editing isn’t really wanted by many people while having a SCM to manage the contents is. I think the Wiki with its extensive history and diffing features gives us a SCMy feel on a web platform. A regular web content management system (wCMS) isn’t really favored either.

It was also quite interesting to see how well versed developers think they are in GuideXML and plain HTML. People are feeling much more confident working in HTML than they do in GuideXML. I guess despite the similarities, developers don’t really know what differences there are and how much of HTML GuideXML actually supports. This is another hint that the current publication system should be reconsidered.

Social Media and new Features

Asking about Social Media surprisingly made many people grab their pitchforks: “social media is stupid, don’t waste your time” and “I hate all of them” were two of the responses participants gave to the question “What Social Media would you like to see more of Gentoo on?”. As outspoken the people disapproving of Social Media are, they aren’t the majority. A lot more participants are asking for an extended Social Media presence of Gentoo, mostly on Google+ (where we actually are already present), Twitter, and Facebook. Other mentions include Reddit (where /r/gentoo already exists), diaspora, identi.ca, and Linkedin. On these sparsely requested sites, I guess there is already a community-founded Gentoo fan or user group, and given the low interest, we shouldn’t invest too much time getting involved there. Instead, the three most-requested sites I mentioned before should be in our focus.

What’s good, what should be changed and why it took a while to get the survey evaluated

In the last three questions, I asked participants to tell us about the things they like most about www.gentoo.org, what things they would change and whether they had any comments regarding any of the other gentoo.org sites such as the Forums, Wiki or Bugzilla. Getting a good picture of the most prevalent opinions was rather hard, I sampled a good few hundreds of the statements and read them, which took a while. Also, I just suck at blogging.

There were also a few very nice troll attempts among the completed surveys; to their authors: Thanks for a good laugh. ;) One hint for your future survey trollings though: Checking “I’m interested in becoming a user” and “I have used Gentoo for longer than five years” are rather good features to classify your posts as spam, so thank you as well for making my job easier there.

Now, here are some comments on the items that I read most frequently:

General comments or spanning all sites

“The headers/layouts of the various Gentoo sites are not the same.”
This is indeed an issue and will be in the focus of any upcoming efforts to improve our websites.
“There should be a site-wide search functionality.”
Again, known issue. We’ll be looking into getting something up here.

www.gentoo.org

“The weekly/monthly newsletter should be back.”
Of course it should. The problem is—you might have guessed it—getting and retaining editorial staff to work on it. A few developers have thought about restarting the GWN/GMN, but I guess a little community support is needed to get things really going again. So if you are interested in helping writing articles, interviewing people, reviewing software, or whatever you can think of that would enrich a newsletter, contact the PR team, tell us about your ideas and maybe we’ll get a new team going.
“I like that www.gentoo.org is browsable without JavaScript. DO NOT CHANGE THAT1!!1″
Simplicity is good, I agree. There is little to no JavaScript on the site as it was made quite a while ago, in a time where browser support for it wasn’t something that could be taken for granted. Now, the web technology has changed a lot in the last decade, as have most people’s expectations of what a usable website is. That means we can not completely ignore the fact that website scripting exists and the possibilities it offers.
So, here are some shocking news: Should we come forward with a new website, expect mild JavaScript usage. Functionality for CLI browsers (i.e. without CSS support as well) will of course be unaffected by this. Speaking of which…
“Browsing the site on a console browser works nicely or doesn’t work well.”
There were quite a few people noting either opinion. Again, any site updates will keep console browsers in mind. Not using a table layout for instance will make the site render linearly in a text browser with less things next to each other, improving readability.

wiki.gentoo.org

A quick note before the remarks regarding the Wiki: It seems that many people are mixing up the community Wiki at gentoo-wiki.com and the official Wiki, launched in November 2011, wiki.gentoo.org. There was a lot of positive feedback about “the” Wiki, some people explicitly mentioned how it helped them over the course of several years, clearly indicating which of the two Wikis they refer to, for others I had to guess. Here are the most common comments that I believe pertain to the official Wiki that the project is actually hosting:

“The community Wiki (gentoo-wiki.com) and the official one should have their contents merged.”
This can not be done for licensing reasons. The two Wikis use similar—yet incompatible—licenses which requires the permission of all contributors involved in creating the contents if we wanted to copy them from or to either Wiki. Seeing that lots of contributions on the community Wiki are only attributed to an IP address, there sadly is no way to contact all these people. (NB: wiki.gentoo.org uses the same license as the official Gentoo documentation)
“The Wiki should have more contents.”
It’s a Wiki, go ahead and start a page on a topic you’re knowledgeable on! :)

packages.gentoo.org

“The site sucks.”
Good news, everyone: We had a GSoC (Google Summer of Code) student working on an improved version and are looking into replacing the site.

forums.gentoo.org

Again, lots of good feedback here, people appreciate the information and help they get regarding any (not just Gentoo-related) issues.

“The search does not work properly.”
The forums team is aware of this. Due to a heavily modified codebase that is in use, there are quite some obstacles getting upstream improvements, including search, implemented.

Conclusions and “thank you”s

So, what did we learn? Our websites do need an overhaul in terms of usability and appearance, repackaging our otherwise nicely done content. That’s what I’ll be working on in the coming months.

Thanks a lot to all the 1392 participants, especially those sharing their verbatim feedback in the last few questions. A few people even have offered help, I will be contacting you in the coming weeks.


General definitions

Here are common definitions of the user groups used throughout this post:

All participants All people who have answered this question.
All developers All people who have answered the first question with “I am a Gentoo developer”.
All users All people who have answered the first question with “I am a Gentoo user”…
Long-time users …and have indicated they have been using Gentoo for more than five years now.
New users …and have indicated they have been using Gentoo for less than a year.
Prospect users All people who have indicated they are interested in becoming Gentoo users.

9 thoughts on “Gentoo Website Survey 2012: Results and Conclusions”

  1. This is an absolutely great report. Thanks for taking the time to do it and even more, for publishing the results!

  2. Great report, thanks a ton! So glad to hear that packages.gentoo.org will be updated (it’s long drove me nuts… and I’m not even sure I mentioned it in the survey).

  3. Good work!
    Thanks for investing the time to get some usable feedback for your page.

    Makes a great project, even greater :)

  4. IIRC I didn’t fill out the survey because I use privoxy with some pretty major filtering, including color-rewriting to light-text-on-dark-background (I hate white or overly light backgrounds, I think partly because I ran CRTs at the high limits of their resolution for years and low-refresh-rate white backgrounds BLINKed! so I got used to dark backgrounds, and partly due to the glare problem from wearing contacts). As such, I figured I don’t see enough unfiltered site presentation to properly evaluate it.

    But it was definitely interesting reading this report. As with the others, thanks for your time.

    Meanwhile, that brings up the accessibility point as well. I know of at least one blind present gentoo dev and one blind previous dev, so I’m sure that will be accounted for, but keep in mind those of us that use non-traditional color-schemes, either browser-forced or filtered thru privoxy or the like, too. (In particular, don’t assume background-images will be displayed, even on “normal” browsers, so set background color too, and ensure proper contrast with foreground color, tho here, my filters will take care of that in most cases.)

    And another vote for easy-does-it on scripting as well, ensuring noscript alternatives where needed, etc.

    I was surprised at how hard on the site gentoo devs are, generally evaluating it rather worse than users. Yet arguably, they’re in a better position to change it if they wanted, yet it hasn’t been done despite the low evaluation. I guess despite the low ratings, devs always had other priorities and despite not liking it, it worked /well/ /enough/, so never /got/ the priority… until now.

    Duncan

  5. The intention behind the code that was in the end used for packages.g.o was to collect repoman or pkgcore-checks results and present it to devs. That information was never added though and the real simplistic design got broken by some kind of css-related discussion between to teams which ended in some demotivated contributors for the site. Thus it was never finished but published in the state it was in. Good to hear that it will be reworked from ground up :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>