Home > Products & Services > Pilot HitList > Case studies

Download the case study (132 kb)

Getting to the heart of the matter of content

Sussex-based Institute of Development Studies (IDS) faced a challenge that is typical in many public-sector organisations when it began to explore how to analyse the collection of development related websites managed by its Information Department. “We must be accountable yet we must promote the widest possible access,” explained Duncan Edwards, senior internet information systems officer at IDS. Funded by public and private sector contributions and focusing on poverty and development issues, IDS links academic research to policy around the world. There are no adverts or subscriptions – as a publishing operation it must prove its worth based on who accesses the content.

The organisation's Information Department is accountable to funders who commission websites to deliver information to target groups. It wanted to use web analytics to understand how content was used and valued. IDS introduced a web analytics system based around Pilot HitList technology and supplements this ongoing data capture and analysis with frequent email and web-based surveys as well as usability testing exercises.

Complex tracking techniques were out of the question. IDS felt that log-in techniques would be off-putting. “We took the decision to avoid requesting a username and password as it would reduce the number of people accessing our information,” Edwards explained. “The reality is that in some developing countries there is distrust of information that isn’t perceived as local. Creating a barrier or a message that we are monitoring visitors would be counter-productive.”

Site design had to be simple in lots of other ways. “Many of our users are in the developing world and have lower bandwidth and less reliable internet connections and facilities than those in developed countries,” Edwards explained. All IDS sites are designed to be accessible to the widest audience so avoid Javascript, Flash and other techniques that can limit browser compatibility. Documents available for download use early pdf file formats and the older MS Word formats.

Edwards identifies a quandary in their remit. “We are asked to deliver information in the most accessible way yet our funders want proof that we are getting through to the people who matter. The web struggles to do both without creating walls or barriers to entry. And putting up such walls leads to distrust.”

Analysing a service
Key initiatives where analytics have been deployed include the Eldis and id21 services, both of which are managed by IDS’s Information Department. Eldis is an internet and email gateway to the world of development information that filters, structures and disseminates the latest thinking on the development process. It was the first service of its kind and has grown to be one of the leading development-focused sites on the internet, currently used by some 65,000 unique visitors per month. Its email services reach 13,000 registered individuals, comprising subscribers to over 30 different email bulletins. Eldis content is built from manually selected and abstracted full text online documents and organisation descriptions. It also includes automatically collected pages from 100 key development websites and email newsletters and bulletins from major agencies. The recently re-designed website and email systems provide a wide range of structures for accessing this pool of information. Eldis serves as an engine to drive forward development knowledge and provides an online space for raising awareness of new development ideas and experience. Critical to this was offering a flexible structure where collaborative activities could be mounted and this flexibility was a priority when the IDS team put in place the content management and analytics reporting systems.

IDS’s content management systems in Eldis and id21 are based around Inmagic DbTextworks and Macromedia Coldfusion. The system is highly flexible, enabling the team to respond quickly and easily to a funder’s request. “Often a sponsor will ask, for example, whether we can put together a searchable database of the latest conferences in a particular area and make it available online,” said Edwards. “We can and this is thanks to the flexibility of the content management and analytics tools we use.”

IDS deployed an entry-level version of Pilot HitList during the development of the id21 service, a fast-track research reporting service funded by the UK Department for International Development (DFID). DFID wanted highly detailed reports. It funded a number of different content themes and wanted to know which individual document, what sort of organisation and from which country each had been requested. “And they wanted this for 300 documents, “ said Edwards. “The need to make both the process of data collection and the drafting of the report manageable prompted us to adopt a packaged analytics solution.”
The team also had to combine analysis from the static, dynamic and email delivered versions of the content. The latter was sourced in real time from a database and had to be tracked differently from an analytics perspective. IDS deployed a consultant to create the tools and reports that extracted the necessary data in a suitable manner. As the need for reports across all areas increased, IDS evaluated a range of software before concluding that an upgrade to the enterprise version of Pilot HitList would best fit its needs. This enabled the organisation to put its analytics data in a more scaleable SQL Server database and mine the information in new ways.

Deriving value from analytics
For IDS' Information Department, continued funding relies on being able to prove that information is reaching audiences. Though a charge is made for printed reports, this is only designed to cover production costs. As Edwards explains, “We don’t derive revenues from the sale of our content – we derive funding from it being accessed and used. This makes analytics and reporting even more important because you can’t sit back and relax in the wake of good sales figures, like many commercial organisations can.”

IDS funders demand high levels of detail, so the team produces a wide range of statistics. Often the level of detail demanded can be very high. According to Colin Cooper, commercial director at ISSEL, who supported the analytics project rollout, Edwards’ team deploy a range of techniques to analyse website usage. This includes regular audience statistics, email and web-based surveys and usability testing exercises. The main analytics measurements remain page views and number of unique visitors but location is also very important. Funders often want to ensure that content is going to users in the developing world and IDS has introduced geolocation technology to break down user IDs to country-level, where possible avoiding potential confusion caused by proxy servers. Reports can outline which documents have been accessed and how frequently, where users go from the home page and the research terms visitors search for.

Edwards’ team has supplemented the core analytics data with surveys that are normally scheduled to provide timely evaluation on a project. “The trouble with surveys is that you normally get feedback from a self-selecting group. Our experience suggests that users who take time to fill in surveys tend to be the enthusiasts rather than the sceptics.”
Cooper argues that the organisation achieves a good balance of reporting. “We’ve deployed Pilot HitList to capture the essential numbers and the IDS team has made intelligent use of tailored online surveys and usability tests to capture more insight, often driven by questions unearthed by the HitList reports.”

Analytics has proved helpful in understanding the inter-relation of IDS sites with those of partners. For example, IDS operates a newsfeed service that takes content from their services and delivers it through others. Through Pilot HitList it can look at the effectiveness of particular newsfeeds, which enables evaluation of other sites. Edwards describes an example where they had one partner site they were funding that didn’t seem to be getting any traffic at all. “Having the numbers to discuss with partners allows you to get past ‘what’ and onto the ‘why’ and ‘how’.”

Edwards concludes by stressing the need for better understanding of web reports. “While a few of our funders still simply seek out a headline figure – normally hits or unique visitors – most are now sufficiently experienced to seek out more insightful data. We’re trying to understand, not simply prove.”

IDS considers itself a leader in its field and believes flexible content management systems, combined with solid analytics and reporting, have proved critical. “We have a competitive advantage because we can meet with funders, say ‘oh yes, we can do that’ and then genuinely deliver it.”