lASSIST NEWSLETTER vol. 4 nos.3&4

3) At present the Canadian Historical Association and the

Canadian Pohtical Science Association represent the

research community on the Advisory Council on Public

Records. The Public Archives is giving serious

consideration to widening the diversity of representation

of the research community

4) The full text of Part I V of the Canadian Human Rights Act

can be found as an appendix to the 1980 Index of Federal

Information Banks which is available for reference in

every Post Office and Canada Manpower Office across

Canada.

5) Kissinger vs Reporters Committee for Freedom of the

Press et al.. Supreme Court of the United States. March

3. 1980. No 78-1088 and 78-1217. The case involved

summary or verbatim transcripts of Kissinger's

conversation notes and telephone notes which he

"unlawfully removed" from the State Department and al

a later date deposited al the Library of Congress with

severe access restriction. In short, the Supreme Court
ruled that the plaintiffs had "no standing" and that only

the National Archives and Records Service and/or the

State Department could pursue Kissinger for return of the

notes.

6) Senate Report to the Federal Records Act of 1950. S Rep
No. 2140. 81st Congress. 2nd Session, at 4<I950).

7) The distribution of SIN numbers by first letter of surnames
for R is 4 94%. The author is currently undertaking a

project to establish an overall mixed numeric-alphabetic

sampling scheme for case files falling under privacy

legislation. This includes a study on the distribution of
first letters of surnames of language, ethnic and
provincial groupings

THE CONDUCT OF USER SURVEYS

Dennis D. McDonald, Ph.D.
King Research, Inc

Washington. DC

I'd like to start by making a few statements in general about

user surveys, and then to discuss their goals, their different

varieties, and finally some methodological pointers

First, there's no such thing as an "end user"

Second, information isn't a commodity like a bar of soap

which can be bought, sold, and priced "over the counter."

Third, don't believe that users and potential users can tell you
pointblank what they really want and need in the way of

information services.

Finally, the more you know about a user before you conduct a

user survey, the better off you'll be

User survey goals can be classified into the following five

categorie:s:

* Input prior to system development.

You may want to find out the needs of a potential user

group before you invest a lot of dollars in the

development of systems or services. This kind of user

survey is difficult to conduct since what people suy they

may need and what they actually end up using may be

two entirely different things.

Information about your competition.

You may be in the process of developing a system or

service, and you may want to assure yourself that related

information products and services, i e., your
"competition." are not satisfying the same needs. This
kind of survey is essentially a form of intelligence

gathering, and conducting it will force you to consider
how your product or service will supplement what is

already available

Identity of current users.

You may have a system in operation, and you may want
to find out not only how and why people are using your
system, but whether they are satisfied with what your
system is supplying This is what most people consider to

be a "user survey."

Potential users.

You may have a system which is operating and satisfying

the needs of a certain core group of users But you want

to expand the system's use You must then seriously

think about potential user populations and how to ask

66 ]



lASSIST NEWSLETTER vol. 4 nos.3&4

them aboul their own needs and what information sources

they currently use I recommend, however, that you

conduct a survey of potential users either after or

concurrenl with your survey of current users. Otherwise,

you may never understand the context within which your

current users operate and this will hamper your ability to

promote your system to new users.

• Evaluation.

You may want to evaluate a system by means of a user

survey In fact, surveys are often conducted as an input to

evaluating many kinds of systems and services, in

addition to those devoted specifically to information

services But it is wise to remember that a survey is not.

in and of itself, an evaluation method; it is simply a tool

of evaluation, and the actual evaluation depends upon

how the results of the user survey are interpreted.

Each of these five goals is related to the different kinds of

user survey you might conduct And accomplishment of these

goals is also related to the four points I made earlier

1 mentioned, for example, that "there is no such thing as an

end user." The term "end user" implies a certain finality, as

if the information your system supplies converges at one

point and "sits there," like the "stop" symbol in a

flowchart If you think aboul the people who have access to

your products and services, you probably hope that the

information you supply do sn't stop with them Ultimately,

you want it to affect their thinking and behavior One of their

potential behaviors is sharing the information you supply with

others, periiaps in a substantially modified form (This can

also feed back into increased use of your services through

"word of mouth" advertising ) So when you survey your

users you want to know not only what they do with the

Information they obtain; you should also find out how ihey

retransmit this information, and to whom they retransmit it

(This is a practical way of identifying potential users.)

I also mentioned that information isn't a commodity, like a

bar of soap This also affects the goals you can accomplish
through a user survey For example, two of the most
important questions you will ask when developing or

modifying your system are "How much are people going to

use it?" and "How much are Ihey willing to pay to use il?" If

you are developing a database which will be accessed online,

you may need to address yourself to two groups: the people

who sit ai the terminal and interact with the system, and the

people who actually use the output of the online interaction;

these two groups are not always the same The kinds of
questions you ask these two groups may be substantially

different The first group you will want to ask aboul their

interaction with the system and how they relate to their users

The second group, the actual users, you'll want to ask what
Ihey do with the information Ihey obtain from the system If

both of these groups have no experience with the type of
system by which your database will be accessed, asking them
hypothetical questions about amount of use and willingness to

pay may very well be counlerproduclive. You would do much
better to ask them about the types of decisions and problems
they have to face on a day-to-day basis. Since information is

only one of the resources people use to solve problems and

make decisions, a detailed understanding of these problems
— what some people call the "situational context"—will be

useful.

KINDS OF SURVEYS

Given the relationship between the user survey goals and the

types of questions which you mighl focus upon in your user

survey, several different kinds of surveys begin to emerge

The first kind of user survey is the one which concentrates on

the user's inleraciion with the system. Here the types of

questions concern such things as;

• Frequency of access

• Type of access channel

• Payment for use

• The nature of the access system

• The physical medium used to supply the information

The ease with which the system can be accessed

• Reasons for satisfaction and frustration with the system

The second kind of user survey concentrates on the

information which is supplied by the system. Here the

relevant questions are:

• The type of problems or decisions which are important to

the user

• The role information plays in solving these problems
• The relative use of different information sources for

solving these problems

• The perceived accuracy and reliability of these

information sources

In the real world, it's impossible in a user survey to

completely separate reactions to ( 1
) the contents of a database

from (2) the system used to obtain access to the contents,

since the two are so often inlenwined The perfect example of

this IS the system which allows the user to manipulate a

database using a very flexible, personalized command
language Essentially, the user is generating a unique product

each time he or she sits down in front of the terminal. In this

case, the actual contents of the database may take secondary

importance to the way in which the user interacts with the

system, since il may be the interaction method which makes
the system unique

METHODOLOGICAL POINTERS

Now, I'd like to discuss some methodological points related

to user surveys. These deal with three main issue areas:

• How the survey should be conducted
• How the sample is designed

• How the questionnaire is designed

[67



lASSIST NEWSLETTER vol. 4 nos.3&4

One assumption I make is thai a survey, by definition,

involves the collection of a standardized set of data elements

fixim a group of individuals A further assumption is thai data

collection may involve an interaction with geographically-

dispersed individual users, instead of, for example, direct

observation of their behavior

Concerning sample design, by far the most important issue is

how best to define and identify a population of users from

which to sample. Sample size is not, as may be commonly
believed, the major issue. The major issue is ensuring that

you contact the right people, the people who are qualified to

answer the questions you have. The more time and effort you

devote to sample development, the more will be your reward.

Some of the practical implications of this view are the

following

First, while it is cheaper to purchase a list of individuals'

names and addresses from a commercial list supplier or

professional association, you may be better off if you develop

your own list, perhaps over time, of people who have used or

who have expressed an interest in your system (This, of

course, will not be sufficient for studying all potential users )

Second, try to obtain complete information on the type of

organization which employs your system's potential user.

Prefer the devlopment of lists with business address, rather

than home address. That way you can subdivide or stratify

your sample by employer type before you conduct your

survey.

As a general rule, try to settle upon the specific population

whose ise you will be studying before you draft your

questionnaire. That way you can be sure that the questions

you want to ask are appropriate for your sample, and vice

versa

Never assume that your users are thoroughly familiar with

your information center or system; always describe in detail

the system or service whose use you are studying.

Remember, your system is only one of the information

sources your user relies upon.

Be prepared to define and explain your questionnaire terms in

great detail This is especially true when you are dealing with

information products and services. Terms such as

"information," "access," and "use" must be defined

concretely if you are to have confidence in your analysis. It's

because of the need to provide extensive definitions that I am
partial to self-administered mail questionnaires, since they

allow the researcher to provide supfKirting definitions and

instructions while being cheaper than personal Interviews

Be sure to pretest your questionnaire with respondents who
are similar to the ultimate survey respondents You'll be

surprised how much pretest respondents may misinterpret

your questions and response categories

Be sure to involve your programmer in questionnaire design.

There is nothing more frustrating than finding you can't

analyze a question in a particular way since the coding and

editing were done afier the questionnaire was filled out.

Devote resources to following up and increasing your

response rate. If it comes to a tradeoff between spending time

and money on increasing the initial sample size, and spending

time and money to follow up your initial mail-out sample, go
with the latter.

Try to avoid designing questionnaires by committee, and set a

definite limit on the number of questionnaire revisions you

can tolerate Remember that no questionnaire is perfect, since

no two individuals are alike and they will want to make
exceptions no matter how detailed your planning is.

Finally, always allow respondents in your survey the

opportunity to express themselves in their own words, even if

the questionnaire is substantially pre-coded; you will be

surprised how valuable individual handwritten comments can

be, especially when thay suggest potential uses and benefits

of your system which you may have overlooked.

CONCLUSION

Be sure to allow your user in the survey to give some

indication of frequency of use. You will want to divide your

analysis into "frequent users" and "infrequent users"; their

preferences may differ significantly

Approach "willingness to pay" questions with great caution

Many individuals are not accustomed to paying directly for

information services. The person whom you want to ask

about payment questions may be the business officer or

administrator with sign-off responsibilities, not the assistant

researcher who will not be paying you directly.

The most important secret to conducting a good user survey

cannot be described in methodological terms Excellent

sampling, proper questionnaire wording, mistake-free survey

processing, and sophisticated analysis cannot overcome an

incomplete or limited understanding of the product or service

whose use you are measuring. Be sure to ask yourself ivM'

y(W are conducting the survey. If you seriously try to answer

this question before you start out you will go a long way to

making sure that your survey helps you answer your most

important questions.