Remember Silly Putty?  It was a really popular toy back in the day.  It could do all sorts of things.  It bounced.  It could be used to glue two things together.  It was actually used by astronauts to secure their tools while in space.  If you left it alone on a warm day it would melt.  And, it was darn near impossible to get out of your clothing and hair.

The coolest property of Silly Putty was that if you flattened it and pressed it against a newspaper, it would transfer the image to the Silly Putty.  For those of you that remember that, we have an analogy to weighting of survey data to share.  Bear with us, this is a bit of a stretch. 🙂

Imagine your survey data set is a flattened handful of Silly Putty.  Your task is to faithfully represent a one-panel comic from the newspaper with it.  If your survey sample is plentiful and covers the image perfectly, this just requires that you are careful as you press it against the comic.  Voila, you’ve represented your universe perfectly!  (Okay, we know it will be a mirror image, but ignore that!)

However, this isn’t really how it worked with Silly Putty or how it works with survey data.  What tended to happen was you didn’t have quite enough Putty to flatten onto the newspaper, or you didn’t quite cover the entire comic with it.  So, you spread it out as best you could.  Then, when you had lifted the image, you stretched the putty a bit to try to make it look like the original.  The problem was that if you stretched the putty in one direction, there tended to be a contraction of it in another.

That is analogous to what we are doing when we try to make a non-random sample match a universe.  We may be lacking enough putty (not enough sample size) or might not be able to get it to perfectly cover the picture (we under-represent some groups).  Through careful weighting (stretching the putty) we can usually get an imperfect, but accurate enough representation of the universe (the image).  If we weight (stretch the putty) too much, we distort the universe (the image).  (That can be really funny with Silly Putty, but it isn’t so funny with research data.)

As “silly” as this sounds, we have found it to be a useful analogy for clients.  Clients often push us to weight data too much.   This is like stretching the Silly Putty so much that you can’t recognize the picture any more.  Well thought out adjustments can make sense if we know what we are shooting for.  We need to know what the universe looks like, just as the Silly Putty user needs to know what the image he/she is seeking to represent looks like.  Stretching it in the dark doesn’t meet with good results.  And, when we weight one group (stretch the putty in one direction), it has the effect of distorting another (contract the putty in another direction).

Weighting is best when we are making subtle adjustments that improve the picture.  Because we almost never have a random sample, it is necessary.  But it can be overdone, and we have to be careful not to stretch the Silly Putty too far.