Archive for the 'Education' Category

Asking about gender and sexual orientation on surveys

When composing questionnaires, there are times when the simplest of questions have to adjust to fit the times. Questions we draft become catalysts for larger discussions. That has been the case with what was once the most basic of all questions – asking a respondent for their gender.

This is probably the most commonly asked question in the history of survey research. And it seems basic – we typically just ask:

  • Are you… male or female?

Or, if we are working with younger respondents, we ask:

  • Are you … a boy or a girl?

The question is almost never refused and I’ve never seen any research to suggest this is anything other than a highly reliable measure.

Simple, right?

But, we are in the midst of an important shift in the social norms towards alternative gender classifications. Traditionally, meaning up until a couple of years ago, if we wanted to classify homosexual respondents we wouldn’t come right out and ask the question, for fear that it would be refused or be found to be an offensive question for many respondents. Instead, we would tend to ask respondents to check off a list of causes that they support. If they chose “gay rights”, we would then go ahead and ask if they were gay or straight. Perhaps this was too politically correct, but it was an effective way to classify respondents in a way that wasn’t likely to offend.

We no longer ask it that way. We still ask if the respondent is male or female, but we follow up to ask if they are heterosexual, lesbian, gay, bisexual, transgender, etc.

We recently completed a study among 4-year college students where we posed this question.  Results were as follows:

  • Heterosexual = 81%
  • Bisexual = 8%
  • Lesbian = 3%
  • Gay = 2%
  • Transgender = 1%
  • Other = 2%
  • Refused to answer = 3%

First, it should be noted that 3% refused to answer is less than the 4% that refused to answer the race/ethnicity question on the same survey.  Conclusion:  asking today’s college students about sexual orientation is less sensitive than asking them about their race/ethnicity.

Second, it is more important than ever to ask this question. These data show that about 1 in 5 college students identify as NOT being heterosexual. Researchers need to start viewing these students as a segment, just as we do age or race. This is the reality of the Millennial market:  they are more likely to self-identify as not being heterosexual and more likely to be accepting of alternative lifestyles. Failure to understand this group results in a failure to truly understand the generation.

We have had three different clients ask us if we should start asking this question younger – to high school or middle school students. For now, we are advising against it unless the study has clear objectives that point to a need. Our reasoning for this is not that we feel the kids will find the question to be offensive, but that their parents and educators (whom we are often reliant on to be able to survey minors) might. We think that will change over time as well.

So, perhaps nothing is as simple as it seems.

The Cost of Not Going to College Is Probably Not As High As You Think

Each year, there are a number of studies that show the same thing:  there has never been a time when the salary gap between high school graduates and college graduates has been higher. According to Pew, this salary gap currently averages $17,500. The College Board puts the gap at $21,100.The implication seems clear:  stay in school, go to college, and reap the benefits.

However, there is actually a lot of nuance to this story and the true causes of this wage gap are rarely discussed. First, the fact that an average college graduate makes, say $20,000 more than a high school graduate entering the workforce does not mean that if you coax a high schooler who was not going to go to college to attend, he/she will make that much more. In fact, you should expect that particular student to make a much lower wage premium. Why?

The data both Pew and the College Board cite suffers from what researchers would call a “self-selection bias.” In short, high school graduates who enter the workforce immediately after graduation aren’t a comparable base of individuals to those who choose to go to college. The result is an apples-to-oranges comparison that makes the economic value of going to college versus going straight to the workforce to seem greater than it actually is.

To get a true measure of the “college premium” we’d have to run an experiment. We’d take a large sample of high school seniors and assign them to either “work” or “college” randomly. The difference in the “work” and “college” group would be the true college premium, and would be much less than the $20,000 that is claimed. (Of course this experiment could never actually happen!)

Why? Because the high school senior who chooses to college has higher earning potential than the one who chooses to work and would earn more even if he/she did not go to college. Similarly, the high school senior who chooses to work rather than college would be expected to make less than the average that current college students make if he/she chose to go to college.

Another example of this same concept would be the salary figures colleges promulgate. The median starting salary for a Stanford graduate is $61,300 per year. The average starting salary for a 4-year college graduate is $45,370.

Does this imply a Stanford education is responsible for a $15,930 starting salary premium compared to an “average” 4-year college? Absolutely not. To understand the Stanford premium, we’d have to take all incoming college students and randomly assign them to colleges. Then, in four years we can compare the average starting salary of graduates and make a credible claim that the Stanford premium is the difference. It will be much less than $15,930. Why? Because the incoming Stanford student has a potential earning power that is higher than the typical incoming college student. Much of the current “Stanford premium” would be due to this self-selection of the student and not to the education they receive at Stanford.

The information that is put out there regarding the college premium has unintended, but serious consequences. First, it pushes many students to choose college who will not gain the salary premium they expect. Many of these students will take substantial loans, may drop out, and will left in a financial mess that takes much of their adult lives to recover from. It is a little known fact that just 53% of those who enroll in a 4-year college actually end up graduating.

Second, this thinking drives many strong students to go to more expensive colleges. Many don’t realize that the salary premiums they will command likely have more to do with who they are than where they choose to attend college. It is likely that the key determinants of a young person’s success will not be where he/she went to college but more their own talents, hard work, and ambition.

Finally, our political leaders jump on statistics such as the college premium. They perpetuate a myth that all students should go to college, establish programs to make this possible, etc. This has contributed to unemployment among college graduates, declines in starting salaries among those who do, a crisis in middle skills employment, and a mismatch of labor to available jobs.

This is not to say a college education is not a worthy pursuit. In fact, it is a good idea for most, and jobs should not be the sole goal of college. However, we don’t do right by high school students by overstating this gap and having a singular mindset that college is the only path to success.

Analysis Paralysis and Big Data

Many of us in the market research field lament our poor timing when choosing when to be born. We went to college and graduate school and entered a field that focuses on data analysis at a time when this was a marginally marketable skill.  Data analysis is a field that has now exploded in its value to employers. Many of us feel we were at the forefront of an eventual “nerd takeover” of the business world.

We can explain why in two words:  “Big Data.” Never has there been more data aching to be analyzed. Consumers used to be tracked in just two ways:  when they bought something and when they took the time to interrupt their dinners to answer a telephone survey in the evening. Now, people are being tracked in ways unimaginable just a few years ago and perhaps in ways they don’t even realize.

The digital trail we leave as we navigate the Internet leaves a powerful and permanent data contrail. It used to be that marketers could learn where we went online.  Now, they can also learn who we are, what our friends do, and where we are when we do various things.

Yet, despite all of this data and attempts to harvest it, marketers often seem no more knowledgeable about their customers than they were a generation ago. This conundrum is often chalked up to a failing of researchers:  we have all this data, yet our researchers just haven’t yet discovered how to separate the signal from the noise.

While this may well be true, there also might be a “hype curve” type phenomenon going on.  The “hype curve” was made most famous by Gartner and is often studied in MBA programs. In short, when we are faced with something disruptive, we tend to overstate its potential.  Then, when the reality of the phenomenon inevitably fails to meet the hype, we adjust our expectations downward… but by too much.  We start to be overly critical of the phenomenon’s potential. In the final phase, the actual potential of the phenomenon establishes itself – somewhere between the initial hype and our revised, downward expectations of it.


The hype curve can be used in many contexts.  I’ve seen it applied in politics.  President Obama came to office amid sky high (and unrealistic) expectations.  As he invariably failed to meet them all, people revised their assessment of his potential too far downward.  Eventually, his performance will be reviewed by historians as being between these two extremes.

I’ve seen the concept applied to music artists. A music artist puts out an incredible first album.  Fans start touting them as the next coming of the Beatles.  Their second album can’t possibly meet these expectations, and when it comes out the group start to fade in popularity. By the time their third album is released, it is time for their popularity to settle at an appropriate level.

They hype curve concept is most commonly applied to technology. A new gadget comes out.  We hear about how it will save the world and change our lives. It fails to meet those expectations, and people start to think that it won’t make much of a difference at all. Over time, the gadget finds its level – it is a useful addition to our lives.  Its reality falls between the initial hype and the revised expectation.

The hype curve example is applicable to Big Data as well, and we are in the early stages of it. Our expectations of what can be done with the incredible amounts of data that are out there are overstated. There will be a point soon when our expectations will be revised downward, and people will start to underestimate what can be done. Eventually, like all other innovations, Big Data will find its level.

So, right now is the perfect time to graduate from college with strong data analysis skills – too late for many of us unfortunately!


Does Class Size Matter?

Reducing class sizes is a commonly discussed goal in education. However, there may not be a more consequential educational issue where the academic research available is a poorer match to anecdotal evidence than the issue of class size.

Ask any teacher, administrator, or parent you know what they would prefer, and almost all of them will say that smaller class sizes are more conducive to learning than larger class sizes. Peruse any higher education website and you will find most try to trumpet their low student to faculty ratio. And, intuitively, it just makes sense that students will learn better if there are fewer of them in a class.

But, there is actually very little academic evidence that class size matters. Our review of the literature indicates that there is some evidence (gathered long ago) that smaller class sizes have an effect at the youngest grade levels, but little or inconclusive evidence that smaller class sizes matter among older students.

Yet a debate rages regarding class sizes. Teacher unions are understandably in favor of lowering class sizes, as this makes the job of the teacher easier and increases the numbers of teachers that need to be hired. Administrators seem to also favor lowering class sizes, but are wary to do so without much evidence indicating that it will improve academic achievement. Politicians favor it as well, as reducing class sizes certainly sounds like ad admirable goal to pursue.

What is undebatable is that there are significant costs involved in decreasing class sizes. Reducing class sizes means building more classrooms, maintaining larger facilities, and hiring more teachers.  The costs of reducing class sizes are potentially large, which is why it is surprising the issue doesn’t have much academic study and thought behind it.

We feel the issue has been oversimplified. Like most things we study, there are likely decreasing returns as class size is reduced. In other words, there is likely an ideal level for class size. There is probably a point where a class size can be too small, as tiny class sizes don’t allow for student-to-student learning and collaboration, small group projects, etc. As class size increases, it likely hits an ideal point, where the learning efficiency of the classroom is maximized. And, invariably, a class size can grow too large, where supervision of students is compromised.

It is possible that the academic studies that are available have not investigated a wide enough range of class sizes and therefore have not been able to spot this ideal point. Since no school district could (by law) change its average class size by more than a few students, academic researchers are likely concentrating on class size differences that are not large enough to show much of an effect.

However, in the debate over class sizes, there is an important issue we have never seen discussed. It is that the ideal class size is likely not the same for all situations. Even within a school, the ideal class size likely varies by the subject taught, the academic capabilities of the students, the grade level, and importantly, the particular strengths and weaknesses of the teacher.

For example, why do we presume that the same class size is needed for English as is needed for Math, or Foreign Languages? Why do we presume that 7th graders need the same class sizes as 12th graders? Or that a first-year teacher will be most efficient teaching the same class size as a proven teacher with 20 years of experience? Or that every student benefits most from the same class sizes?

We ignore the variability that is inherent in the process.  And, we don’t give our school managers (School Principals) much leeway in how they can manage their resources to take into account this variability.

We’d like to see Principals given a lot more latitude over how to best utilize their staff. In any organization whose success is dependent on the capabilities and productivity of its workers, the main task of a manager is to understand his/her staff’s capabilities and knowing how to properly deploy human resources.

Currently, Principals are given almost no latitude regarding class sizes. The Principal is forced to take a cookie cutter approach – with all teachers being assigned virtually the same number of students. A teacher is largely given the same responsibility on his/her first day on the job as his/her last day. Regardless of his/her subject, experience level, talents, teaching style, grade level, etc.  The teaching staff is the most important asset a Principal has to achieve academic excellence, and it is time to give Principals more responsibility in this area.

Class size absolutely matters. Just not in the same way and same level for every school, teacher, and student.

Does a Focus on One Genre of Music Hurt Music Education?


It is no secret that music education in America faces many threats. Curricula have been narrowed and a focus on testing in core subjects has diverted many educational resources to science and math education. Some would say this is for the better. Others would say that our children need to be well-rounded, and arts and music education are central to maintaining our culture. Plenty of studies underscore the importance of music education and its beneficial effects on learning for other subjects.

Kids and teens have an almost inbred interest in music. Thanks to the wonders of file sharing, Pandora, Spotify, and iTunes, most of them have thousands of songs on their playlists. Having ear buds and cords coming out of their ears is as common to them as wearing shoes is for the rest of us.

Ask any teen what is on their song list and you’ll find a wide range of genres and artists. It seems that tastes in music have broadened quite a bit over the past couple of generations. The only thing you’ll find in common on these playlists is that one genre is specifically missing from most of them:  Classical music.

The decline in music education comes both from the supply side (schools not offering as many opportunities) and the demand side (kids not being as interested in playing instruments and parents not caring enough to press to save the programs). Perhaps the decline in participation is inevitable, but I have a feeling that if enough students and parents wanted them and clamored for them, music programs would be expanding and not contracting.

A key contributor to this problem may be the types of instruments we offer to students. Ask any teenager to name which instrument they would play if they had the talent and opportunity. The vast majority would play back to you what they are hearing on their iPods: electric guitars, keyboards, bass, drum kits, ever turntables and mixers. How many of them would respond with instruments like the violin, the cello, or the clarinet?

Yet for some reason those are the types of instruments we offer students in school. Almost exclusively. Why is that? I am not quite sure when it happened, but a long time ago classical music interests completely took over the direction of music education in this country.

Ray Charles (quoting someone else I believe) once said that there are really only two types of music: good music and bad music. He is right. We need to get over a snobby perception that the local Philharmonic is somehow producing better and more culturally significant music than the local hip-hop artist or the busker playing for dollars in the subway.

What matters for our children is that they learn to create and have an opportunity to work to improve their skill.  And ultimately contribute to our culture.

Music educators should learn from the changes in physical education over the past twenty years. In many districts, the physical education program is no longer seen as “play time” and as an opportunity to give students a quick break from their studies. Physical education is now seen as a way to expose young people to wellness education and to learn the value of teamwork. Its goal is to expose students to a healthy lifestyle and hopefully to impress the value of keeping fit well beyond their school years.

Why shouldn’t music education do something similar? We should be trying to spark young people’s interest in music. We can only succeed in this by incorporating instruments they like to hear and genres relevant to them into the curriculum. Expose them to a wide range of genres and help them understand the cultural significance of music. Classical music’s historical background is important to understand as it has played a key role in history. But, so has jazz, big bands, blues, folk, rock and roll, and hip-hop. In fact, a case could be made that these genres are not only more interesting to students than classical, but more directly tied to our history and culture because they are more recent.

It isn’t hard to see that these genres have a strong chance at igniting a lifelong interest in music and creativity, as they are what young people flock to on their own time. Our goal should be to get students motivated to play and appreciate music across their entire lifetimes.

We can only meet this goal by getting over a perception that some types of genres and instruments are superior to others. This won’t happen as long as the one genre children have an almost universal dislike for is the only one we teach.

Just ask them: 9 out of 10 high school students are above average


We often have a need to ask a question that relates to academic performance in studies – so we can analyze results across a range of academic achievement. We necessarily have to rely on self-reports, and kids and teens tend to paint a fairly optimistic performance of their academic performance.

I used to think that this sort of optimism associated with your grades tended to result from faulty memory more than anything intentional. I know that my high school GPA and my college track times have miraculously improved as I have aged. But our results suggest that this isn’t the case – that students will greatly overstate their academic performance while they are still in high school.

The question we ask is straightforward: Which best describes your academic performance so far?

What we find is the following …

  • 7% of students will say they are in the top 1% of the class
  • 17% will say they are in the top 5%
  • 38% will say they are in the top 15%
  • 61% will say they are in the top 25%
  • 90% will say they are in the top 50%

So, 90% of high school students feel they are above average academically.

Is this a problem? I tend to think not – having confidence and a healthy sense of self-worth can be a good thing as children move out on their own. However, when this inflated sense of performance moves towards narcissism or unrealistic expectations it can be setting our children up for failure.

It can be challenging to ascribe a cause to this. Most commentators agree that the Millennial generation is characterized as being over-protected and having adults in their lives who continually reinforce how special they are. Grade inflation in schools and colleges can engender this feeling. Youth sports have moved to encouraging and reinforcing participation at least as much as rewarding successful competition. I suppose this all yields a generation with a healthy ego, but not necessarily one that has learned to deal with failure.

When searching for causes, we need to look towards parents as well, as they set the context for their children. In a parallel study, we have shown that parents are even more likely to feel their children are above average than the children themselves are. Among parents …

  • 11% will say their child is in the top 1% of the class
  • 40% will say their child is in the top 5%
  • 63% will say their child is in the top 15%
  • 84% will say their child is in the top 25%
  • 93% will say their child is in the top 50%

Again, while there may be positives in parents being optimistic regarding the abilities of their children, it can also be a cause of complacency. Why does the US lag other countries in test performance? Perhaps when this many parents overstate their children’s academic achievement it is difficult to create any urgency behind the issue. It is quite common on surveys to see parents state that the nation’s schools are doing a poor job, but their local district is outstanding.

“Well, that’s the news from Lake Wobegon, where all the women are strong, all the men are good looking, and all the children are above average.” –Garrison Keillor

Whose Job is it to Close the Gap?


There have been many studies released, from very credible sources, that indicate that a college education clearly pays back. A May 2014 New York Times article indicates that the pay gap between college graduates and non-graduates is widening, even as more students attend college. The College Board has indicated that both individuals and society as a whole benefit from increased levels of education. Pew Research has shown that although the pay gap is increasing, Americans are beginning to question the value of higher education and its affordability.

Today’s colleges face many challenges in helping prepare students for the workforce. As more students attend college and costs continue to rise, higher education institutions will be under increasing pressure to prepare students for the workforce. Gaps in workforce preparedness contribute negatively to employers’ views of graduates, the reputation of colleges, and the well-being of young adults. There is a sense that college curricula are struggling to keep pace with the changing needs of the workforce.

Crux Research recently conducted a study for Chegg which focused on workforce preparedness. We surveyed large samples of students, college faculty, and employers to explore beliefs around accountability and ownership in creating a hirable, attractive, ready-to-work population from U.S. colleges and universities.

This study sheds new light on issues of workforce preparedness, the unique perspectives of faculty and employers, and the need for a new approach to the way faculty and employers work together.

A summary of results of the project can be found at Chegg’s website here.

Crux Research to Present at 2014 MRA Insights + Strategies Conference

Crux Research, along with Bridgemark Solutions, will be presenting at the 2014 MRA Insights & Strategies Conference in Chicago, which runs from June 4-6. Our session, (“What Do Clients and Suppliers REALLY Think About Each Other?”) will take place on Thursday June 5. We anticipate an interactive session, as discussing the changes in the market research field and what they mean to the supplier/client relationships that are at the core of our industry.

Below is our session description:

How do clients really choose which research suppliers to work with?What makes a good client from the supplier’s standpoint?In conjunction with Bridgemark Solutions, Crux Research recently conducted an extensive industry poll among 290 individuals working in the market research field. This session will look closely at both sides of the industry to shed light on what really makes these relationships tick in order to help facilitate more efficient and rewarding relationships between suppliers and clients. This will be a highly interactive session and attendees will be encouraged to share their own experiences and insights.

Click here to register for the event. We hope to see you in Chicago!

Visit the Crux Research Website

Enter your email address to follow this blog and receive notifications of new posts by email.