Staff from the University of Oxford's Centre on Skills, Knowledge and Organisational Performance quizzed 723 sixth-formers about their attitudes to university and graduate debt.
The responses of the Year 13 students indicate a more hard-headed approach to university applications than in the past as tuition fees treble to a maximum of ?9,000 a year this autumn.
Four in 10 school-leavers say they are "concerned" or "very concerned" about the expected level of debt that would result from their taking out a student loan to pay for a university course.
More than three-quarters (78 per cent) of those questioned say they expect graduates to earn a higher salary than non-graduates. Of those who think graduates will earn more, 19 per cent are not applying for higher education courses.
However, of the 22 per cent who do not think graduates will earn more than non-graduates, nearly half (47 per cent) are not applying to university.
Helen Carasso, the study's co-author, said: "All the indications are that, under the new arrangements for fees and funding, prospective undergraduates will be very selective when applying to university.
"This may mean less of them are willing to go through the clearing process and accept an offer of a course or institution that was not on their original shortlist."
Despite concerns about debt, however, financial considerations are not the main factor for all school-leavers considering higher education, the study finds.
Only 28 per cent say their decision about a course was based on how much they were likely to earn after graduation.
Meanwhile, one-fifth of sixth-formers questioned say they do not know or have not thought about the level of debt they will accrue as a result of going to university.
Many students are also not able to comprehend the future level of student debt because the amount of money involved seems so large, the study finds.
Co-author Hubert Ertl, lecturer in higher education at Oxford's department of education, said: "Many of the sixth-formers we questioned were not clear about the details of the financial costs and benefits of a higher education degree, and it seems they postpone concerns about debt to a later point in time.
"However, they were clear that higher fees have increased the pressure on them to make the right decisions concerning what and where they study."
The findings have been published a day before this year's A-level results are released, heralding the start of this year's clearing window.