California Surveyors Forums

end result is it the actual test takers for this years exam had issues, it needs to be put in writing to the Board. If enough have the same issue (more than 1), I believe the Board staff will try to address the issue in future exams. For license's if you want to make a difference, can keep an open mind, contact the Board and ask to be involved in the process, it is a educational eye opening experience.

RAM wrote: For license's if you want to make a difference, can keep an open mind, contact the Board and ask to be involved in the process, it is a educational eye opening experience.

Boy, this is so true. I would highly recommend this to anyone that has strong opinions about the test. It is really an educational experience. You get to meet some great people too.

Sorry to chime in so late on this topic. For those of you who are not aware, I serve CLSA (the CLSA President, Executive Committee, Board of Directors, and membership) as the BPELSG liaison. In that capacity, I attend most BPELSG and LS TAC meetings and communicate often with BPELSG staff. I have been involved with exam grading; taught college, LS Review, and my organization's internal land surveying classes; and served as CLSA Education Committee Chair while that committee developed the LS Exam Review Class guidelines for Chapters to hold LS Review classes. As CLSA EdComm chair, I also prepared several years of extensive analyses of exam candidate pass rates including various aspects such as review class participation and prior licensure as a PE.
The OP contacted me late on the night of the exam (November 2nd) to express his concerns to me about the California State Specific PLS Exam as described above. I attended the LS TAC and BPELSG meetings on November 4th and 5th. On the 5th (the same day as this post started), I expressed the concerns the OP and others had to BPELSG during their session. The Board and BPELSG staff were GENUINELY CONCERNED about the issues raised. Various suggestions for improvement were made and discussed, and BPELSG staff are evaluating process improvement options. We all (CLSA, members of the land surveying profession, the BPELSG Board and BPELSG Staff) want an exam that is fair, consistent, and protects the public. As a PLS candidate, the OP desires the same thing. And yes, we are all members of the public as well as licensed professionals.

Ricard - I was aware of your concerns as well, and they were shared with BPELSG.
Mr. Hixon, & Mr. Maffia - Having the reference maps and deeds on screen is not about saving $10, it is about maintaining exam security and protecting the funds expended on exam development. Preparing an entirely new exam would cost perhaps $4 million. If an exam was compromised, would you agree that every LS in the state be billed $1,000 to help cover the cost of developing a new exam?
Mr. Kendall - if last fall's issues were brought to BPELSG's attention, perhaps this would not have been an issue this time around. Let's hope it is resolved in the near future. I urge anyone else who experienced issues with the exam to contact BPELSG staff, or me if they are more comfortable doing so.
Mr. Page - Although you raise some valid points, and have support from others who are less familiar with the exam process, I can't completely buy in to some of the rhetoric, despite the imaginative imagery (gold painted paperweight). Having a multiple choice exam is not "about saving the state $$$, period." It is about having an exam that is fair, consistent, and protects the public, and yes, it helps protects BPELSG and all of us licensees from exam subversion as well. I find it interesting that you appear to be adamantly opposed to the multiple choice format. Weren't you licensed in 1996, where a good portion of the morning session was multiple choice?
LS 8750 - Please re-read the original post. The OP was not "sniveling", he was raising his concerns in a professional manner, and trying to determine if other testing centers had similar issues in the hopes of improving the process. Feel free to apologize to the OP. This thread is important, it should not be deleted.
DMI, RAM, and Mr. Reading - Thank you for the voices of experience and reason in this discussion.
Coast - You should have heard me bitching about the Board after the 1994 exam. Then I saw my answers in appeal... Dumb mistakes... I was qualified, but in too much of a hurry. Over the last 2 decades the pass rate has varied from less than 2% (the year of the nine) to over 34%. If examinees and their mentors understand what they are going to be tested on, there is far less complaining and far better results. The first year of the NCEES PS exam, California candidates had much higher passing percentage than the national average.
Licensed professionals - It is up to you to mentor your people, to learn about the exam process, to provide your staff guidance, and to sign for those people who you believe are truly qualified to sit for the exam! New licensees are the responsibility of the profession - when you are completing the reference form, don't shrug your shoulders and say "Let the exam determine if they are qualified..."

Best regards,

Rob McMillan, PLS
CLSA BPELSG Liaison

So what's any different now, exam security-wise, from all those decades when hardcopies were the norm?

Take their driver's licenses; return them when they return the papers. Done.

^ yup

Rob,

Multiple choice can be a useful part of a professional exam. It cannot comprise the whole of a valid professional level exam format. I don't recall how much, if any (if any, it was a minor portion) of the exam I took in 1996 was comprised of multiple choice, but I had also taken the OR LS the previous year (LS 2708, allowed to lapse several years ago). At that time OR used 6 hours of the NCEES exam and had a 2 hour state-specific. The NCEES was, as now, all multiple guess and the state-specific was essay or other show and communicate your work type questions. At that time, OR not only provided licensees with notice of whether or not they passed, but also provided their percentage scores as well.

I suppose that it's possible that I am some super amazing genius surveyor and neither I nor anyone else recognizes it, but it's more likely that in addition to better than minimally competent knowledge and good preparation for the exam, that I gleaned several points from the process of eliminating improbable answers and a few good guesses. My scores on the NCEES portions were 98% and 95%, and I finished each in 1/3 to 1/2 the time allotted. My score on the state-specific, in which I needed to communicate my conclusions and the reasoning behind them was 82%. Without bragging, I know that I am, and was then able to communicate more effectively than the average surveyor, which means that the 82% was probably a pretty accurate measure of my performance on a professional level exam.

Many of the questions on the NCEES were pretty simple for a professional level exam, but even so, had they been given as open ended questions, my scores likely would have been 5% to 10% less, or much closer to the score I earned to problems in which there was no possibility of gaining points by knowing just enough to eliminate some answer options and making good guesses, and for which I had to demonstrate knowledge, good reasoning, and correct conclusions.

Even if the questions themselves are of an equal difficulty to those that would be given as short answer or essay questions, an examinee will inherently perform 5% to 10% better on the format where the correct answer is given along with a few distractors. That in itself is enough to lift the just-not-competent to the just-competent level without throwing out questions and lowering the cut scores. When you take an exam that already does not have questions of great difficulty (most comments I've seen of past CA PLS exams were along the lines of "no overly difficult questions, just not enough time to complete entire exam"), and then provide a short list of possible answers with the qualifier that one is correct, you have greatly reduced the time pressure and have made it possible for those that almost but not quite know the material well to mark the correct answer where they would not have been able to provide a completely correct answer had it not been placed before them.

When the cut score for such an exam is lowered to near 50%, the results do not adequately serve to protect the public by ensuring that those who pass have actually achieved a professional level of knowledge, and more importantly, a professional level to reason and to convey conclusions.

Having the lowest set of educational and experience requirements to take the exam among the states, together with an exam that no longer adequately measures the professional attributes that such an exam should measure, and lacking any kind of continuing education or professional development requirement of existing licensees, I fear that we are going to witness the decline of the profession.

Some have expressed that they fear the decline of the profession due to lesser numbers of licensees which would translate to lesser influence among related professions. I don't see decreasing numbers being nearly as destructive as the attempt to keep the numbers up by making it easier to gain a license. The decline will be in that a level of expertise that others should be able to look to surveyors for will not be there. Every problem that surveyors find will be one that the client will be sent to either a lawyer or an engineer to fix.

Rather than the fear that civil engineers will be clamoring for the authority to perform surveys again because of the lack of licensed surveyors, they are going to be clamoring for that authority because nothing that the licensed surveyor is expected to know will be very difficult and will have been effectively reduced to matters that could fit into the traditional core of the engineer's skill set - math, measurement, and looking up a set of specifications to decide whether or not to consider various types of boundary evidence.

I'm glad that the Board/TAC are looking for a solution to the specific complaints of the OP, but that does not address more fundamental problems.

You have an impressive set of credentials to speak on the topic. Without any sarcasm, I admire the time and effort that you put into the betterment of the profession, and I think you know that. I was on the ad hoc committee put together by BPELSG mad up of, IIRC, 5 practicing surveyors from around the state that was given a presentation of the all multiple choice CBT format before it was adopted. Ostensibly the committee was formed to give advice as to how, to what extent, and whether the format should be adopted. The reasoning given for adopting was 1) cost savings, 2) improved security, 3) defensibility. It was very clear that Ric & Pat wanted to move forward with this, and somewhat evident that they would regardless of the recommendations of this committee.

I pointedly asked if the purpose of the committee was to provide advice that would be given very serious consideration, or if it was simply to get agreement to be able to say that they had the buy in of the profession. We were assured that our opinions were important and would be incorporated into the decision making.

There was no denying the cost savings or security aspects. Most of the discussion centered on the possibility of creating an exam in this format of sufficient difficulty, whether questions could be devised to adequately measure reasoning ability, and of the appropriateness of eliminating all portions of the exam the measured communications. By the end of the meeting, one guy was all for the new format, another was mostly sold on the concept but wanted additional information as to how the concerns raised would be addressed, and the rest of us were dubious as to whether a multiple choice format could be used as the sole format to adequately measure minimal competence as a professional. We were told that there would be at least one follow up meeting to address the concerns, probably about 3 months hence.

The next I heard anything about it, Ric was announcing a few months later at the Fresno Conference that the all multiple guess format had been adopted. I checked with each of the other members on that committee to see if they had participated in a follow up meeting. There was no such meeting.

I have no doubt that the LS's who participate in formulating the questions and answers do so with the intent of making the best exam that they can, but the format is inadequate to measure reasoning, incompetent to measure communication, and considering the manipulations of the cut score, inadequate to reliably measure knowledge.

You're right when you say that it's not all about the costs. It's also about boosting numbers of licensees.

Lee Hixson wrote:So what's any different now, exam security-wise, from all those decades when hardcopies were the norm?

Take their driver's licenses; return them when they return the papers. Done.

Although I would like to agree with that, exam subversion has gotten a lot more sophisticated. With cameras that can be hidden inside a common looking pen, all sorts of communication devices, some of which can be disguised as something else if someone comes in having planned and prepared to obtain copies of the exam, the proctors would need to hold much more than just driver's licenses. They would need to hold all electronics, allowing only certain calculators after inspection. They would need to employ electronic countermeasures such as jammers or signal detectors. They would need to inspect all items brought it to determine whether they were the innocuous objects they appear to be or are actually a covert camera or communication device.

The driver's license idea would fall apart the first time someone cries that they were improperly denied the opportunity to take the exam because they did not have a driver's license. The list of acceptable IDs would likely end up being so broad as to be almost meaningless. There is probably some workable means of providing for collateral security, but it would take a good deal of thought to arrive at the solution which is effective yet precludes the possibility or likelihood of a complaint that would be given legitimacy by a court.

Much of the other concerns continue whether there are papers to hand in or not. Pictures can be taken of computer screens nearly as well as of papers.

Much of the security concern with the hardcopy exams was in the transport of boxes of exams to and from the testing sites. With printing and transport, there were several points at which exam materials could get stolen or lost. I recall someone telling a story of a proctor having been in a traffic accident several years ago and exam materials getting scattered about the scene. A very rare occurrence, but security concerns nonetheless.

A computer based format eliminates a good deal of concern associated with transport to and from exam locations and grading sessions.

I'm not opposed to a computer based format, just the all multiple guess format. I don't see why a computer based format cannot also include fill-in-the-blank questions, short answer questions, essay questions, and other question formats that can reliably measure an examinee's reasoning abilities and communications skills. Oh wait, I remember. It would require that BPELSG again form a grading team of live LSs to collectively discuss and grade the answers. That costs $$$$.