Responses

Download PDFPDF

Review
Performance validity test failure in clinical populations—a systematic review
Free
Compose Response

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
Author Information
First or given name, e.g. 'Peter'.
Your last, or family, name, e.g. 'MacMoody'.
Your email address, e.g. higgs-boson@gmail.com
Your role and/or occupation, e.g. 'Orthopedic Surgeon'.
Your organization or institution (if applicable), e.g. 'Royal Free Hospital'.
Statement of Competing Interests

PLEASE NOTE:

  • A rapid response is a moderated but not peer reviewed online response to a published article in a BMJ journal; it will not receive a DOI and will not be indexed unless it is also republished as a Letter, Correspondence or as other content. Find out more about rapid responses [https://authors.bmj.com/after-submitting/rapid-responses/].
  • We intend to post all responses which are approved by the Editor, within 14 days (BMJ Journals) or 24 hours (The BMJ), however timeframes cannot be guaranteed. Responses must comply with our requirements and should contribute substantially to the topic, but it is at our absolute discretion whether we publish a response, and we reserve the right to edit or remove responses before and after publication and also republish some or all in other BMJ publications, including third party local editions in other countries and languages
  • Our requirements are stated in our rapid response terms and conditions and must be read. These include ensuring that: i) you do not include any illustrative content including tables and graphs, ii) you do not include any information that includes specifics about any patients,iii) you do not include any original data, unless it has already been published in a peer reviewed journal and you have included a reference, iv) your response is lawful, not defamatory, original and accurate, v) you declare any competing interests, vi) you understand that your name and other personal details set out in our rapid response terms and conditions will be published with any responses we publish and vii) you understand that once a response is published, we may continue to publish your response and/or edit or remove it in the future.
  • By submitting this rapid response you are agreeing to our terms and conditions for rapid responses [https://www.bmj.com/company/journals-terms-and-conditions-for-rapid-responses/] and understand that your personal data will be processed in accordance with those terms and our privacy notice [https://www.bmj.com/company/your-privacy/].
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.

Vertical Tabs

Other responses

Jump to comment:

  • Published on:
    Response to Larrabee et al
    • Laura McWhirter, Clinical Research Fellow University of Edinburgh
    • Other Contributors:
      • Craig Ritchie, Professor of Psychiatry of Ageing
      • Jon Stone, Professor of Neurology
      • Alan Carson, Professor of Neuropsychiatry

    We read Larrabee and colleagues’ e-letter response to our systematic review on Performance Validity Testing (PVT). Whilst we welcome debate, and we recognize that some clinicians will disagree with our conclusions, we were disappointed that they misrepresented our paper in formulating their response:

    1. The authors state “Throughout the paper, the authors refer to PVTs as “effort tests”, a characterization that is no longer in use in the United States..”. In reality we used the term “effort test” only twice in our paper; in the introduction: “(PVTs), also historically called effort tests” and once in the methods in describing our search terms. By contrast we use the term PVT on 45 occasions.

    2. We are concerned that they then go on to misrepresent the results of our review. We found a wide variation in results in different clinical groups and in different tests. We noted that failure rates for some groups and some tests exceeds 25%. We did not conclude that all failure rates were as high as this, but rather that failing a PVT was not a rare phenomenon but was reasonably common in a range of clinical groups.

    We presented results to support our conclusion that the PVT literature is problematic with regards to blinding to diagnosis and potential for selection bias.

    We also uphold our speculation that an alternate explanation for failure on forced choice tests at above chance cutoffs may result from attentional deficit related to other symptoms. W...

    Show More
    Conflict of Interest:
    None declared.
  • Published on:
    Response to Kemp et al.
    • Laura McWhirter, Consultant Neuropsychiatrist University of Edinburgh
    • Other Contributors:
      • Craig Ritchie, Professor of Psychiatry of Ageing
      • Jon Stone, Professor of Neurology
      • Alan Carson, Professor of Neuropsychiatry

    We read with interest Kemp and colleagues response to our recent systematic review on Performance Validity Testing (PVT). In response to specific criticisms raised:

    1- The searches and data extraction were conducted by one investigator. We agree this is a potential limitation although only if papers were missed, or data was erroneously transcribed, and it can be demonstrated this would have changed the conclusions. Although Kemp and colleagues place great weight on this, the evidence they put forward to support their contention was limited. Of the four citations in their letter, reference 2 and reference 4 were in fact included (see our supplementary tables and our reference 57)(1,2). Reference 3 was, by coincidence, published simultaneously with our manuscript submission and was not available to us(3).

    Reference 1 did not fit the terms of our search strategy and was not included although it would have been eligible(4). It was an unblinded study of the ‘coin in hand test’, a brief forced choice screening test for symptom exaggeration, administered to 45 patients with mixed dementias. It found 11% scored at or above a 2 error cut off and the authors proposed a new set of cut offs for interpretation; it was in keeping with our conclusions. We’d be happy to consider any other specific omissions or quality assessment issues not discussed which the authors consider would have altered the conclusions of the review.

    2- The authors criticise our understanding...

    Show More
    Conflict of Interest:
    None declared.
  • Published on:
    Letter: Response to McWhirter et al (2020)
    • Glenn J. Larrabee, Licensed Psychologist Independent Practice Sarasota FL USA
    • Other Contributors:
      • Kyle B. Boone, Licensed Psychologist
      • Kevin J. Bianchini, Licensed Psychologist
      • Martin L. Rohling, Professor and Licensed Psychologist
      • Elisabeth M.S. Sherman, Licensed Psychologist

    McWhirter et al. (2020) reviewed the published literature on Performance Validity Tests (PVTs), concluding that high false positive (Fp) rates were common in clinical (non-forensic) samples, exceeding 25%. In their discussion, they stated: “The poor quality of the PVT evidence base examined here, with a lack of blinding to diagnosis and potential for selection bias, is in itself a key finding of the review.” They also conclude that the use of a forced choice format with cut scores that are significantly above chance on two alternative forced choice tests (e.g., TOMM), raises questions about the utility of the forced choice paradigm, essentially characterizing these PVTs as “floor effect” procedures. As such, McWhirter et al. then argued that failure at above chance cutoffs represents “functional attentional deficit in people with symptoms of any sort,” rather than invalid test performance due to intent to fail.

    Throughout the paper, the authors refer to PVTs as “effort tests”, a characterization that is no longer in use in the United States, in part because PVTs require little effort to perform for persons experiencing significant cognitive impairment (1). Rather, PVTs have been defined as representing invalid performance that is not an accurate representation of actual ability. Continuing to refer to PVTs as “effort tests” allows McWhirter et al. to more easily mischaracterize the tests as sensitive attentional tasks affected by variable “effort” rather than measur...

    Show More
    Conflict of Interest:
    All contributors provide neuropsychological consultation in medicolegal matters.
  • Published on:
    Response to McWhirter et al
    • Steven Kemp, Consultant Clinical Neuropsychologist Leeds Beckett University
    • Other Contributors:
      • Narinder Kapur, Consultant Clinical neuropsychologist

    Dear Editor

    Response to McWhirter et al (2020):

    In their article, Performance validity test failure in clinical populations - a systematic review, McWhirter and colleagues (2020) present the ‘base rates’ of performance validity test (PVT) failure (or what are commonly referred to as effort tests) and offer an analysis of PVT performance from their perspective as neurologists and neuropsychiatrists.

    As a group of senior practicing clinical neuropsychologists, we are pleased that they have drawn attention to an important issue, but we have significant concerns about the methodology used and with several of the conclusions drawn within the review. We present this response from the perspective of U.K. neuropsychology practice, and as practitioners involved in research and formulating clinical guidance on the use of PVTs. In preparing this response, we were aware of parallel concerns of our U.S. counterparts (Larrabee et al) but we have submitted separate responses due to the word limit.

    The systematic review methodology used by McWhirter et al. has resulted in a limited number of papers being included, and there is no indication of the quality of the studies included. All of the literature search and analytic procedures appear to have been undertaken by one person alone, hence there was no apparent control for human error, bias, omission or inaccurate data extraction. Also, it is unclear to us to what extent McWhirter and colleagues had the knowle...

    Show More
    Conflict of Interest:
    None declared.