Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model

Gavin W. Fulmer, Hye-Eun Chu, David F. Treagust, Knut Neumann

    Research output: Contribution to journalArticleResearchpeer-review

    Abstract

    Two-tier multiple-choice (TTMC) items are used to assess students’ knowledge of a scientific concept for tier 1 and their reasoning about this concept for tier 2. But are the knowledge and reasoning involved in these tiers really distinguishable? Are the tiers equally challenging for students? The answers to these questions influence how we use and interpret TTMC instruments. We apply the Rasch measurement model on TTMC items to see if the items are distinguishable according to different traits (represented by the tier), or according to different content sub-topics within the instrument, or to both content and tier. Two TTMC data sets are analyzed: data from Singapore and Korea on the Light Propagation Diagnostic Instrument (LPDI), data from the United States on the Classroom Test of Scientific Reasoning (CTSR). Findings for LPDI show that tier-2 reasoning items are more difficult than tier-1 knowledge items, across content sub-topics. Findings for CTSR do not show a consistent pattern by tier or by content sub-topic. We conclude that TTMC items cannot be assumed to have a consistent pattern of difficulty by tier—and that assessment developers and users need to consider how the tiers operate when administering TTMC items and interpreting results. Researchers must check the tiers’ difficulties empirically during validation and use. Though findings from data in Asian contexts were more consistent, further study is needed to rule out differences between the LPDI and CTSR instruments.
    LanguageEnglish
    Pages1-1-1-16
    Number of pages16
    JournalAsia-Pacific science education
    Volume1
    DOIs
    Publication statusPublished - 2015

    Fingerprint

    science
    diagnostic
    classroom
    Korea
    Singapore
    student

    Bibliographical note

    Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

    Keywords

    • science education
    • two-tier items
    • rasch measurement models
    • optics
    • scientific reasoning

    Cite this

    @article{0cac69743cce487da2ffb36f3fda350f,
    title = "Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model",
    abstract = "Two-tier multiple-choice (TTMC) items are used to assess students’ knowledge of a scientific concept for tier 1 and their reasoning about this concept for tier 2. But are the knowledge and reasoning involved in these tiers really distinguishable? Are the tiers equally challenging for students? The answers to these questions influence how we use and interpret TTMC instruments. We apply the Rasch measurement model on TTMC items to see if the items are distinguishable according to different traits (represented by the tier), or according to different content sub-topics within the instrument, or to both content and tier. Two TTMC data sets are analyzed: data from Singapore and Korea on the Light Propagation Diagnostic Instrument (LPDI), data from the United States on the Classroom Test of Scientific Reasoning (CTSR). Findings for LPDI show that tier-2 reasoning items are more difficult than tier-1 knowledge items, across content sub-topics. Findings for CTSR do not show a consistent pattern by tier or by content sub-topic. We conclude that TTMC items cannot be assumed to have a consistent pattern of difficulty by tier—and that assessment developers and users need to consider how the tiers operate when administering TTMC items and interpreting results. Researchers must check the tiers’ difficulties empirically during validation and use. Though findings from data in Asian contexts were more consistent, further study is needed to rule out differences between the LPDI and CTSR instruments.",
    keywords = "science education, two-tier items, rasch measurement models, optics, scientific reasoning",
    author = "Fulmer, {Gavin W.} and Hye-Eun Chu and Treagust, {David F.} and Knut Neumann",
    note = "Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.",
    year = "2015",
    doi = "10.1186/s41029-015-0005-x",
    language = "English",
    volume = "1",
    pages = "1--1--1--16",
    journal = "Asia-Pacific science education",
    issn = "2364-1177",
    publisher = "Springer, Springer Nature",

    }

    Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model. / Fulmer, Gavin W.; Chu, Hye-Eun; Treagust, David F.; Neumann, Knut.

    In: Asia-Pacific science education, Vol. 1, 2015, p. 1-1-1-16.

    Research output: Contribution to journalArticleResearchpeer-review

    TY - JOUR

    T1 - Is it harder to know or to reason? Analyzing two-tier science assessment items using the Rasch measurement model

    AU - Fulmer, Gavin W.

    AU - Chu, Hye-Eun

    AU - Treagust, David F.

    AU - Neumann, Knut

    N1 - Version archived for private and non-commercial use with the permission of the author/s and according to publisher conditions. For further rights please contact the publisher.

    PY - 2015

    Y1 - 2015

    N2 - Two-tier multiple-choice (TTMC) items are used to assess students’ knowledge of a scientific concept for tier 1 and their reasoning about this concept for tier 2. But are the knowledge and reasoning involved in these tiers really distinguishable? Are the tiers equally challenging for students? The answers to these questions influence how we use and interpret TTMC instruments. We apply the Rasch measurement model on TTMC items to see if the items are distinguishable according to different traits (represented by the tier), or according to different content sub-topics within the instrument, or to both content and tier. Two TTMC data sets are analyzed: data from Singapore and Korea on the Light Propagation Diagnostic Instrument (LPDI), data from the United States on the Classroom Test of Scientific Reasoning (CTSR). Findings for LPDI show that tier-2 reasoning items are more difficult than tier-1 knowledge items, across content sub-topics. Findings for CTSR do not show a consistent pattern by tier or by content sub-topic. We conclude that TTMC items cannot be assumed to have a consistent pattern of difficulty by tier—and that assessment developers and users need to consider how the tiers operate when administering TTMC items and interpreting results. Researchers must check the tiers’ difficulties empirically during validation and use. Though findings from data in Asian contexts were more consistent, further study is needed to rule out differences between the LPDI and CTSR instruments.

    AB - Two-tier multiple-choice (TTMC) items are used to assess students’ knowledge of a scientific concept for tier 1 and their reasoning about this concept for tier 2. But are the knowledge and reasoning involved in these tiers really distinguishable? Are the tiers equally challenging for students? The answers to these questions influence how we use and interpret TTMC instruments. We apply the Rasch measurement model on TTMC items to see if the items are distinguishable according to different traits (represented by the tier), or according to different content sub-topics within the instrument, or to both content and tier. Two TTMC data sets are analyzed: data from Singapore and Korea on the Light Propagation Diagnostic Instrument (LPDI), data from the United States on the Classroom Test of Scientific Reasoning (CTSR). Findings for LPDI show that tier-2 reasoning items are more difficult than tier-1 knowledge items, across content sub-topics. Findings for CTSR do not show a consistent pattern by tier or by content sub-topic. We conclude that TTMC items cannot be assumed to have a consistent pattern of difficulty by tier—and that assessment developers and users need to consider how the tiers operate when administering TTMC items and interpreting results. Researchers must check the tiers’ difficulties empirically during validation and use. Though findings from data in Asian contexts were more consistent, further study is needed to rule out differences between the LPDI and CTSR instruments.

    KW - science education

    KW - two-tier items

    KW - rasch measurement models

    KW - optics

    KW - scientific reasoning

    U2 - 10.1186/s41029-015-0005-x

    DO - 10.1186/s41029-015-0005-x

    M3 - Article

    VL - 1

    SP - 1-1-1-16

    JO - Asia-Pacific science education

    T2 - Asia-Pacific science education

    JF - Asia-Pacific science education

    SN - 2364-1177

    ER -