Publishing research using outdated methods












2















I'm currently an Economics MA student doing a referee report on a paper that employs a dated empirical method.Specifically I'm looking at a paper on an application of the Almost Ideal Demand System* for gas prices in 2012.



It should be noted that this method was suggested in 1980 but then improved upon in 1998 by other researchers (called the Quadratic Almost Ideal Demand System) after discovering certain issues with it.



This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.



This is concerning because it shows that published researchers make mistakes and dont review all relevant research before publishing. I can imagine that in the hard sciences and medicine this would happen also (which is especially concerning knowing that those on the cutting edge could be decades behind in knowledge which has been around for a while).



Is this acceptable? If so, why?





* for those interested i made a video teaching the topic here: https://www.youtube.com/watch?v=zsTknpqUzh4










share|improve this question




















  • 1





    That's a pretty specific description of the paper - the authors would almost certainly recognize it if they read this, and then they'll know that you are the referee (and with your YouTube channel I bet they can tie you back to your real self). So I think you're putting your anonymity as a referee in jeopardy with this post.

    – Nate Eldredge
    3 hours ago













  • @NateEldredge im a student. Chill

    – EconJohn
    3 hours ago






  • 1





    Your decision, of course, but as a student, you're likely more vulnerable to potential retaliation than an established researcher would be.

    – Nate Eldredge
    3 hours ago











  • @NateEldredge assuming these people are adults with real lives and real academic dialouge and experience im good. This is fairly common assignment in graduate degree programs.

    – EconJohn
    3 hours ago













  • @NateEldredge this post isnt even about a specific paper. It could be said about the whole body of literature of demand system estimation.

    – EconJohn
    3 hours ago


















2















I'm currently an Economics MA student doing a referee report on a paper that employs a dated empirical method.Specifically I'm looking at a paper on an application of the Almost Ideal Demand System* for gas prices in 2012.



It should be noted that this method was suggested in 1980 but then improved upon in 1998 by other researchers (called the Quadratic Almost Ideal Demand System) after discovering certain issues with it.



This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.



This is concerning because it shows that published researchers make mistakes and dont review all relevant research before publishing. I can imagine that in the hard sciences and medicine this would happen also (which is especially concerning knowing that those on the cutting edge could be decades behind in knowledge which has been around for a while).



Is this acceptable? If so, why?





* for those interested i made a video teaching the topic here: https://www.youtube.com/watch?v=zsTknpqUzh4










share|improve this question




















  • 1





    That's a pretty specific description of the paper - the authors would almost certainly recognize it if they read this, and then they'll know that you are the referee (and with your YouTube channel I bet they can tie you back to your real self). So I think you're putting your anonymity as a referee in jeopardy with this post.

    – Nate Eldredge
    3 hours ago













  • @NateEldredge im a student. Chill

    – EconJohn
    3 hours ago






  • 1





    Your decision, of course, but as a student, you're likely more vulnerable to potential retaliation than an established researcher would be.

    – Nate Eldredge
    3 hours ago











  • @NateEldredge assuming these people are adults with real lives and real academic dialouge and experience im good. This is fairly common assignment in graduate degree programs.

    – EconJohn
    3 hours ago













  • @NateEldredge this post isnt even about a specific paper. It could be said about the whole body of literature of demand system estimation.

    – EconJohn
    3 hours ago
















2












2








2








I'm currently an Economics MA student doing a referee report on a paper that employs a dated empirical method.Specifically I'm looking at a paper on an application of the Almost Ideal Demand System* for gas prices in 2012.



It should be noted that this method was suggested in 1980 but then improved upon in 1998 by other researchers (called the Quadratic Almost Ideal Demand System) after discovering certain issues with it.



This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.



This is concerning because it shows that published researchers make mistakes and dont review all relevant research before publishing. I can imagine that in the hard sciences and medicine this would happen also (which is especially concerning knowing that those on the cutting edge could be decades behind in knowledge which has been around for a while).



Is this acceptable? If so, why?





* for those interested i made a video teaching the topic here: https://www.youtube.com/watch?v=zsTknpqUzh4










share|improve this question
















I'm currently an Economics MA student doing a referee report on a paper that employs a dated empirical method.Specifically I'm looking at a paper on an application of the Almost Ideal Demand System* for gas prices in 2012.



It should be noted that this method was suggested in 1980 but then improved upon in 1998 by other researchers (called the Quadratic Almost Ideal Demand System) after discovering certain issues with it.



This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.



This is concerning because it shows that published researchers make mistakes and dont review all relevant research before publishing. I can imagine that in the hard sciences and medicine this would happen also (which is especially concerning knowing that those on the cutting edge could be decades behind in knowledge which has been around for a while).



Is this acceptable? If so, why?





* for those interested i made a video teaching the topic here: https://www.youtube.com/watch?v=zsTknpqUzh4







journals academic-history






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 5 hours ago







EconJohn

















asked 6 hours ago









EconJohnEconJohn

1696




1696








  • 1





    That's a pretty specific description of the paper - the authors would almost certainly recognize it if they read this, and then they'll know that you are the referee (and with your YouTube channel I bet they can tie you back to your real self). So I think you're putting your anonymity as a referee in jeopardy with this post.

    – Nate Eldredge
    3 hours ago













  • @NateEldredge im a student. Chill

    – EconJohn
    3 hours ago






  • 1





    Your decision, of course, but as a student, you're likely more vulnerable to potential retaliation than an established researcher would be.

    – Nate Eldredge
    3 hours ago











  • @NateEldredge assuming these people are adults with real lives and real academic dialouge and experience im good. This is fairly common assignment in graduate degree programs.

    – EconJohn
    3 hours ago













  • @NateEldredge this post isnt even about a specific paper. It could be said about the whole body of literature of demand system estimation.

    – EconJohn
    3 hours ago
















  • 1





    That's a pretty specific description of the paper - the authors would almost certainly recognize it if they read this, and then they'll know that you are the referee (and with your YouTube channel I bet they can tie you back to your real self). So I think you're putting your anonymity as a referee in jeopardy with this post.

    – Nate Eldredge
    3 hours ago













  • @NateEldredge im a student. Chill

    – EconJohn
    3 hours ago






  • 1





    Your decision, of course, but as a student, you're likely more vulnerable to potential retaliation than an established researcher would be.

    – Nate Eldredge
    3 hours ago











  • @NateEldredge assuming these people are adults with real lives and real academic dialouge and experience im good. This is fairly common assignment in graduate degree programs.

    – EconJohn
    3 hours ago













  • @NateEldredge this post isnt even about a specific paper. It could be said about the whole body of literature of demand system estimation.

    – EconJohn
    3 hours ago










1




1





That's a pretty specific description of the paper - the authors would almost certainly recognize it if they read this, and then they'll know that you are the referee (and with your YouTube channel I bet they can tie you back to your real self). So I think you're putting your anonymity as a referee in jeopardy with this post.

– Nate Eldredge
3 hours ago







That's a pretty specific description of the paper - the authors would almost certainly recognize it if they read this, and then they'll know that you are the referee (and with your YouTube channel I bet they can tie you back to your real self). So I think you're putting your anonymity as a referee in jeopardy with this post.

– Nate Eldredge
3 hours ago















@NateEldredge im a student. Chill

– EconJohn
3 hours ago





@NateEldredge im a student. Chill

– EconJohn
3 hours ago




1




1





Your decision, of course, but as a student, you're likely more vulnerable to potential retaliation than an established researcher would be.

– Nate Eldredge
3 hours ago





Your decision, of course, but as a student, you're likely more vulnerable to potential retaliation than an established researcher would be.

– Nate Eldredge
3 hours ago













@NateEldredge assuming these people are adults with real lives and real academic dialouge and experience im good. This is fairly common assignment in graduate degree programs.

– EconJohn
3 hours ago







@NateEldredge assuming these people are adults with real lives and real academic dialouge and experience im good. This is fairly common assignment in graduate degree programs.

– EconJohn
3 hours ago















@NateEldredge this post isnt even about a specific paper. It could be said about the whole body of literature of demand system estimation.

– EconJohn
3 hours ago







@NateEldredge this post isnt even about a specific paper. It could be said about the whole body of literature of demand system estimation.

– EconJohn
3 hours ago












3 Answers
3






active

oldest

votes


















2














What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.



Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.



Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.






share|improve this answer































    2














    What you call "an outdated method" another may call "the well-understood method".



    In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis1.



    I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.





    1 In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)






    share|improve this answer

































      1















      I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?




      You're the referee, so you tell us!



      As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identify that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).






      share|improve this answer























        Your Answer








        StackExchange.ready(function() {
        var channelOptions = {
        tags: "".split(" "),
        id: "415"
        };
        initTagRenderer("".split(" "), "".split(" "), channelOptions);

        StackExchange.using("externalEditor", function() {
        // Have to fire editor after snippets, if snippets enabled
        if (StackExchange.settings.snippets.snippetsEnabled) {
        StackExchange.using("snippets", function() {
        createEditor();
        });
        }
        else {
        createEditor();
        }
        });

        function createEditor() {
        StackExchange.prepareEditor({
        heartbeatType: 'answer',
        autoActivateHeartbeat: false,
        convertImagesToLinks: true,
        noModals: true,
        showLowRepImageUploadWarning: true,
        reputationToPostImages: 10,
        bindNavPrevention: true,
        postfix: "",
        imageUploader: {
        brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
        contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
        allowUrls: true
        },
        noCode: true, onDemand: true,
        discardSelector: ".discard-answer"
        ,immediatelyShowMarkdownHelp:true
        });


        }
        });














        draft saved

        draft discarded


















        StackExchange.ready(
        function () {
        StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2facademia.stackexchange.com%2fquestions%2f125685%2fpublishing-research-using-outdated-methods%23new-answer', 'question_page');
        }
        );

        Post as a guest















        Required, but never shown

























        3 Answers
        3






        active

        oldest

        votes








        3 Answers
        3






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes









        2














        What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.



        Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.



        Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.






        share|improve this answer




























          2














          What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.



          Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.



          Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.






          share|improve this answer


























            2












            2








            2







            What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.



            Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.



            Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.






            share|improve this answer













            What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.



            Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.



            Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered 5 hours ago









            DrecateDrecate

            5,15012241




            5,15012241























                2














                What you call "an outdated method" another may call "the well-understood method".



                In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis1.



                I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.





                1 In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)






                share|improve this answer






























                  2














                  What you call "an outdated method" another may call "the well-understood method".



                  In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis1.



                  I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.





                  1 In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)






                  share|improve this answer




























                    2












                    2








                    2







                    What you call "an outdated method" another may call "the well-understood method".



                    In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis1.



                    I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.





                    1 In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)






                    share|improve this answer















                    What you call "an outdated method" another may call "the well-understood method".



                    In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis1.



                    I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.





                    1 In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)







                    share|improve this answer














                    share|improve this answer



                    share|improve this answer








                    edited 4 hours ago

























                    answered 4 hours ago









                    eykanaleykanal

                    41.9k15101205




                    41.9k15101205























                        1















                        I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?




                        You're the referee, so you tell us!



                        As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identify that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).






                        share|improve this answer




























                          1















                          I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?




                          You're the referee, so you tell us!



                          As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identify that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).






                          share|improve this answer


























                            1












                            1








                            1








                            I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?




                            You're the referee, so you tell us!



                            As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identify that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).






                            share|improve this answer














                            I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?




                            You're the referee, so you tell us!



                            As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identify that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).







                            share|improve this answer












                            share|improve this answer



                            share|improve this answer










                            answered 27 mins ago









                            BenBen

                            13.5k33461




                            13.5k33461






























                                draft saved

                                draft discarded




















































                                Thanks for contributing an answer to Academia Stack Exchange!


                                • Please be sure to answer the question. Provide details and share your research!

                                But avoid



                                • Asking for help, clarification, or responding to other answers.

                                • Making statements based on opinion; back them up with references or personal experience.


                                To learn more, see our tips on writing great answers.




                                draft saved


                                draft discarded














                                StackExchange.ready(
                                function () {
                                StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2facademia.stackexchange.com%2fquestions%2f125685%2fpublishing-research-using-outdated-methods%23new-answer', 'question_page');
                                }
                                );

                                Post as a guest















                                Required, but never shown





















































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown

































                                Required, but never shown














                                Required, but never shown












                                Required, but never shown







                                Required, but never shown







                                Popular posts from this blog

                                GameSpot

                                日野市

                                Tu-95轟炸機