Brett Buttliere

Imagine having argued against AI like a year ago :D

seems to be built with an assumption of small samples, since anything past .0001 or whatever is discounted in evidential value in some way

yes, how did they land on this curve, can you explain? seems to penalize big time past .045, it is because of the formula they apply?

Notice that *small* p values --- those with high evidential weight --- are mapped to 0, and values near the significance criterion are mapped to arbitrarily large values. This means that positive evidence is destroyed, and negative evidence is given very heavy weight! Example: suppose you... 5/?

Replication and Mere Replication - Michael Mulkay, G. Nigel Gilbert, 1986

I guessed you’d bring this up.

You should read more about replication. Scientists really don’t do it, you know. They never did, in the real world.
journals.sagepub.com/doi/abs/10.1...

agreed, and if you look closely at replications most often they are done by people who are not connected with the research field. often being done in fact by teams that actively doubt the research field, often they are not piloted and problems are found. mostly just not done how orig authors would

Adopting standard variable labels solves many of the problems with sharing and reusing data - Brett Buttliere, 2021 Datasets and analysis scripts are becoming more available online, but most datasets are still unclear and difficult to use due to poor meta-data. Adopting stand...

have you implemented any WM specific variable labels or meta.data? basically, if you standardize your labels and meta data you can combine datasets if they measure the same or similar things. big improvement and could be useful for you journals.sagepub.com/doi/full/10....

when i first started asking her questions i had lots of people defending such behavior but now it is crickets

notable silence from @elisabethbik.bsky.social who is normally so vocal! :D

i was wondering why it is was so hidden behind the paywall

i was wondering recently how you feel about our QReP paper now that you have had like 6 8 months to think about it

in theory they are good but in practice they are often used incorrectly. for instance by people who have no experience in the area, often times they are used so that the team only has to try one time, often they dont involve any piloting of the study, and as a result they fail more often

@elisabethbik.bsky.social when asked cant provide any ethical approval for her most famous 2016 study, or any information on her false positive rate

no open analysis scripts, no open data, no science

@nature.com @science.org @einsteinberlin.bsky.social @plos.org @plosbiology.org @error.reviews

I am sure @science.org @cos.io @nature.com @currentbiology.bsky.social @plosbiology.org also have an interest in your setting an example in this regard and making clear that you did not actually get ethical approval for the study, and you did not take any precautions about false positives?

opportunity to respond or explain i mean. for all i know all of the different colors are changed to make the differences and conditions more obvious.

slight difference, it looks kind of suspicious i agree but im pretty sure most ethics boards would also agree until there is some opportunity and investigation, it probably shouldnt be on social media

i think if it is just an example well stupid and bad but not really fraudulent

I have a paper where we ran a study with the same pictures of cartoon people but in some of them the skintone changed and flipped. and also there is a study i know where the same words were presented but in different colors and this is demonstrated in the paper. how this different?

I was also wondering, do you not also publish your results when you don't find any problematic images? i.e., your negative results, or do you only post your positive results?

and what effect do you think that has on your results or empirical conclusions?

thx for transparency

I am sure @science.org @cos.io @nature.com @currentbiology.bsky.social @plosbiology.org also have an interest in your setting an example in this regard and making clear that you did not actually get ethical approval for the study, and you did not take any precautions about false positives?

can you link to the ethical approval for your 2016, most famous study?

Id also like information on your false + rate, and what steps you take to safeguard people who might be falsely accused.

I assume you have this material prepared, since it would be needed for any real ethics approval .. ?

why is it not written as a paper or as part of your open materials and analysis scripts?

isnt this the sort of thing that ai is just going to do as a part of review?

how does it in your eyes substantially change the interpretation or evaluation of the paper? to me that picture means nothing but i do not have any exp in area.

did it change anything or is it just example?

haha, glad you remember me and that i didn't start with nice to meet you. Merry Christmas - why not answer the questions though

you really trying to get famous eh? :D I wonder about you - can you say some words about how you safeguard against false positives?

do you use a standard analysis script, or how do you keep yourself from seeing false things in your excitement to find fraud?

what about your conflict of interest?

can you link to the ethical approval for your 2016, most famous study?

Id also like information on your false + rate, and what steps you take to safeguard people who might be falsely accused.

I assume you have this material prepared, since it would be needed for any real ethics approval .. ?

i drive an old bmw, i really dont think i want anything newer - the thunk of the car door is just unmatched by todays standards and newer cars they ding at you if you go too fast, unpleasant

you dont consider it questionable or problematic to not mention these results in a talk about reproducibility?

how is it different in your eyes about an original researcher talking about their work without mentioning failed replications of it?

I didnt see simine mention those results in her talk even though she surely knew about them (she started collabra)... i did see her talk a lot about the studies that these studies show were wrong though.. funny how that works no? but ignore it and you will be able to retire feeling good

many labs 5 finds a similar 30% but notice how they don't talk about it in the abstract.. basically their huge studies came to the wrong conclusion

also ML5 was the last many labs ... hm but ok just ignore it and you will feel good while the field burns

30% of failed original replications are successful when run again and over 60% do not fall within the confidence interval of the original replication.