Category Archives: Research
If a journal does not hold constant the number of citable items it publishes year after year, then its Impact Factor will always decline if the rate of citable items published in a two year period increases faster than the one year rate of cites to those citable items. Hence:
- Using the Impact Factor to compare a journal’s impact year after year may only make sense (if at all) if that journal holds constant the number of citable items it publishes year after year.
- Megajournals (potentially) do not hold constant the number of citable items they publish year after year.
- Therefore, the Impact Factor does not apply to megajournals.
Background & Explanation
The Impact Factor (IF) is a simple average. It’s the number of cites that accumulated in Year A to citable items (articles, etc.) a journal published in the previous two years, and then divided by the count of those citable items. To illustrate, to arrive at a 2015 IF for some journal, we sum the number of cites in year 2015 to the citable items a journal published in years 2013 and 2014, and then divide that by the count of those citable items.
There are many reasons why the IF is no longer a great (or appropriate) metric for evaluating or ranking journals (a web search will return many sources that criticize the IF). It was a useful metric when it was invented in the 1960s. Academic librarians, and others, even at that time, had to manage an increasing glut of scholarly journals and using the IF, aided with a decent understanding of Bradford’s Law, helped librarians manage print journal collections. It’s no longer necessary to use the IF for that, for a number of reasons that can be discussed at some other time. However, beyond such a scenario, the IF suffers because it is also an artifact of its time: the computational power and data storage capabilities in the 1960s were, obviously, limited and thus some selection process helped reduce the amount of data that needed crunching. Such selection shortcuts are less necessary now.
Therefore, these days the IF is not simply misused, it is also largely obsolete, and this is certainly true for megajournals like PLOS ONE and others. Specifically, because the IF is a simple average, using it to compare a single journal’s IF year after year only makes sense if that journal publishes a fairly constant number of citable items year after year: that is, if the denominator (count of citable items) in the IF stays [fairly] constant year after year while the numerator (citations to citable items) is left to vary. For most journals for most of modern history, this has been the norm. That is, journals generally publish X amount of articles per week, per month, or per quarter, and generally maintain this type of activity across time. Journal publishers do this because for most of modern history (i.e., before the web), they published in a print format and suffered the constraints that the economies of print placed on them (e.g., page constraints due to page costs). Hence, binding articles and disseminating them via an (e.g., weekly, monthly, or quarterly) issue made the best economic sense (limitations of scale in a print only era). Even though most journals have migrated to online formats and an increasing number no longer print physical copies, this remnant from the past, the constraint on pages per year and a set number of issues per year, remains a standard way of doing things, even if such a constraint is not placed on them by a publisher.
It could be that this way of managing a journal and publishing serially benefits certain kinds of work flows, and these benefits may be more pronounced for smaller research communities and journals, but this way of disseminating scholarly articles is not a necessity in digital publishing. Yet, the standard way of doing things is ingrained and many journals that are born online or born digital continue to publish periodically: that is, a fairly constant set of articles per issue and issues per year. Even some early journal software platforms, like Open Journal Systems, were built to accommodate this as a part of the software’s basic framework. What’s interesting, then, are those born digital journals that think past this assumption and have removed themselves from the constraints of printing. Enter the megajournal.
I’m fairly neutral (if not skeptical) about the existence of megajournals like PLOS ONE. I think those of us who study scholarly communication need to continue to flush out the possible implications with this kind of publishing and these kinds of platforms (that’s the job). That is, we don’t know enough yet to say whether these entities are good or bad for science. However, in the last several years, the IF for PLOS ONE has declined and several popular sources have drawn, what I think, are hasty generalizations about what that decline means. The only accurate thing we can say, right now, about the decline is that it is a result of publishing works, over a two year period, at a rate that is faster than the rate of incoming cites that accumulated in a one year period (for the 2-year IF). To illustrate this, below is a table showing the counts of cites, citable items, and 2-year IF scores taken from Thompson Reuters’ Journal Citation Reports for the journal PLOS ONE. I’ve added two additional columns that show the percentage change in the number of cites and the percentage change in the number of citable items. As we can see, the 2 year IF score declines each year simply because the percentage increase for the number of citable items published is larger than the percentage increase in the number of cites to those items:
|Year||Cites||Cites Perc Δ||Citable Items (CI)||CI Perc Δ||2-yr IF||CI Perc Δ > Cites Perc Δ|
(By the way, there’s not enough data in the table above to say anything remotely conclusive about trends, no matter what anyone says.)
I’m looking forward to the 2015 release of the Journal Citation Reports simply because it will be interesting to see the reactions to PLOS ONE‘s score. A couple of summers ago (and the one before that), when the 2014 scores were released, there were a number of blogs that predicted the decline or the fall of PLOS ONE because its IF fell again. Even though some of those posts explain the IF and refer to a decreasing number of submissions to or publications by PLOS ONE, which may be true but may be true for reasons such as the size of the scientific community, funding sources, etc. and not to authors’ reactions to a decline in PLOS ONE‘s IF, no one, as far as I can tell, actually refers to the simple problem with the moving parts of the IF as a fraction, and by entailment, why the rate of cites might be lower than the rate of citable items. That latter issue leads to many more interesting and illuminating questions.
This post originally appeared on my personal blog.
The Myth: Digital Divide has a small literature. Pretty much, almost every book or paper on the topic will say this. I used to believe that not enough work has been done on Digital Divide, until I started studying for my qualifying exam. Fortunately or unfortunately I found out that the literature is actually very large. The problem is that the digital divide research is spread throughout all kinds of disciplines, such as: ICT4D, Community Informatics, HCI, Social Informatics, Sociology and Communication studies. In fact, the literature is not new, because it goes way back when academics were studying the diffusion of telephones and televisions.
The Problem: Quantitative approaches are addressed to answer the wrong questions. A lot of the research done on digital divide is done quantitatively. They rely on the data collected by International Telecommunication Union, World Bank and other agencies. And what these researches do is to identify a digital gap and try to correlate that gap with some sort of social, economic or political issue. For example, there is a cross country study done by Luis Andres, he says that, based on his quantitative analysis, in order to bridge the digital gap we need to liberalize the telecommunication market to promote internet provider competition. I agree, but Brazil has had this free market for about 15 years, and we still have a vast digital divide. So, obviously, this is not an issue for Brazil, something must be happening that is keeping the divide wide. What I’m trying to say here is that in order to fully understand and propose meaningful solutions, the digital divide research requires local and context based research. It doesn’t matter if it’s quantitative or qualitative, I don’t want to get into this argument, but we need to understand that each country has its own set of policies, people have different cultural backgrounds, so solutions need to be tailored and not based on general auto analysis.
The Challenge: “How to talk to policymakers?”. Policymakers of the digital divide tend to have a technological deterministic perspective. They focus on single factors, such as “access”, because they are convenient since they are easy to measure. These simple measures can be used to influence public opinion since lay people can relate to them. Policymakers also need to justify allocation of resources, which is easier to do when they can create benchmarks (Barzilai-Nahon, 2006). So policymakers are strung up on numbers, and how can we show them that subjective factors such as education and training can be of much better value to promote the digital inclusion than pure access? I don’t want to blame policymakers for approaching the digital divide quantitatively, but I’d like to leave this challenge for us, digital divide scholars, to realize a way to start conversations with people that can only see numbers.
Barzilai-Nahon, Karine. 2006. “Gaps and Bits: Conceptualizing Measurements for Digital Divide/s.” Information Society 22:269-278.
As many of you know, I am now directing a Social Informatics (SI) Group in a School of Informatics and Computing (SoIC) at Indiana University Bloomington. The SI group is quite unique in Informatics/Computer Science/Information Studies, it that is has chosen to oriented itself explicitly to the field of Science, Technology, and Society (STS, also referred to as Science and Technology Studies). I am also thinking about retirement in the next 3-5 years. Being in these situations has shaped the research agenda that follows.
My current research is all framed generally within Socially Robust and Enduring Computing. SREC is based on the notion that developing a notion of social robustness, comparable to the technical notion of robustness in Computer Science, is a goal worth pursuing. I have developed SREC with colleagues in Trento, Italy.
My main research time commitment at the moment is to a writing project on Value(s) with Maurizio Teli, a young researcher at the Foundation in Trento, where I spend a couple months every year. My interest in this area grew out of efforts to identify the forms whereby and the extent to which computing professionals are responsible ethically for the current economic and social crisis set off in finance. Maurizio’s and my value(s) project is a continuation of this work on the crisis and is linked to the project of David Graeber in Debt: The First 5000 Years, itself a work that builds on much of the recent anthropology of value. That is, we want to give a similar account of the ways in which value and values are and should be treated and thought about in the reproduction of current social formations. Such an account is made necessary by the ways in which contemporary reproduction is increasingly detached from the prior industrial dynamics but which has not yet established a new dynamic. In our view, establishing new social formation reproduction dynamics requires identification of new values, new institutions for pursuing those values, and new means to measure especially value relating to the success or failure of establishing these new values and institutions. A major point we wish to make regards the increasingly larger role in these new dynamics we see being played by common pool resources, the focus of Eleanor Ostrom, winner of the 2009 Nobel Prize in economics, and, until she died last Spring, a valued colleague here in Bloomington.
It is my hope that this writing will be paralleled by a research and demonstration project in Trentino on new systems, including information systems, for supporting the independent living of Seniors. This Suitcaseproject will build on my previous work in disability studies and technology, as well as more general ethnography in this region. Another aspect of the Trento ethnography is an attempt to understand what has made the region relatively hospitable to Participatory Design. PD is the focus of what I hope to and expect will be my last permanent contribution to the curriculum in the SoIC. In addition, I am working on another, related writing project, a text on Organizational Informatics, with Stefano De Paoli, another researcher. This text will incorporate much of the work behind my 2011 AAA paper in the business anthropology sessions as well as my current teaching, including my course on the Ethnography of Information.
A final areas of research, this time in collaboration with two SoIC graduate students, Nic True and Shad Gross, is on Massive Multiplayer Online Games (MMOGs). In this work, we engage the current interest in Big Data, intending to show how some of the epistemological shortcomings in its standard approaches can be address when it is triangulated with ethnography. In our case, we argue that a preliminary ethnography of gaming can provide clearer direction regarding what we should be looking for in the automated analysis of large corpora of game play data. This work is directly related to the effort in SoIC to create a professional masters degree in Big Data.
Presented in this way, it should be easy to see, as I said initially, the multiple ways in which this research agenda is a function of my current position. While I have participated in the AAA meetings and CASTAC occasionally since I went to Indiana in 2004, this occasional connection has not been enough to justify systematic orientation of my research toward anthropology. Ironically, when I studied the careers of anthropologists interested in STS in the 1980s, I found a similar phenomenon; there were few if any examples of individuals who developed these interests while sustaining strong connections with academic anthropology. I should mention that my efforts to interest Indiana University Bloomington Department of Anthropology scholars in this type of work has born little fruit.
I mention these things as a warning: Interest in the anthropology of science, technology, and computing is not automatically, or even generally, a good way to build a career in anthropology. Working in and through vehicles like CASTAC should thus be understood as essential to the work of anthropologists who wish to continue to do so.
This blog post can also be seen at: http://blog.castac.org/2012/11/on-building-social-robustness/