JQII Evaluation Process
Journal Quality Indicator of India (JQII) is a mechanism devised to determine the scientific value of Indian journals. It is a proxy indicator of the impact of the journals in Physics-Chemistry and Biology disciplines. This proxy indicator cannot be used as a parallel indicator for Thompson Reuter’s journal citation report. However, it does provide an opportunity to identify the scientific value of those journals which are yet to index in JCR. A glance at the impact factors of the journals covered by JCR reveals that a considerable number of Indian journals are yet to be included in JCR database. Therefore, there is a need for developing an alternative method for determining the quality of Indian scientific and technical journals. The mechanism of the scientific journal value assessment that we adopted here is based on certain criteria. As these criteria are mostly qualitative in nature, so it is named as Journal Quality Indicator. Additionally, we have attempted to convert the qualitative features of any journal into quantitative values by attributing the appropriate scores. We have conceptually grouped our criteria into three general categories, which are as follows:
1) Basic criteria,
2) Essential criteria and,
3) Subsidiary criteria.
Journals which have at least an ISSN number have been considered as eligible to be included in this database. All the evaluation is based on the characteristics of publications observed during the last five years. We tried to include as many factors as possible, which are likely affecting the quality of publishing. Before discussing the criteria and the evaluation process in detail, following issues should be kept in mind:
  • The scoring system we devised here is tentative and should never be considered as final, but it is relatively free from the rater bias because it can be applied by anyone. Over the time, the experience gathered from the actual handling of data may lead to further evolution of the approach.
  • By publishing an article, authors usually gain some value (e.g. better career opportunities and advancement, CV fortification, etc.), but authors should also be aware of the costs and quality control involved in the publishing process, as that also plays a role when judging about the value gained from that publication.
  • In the scholarly communication process, transparency in publication is essential, as it is often subject to human and computer flaws. No publisher is born experienced or perfect; nor is the author born knowledgeable. So, there is always a place and need for improvement for both the authors and the publishers.
  • Among the following factors for measuring quality, there are no neutral factors. Every factor has a relative weight, as well as positive and negative value. Just because a publisher or publication have negative score does not necessarily implies that the publisher and publication is poor.
Basic Criteria
  1. ISSN: As standard number is necessary for a document to be considered a part of formal scholarly communication process, we do not assign any specific score for a journal having an ISSN number. However, if the journal appears with same ISSN number for print and online version, we have assigned (-0.01) number in this case. On Line First Check if the journal publishes articles as ‘Online First’, i.e. before the print edition. If the online first version contains separate online ISSN, DOI, however, without page numbers, volume number, etc., we assign additional score of 0.01 points.
  2. Name of the Journal: On searching the name of the journal, if the name of journal appears as the verbatim of a reputed journal or comparatively old journal, then negative score of (-0.01) is given. Furthermore, if the journal is found multidisciplinary in its coverage against its nomenclature (covers other broad subject areas), then negative score of (-0.01) has been assigned. In the same sequence, we check the scope and mission of the journal mentioned in the journal documentation, if it does not match then negative score of (-0.01) has been assigned, otherwise no score.
  3. Brief information about a Journal: Under this criterion, we have examined, whether a journal mentions a short description including history, scope and subject coverage in its documentation. Although, no score has been given for the availability of such information, non-availability of this information, however, was treated as a negative feature of the journal and for those cases negative score of (-0.1) have been assigned.
  4. Longevity: We identified the starting year of a journal in order to calculate its years of existence in the scholarly communication process. Following the theory of survival of the fittest, we believe that only quality journals can survive for a long time. Therefore, the scoring for year of existence of a journal is 0.5 points for 10 years of existence, 1 point for 30 years of existence and 1.5 points for >50 years of existence has been assigned.
  5. Presence over internet: In present digital era, the users are computer savvy and easy access of information from anywhere has become need of them. So the presence of the journal over internet is considered very important characteristics of a journal. Therefore, if the journal have a web address then 0.01 score and if not negative score (-0.01) score has been assigned.
    1. Web address functional or not: If the links of the web address provided of a journal are functional or not? On randomly checking, if all links found functional especially for archival links then positive score added. In case such links found non-functional up to 30% then negative score of (-0.01) has been assigned.
  6. Accessibility: Journal is available in which format? Print only, online only, or in both formats, or online ahead of print. If the journal is available only in print form then negative score of (-0.01) has been assigned. However, if the journal is available in print and electronic both and articles can be accessible on web in full text, positive score (0.01) added. The accessibility of a close access journal has been confirmed through the tab .pdf/.html of article.
  7. Availability: Whether the journal is open access or close access? If, it is available in open access, then positive score of 0.1, and if, close access, then (- 0.1) has been allotted.
  8. Pricing: The pricing type of journal is checked, whether the journal is available for free or priced; or online free but print subscription based or both subscriptions based or APC based?
    1. If a journal charges Article Processing Charge (APC), whether or not they have clearly mentioned the amount of APC and the mode of payment in their documentation, is also checked. In case such information is available on the journal’s website, we further check whether the account number where publisher is asking APC charges, belongs to journal’s name or it is a personal account of someone else. If it is found that APC information is incomplete, hidden amount of APC or publisher is asking money to someone’s individual account, a negative score is added in the scoring process.
Essential Criteria
In this section we have attempted to evaluate journal through its four most important characteristics: Continuity in Publication, Review Policy, Editor & Editorial Board, and Indexing of Journal in Databases & Citation Profile. Status of the Journal: Whether the journal is live, ceased, or suspended. If ceased or suspended, then the year of suspension/termination has been identified and no further evaluation process has been conducted.
  1. Promptness in publication: Regularity in frequency and promptness are essential in publishing. To check the regularity of the journal, we considered the total publications of the journal for the last 5 years and checked upon the following points:
    1. Missing issues: Check, if there are any missing issues. For every missing issue, 0.01 point has been deducted.
    2. Combined issues: Check, if there are any combined issues. For every combine issues 0.01 point has been deducted.
    3. Late publication: Check, whether the latest issues came out with the stated frequency or not. For every late appearance 0.01 point has been deducted.
  2. Review Policy: In order to check review policy, we have examined the following:
    1. If the journal does not clearly mention the review policy of the journal in the documentation, a negative score of (-0.1) points has been assigned. otherwise no score.
    2. During our study, we found that in prestigious/reputed journals, (of physics-chemistry and biology) the review policy is mentioned for all i.e. for reviewers (how to review); for authors (how the paper will be reviewed); for editors (how to handle the process of review). This kind of detailed description of review policy for all brings transparency to the review process and increases the authenticity of the journal. Therefore, if the type of review policy is mentioned only in one word (like peer-review, double blind, blind, etc.), then no points were added. However, if such policy is described in detail, (e.g. in a paragraph or two) we have added 0.01 point and if it is available in short, then no score has been assigned. Additional value of 0.01 points has been assigned, if the journal explained review policy separately for authors and reviewers and has explained the process in sufficient details, thus making it clear for all.
  3. Nature of review policy: Clear Review policy is very important for the long life of a journal. What type of review policy is practiced by the journal, i.e. peer review/ blind peer review/ double blind peer review etc. have been checked. In fact, it is very difficult to ascertain if the journal adheres to the review policy mentioned in journal documentation. Because of this, we have cross-checked the claims of journals about the review policy with the data (see below) obtained on 20 randomly selected articles (systematic sampling) published during the last 5 years. The mean value and its standard deviation have been calculated on the basis of results and it was also checked if any of the articles is a statistical outlier on any of the selected measures.
    1. Cross checking of the review policy:
      1. Text length/ Abstract Length/ Reference number: In order to validate the claim of the journals regarding their review policy, we have checked their claim thoroughly by comparing mutual difference between two article of a journal in terms of total pages length of an article, number of words in abstract and number of references. Here we have applied quartile test (by choosing 20 randomly selected articles through systematic sampling, if the abstract length, article length and reference number of 50% articles comes below the corresponding lower bound of the quartile then negative score of (-0.1) has been assigned, otherwise 0.1 added. Here higher bound has not been considered for scoring.
  4. Articles per issue: In order to establish the average number of articles published in the last 5 years, we have counted the total number of articles published in all issues and then divided it with total number of issues. The score of articles per issue has been calculated as follows:
    Frequency Total Articles Score
    Monthly 0-4 0.02
    4-10 0.03
    10-15 0.01
    >15 0
    Four Issue per year/Quarterly 0-4 0.01
    5-10 0.02
    11-15 0.03
    16-25 0.01
    >25 -
    Three issues per year 0-4 0.005
    5-10 0.01
    11-15 0.02
    16-25 0.03
    25-40 0.01
    >40 -
    Half-yearly 0-4 0.001
    5-10 0.005
    11-15 0.01
    16-25 0.02
    25-40 0.03
    >40 -
    Yearly 0-4 0.001
    5-15 0.005
    16-25 0.01
    26-40 0.02
    >40 0.001
  5. Originality/plagiarism in the article: In order to check the originality, we have considered 20 articles randomly from the 5 years time span. iThenticate has been used to determine the percentage of duplication. If percentage of duplication shows less than 20% then 0.5 added. However, if such duplication shows more than 30% of an individual article (excluding self-citation, quotations, references), negative score of (-0.1) has been assigned.
  6. Editorial Quality of Journal: The role of Chief-Editor is important in a journal publishing. Editors of scientific journals are responsible for the selection of quality articles and their delivery, for making decision about acceptance/rejection, etc., which ultimately reflects quality. In order to perform this role in an effective manner, one has to be an expert in the given subject, but also has to have the expertise in handling such responsibilities. By following this rationale, we have attempted to determine how much the Editors, and especially Chief-Editors, are eligible to perform such a responsible job and whether they have expertise for it. During our investigation, we determined that the Chief-Editors in some APC based journals were also the owners of that journal, and the same/very similar ‘Editorial Boards’ serve on multiple journals. Such type of engagement seems unrealistic. Therefore, cross-checking of such issues have been conducted in respect to these parameters:
    1. Identity of Chief-Editor: If the name of the Chief-editor is not mentioned in the documentation, then negative score of (-0.01) has been assigned and if mentioned, then no score.
    2. Designation & Organization: Besides editing, what other post is held by the Chief-editor of the journal? Although, we have not correlated professional status with any scoring system, this will, however, help to determine the professional status of the Editors. If the affiliation of the Chief-editor is academic and working as at least Associate Professor rank, then positive score of 0.01 has been augmented otherwise no score.
    3. Name searchable through a search engine or not: If the name of the chief editor is searchable, did the results indicate their personal profile in Social Networking Sites, or did the profile show their scholarly activity? If such hit appeared only for Social networking sites only then negative score of (-0.01) has been assigned. In the other case, if the name of the chief editor appears except from social networking websites, then positive score of 0.01 is added.
    4. Publication Profile: Whether the publication list of the chief-editor is available or not. If available then additional score of 0.01 has been assigned and if not available then no deduction has been made.
    5. Publication in indexed journal or not: If it is available, a randomly selected 10% papers has been cross-checked to confirm, and if such papers are from ISI/WoS indexed journals, a positive score of 0.01 has been assigned. Otherwise, no points were given.
    6. Paper in the same journal: If more than 10% papers of the chief editor are published in the same journal edited by him/her then negative score of (-0.01) will be there.
    7. H-index of the Chief editor: Furthermore, we have checked, if the editor has an h-index. If yes, then 0.1 score was added, otherwise no value is added.
    8. Expertise (in years) & Subject Expertise: How long is the chief- editor performing the job of journal editing and whether he/she belongs to the subject field of the journal? When such expertise was more than 5 years, a positive score of 0.01 has been assigned.
    9. Editors in many journals: If same editor belongs to two or more than two journals as a chief editor then negative scoring (-0.1) has been made.
  7. Editorial Board Quality : Journal Editing is a joint effort and Editorial Board plays an important role in the editing process. Therefore, examining the role of editors, the proportion of Indian and foreign authors, and the countries from which they belong, are important to examine. After confirming the identities, as they are mentioned in a journal, we have assigned scores for each of the following sub-points.
    1. Editors for different Sections: In order to maintain quality, reputed journals appoint different editors for handling different types of scholarly jobs for different sections. We assume such quality is an important feature for measuring publisher’s concern for quality publishing. Therefore, a positive score of 0.01 has been assigned for such provision.
    2. Geographic Diversity of Editorial Members: Here, we examined the percentage of the foreign members out of the total members in various sections of editorial activity. However, we have not considered foreign members who are on the Advisory Board. To assign the score in this field, 0.01% of the resultant percentage-value is calculated. If not found any foreign members in editorial board, negative scoring (-0.01) is assigned.
    3. Country of origin of the Editorial Board Members: Names of the countries to which foreign members of the editorial board belong to. Following scores have been allotted for each member from a different country.
      Country List Score
      USA, European Union Countries or non-Eastern European countries, Canada, Japan 0.03
      China, Malaysia, Denmark 0.02
      Others 0.01
    4. Same organization: How many editors/editorial board members are from the same institutions? For scoring of this criteria, If there are >50% members are from same organizations then negative score of (- 0.03), if in between 21-50% then 0.02. And if <20% but greater than 10% then 0.01 and within 10% then there would be no deduction.
    5. Any falsified information about editorial board: On cross checking, if any falsified information related to designation, affiliation etc. about the editorial members is found, then negative score of (-0.01) is added. (Randomly selected 10 names have been considered for cross checking and scoring).
    6. Address of Editorial Board is searchable through Search Engine or not: If the full identity (designation, Affiliation and contact details, etc.) of board members is obtained or searchable, +0.01 is added and if not then negative score of (-0.01) has been assigned.
  8. Attractability, Readability & Visibility
    1. A considerable number of indexing-abstracting databases is now available. Among them, a few are subject specific, while others have wide subject coverage. In this part, we have attempted to identify prominent databases and then categorized them into four categories, as shown in the table below.
      Category A Category B Category C Category D
      General Subject General Subject General Subject General Subject
      Thompson Reuter’s JCR Physics Abstract, INSPEC SCOPUS COMPENDEX, Google Scholar, ProQuest Others
      General Subject General Subject General Subject General Subject
      Thompson Reuter’s JCR Chemical Abstract SCOPUS CAS Google Scholar, ProQuest Others
      General Subject General Subject General Subject General Subject
      Thompson Reuter’s JCR Biological Abstract, BIOSIS, Zoological Record SCOPUS, SJR PubMed, Medline, Ecological Abstract, Embase Biological Abstract Google Scholar, ProQuest, Indian Science Abstract, Indian Citation Index, CAB Animal Science Abstract, Herbage Abstract, Plant Breeding Abstract, Proto Zoological Abstract, Review of Aromatic and Medicinal Plant, Weed Abstract Ulrich Periodical Directory, DOAJ, EBSCO Indexing System, J-Gate Global Health Databases, Paryavaran Abstract,
    2. Citations of an article in the last 3 years: We measure citation of articles in a diachronous way by using Publish or Perish. Total number of citations received during last three years (2011 to 2013) has been counted. The 1% of the average citations per article has been used for assigning score.
Subsidiary Criteria
  1. Journal Quality
    1. Standard pattern of Articles: Content of the article coded under various headings makes it readable and attractive. We checked, if the articles published in the journal follows a standard pattern (Introduction, Review, Aim-Objective, Method, Result, Discussion) for the content or not? If most of the articles (>90%) follow standard pattern then no score and if not then negative score of (-0.01).
    2. Author Guidelines: Guidelines for preparing manuscript for the author is available or not? If author guideline is available in the documentation of the journal then no score, however if the author guideline is not available then negative score of (-0.01) has been assigned.
    3. Original or Copied: We have checked if the guidelines available are original or copied from somewhere else. If it is found copied, then negative score of (-0.01) point has been assigned.
    4. Article Submission: We checked the procedure of article submission to a journal, to determine if it is ESS, hard copy, or email to the editor. If there is ESS process is available for article submission, then positive score of 0.01 has been assigned, however, if the submission is allowed through hard copy only then negative score of (-0.01) has been assigned.
    5. Article Tracking Process: Whether author can track his/her submission. If such tracking facility is available, we added 0.01 points.
    6. Acceptance and Rejection rate: Check whether the journal mentions acceptance or rejection rate or not. If available, a positive score of 0.01may be added and if not, then negative score of (-0.01) has been assigned.
  2. Article Quality:
    1. Availability of Expressive titles of the Article: We checked, if the titles of the articles published in the journal are expressive or not? If most of the articles (>90%) have expressive titles then no score and if not then negative score of (-0.01) has been assigned.
    2. Keywords of the article: We have checked, if the articles published in the journal contains keywords or not? If most of the articles (>90%) have keywords then no score and if not then negative score of (-0.01) is given.
    3. Author's Address& Email: We have checked, if the articles published in the journal contains the authors address or not? If most of the articles (>90%) have author address and email then no score and if not then negative score of (-0.01) is given.
    4. Article publishing History: We checked, if the each published article contains information about the date of submission, the date of revision, and the date of acceptance. If the articles (>90%) have article publishing history then 0.01 score and if not then negative score of (-0.01).
    5. Time lag in publication: This type of information can only be judged when publication contains article publishing history. This type of information is important for an author; as such information indicates how much time the journal takes to publish an article. Nowadays, the development of concepts like ‘Rapid Publication System’ creates a new challenge for the publishers to publish articles on time. Review of articles is a human centric process and involvements of reviewer are a volunteer service. Therefore, keeping the review process on schedule is always a challenging job for the editors. We have found that most of the reputable publishers generally take 4-6 months for completing this process and it seems unrealistic to conduct a proper review in a shorter time period. In order to score this criteria, if the time lag in publication for 20 articles is more than 3 months (for online) and 6 months (for print) then negative score of (-0.01) may be added.
    6. Digital Object Identifier (DOI)/QR Code: Whether journal uses DOI or QR code for individual articles or not? When a DOI or QR code is assigned to each article, additional 0.01 points were added and if not then negative score of (-0.01) has been assigned.
    7. Rapid Publication Facility: If the journal have a facility of publishing article in less than 4-5 weeks then negative score of (-0.01) has been assigned, otherwise no score.
  3. Data handling in Journal: Software used by journals to handle different type of data.
    1. Text handling: Whether the journal allows authors to submit manuscript in multiple formats or not. No points were assigned to any specific submission process in this regard.
    2. Graphics handling: What are the acceptable formats for submitting graphics (images, tables and graphs, etc.)? Does the journal accept embedded picture, or picture must be submitted separately? No points were assigned to any specific submission process in this regard.
    3. Color illustration: We checked, whether color illustrations are accepted by the journal or not. If such provision is available, is there a charge for color illustration? If colour illustrations are accepted for the print edition then positive score of 0.01 otherwise no score.
    4. Charges for color illustrations: If the charge, accepted for color illustrations is very high (>1000 INR/per object) then -0.01 otherwise no score.
    5. Scholarly forms of text: Number of scholarly forms of text accepted by the journal and their names.
  4. Search function: We checked, if the searching function is available or not. If yes, what kind of search function is included at the journal’s website: simple or advanced? If search facility is available then positive score of 0.01 has been assigned and if no search facility is available then negative scoring of (-0.01) has been assigned.
    1. Simple and Advanced Search Facility: As this feature significantly improves the user experience, we assigned 0.01 points if advanced search facility is available then additional score of 0.01 has been assigned.
  5. Journal usage statistics: Statistics of journal usage are available or not? If journal usage statistics is available then additional score of 0.01 has been assigned otherwise no score.
  6. Publication Ethics: Whether the journal has any ethics statements or not? If yes, was the statement devised by the journal itself or standard code? If Publication ethics is available in the documentation of the journal and it is a standard code of ethics then positive score of 0.1 has been assigned, otherwise negative score of (- 0.01).
  7. License Agreement of Article is available for OA: In this criterion we checked whether the journal has any license agreement for open access articles (eg. Creative Commons)? If yes, then positive core of 0.01 has been assigned, but in case no such license agreement available then negative score of (-0.01) has been assigned.
  8. Excessive advertising: We checked if the journal has excessive advertisement to that extent so that it affects the content access of the journal then negative score of (- 0.01) point will be assigned.
  9. Email id of the editor: Email id of the chief editor/journal for contact is available or not in the documentation of the journal. If the email address of the Editor-in-Chief or journal is available for contact then no score and if not mentioned then negative score of (-0.01) is given.
  10. Commercial server for email: If the email address of the Chief-Editor is from free server then 0.01 score and if it is from commercial server then no score.
  11. Publishing Body Information: We studied reputation of a journal in terms of reputation of its publishing & sponsoring body. We assume that a well reputed organization should have well documented history; it must regularly perform various scholarly activities, such as conducting conference-workshop-training programs, etc. Furthermore, its ranking in SE must be higher in position and the institution’s web server should have a considerable number of in-links (inbound links).
    1. Name of the Publisher: We studied whether or not the name of the publishing body of journal is clearly mentioned as ‘Published by’. If the name is not clearly mentioned in the journal’s documentation, a negative value of 0.1 is assigned.
    2. Collaboration with ISI indexed publisher: When any publisher has joint collaboration with ISI Indexed publishers additional 0.1 points is added.
    3. Nature: We have tried to identify whether the publishing body of the journal is Government/ Semi-government/Private/Society in nature. For Society and Private nature organizations, we have checked the registration, purpose of establishment, etc. Availability of such information leads to positive score of 0.01 however in case of unavailability of such information we have assigned negative score of (-0.01).
  12. Activity of Publisher: This criterion assumes checking if the publishing establishment performs other activities besides publishing a journal. If it does, we checked what those activities are. Performing any other activities has been awarded with an additional score of 0.1.
  13. Archiving policy followed by the journal: We checked if the journal adopted any archiving policy or not (eg. SHERPA/Romeo or any other)? If the publisher is not listed in Sherpa/Romeo project then negative score of -0.02 has been assigned. if the journal is listed in SHERPA/Romeo then 0.03 for Green, 0.02 for Blue and 0.01 for yellow and no score for white is assigned for archiving policy of the journal.
  14. Server/Website visibility and clarity:
    1. Top Page hit of Google: Top page hit of Google: The purpose of this criterion is to gather information about the reputation of a journal and their publisher. It indirectly helps to understand whether the publisher is well-known, moderately known, or commonly unknown. Of the top 10 search engine results, we checked the rank of publisher’s name along with the journal’s name. If their name appeared among the first two results (in the cases when the name of the journal is not a verbatim or mimic of some foreign journal), we have assigned 0.01 points.
    2. Total hits from Google: Total number of hits observed out of the first ten results for the publishing body in search engine is checked here. A point of 0.01 is augmented when the names appear in more than one hit (results from Wikipedia, etc.).
    3. Number of in-links: By putting a simple command link: domain.name, we have counted the total number of in-links of the publisher’s host. For every 20 in-links we have augmented 0.01 points.