TOIT Reviewer Instructions

This page is intended to describe the process of reviewing for TOIT. Reviewers have a good deal of flexibility in what/how they write the review. However, the editorial board asks that the reviewer follow the instructions below and remember two things: a) the authors will read the reviews, so try to be constructive and polite, and b) try to be clear, as in almost all cases the authors will craft their future efforts on the reviewed work based on these comments.

Review Criteria

Each reviewer should assess whether the paper should be published in TOIT. What makes a paper publishable in TOIT? There are really two criteria. First, the paper content must be consistent with the subject matter of TOIT. Reviewers (and authors) are directed to the subject page for further detail, but generally one should view any paper dealing with the middleware, applications, and public policy associated with the Internet as being in scope. When in doubt, talk to the journal associate editors or its Editor-in-Chief.

The second part of the assessment asks, "Is this paper good science?" This assessment should provide at least answers to the following questions:

  1. Is the problem addressed/studied in this paper well defined and articulated?
  2. Is the problem addressed/studied interesting enough to warrant publication?
  3. Is the methodology used appropriate for the problem addressed/studied?
  4. If this is a theoretical paper, does the level of rigor to support the result? Are the theoretical devices (e.g., proofs) correct?
  5. If this is an empirical paper, are the measurements or analysis properly applied and the correct conclusions drawn? Are there hidden factors that may invalidate the drawn conclusions?
  6. If this is a policy paper, does the paper offer insight into the policy issues and alternatives, and/or provide well reasoned critiques of proposals?
  7. If this is a survey paper, is the coverage exhaustive, the presentation clear and accessible, and is the topic timely?

The answers to these questions will indicate whether the paper usefully informs the TOIT technical community on some issue or problem of current importance (and is thus potentially publishable). If it does, and does so clearly, completely, and convincingly, then the paper is probably appropriate for publication. It commonly takes one or more rounds of edits to direct the paper into publishable form (see editorial process). However, it is common that the reviewer is able to determine whether it is likely that the paper will ever be able reach that point on the first reading.

Re-review policy: Each reviewer is responsible for assessing the quality of paper revisions provided by the authors (via the editor). Each revised paper should be reviewed to a level commensurate with the modifications. The re-review should contain a recommendation (as below--accept without changes, minor revision, ...) and include commentary identifying any remaining issues. The reviewer is free to downgrade a recommendation (potentially to reject) should he/she feel that the revisions decreased the quality or appropriateness of the paper.

Report format

The review should be returned to the editor by the deadline provided in the initial invitation. The review should be written in ASCII text and included in an email (although some editors may allow other formats such as Microsoft Word). The reviewers should remain anonymous. Note that some editors may edit for clarity or structure before forwarding to the EIC and authors, but will not change its intent. The review itself is commonly 100-300 lines of text, but in some cases can considerably longer. Shorter reviews are often a sign that the paper has not been read deeply, and are strongly discouraged.

The report should include, at a minimum:

  1. A 3-4 sentence overview of the work including its area, contribution, methodology, and results.

  2. Classification of the reviewer and paper:

    • This paper is primarily a/an [ EMPIRICAL | THEORETICAL | PUBLIC POLICY | SURVEY ] paper.

    • The reviewer is [ AN EXPERT | KNOWLEDGEABLE IN | AWARE OF | UNINFORMED OF ] the subject area.

    • The methodology, analysis, description of related work, and technical explanations in this paper are [ COMPLETELY SOUND | SLIGHTLY FLAWED | CONSIDERABLY FLAWED | INCORRECT ].

    • The content of this paper is [ PERFECTLY ALIGNED | RELATED | TANGENTIAL| UNRELATED ] with/to TOIT subject areas.

    • The editorial quality of the current version of this paper is [ FLAWLESS | EXCELLENT | FAIR | POOR ].

    • The contribution made by the current version of this paper is [ GROUNDBREAKING | IMPORTANT | ORDINARY | SMALL | NONEXISTENT ].

  3. An overall recommendation for the paper. These include:

    • Accept without modification - the paper is publishable in its present form. It is extraordinarily rare that a paper receives this rating on initial submission.
    • Minor revision - the paper requires minor modifications to the text and/or concepts. Minor revisions generally include small editorial changes, addressing specific technical comments (with a paragraph or less), or the introduction of small amounts of additional detail on existing content. Receiving this rating does not ensure that the paper will be accepted for publication in TOIT, but is indicative of the editor's belief that the paper will be published.
    • Major revision - the paper needs major modifications to be publishable at TOIT. Generally, any review that requires the introduction of new experiments or introduces substantially new content or exposition, or requires substantial editorial cleanup would be in this class. Receiving this rating does not ensure that the paper will be accepted for publication in TOIT.
    • Reject - the paper is not appropriate for publication in TOIT. No further modifications will be accepted or reviewed.
  4. Detailed comments for the reviewer on the paper. These comments should be directed toward making the paper publishable, if possible. The reviewer should clearly identify any conceptual, technical, or editorial issues that make the paper unpublishable. Any related work that should be cited or referenced should be identified. This is the main body of the review.